시뮬레이션 기반 평가에서 프로세스 데이터를 활용한 협력적 문제해결력 유형 분석 = Collaborative Problem Solving Type Analysis Using Process Data in Simulation-based Assessment

국문 초록 (Abstract)

이 연구는 우리나라 피험자의 협력적 문제해결력의 유형 및 특성을 발견하는 것을 목적으로 한다. 이에 PISA 2015의 협력적 문제해결력 평가에서 기록된 프로세스 데이터를 K-medoids 클러스터링 방법으로 분석하였다. 그 결과 협력적 문제해결력의 세 가지 유형을 발견하였다. 이는 응답까지 도달하는 시간을 다르게 할애하는 신속한 유형과 숙고적인 유형, 많은 상호작용 시도를 하면서 긴 시간을 활용하는 유형이었다. 그리고 군집별·수준별 정답률과 협력적 문제해결력 점수의 분포를 확인하고, 모든 군집에서 다양한 성취 수준의 분포가 나타남을 확인하였다. 그리고 이러한 차이에 따라 교육적·평가학적 개입의 방안을 모색하고 평가학적 함의를 논의하였다.

다국어 초록 (Multilingual Abstract)

This study aims to identify the types and characteristics of collaborative problem-solving skills among test takers in South Korea. For this, process data from the PISA 2015 collaborative problem-solving assessment was analyzed using the K-medoids clustering method. The analysis revealed three distinct types of collaborative problem-solving skills: a fast type that allocates less time to responses, a deliberative type, and a type that engages in a higher number of interactions over a longer period, even when achievement levels are similar. We also analyzed the distribution of correct answer rates and collaborative problem-solving competency scores by cluster and level, confirming that various achievement levels were represented across all clusters.
Based on these differences, educational and evaluative intervention strategies were explored, and the implications for assessment were discussed.

참고문헌 (Reference)

1 소경희, "학교교육의 맥락에서 본 ‘역량(competency)’의 의미와 교육과정적 함의" 25 (25): 1-21, 2007

2 함은혜, "프로세스데이터를 활용한 과제참여도 측정가능성 탐색 : PISA2015 과학성취도평가의 문항응답시간을 중심으로" 35 (35): 23-48, 2022

3 신효정, "프로세스 데이터를 활용한 응답속도와 정확도의 관계 분석 : PISA 2015 미국과 한국의 사례를 중심으로" 34 (34): 587-614, 2021

4 임효진 ; 이순영, "컴퓨터 기반의 협력적 문제해결력 성취를 예측하는 학생과 학교 및 ICT 요인 : 다층모형 분석을 중심으로" 22 (22): 457-471, 2018

5 김연후, "컴퓨터 기반 평가에서 프로세스 데이터를 활용한 피험자의 문제해결 행동 분석: 대표 액션 시퀀스의 추출 및 연결 양상을 중심으로" 서울대학교 대학원 2023

6 김성숙 ; 임효진 ; 정혜경, "우리나라 학생들의 PISA 2015 협력적 문제해결력과 협력적 태도에 대한 관계 비교" 21 (21): 155-179, 2018

7 안덕선, "시뮬레이션 교육과 평가" 3 (3): 1-, 2014

8 김연후, "딥러닝 기반 오토인코더를 활용한 협력적 문제해결 행동 유형 탐색" 86 : 17-56, 2024

9 Kline, R. B, "구조방정식모형 원리와 적용" 학지사 2010

10 Bergner, Y., "Visualization and confirmatory clustering of sequence data from a simulation based assessment task" 177-184, 2014

11 Jiang, Y., "Using process data to understand problem-solving strategies and processes for drag-and-drop items in a large-scale mathematics assessment" 9 (9): 1-31, 2021

12 Zhu, M., "Using networks to visualize and analyze process data for educational assessment" 53 (53): 190-211, 2016

13 Hubley, A. M., "Understanding and investigating response processes in validation research" Springer 1-12, 2017

14 Asseburg, R., "Too hard, too easy, or just right? The relationship between effort or boredom and ability-difficulty fit" 55 (55): 92-104, 2013

15 Chung, G. K. W. K., "The use of computer-based collaborative knowledge mapping to measure team processes and team outcomes" 15 (15): 463-493, 1999

16 Silm, G., "The relationship between performance and test-taking effort when measured with self-report or time-based instruments : A meta-analytic review" 31 : 100335-, 2020

17 Arslan, B., "The effect of drag-and-drop item features on test-taker performance and response strategies" 39 (39): 96-106, 2020

18 Dalmaijer, E. S., "Statistical power for cluster analysis" 23 (23): 1-28, 2022

19 Dong, G., "Sequence data mining. (Vol. 33)" Springer 2007

20 OECD, "Research and Innovation in PISA"

21 Levy, R., "Psychometric and Evidentiary Advances, Opportunities, and Challenges for Simulation-Based Assessment" 18 : 182-207, 2013

22 Klerk, S., "Psychometric analysis of the performance data of simulation-based assessment : A systematic review and a Bayesian network example" 85 : 23-34, 2015

23 Provasnik, S., "Process data, the new frontier for assessment development : rich new soil or a quixotic quest?" 9 (9): 1-17, 2021

24 Bergner, Y., "Process Data in NAEP : Past, Present, and Future" 44 (44): 706-732, 2019

25 Asparouhov, T., "Plausible values for latent variables using Mplus"

26 Li, S, "Patterns of action transitions in online collaborative problem solving : A network analysis approach" 17 : 191-223, 2022

27 교육부, "PISA2015(제5권) 협력적 문제해결력 주요 내용"

28 김혜숙 ; 함은혜, "PISA 2015 협력적 문제해결력에 영향을 미치는 학생 및 학교 수준 특성 분석-과학성취도와의 비교를 중심으로" 22 (22): 199-224, 2019

29 이소연 ; 구남욱, "PISA 2015 협력적 문제해결력 영역 결과에 나타난 우리나라 학생들의 성취 특성 분석 : 성취 상위국과의 비교를 중심으로" 52 (52): 213-237, 2020

30 우한솔, "PISA 2015 교사인식 변인이 협력적 문제해결력에 미치는 영향 : 과학성취도와 비교를 중심으로" 71 (71): 441-471, 2023

31 OECD, "PISA 2015 technical Report"

32 OECD, "PISA 2015 collaborative problem solving framework. PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic, Financial Literacy and Collaborative Problem Solving"

33 OECD, "PISA 2015 Results (Volume Ⅴ): Collaborative Problem Solving" PISA, OECD Publishing 2017

34 "OECD"

35 Ma, Y., "Identifying collaborative problem-solver profiles based on collaborative processing time, actions and skills on a computer-based task" 30 : 465-488, 2023

36 Kroehne, U., "How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items" 45 (45): 527-563, 2018

37 OECD, "How deos PISA Measure Student’s Ability to Collaborate?"

38 Jiao, H., "Handbook of diagnostic classification models: Models and model extensions, applications, software packages" 421-436, 2019

39 Berkhin, P., "Grouping multidimensional data" Springer 25-71, 2006

40 Goldhammer, F., "From byproduct to design factor : On validating the interpretation of process indicators based on log data" 9 : 1-25, 2021

41 Borgonovi, F., "From asking to observing. Behavioural measures of socio-emotional and motivational skills in large-scale assessments" 112 : 102874-, 2023

42 Papanastasiou, E. C., "Examining successful and unsuccessful time management through process data : two novel indicators of test-taking behaviors" 12 (12): 3-16, 2024

43 Mislevy, R. J., "Evidence-Centered Design for Simulation-Based Assessment" 178 : 107-114, 2013

44 Bholowalia, P., "EBK-means : A clustering technique based on elbow method and k-means in WSN" 105 (105): 17-24, 2014

45 OECD, "Description of the Released Unit from the 2015 PISA Collaborative Problem-Solving Assessment, Collaborative Problem-Solving Skills, and Proficiency Levels"

46 Shingala, M. C., "Comparison of post hoc tests for unequal variance" 2 (2): 22-33, 2015

47 Sonagara, D., "Comparison of basic clustering algorithms" 3 (3): 58-61, 2014

48 De Boeck, P., "Collaborative Problem Solving : Processing Actions, Time, and Performance" 10 (10): 1-9, 2019

49 Tibshirani, R., "Cluster validation by prediction strength" 14 (14): 511-528, 2005

50 Han, A., "Behavioral patterns in collaborative problem solving : a latent profile analysis based on response times and actions in PISA 2015" 11 (11): 1-28, 2023

51 Zhu, M., "Behavior Sequence Patterns in Collaborative Problem Solving : A Multiple-perspective Analysis" International Society of the Learning Sciences 2024 : 205-208, 2024

52 Clarke-Midura, J., "Assessment, Technology, and Change" 42 (42): 309-328, 2010

53 James, G., "An introduction to statistical learning (Vol. 112)" Springer 2013

54 Sundre, D. L., "An exploration of the psychology of the examinee : Can examinee self-regulation and test-taking motivation predict consequential and non-consequential test performance?" 29 (29): 6-26, 2004

55 Cukurova, M., "Adaptive and Adaptable Learning" 63-569, 2016

56 Park, H. S., "A simple and fast algorithm for K-medoids clustering" 36 (36): 3336-3341, 2009