Relationship Between Variables of pre- and post-Randomized Questions in Computer-based Tests

감비성 사단법인 미래융합기술연구학회 2021 아시아태평양융합연구교류논문지 Vol.7 No.3

Cheating during assessments can be a result of poor physical arrangement of exam settings. In multiple-choice question of computer-based tests (CBT), shuffling items is used to deter students from copying answers from their colleagues. Randomizing questions and shuffling items in multiple-choice questions are meant to set a copying-disabled environment for students. This study aims to statistically verify that shuffling is suitable for preventing cheating using quantitative data. Since object awareness may compromise the effectiveness of a survey method, we used quantitative data from the students’ selection of items without them being aware and to further avoid compromising the data. A total of 472 CBT data from two academic semester years of 2013 and 2016 were collected from first-year students enrolled in two exams of Cell Regulation Mechanism and Basic Human Structure. Participants data collected from test score, multiple-question selection number, and seat number. The question item number of each student was compared with the item selection of five nearby students seated in different positions. Item selection data from students enrolled in Basic Human Structure exam compared to five nearby colleagues showed a correlation coefficient of less than 0.8, indicating that there were no cases of cheating confirmed. In the case of Cell Regulation Mechanism exam, by ignoring the front seat position, (considered limitation of this study) the measurement of comparing individual students to five nearby colleagues showed a correlation coefficient above 0.8 among 13 students (n=13), about 5.8% of total students, meaning 5% of students either cheat or selected items based on their knowledge. Practically, by simply looking at the monitor of a colleague, justifying cheating is difficult; however, the measurement of quantitative data produced by the students’selection of items showed that question-shuffling and item randomization reduce cheating among students. The results of this study is also intended to introduce the feeling of a safer environment by justifying that shuffling would deter cheating.

3학년 학생의 역량수준과 관련 요소

감비성,이상엽,임선주,Kam, Beesung,Lee, Sang Yeoup,Im, Sun Ju 연세대학교 의과대학 2013 의학교육논단 Vol.15 No.2

The purpose of this study was to assess third-year medical students' competency for development or revision of the undergraduate curriculum and assessments. One hundred and twenty-seven third-year medical students at the Pusan National University were included in the study. After third- and fourth-year students took a common written examination, clinical performance examination (CPX), and objective structured clinical examination (OSCE) with common items as a summative assessment, the third-year students' competency was compared with 132 forth-year students' results. The correlation of the written examination and CPX/OSCE was analysed, and the summative results were compared with the grade point average (GPA) through the second year, CPX/ OSCE in the second year, and GPA in the clerkship. On the written examination, the third-year students' mean score was lower than the fourth-year students' by over 11 points, whereas the gap in the CPX/OSCE was 4 points and there was no difference in the OSCE. There was a moderate correlation between the written examination and the CPX/OSCE scores (R=0.371, p<0.01). The written examination was highly correlated with GPA through the second year, which mainly evaluated medical knowledge (R=0.771, p<0.01). A relatively high correlation was observed between CPX/OSCE scores and GPA in the clerkship (R=0.641, p<0.01). The summative CPX/ OSCE scores showed a moderate correlation with formative CPX/OSCE scores in the second year (R=0.464, p< 0.01). The third-year students' score was quite low on the written examination and slightly low on the CPX/OSCE compared to that of the fourth-year students. The written examination and CPX/OSCE cannot replace each other and should be combined with other methods of evaluation to measure competency. Early OSCE and workplacebased assessment should be useful in the early assessment of clinical skills competency.

Relational algorithm interpreting symptoms to demographic sign of patient medical history

감비성 사단법인 인문사회과학기술융합학회 2019 예술인문사회융합멀티미디어논문지 Vol.9 No.1

Doctor-patient communication is a skill to be thought in medical education. Challenges in performing an effective communication pressurizes medical students to focus on how to earn accurate patient history by using different methods of communication which is a challenging issue toward quality of care. Time pressure of waiting patients, irrelevant patient dialogue requiring sympathy, less accurate patient response, none categorized set of quick answers which needs to be specified and might be totally irrelevant to the care are sets of examples of frustration in communication within clinical care. Although many studies through machine learning, artificial intelligent and virtual reality addressed communication challenges, there are still rooms yet to establish an accurate communication channel for improving quality of care. This paper introduces a new method as a domain concept to interpret sign values as graphical interface for medical care providers so that information exchange with an enhanced and clear history taking symptoms relevant to both patients and disease category for an improved quality care environment. Designing a medical terminology interpretation software that relates medical history to graphical interface of various and easy to understand signs to reduce communication error of patients having difficulty relating different signs to their existing symptoms.

부산 ․경남 컨소시엄에서 임상수행평가의 시행 경험

감비성,오영림,이상화,노혜린,함종렬,임선주 한국의학교육학회 2013 Korean journal of medical education Vol.25 No.4

Purpose: The purpose of this study is to judge the quality of clinical skills assessment in Busan-Gyeongnam Consortium. Methods: Fourth grade medical school students (n=350 in 2012 and n=419 in 2013) in the Busan-Gyeongnam Consortium were included in the study. The examination was consisted of 6 clinical performance examination (CPX) and 6 objective structured clinical examination (OSCE) stations. The students were divided into groups to take the exam in 4 sites during 3 days. The overall reliability was estimated by Cronbach α coefficient across the stations and the case reliability was by α across checklist items. Analysis of variance and between-group variation were used to evaluate the variation of examinee performance across different days and sites. Results: The mean total CPX/OSCE score was 67.0 points. The overall α across-stations was 0.66 in 2012 and 0.61 in 2013. The α across-items within a station was 0.54 to 0.86 in CPX, 0.51 to 0.92 in OSCE. There was no significant increase in scores between the different days. The mean scores over sites were different in 30 out of 48 stations but between-group variances were under 30%, except 2 cases. Conclusion: The overall reliability was below 0.70 and standardization of exam sites was unclear. To improve the quality of exam,case development, item design, training of standardized patients and assessors, and standardization of sites are necessary. Above of all, we need to develop the well-organized matrix to measure the quality of the exam.

Automatic Generation of Author Output based on the Iterative Clustering Method

감비성 사단법인 미래융합기술연구학회 2020 아시아태평양융합연구교류논문지 Vol.6 No.5

The structural differences and requirements of diverse techniques in searching resources, disambiguation, and bias in authors’ names for bibliographic searching introduce crafting inaccuracies. On the other hand, usage of external instances of searching such as mail and venue is not sufficient for clearance in searching bibliographic data. This paper proposes iterative clustering method and labeling procedure to search a distinguished set of authors’ affiliations zipped into a function to be performed on three databases having different structures. A distinguished algorithm was created using AI methodology to generate best outcomes toward reduction of bias in author’s names, resulting in the enhancement of accuracy in bibliographic data. This technique also reduced the requirement of diverse manual inputs in searching resources for finding out author’s national outputs. A total of 31 authors (1076 publications) were selected manually and compared against searches of three database sets RISS (Research Information System Store), KMBASE (Korean Medical Database), and DBPIA (Korea Association of Representatives) for a total of 759 found publications. 90.2% of national publication search results confirmed to be accurate.

How question shuffling influences correlation of student item selections in computer-bases test

감비성 사단법인 인문사회과학기술융합학회 2018 예술인문사회융합멀티미디어논문지 Vol.8 No.6

Handling Computer Based Test (CBT) in large classroom requires more of test reliability. Because of limited space assigned to each student, computer monitors are positioned close together so that reliability of exam goes under question also the environment makes student vulnerable to copy from their neighbors. A method to deter students from copying answers of their colleagues is to shuffle items in multiple choice questions. Objective of this study is compare student selected items and questions before and after randomizing assessment of multiple choice test to emphasize that reordering questions can set an environment to discourage students from copying answers of their colleagues in an exam room. This study used existing data from students enrolled in circulation system assessment of Pusan National University School of Medicine over two academic semester year of 2013 and 2016 and measured for correlation coefficient of pre and post randomization. Statistically significant relationship between selected items were found in favor of item shuffling in CBT.

해부학실습에서 동료평가 및 자기평가와 해부학 학업성취도의 관련성

감비성(Bee Sung Kam),김민정(Min Jeong Kim),주성일(Seung Il Joo),윤소정(So Jung Yune),임선주(Sun Ju Im),이상엽(Sang Yeoup Lee),윤식(Sik Yoon),백선용(Sun Yong Baek) 대한체질인류학회 2018 해부·생물인류학 (Anat Biol Anthropol) Vol.31 No.1

해부학실습과정은 소그룹 단위로 실습이 진행되면서 팀워크 조성, 성실성, 능동적 학습, 의사소통으로 효율적인 협동학습을 필요로 하며, 이는 전문직업성의 주요한 요소에 해당한다. 본 연구의 목적은 해부학실습 과정에서의 전문직업성을 동료와 본인이 평가하여 학업성취도와의 상관관계를 확인하는 것이다. 2015년 부산대학교 의학전문대학원 1학년 학생의 ‘인체의 구조와 기능’ 과정에서 해부학실습에 참여한 108명의 학생들 대상으로 설문지를 통한 동료평가와 자기평가를 실시하였다. 동료평가와 자기평가의 중앙값을 중심으로 상하로 나누어 높은 동료평가/높은 자기평가, 높은 동료평가/낮은 자기평가, 낮은 동료평가/높은 자기평가, 낮은 동료평가/낮은 자기평가 4개의 집단으로 분류하였다. 동료평가와 자기평가에 따른 4개 집단 중에서 높은 동료평가 집단이 낮은 동료평가 집단에 비해 학업성취도 중에서 과정시험 (F=3.24), 학점 (F=3.54), 예습시험 (F=3.94), 실기시험 (F=3.60) 성적에서 다른 집단과 유의한 차이를 보였다. 학업성취도를 구성하는 성적변인 사이에는 전반적으로 높은 상관계수를 보였다. 그러나 성적변인 중에서 실습시험 성적은 다른 변인과의 상관관계에서 집단에 따라서 차이를 보였다. 실습시험과 다른 변인 사이의 상관관계가 가장 높은 집단은 높은 동료평가/높은 자기평가 집단이었으며, 이어서 낮은 동료평가/ 낮은 자기평가 집단, 낮은 동료평가/높은 자기평가 집단이었으며 높은 동료평가/낮은 자기평가 집단이 상대적으로 낮은 상관관계를 보였다. 결론적으로 동료평가와 자기평가의 상대적 차이에 따른 집단을 대상으로 학업성취도에서의 유의한 차이를 확인하여 이러한 분류체계가 전문직업성의 작업습관에 대한 평가도구로 활용이 가능하며, 동료평가와 자기평가가 모두 높은 집단의 경우는 맥락적 학습에 보다 잘 적응하는 집단으로 생각되었다. Gross anatomy, with cadaver laboratory dissection, is in a unique position to preside over a rich number of activities such as the team work, integrity, active learning, communication in the small group-all aspects of professional conduct. The purposes of this study are to investigate the correlation of professionalism with academic performance and the characteristics of groups of students in the performance variables. First-year Pusan National University School of Medicine students (n=108) from the Class of 2015 taking the gross anatomy course were required to do the self- and peer assessment about the professional behaviors of each of the six members of their dissecting group. The students were classified into one of four subgroups based on their deviation from the self- and peer assessment medians, such as high peer/high self (HP/HS), high peer/low self (HP/LS), low peer/high self (LP/HS), low peer/low self (LP/LS). There were significant higher scores in the high peer groups (HP/HS, HP/LS) in comparison with low peer groups (LP/HS, LP/LS) in the academic performance of end-of-term (F=3.24), credit (F=3.54), pre-Lab (F=3.94), practical examination (F=3.60) scores. Significant correlations among academic performance variables were observed generally. There were some differences in the correlation in the practical examination and other variables. The relatively high correlation between practical examination and other variables is HP/HS, followed by LP/LS, LP/HS and then HP/LS. In conclusion, peer- and self assessment subgroups showed a significant differences in the academic performance. This assignment would be available to evaluate work habits of professionalism in the gross anatomy laboratory. It is suggested that HP/HS group was more adaptable in the contextual leaning the gross anatomy course.

CBT(Computer-Based Testing) 시험의 부정행위 방지를 위한 랜덤(Random) 보기 기반 학습자 능력평가 시스템

이정미,감비성 사단법인 인문사회과학기술융합학회 2018 예술인문사회융합멀티미디어논문지 Vol.8 No.1

Today Computer Based Tests (CBT) are used more frequently than Paper Based Tests (PBT) however CBTs are often taken place in classrooms with not much space for computer system so that computer screens are very close together. In this case dishonesty during exam in Computer Based Test (CBT) is unavoidable because the environment itself is open for copying answers from other students. Randomizing questions in an examination is a method that has been used mainly to deterrent student. In this study we argue that if test question reordered with a randomizing method and question items shuffled it effects setting environment in which disables student from copying answers from colleagues. A survey of test validity from pre and post randomization of CBT assessment conducted on 3rd grade student of Pusan National University School of Medicine. Satisfaction of students recorded in 83.3%, Although 75% of answers recorded of reduction in copying answers from colleagues after randomizing CBT. We conclude that reordering questions and shuffling question items are a valid method to examination in large or classrooms. Important factor is not only reduction of dishonesty but also fairness of test scores by performing randomization in computer assessment. 의학교육에 있어 디지털기술의 한가지인 CBT(Computer-Based Testing)는 관리자 차원에서의 과제 생성과 관리, 수정 등 아이템들의 검토와 조절이 쉬우며 정확한 자신의 평가능력 수준을 제시해 주는 등 많은 강점을 가지고 있다. 하지만 이러한 많은 장점들에도 불구하고, 학생들의 여러 종류의 부정 시험(A Deterrent for Cheating)과 관리자 영역의 해킹 등이 문제시 되어 CBT의 형평성과 공정성을 담보하기 위한 보완점들이 요구되어 지고 있다. 특히, CBT가 이루어지는 시험실 공간이 좁다보니 PC가 학생들 간에 너무 가까이 있어 커닝할 여지가 많다는 우려의 목소리가 높았다. 이에 본 논문에서는 CBT 수행 시 타인의 모니터를 보고 답안을 베끼는 부정 행위를 방지하기 위해 랜덤(Random)방식으로 시험 문제와 보기 지문을 나열하는 “랜덤(Random) 보기 기반 학습자 능력평가 시스템(The practise of randomizing multiple choice items in computer-based test)”을 개발하여 의학전문대학원 3학년 CBT 수행 시에 적용하였다. 그 결과, CBT에 대한 만족도로 아주 만족한다, 만족한다의 긍정적인 응답이 83.3%, 랜덤방식 출제 후 부정행위 감소 효과에 대해 감소되었다가 75%로 랜덤(Random) 보기 기반 학습자 능력평가 프로그램이 부정 행위를 막아주는 역할을 해내었음을 확인할 수 있었다.

정보기술을 활용한 의학교육시스템의 설계와 개발: 부산대학교 의학전문대학원 사례

임선주,이상엽,백선용,우재석,감비성,Im, Sun Ju,Lee, Sang Yeoup,Baek, Sun Yong,Woo, Jae Seok,Kam, Beesung 연세대학교 의과대학 2014 의학교육논단 Vol.16 No.1

The development of information technology (IT) has led to changes in medical education. IT has been used for e-learning and e-teaching, e-assessment, e-logistics, and e-administration. Pusan National University School of Medicine has developed its own educational information system using IT to support students' learning and assessment and to manage curricular activities. Based on our experience, we propose six suggestions for designing new software, specifically regarding simplifying the design for users, communication with the programmer, a rapid cycle from design to implementation, orientation support for users, backup and security support, and obtaining patents. Collaborating with the Department of Medical Informatics within the School of Medicine has advantages, and an alliance among medical schools can simplify the development of software. In any case, curricular innovation should precede the introduction of technology because all technologies should support curricular goals.

가정의학 임상실습과정에 대한 구조화된 평가

박은주,이상엽,임선주,윤소정,감비성,백선용,김윤진,우재석,이정규,정동욱,조영혜,이유현,탁영진,Park, Eun Ju,Lee, Sang Yeoup,Im, Sun Ju,Yune, So Jung,Kam, Beesung,Baek, Sun Yong,Kim, Yun-Jin,Woo, Jae Seok,Lee, Jeong-Gyu,Jeong, Dong-Wook,Cho, Yo 연세대학교 의과대학 2017 의학교육논단 Vol.19 No.1

This study was conducted to analyze the strengths and weaknesses of a 3-week family medicine clerkship program based on the results of an online survey taken by the students (N=127) and a structured interview with a focus group (n=10), aimed to improve the quality of the clerkship program. The online survey contained questions pertaining to goals, schedule, contents, arrangement, atmosphere, environment, evaluation, and satisfaction regarding the clerkship. The focus group interview addressed the schedule and achievements of the program. Scores were reported on a 5-point Likert scale. Most students were highly satisfied with the overall quality of the clerkship. The structured interview results showed that 97.6% of the clerkship program was executed according to the schedule. The focus group reported a perfect score of 5 points on several measures including: accomplishment of the educational goals of the family medicine clerkship, providing many chances to obtain medical histories and perform physical examinations on real patients, experience with various symptoms and diseases, positive attitudes of faculty members when teaching, notification of the guidelines for evaluation beforehand, well-constructed and effective clerkship schedule, and reflection of student feedback. However, the focus group gave low scores on: support for health accidents of students, access to patient information, enough opportunities to practice clinical skills, appropriate rest facilities for students, and fairness of clerkship evaluation process. In conclusion, the structured evaluation performed after the 3-week clerkship program motivated students and helped them ensure an efficient clerkship. This structured evaluation also suggested basic data to make the professor who is subject of the assessment. This study shows that structured assessment is an effective method which can be used to improve the quality of clerkships.