Medical student selection process enhanced by improving selection algorithms and changing the focus of interviews in Australia: a descriptive study

Boaz Shulruf,Gary Mayer Velan,Sean Edward Kennedy 한국보건의료인국가시험원 2022 보건의료교육평가 Vol.19 No.-

Purpose The study investigates the efficacy of new features introduced to the selection process for medical school at the University of New South Wales, Australia: (1) considering the relative ranks rather than scores of the Undergraduate Medicine and Health Sciences Admission Test and Australian Tertiary Admission Rank; (2) structured interview focusing on interpersonal interaction and concerns should the applicants become students; and (3) embracing interviewers’ diverse perspectives. Methods Data from 5 cohorts of students were analyzed, comparing outcomes of the second year in the medicine program of 4 cohorts of the old selection process and 1 of the new process. The main analysis comprised multiple linear regression models for predicting academic, clinical, and professional outcomes, by section tools and demographic variables. Results Selection interview marks from the new interview (512 applicants, 2 interviewers each) were analyzed for inter-rater reliability, which identified a high level of agreement (kappa=0.639). No such analysis was possible for the old interview since it required interviewers to reach a consensus. Multivariate linear regression models utilizing outcomes for 5 cohorts (N=905) revealed that the new selection process was much more effective in predicting academic and clinical achievement in the program (R2=9.4%–17.8% vs. R2=1.5%–8.4%). Conclusion The results suggest that the medical student selection process can be significantly enhanced by employing a non-compensatory selection algorithm; and using a structured interview focusing on interpersonal interaction and concerns should the applicants become students; as well as embracing interviewers’ diverse perspectives.

An expert-led and artificial intelligence system-assisted tutoring course to improve the confidence of Chinese medical interns in suturing and ligature skills: a prospective pilot study

Ying-Ying Yang,Boaz Shulruf 한국보건의료인국가시험원 2019 보건의료교육평가 Vol.16 No.-

PurposeLack of confidence in suturing/ligature skills due to insufficient practice and assessments is common among novice Chinese medical interns. This study aimed to improve the skill acquisition of medical interns through a new intervention program. MethodsIn addition to regular clinical training, expert-led or expert-led plus artificial intelligence (AI) system tutoring courses were implemented during the first 2 weeks of the surgical block. Interns could voluntarily join the regular (no additional tutoring), expert-led tutoring, or expert-led+AI tutoring groups freely. In the regular group, interns (n=25) did not receive additional tutoring. The expert-led group received 3-hour expert-led tutoring and in-training formative assessments after 2 practice sessions. After a similar expert-led course, the expert-led+AI group (n=23) practiced and assessed their skills on an AI system. Through a comparison with the internal standard, the system automatically recorded and evaluated every intern’s suturing/ligature skills. In the expert-led+AI group, performance and confidence were compared between interns who participated in 1, 2, or 3 AI practice sessions. ResultsThe end-of-surgical block objective structured clinical examination (OSCE) performance and self-assessed confidence in suturing/ligature skills were highest in the expert-led+AI group. In comparison with the expert-led group, the expert-led+AI group showed similar performance in the in-training assessment and greater improvement in the end-of-surgical block OSCE. In the expert-led+AI group, the best performance and highest post-OSCE confidence were noted in those who engaged in 3 AI practice sessions. ConclusionThis pilot study demonstrated the potential value of incorporating an additional expert-led+AI system-assisted tutoring course into the regular surgical curriculum.

What impacts students’ satisfaction the most from Medicine Student Experience Questionnaire in Australia: a validity study

Pin-Hsiang Huang,Boaz Shulruf,Gary Velan,Greg Smith,Melanie Fentoullis,Sean Edward Kennedy,Karen Jane Gibson,Kerry Uebel 한국보건의료인국가시험원 2023 보건의료교육평가 Vol.20 No.-

Purpose This study evaluated the validity of student feedback derived from Medicine Student Experience Questionnaire (MedSEQ), as well as the predictors of students’ satisfaction in the Medicine program. Methods Data from MedSEQ applying to the University of New South Wales Medicine program in 2017, 2019, and 2021 were analyzed. Confirmatory factor analysis (CFA) and Cronbach’s α were used to assess the construct validity and reliability of MedSEQ respectively. Hierarchical multiple linear regressions were used to identify the factors that most impact students’ overall satisfaction with the program. Results A total of 1,719 students (34.50%) responded to MedSEQ. CFA showed good fit indices (root mean square error of approximation=0.051; comparative fit index=0.939; chi-square/degrees of freedom=6.429). All factors yielded good (α>0.7) or very good (α>0.8) levels of reliability, except the “online resources” factor, which had acceptable reliability (α=0.687). A multiple linear regression model with only demographic characteristics explained 3.8% of the variance in students’ overall satisfaction, whereas the model adding 8 domains from MedSEQ explained 40%, indicating that 36.2% of the variance was attributable to students’ experience across the 8 domains. Three domains had the strongest impact on overall satisfaction: “being cared for,” “satisfaction with teaching,” and “satisfaction with assessment” (β=0.327, 0.148, 0.148, respectively; all with P<0.001). Conclusion MedSEQ has good construct validity and high reliability, reflecting students’ satisfaction with the Medicine program. Key factors impacting students’ satisfaction are the perception of being cared for, quality teaching irrespective of the mode of delivery, and fair assessment tasks which enhance learning.

Assessment methods and the validity and reliability of measurement tools in online objective structured clinical examinations: a systematic scoping review

Jonathan Zachary Felthun,Silas Taylor,Boaz Shulruf,Digby Wigram Allen 한국보건의료인국가시험원 2021 보건의료교육평가 Vol.18 No.-

The coronavirus disease 2019 (COVID-19) pandemic has required educators to adapt the in-person objective structured clinical examination (OSCE) to online settings in order for it to remain a critical component of the multifaceted assessment of a student’s competency. This systematic scoping review aimed to summarize the assessment methods and validity and reliability of the measurement tools used in current online OSCE (hereafter, referred to as teleOSCE) approaches. A comprehensive literature review was undertaken following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. Articles were eligible if they reported any form of performance assessment, in any field of healthcare, delivered in an online format. Two reviewers independently screened the results and analyzed relevant studies. Eleven articles were included in the analysis. Pre-recorded videos were used in 3 studies, while observations by remote examiners through an online platform were used in 7 studies. Acceptability as perceived by students was reported in 2 studies. This systematic scoping review identified several insights garnered from implementing teleOSCEs, the components transferable from telemedicine, and the need for systemic research to establish the ideal teleOSCE framework. TeleOSCEs may be able to improve the accessibility and reproducibility of clinical assessments and equip students with the requisite skills to effectively practice telemedicine in the future.

Comparison of the effects of simulated patient clinical skill training and student roleplay on objective structured clinical examination performance among medical students in Australia

Silas Taylor,Matthew Haywood,Boaz Shulruf 한국보건의료인국가시험원 2019 보건의료교육평가 Vol.16 No.-

PurposeOptimal methods for communication skills training (CST) are an active research area, but the effects of CST on communication performance in objective structured clinical examinations (OSCEs) has not been closely studied. Student roleplay (RP) for CST is common, although volunteer simulated patient (SP) CST is cost-effective and provides authentic interactions. We assessed whether our volunteer SP CST program improved OSCE performance compared to our previous RP strategy. MethodsWe performed a retrospective, quasi-experimental study of 2 second-year medical student cohorts’ OSCE data in Australia. The 2014 cohort received RP-only CST (N=182) while the 2016 cohort received SP-only CST (N=148). The t-test and analysis of variance were used to compare the total scores in 3 assessment domains: generic communication, clinical communication, and physical examination/procedural skills. ResultsThe baseline characteristics of groups (scores on the Australian Tertiary Admission Rank, Undergraduate Medicine and Health Sciences Admission Test, and medicine program interviews) showed no significant differences between groups. For each domain, the SP-only CST group demonstrated superior OSCE outcomes, and the difference between cohorts was significant (P<0.01). The superiority of volunteer SP CST over student RP CST in terms of OSCE performance outcomes was found for generic communication, clinical communication, and physical examination/procedural skills. ConclusionThe better performance of the SP cohort in physical examination/procedural skills might be explained by the requirement for patient compliance and cooperation, facilitated by good generic communication skills. We recommend a volunteer SP program as an effective and efficient way to improve CST among junior medical students.

Factors predicting rural location employment intent and choice among medical students and graduates

Guilfoyle Charis,Huang Pin-Hsiang,Forster Lesley,Shulruf Boaz 한국의학교육학회 2022 Korean journal of medical education Vol.34 No.4

Purpose: Workforce shortage is a contributing cause of health inequality in rural Australia. There is inconclusive evidence demonstrating which factors cause doctors to choose rural practice. This study’s objective is to determine predictive factors for medical students’ intent to work rurally and for graduates’ current rural employment location choice.Methods: This prospective cohort study, utilized data gathered from the University of New South Wales about students and graduates who had spent one or more years in a Rural Clinical School. Participants were final year students and graduates already working in Australia. Stepwise logistic regression was used to determine predictive factors for the two outcomes.Results: Predictors for student intent to work rurally are rural background (odds ratio [OR], 7.16; 95% confidence interval [CI], 2.59–19.53), choosing to study at the Rural Clinical School (OR, 8.72; 95% CI, 1.32–57.63), and perceiving rural areas as opportunistic for career advancement (OR, 1.69; 95% CI, 1.15–2.49). Predictors for graduates currently working in a rural location are Bonded Medical Program participation (OR, 6.40; 95% CI, 1.15–35.59) and personal altruism (OR, 1.91; 95% CI, 1.02–3.57).Conclusion: While intent is predicted by having a rural background, choosing to study at the Rural Clinical School and perception of rural areas as having positive career opportunities, a current rural workplace location among graduates is predicted by holding a bonded medical position and a desire to serve an under-resourced population. Maintaining the Bonded Medical Program and clear communication regarding training pathways may increase numbers of rural doctors.

Definition of professionalism and tools for assessing professionalism in pharmacy practice: a systematic review

Huda Dubbai,Barbara-Ann Adelstein,Silas Taylor,Boaz Shulruf 한국보건의료인국가시험원 2019 보건의료교육평가 Vol.16 No.-

PurposeIn contemporary pharmacy, the role of pharmacists has become more multifaceted, as they now handle a wider range of tasks and take more responsibility for providing patient care than 20 years ago. This evolution in pharmacists’ responsibilities has been accompanied by the need for pharmacists to display high-quality patient-centred care and counselling, and to demonstrate professionalism, which now needs to be taught and assessed as part of pharmacy education and practice. This study aimed at identifying definitions of professionalism in pharmacy practice and critically evaluating published instruments for assessing professionalism in pharmacy practice. MethodsWe searched the medical literature listed in Scopus, MEDLINE, and PsycINFO databases from 1 January 2000 to 31 December 2018. All papers meeting our selection criteria were reviewed and summarised into a clear review of professionalism requirements in pharmacy practice. Details of the instruments measuring professionalism were reviewed in detail. ResultsThere is no accepted simple definition of professionalism, although we identified several theoretical and policy frameworks required for professional pharmaceutical practice. We identified 4 instruments (the Behavioural Professionalism Assessment Instrument, Lerkiatbundit’s instrument, the Pharmacy Professionalism Instrument, and the Professionalism Assessment Tool that build on these frameworks and measure professional practice in pharmacy students. These were found to be reliable and valid, but had only been used and tested in student populations. ConclusionGiven the increasingly broad role of community pharmacists, there is a need for assessments of professionalism in practice. Professionalism is a complex concept that is challenging to measure because it has no standardised definition and the existing literature related to the topic is limited. Currently available instruments focus on measuring the development of the elements of professionalism among pharmacy students, rather than pharmacists.

Development and validation of the student ratings in clinical teaching scale in Australia: a methodological study

Pin-Hsiang Huang,Anthony John O’Sullivan,Boaz Shulruf 한국보건의료인국가시험원 2023 보건의료교육평가 Vol.20 No.-

Purpose This study aimed to devise a valid measurement for assessing clinical students’ perceptions of teaching practices. Methods A new tool was developed based on a meta-analysis encompassing effective clinical teaching-learning factors. Seventy-nine items were generated using a frequency (never to always) scale. The tool was applied to the University of New South Wales year 2, 3, and 6 medical students. Exploratory and confirmatory factor analysis (exploratory factor analysis [EFA] and confirmatory factor analysis [CFA], respectively) were conducted to establish the tool’s construct validity and goodness of fit, and Cronbach’s α was used for reliability. Results In total, 352 students (44.2%) completed the questionnaire. The EFA identified student-centered learning, problem-solving learning, self-directed learning, and visual technology (reliability, 0.77 to 0.89). CFA showed acceptable goodness of fit (chi-square P<0.01, comparative fit index=0.930 and Tucker-Lewis index=0.917, root mean square error of approximation=0.069, standardized root mean square residual=0.06). Conclusion The established tool—Student Ratings in Clinical Teaching (STRICT)—is a valid and reliable tool that demonstrates how students perceive clinical teaching efficacy. STRICT measures the frequency of teaching practices to mitigate the biases of acquiescence and social desirability. Clinical teachers may use the tool to adapt their teaching practices with more active learning activities and to utilize visual technology to facilitate clinical learning efficacy. Clinical educators may apply STRICT to assess how these teaching practices are implemented in current clinical settings.

Effects of virtual reality training on decreasing the rates of needlestick or sharp injury in new-coming medical and nursing interns in Taiwan

Szu-Hsien Wu,Chia-Chang Huang,Shiau-Shian Huang,Ying-Ying Yang,Chih-Wei Liu,Boaz Shulruf,Chen-Huan Chen 한국보건의료인국가시험원 2020 보건의료교육평가 Vol.17 No.-

Purpose: Senior nursing and medical interns’ lack of familiarity and confidence with respect to practicing universal precaution for theprevention of occupational needlestick or sharp injuries may harm themselves. Trainees’ self-reported needlestick or sharp injury ratewas known to be especially high during the first 2 months of internship in Taiwan. This prospective cohort study aimed to assess the effect of newly developed virtual reality (VR) game, which uses Gagne’s learning model to improve universal precaution for needlestickor sharp injury prevention and decrease the rates of needle stick or sharp injuries in new-coming medical and nursing interns in Taiwan. Methods: From 2017 to 2019, the VR system was developed and applied in training of 59 new-coming nursing and 50 medical interns. Occupational needlestick or sharp injury prevention was sought to be achieved through a game of right and wrong choices for safe orunsafe universal precaution behaviors. Results: In comparison with medical interns, a higher proportion of nursing interns had past experiences of deep occupational needlestick or sharp injury. Before VR training, the familiarity and confidence for needlestick or sharp injury prevention were higher amongnursing interns than medical interns. Trainees with past experiences of deep needlestick or sharp injury exhibited better performanceon the accuracy rate and time needed to complete 20 decisions than those without past experiences in VR practice. All trainees showedan improved performance after VR training. A high proportion of trainees reported that the VR-based training significantly decreasedtheir anxiety about needlestick or sharp injury prevention. Conclusion: This self-developed VR game system using Gagne’s flow improved universal precaution for needlestick or sharp injuryprevention and reduced the needlestick or sharp injury rates in the first 2 months of nursing and medical internship.

Dreyfus scale-based feedback increased medical students’ satisfaction with the complex cluster part of a interviewing and physical examination course and improved skills readiness in Taiwan

Shiau-Shian Huang,Chia-Chang Huang,Ying-Ying Yang,Shuu-Jiun Wang,Boaz Shulruf,Chen-Huan Chen 한국보건의료인국가시험원 2019 보건의료교육평가 Vol.16 No.-

PurposeIn contrast to the core part of the clinical interviewing and physical examination (PE) skills course, corresponding to the basic, head-to-toe, and thoracic systems, learners need structured feedback in the cluster part of the course, which includes the gastrointestinal, neuromuscular, and musculoskeletal systems. This study evaluated the effects of using Dreyfus scale-based feedback, which has elements of continuous professional development, instead of Likert scale-based feedback in the cluster part of training in Taiwan. MethodsInstructors and final-year medical students in the 2015-2016 classes of National Yang-Ming University, Taiwan comprised the regular cohort, whereas those in the 2017-2018 classes formed the intervention cohort. In the intervention cohort, Dreyfus scale-based feedback, rather than Likert scale-based feedback, was used in the cluster part of the course. ResultsIn the cluster part of the course in the regular cohort, pre-trained standardized patients rated the class climate as poor, and students expressed low satisfaction with the instructors and course and low self-assessed readiness. In comparison with the regular cohort, improved end-of-course group objective structured clinical examination scores after the cluster part were noted in the intervention cohort. In other words, the implementation of Dreyfus scale-based feedback in the intervention cohort for the cluster part improved the deficit in this section of the course. ConclusionThe implementation of Dreyfus scale-based feedback helped instructors to create a good class climate in the cluster part of the clinical interviewing and PE skills course. Simultaneously, this new intervention achieved the goal of promoting medical students’ readiness for interviewing, PE, and self-directed learning.