교육, 훈련프로그램 반응평가를 위한 평가범주와 질문지 개발준거 = Dimension Classification and Questionnaire Design Criteria for Reaction Evaluation of Educational/Training Programs

다국어 초록 (Multilingual Abstract)

The purposes of the present study were to classify the dimensions of information sought using reaction evaluations and to establish design criteria for developing reaction questionnaires of educational/training program. There are major phases of the study: (1) classification of the dimensions and questionnaire design criteria for using reaction evaluations; (2) validation of the classified dimensions and design criteria of questionnaire by subject matter experts; and (3) assessment of a sample of educational/training reaction questionnaires currently used in educational/training centers of Korea. The findings of the study were as follows: eleven dimensions for reaction evaluation of educational/training programs were identified and classified by purposes. Five overall design criteria, each consisting of several sub-criteria, were judged important factors in designing questionnaires for reaction evaluation. The criteria include: introduction, directions, and closing statements; question format; question construction; questionnaire layout; and data analysis. It was concluded that a well-designed questionnaire for reaction evaluation of educational/training programs integrates the proper application of design criteria with appropriate reaction dimensions. Most questionnaires for reaction evaluation of educational/training programs used in educational/training centers of Korea consisted of questions representing only 3-5 dimensions. The questionnaires varied in form and length. Few of them poorly utilized the established questionnaire design criteria. With appropriate dimensions of reaction evaluation and design criteria for constructing questionnaire of educational/training program evaluation, program evaluation professionals can make reaction evaluations of program more useful and valuable in educational/training programs.

참고문헌 (Reference)

1 황정규, "학교학습과 교육평가(개정판)" 교육과학사 1998

2 강이철, "기업교육 프로그램 평가. 나일주, 임철일, 이인숙 (편저) (2003)" 학지사 225-249, 2003

3 정재삼, "교육프로그램평가" 교육과학사 2004

4 Easterby-Smith,M, "Turning course evaluation from an end to a means" 17 (17): 25-27, 1985

5 Richey,R.C, "Trends in instructional design:Emerging theory-based models" 8 (8): 96-110, 1995

6 McGehee, "Training in business and industry" 1961

7 Carnevale, "Training in America The organization and strategic role of training" Jossey-Bass 1990

8 Robinson, "Training for impact How to link training to business needs and measure the results" Jossey-Bass 1989

9 Ostroff,C, "Training effectiveness measures and scoring schemes:A comparison" 44 (44): 353-374, 1991

10 7th ed, "The practice of social research (7th ed.)." Wadsworth 1995

11 Holton,E.F.III, "The flawed four-level evaluation model" 7 (7): 5-21, 1996

12 Payne, "The art of asking questions" Princeton Princeton University Press 1951

13 Gordon,S.E, "Systematic training program design:Maximizing effectiveness and minimizing liability" PTR Prentice Hall 1994

14 Neuman, W. L., "Social research methods: Qualitative and quantitative approach" Allyn and Bacon 1997

15 March 24-26, "Research & theory to practice Paper presented at the International Society for Performance Improvement" 1998

16 McBean, "Questionnaire design for student measurement of teaching effectiveness" 273-288, 1982

17 Worthen, "Program evaluation: Alternative approaches and practical guidelines (2nd ed.)" Longman 1997

18 Dixon,N.M, "New routes to evaluation" 50 (50): 82-85, 1996

19 Jick,T.D, "Mixing qualitative and quantitative methods:Triangulation in action" 24 (24): 603-611, 1979

20 Hayes,B.E, "Measuring customer satisfaction:Development and use of questionnaire" ASQC Quality 1992

21 Beer, "Levels of evaluation" 335-345, 1986

22 & Janak, "Kirkpatrick‘s levels of training criteria" 331-342, 1989

23 Dyer,S, "Kirkpatrick's mirror" 18 (18): 31-32, 1994

24 & Ban, "Issues in training evaluation" 299-314, 1993

25 Thiagarajan,S, "Instructional product verification and revision:20 questions and 200 speculation" 26 (26): 133-141, 1978

26 Dolezalek,H, "Industry report 2004:Training magazine's 23rd annual comprehensive analysis of employer-sponsored training in the United States" 41 (41): 20-37, 2004

27 Phillips, J. J., "Handbook of training evaluation and measurement methods: Proven models and methods for evaluating any HRD program (3rd ed.)." Gulf 1997

28 Cayer, "Handbook of training and development for the public sector" Jossey-Bass 1993

29 In P, "Handbook of survey research" 1983195-230

30 Boyle, "Handbook of adult education" Macmillan 1970

31 Schouborg,G, "Flex:A flexible tool for continuously improving your evaluation of training effectiveness" HRD 1993

32 Warr, "Evaluation of management training" 1970

33 Kaufman, "Evaluation beyond Kirkpatrick" 35 (35): 8-12, 1995

34 Hamblin,A.C, "Evaluation and control of training" McGraw-Hill 1974

35 Phillips,S, "Evaluation" Library Association 1993

36 Kirkpatrick, D. L., "Evaluating training programs: The four levels (2nd ed.)." Berrett-Koehler 1998

37 Rosenberg, "Evaluating training programs for decision making" 57-74, 1987

38 Roberts,A, "Evaluating training programmes" 26 (26): 18-23, 1990

39 Mattoon,J.S, "Evaluating training and educational programs:A review of the literature" Air Force Material Command 1992

40 Keller,J.M, "Evaluating diversity training:17 ready-to-use tools" Pfeiffer 1996

41 Brown, "Evaluating corporate training: Models and issues" Kluwer Academic 1998

42 Forsyth, "Evaluating a course: Practical strategies for teacher, lecturers and trainers" 1999

43 Swisher,R, "Criteria for the design of mail questionnaire" 21 (21): 159-165, 1980

44 Cronbach,L.J, "Course improvement through evaluation" 64 (64): 673-683, 1963

45 Basarab,D.J, "Assessing the quality of training evaluation studies" 33 (33): 19-22, 1994

46 Weisberg, "An introduction to survey research, polling, and data analysis" Sage 1996

47 Sugrue, "American Society for Training & Development's 2004 state of the industry report American Society for Training & Development" American Society for Training & Development 2004

48 Fisk,C.N, "ASTD trainer's toolkit:Evaluation instruments" Pfeiffer 1991

49 & Root, "A practical approach to evaluating corporate training programs" 1992

50 Molenda, "A guide to human resource development" McGraw-Hill 266-293, 1996

51 Covert,R.W, "A checklist for developing questionnaires" 5 (5): 74-78, 1984