Development of design principles based on learning analytics for online collaborative learning in higher education

다국어 초록 (Multilingual Abstract)

Instructors in online collaborative learning (OCL) face the challenge of being unable to monitor and closely observe learners’ learning processes constantly. However, the online environment inevitably records all learner activities as log data. This data, when analyzed through learning analytics, can provide meaningful insights into learners’ collaboration patterns and interactions. Leveraging learning analytics enables personalized instructional interventions, facilitating the successful execution of collaborative learning. Within higher education, the evolution of learning analytics has occurred chiefly in conjunction with e-learning infrastructures; however, it faces challenges in gaining stakeholder trust and the incorporation of effective practices to improve teaching and learning methodologies. Although the technology behind learning analytics has significantly advanced, its application in real-world settings demonstrates a relatively low maturity. As a result, there is a strong need for guidelines that can support instructors with course design from a technology integration perspective.
This study addresses these challenges by developing principles for designing OCL grounded in learning analytics within higher education. It outlines two primary purposes: from a macro perspective, to broaden the knowledge base of OCL and learning analytics based on established educational theories, and, from a micro perspective, to formalize key concepts that link detailed tracking data and learning artifacts found in learner logs, allowing the implementation of specific design principles based on learning analytics for OCL in higher education. Therefore, this study’s research questions were aimed at developing such a design principle and assessing its effectiveness.
This study was conducted in several stages using Design and Development Research (DDR). It began with systematic reviews to find conceptual components, design principles, and guidelines relevant to OCL and learning analytics in higher education. Once determined, five experts in educational technology and higher education were involved in a rigorous validation process to establish their internal validity. The final conceptual components, design principles, and guidelines were developed over several rounds of the Delphi method and validation reviews, ensuring the robustness of the findings. Furthermore, formative assessment for external validation was conducted by collecting and analyzing feedback on a prototype designed to incorporate the principles and guidelines based on learning analytics for OCL. Participants included both higher education learners and instructional designers.
The findings identified four conceptual components and developed 15 design principles and 44 guidelines based on learning analytics for OCL in higher education. This study’s significance is found in its provision of design principles based on educational theories. The study includes both theoretical and practical implications in higher education and the design aspects of OCL from a learning analytics perspective, making significant contributions to the education field.

국문 초록 (Abstract)

온라인 협력학습에서 교수자가 학습 상황에 상주하며 학습자들의 과정을 면밀히 관찰하는 것에는 한계가 있다. 그러나 온라인 환경에서 학습자들의 모든 활동은 로그 데이터로 저장되며, 이 데이터는 학습분석을 통해 협업 패턴과 상호작용을 이해하는 데 중요한 통찰(insight)을 제공할 수 있다. 즉, 온라인 협력학습에서 학습분석을 활용하는 것은 개별화된 교수 개입이 가능해짐을 의미하고 이는 협력학습의 성공적인 수행을 지원한다.
그러나 고등교육에서 학습분석은 독립적인 시스템으로 존재하기보다는 기존의 e-러닝 인프라와 통합된 형태로 발전해 왔다. 이러한 고등교육 온라인 학습에 학습분석을 적용하는 것은 고등교육 이해관계자의 지지 기반이 미약하고, 교수-학습 과정 개선에 필요한 실천 및 연구가 부족하다는 문제가 존재한다. 이로 인해 학습분석 기술은 고도로 발전했으나 현장에서의 적용은 아직 초기 단계에 머물러 있다. 따라서 교수자의 수업 설계를 보조하고 지원할 수 있는 구체적인 설계지침이 요구된다.
본 연구는 고등교육에서 학습분석 기반의 온라인 협력학습 설계원리를 개발하는 것을 목적으로 한다. 이를 위해 교육학 이론을 중심으로 온라인 협력학습과 학습분석의 지식 기반을 확장하고자 하며, 학습자 로그(log)에서 발견되는 데이터와 학습 아티팩트(artifact)를 연결하는 핵심 개념을 탐구함으로써 구체적인 설계원리를 도출하고자 한다. 이에, 본 연구에서는 고등교육에서 학습분석 기반 온라인 협력학습 설계원리를 개발하고, 개발된 설계원리의 효과를 확인하는 것을 연구문제로 설정하였다.
연구문제를 탐구하기 위하여 사용한 방법론은 설계․개발 연구(Design and Development Research; DDR)를 기반으로 하였으며, 체계적 문헌 고찰을 통해 고등교육에서의 온라인 협력학습 및 학습분석의 개념적 구성요소, 설계원리, 및 설계지침을 도출하였다. 이후 교육공학 및 교육정책 전문가의 델파이 조사 및 내적 타당성 검토 과정을 거쳐 최종적인 개념적 구성요소, 설계원리 및 설계지침을 확정하였다. 또한, 외적 타당성을 확보하기 위하여 학습분석 기반 온라인 협력학습의 설계원리 및 설계지침을 종합하여 설계된 프로토타입에 대한 교수설계자와 고등교육 학습자의 반응을 조사하고 분석하였다.
연구 결과, 고등교육에서 학습분석 기반 온라인 협력학습의 개념적 구성요소는 총 네 가지로 분류되었으며, 15가지 설계원리와 44개의 설계지침이 개발되었다. 본 연구는 고등교육 맥락에서 교육학 이론에 기반하여 학습분석 기반 온라인 협력학습 설계원리를 제시하였다는 점에서 의의가 있다. 이는 고등교육 분야에서 온라인 협력학습과 학습분석의 설계에 관한 이론 및 실천적 함의를 가진다.

목차 (Table of Contents)

• Chapter 1. Introduction 1
• 1.1. Background of the Study 1
• 1.2. Purpose of the Study 5
• 1.3. Research Questions 11
• Chapter 2. Literature Review 13
• 2.1. Online Collaborative Learning in Higher Education 13
• 2.2. Learning Analytics in Online Collaborative Learning 39
• Chapter 3. Methodology 62
• 3.1. Research Design and Approach 62
• 3.2. Procedures 65
• 3.3. Development of Design Principles 67
• 3.4. Internal Validation 71
• 3.5. External Validation 85
• Chapter 4. Results 99
• 4.1. The Initial Design Principles 99
• 4.2. Internal Validation Results 109
• 4.3. External Validation Results 155
• 4.4. The Final Design Principles 192
• Chapter 5. Discussion 204
• 5.1. Design of Online Collaborative Learning Based on Learning Analytics in Higher Education 205
• 5.2. The Effectiveness of Online Collaborative Learning Based on Learning Analytics in Higher Education 211
• Chapter 6. Conclusion 215
• 6.1. Summary 215
• 6.2. Conclusion 217
• 6.3. Limitations and Recommendations 219
• References 221
• Appendices 269
• Abstract in Korean 324

참고문헌 (Reference)

1. Fostering Interaction in synchronous online class sessions with foreign language learners In R. E. Ferdig, E. Baumgartner, R. Hartshorne, R. Kaplan-Rakowski, C. Mouza, (Eds.) Teaching, technology, and teacher education during the COVID-19 pandemic Stories from the field, Bauer, E., pp. 174-178). Association for the Advancement of Computing in Education (AACE)., , 2020

2. External validity, Kikuta, K., Denly, M., Annual Review of Political Science, 24, 365-393, , 2021

3. The power of feedback, Timperley, H., 77, 81-112, , 2007

4. Towards usable E-health, Lopez, K. D., 8(2), 470-490, , 2017

5. Collaborative learning analytics In, A. F. Wise, Shum, S. B. Cress, Knight, S., C. Rosé, J. OshimaEds, International handbook of computer-supported collaborative learning (pp. 425-4`43 Cham, Switzerland Springer, , 2021

6. Why people prefer unequal societies, Sheskin, M., Bloom, P., 1(4), 1-7, , 2017

7. Total learning architecture 2019 report, Johnson, A., Hayden, T., Smith, B., Advanced Distributed Learning (ADL) Initiative, , 2020

8. A reference model for learning analytics, Dyckhoff, A. L., Thüs, H., Schroeder, U., 4(5-6), 318-331, , 2012

9. Developing an action model for WBI design, Chung, H. M., 17(2), 27-52, , 2001

10. A survey of value sensitive design methods, Borning, A., Hendry, D. G., Foundations and Trends® in Human– Computer Interaction, 11(2), 63-125, , 2017

11. Building an ontology of learning analytics, Sheridan, D., Gardner, L., In Proceedings Pacific Asia Conference on Information Systems (PACIS) 2018. Yokohama, Japan, , 2018

12. Exploratory analysis in learning analytics, De-Freitas, S., Technology, Knowledge and Learning, 21, 5-19, , 2016

13. Towards a micro-ecological approach to CSCL, Mercier, E., 14, 219-235, , 2019

14. Active learning Cooperation in the classroom, Johnson, D. W., 47, 29-30, , 2008

15. Design principles for industrie 4.0 scenarios, Pentek, T., Otto, B, pp. 3928-3937, , 2016

16. Smart technology for self-organizing processes, Bereiter, C., Smart Learning Environments, 1(1), 1-13, , 2014

17. Social presence Conceptualization and measurement, Weidlich, J., Xu, K., 34, 139-170, , 2021

18. Computer supported collaborative learning A review, Muukkonen, H., Rahikainen, M., Hakkarainen, K., Lipponen, L., Netherlands University of Nijmegen, , 1999

19. Scope statements Imperatives for evaluating theory, Cohen, B. P., 50(3), 288-301, , 1985

20. Infusing cooperative learning in distance education, Ivone, F. M., 24(1), 1-15, , 2020

21. Joining together Group theory and group skills Eds., Johnson, F., 4th Englewood Cliffs, NJ Prentice Hall, , 2017

22. Online teaching amidst COVID-19 in India An outlook, Bhaskar, P., Vinay, M., 15(2), 105-111, , 2020

23. Active learning Creating excitement in the classroom, Eison, J. A., ASHE-ERIC Higher Education Report, Washington, DC George Washington University, , 1991

24. Unravelling cooperative learning in higher education, Ang, C. S., Research in Social Sciences and Technology, 5(2), 22-39, , 2020

25. Guiding collaborative revision of science explanations, Kidron, A., Linn, M. C., 14(3), 291-324, , 2019

26. Use of qualitative methods to enhance content validity, May, B. A., Nelson, C. A., Nursing Research, 39(3), 172-175, , 1990

27. Penetrating the fog Analytics in learning and education, Long, P., EDUCAUSE Review, 46(5), 30-40, , 2011

28. Interactivity in multimedia learning An integrated model, Schwartz, R. N., Plass, J. L., 26(5), 1024-1033, , 2010

29. Augmenting learning analytics with multimodal sensory data, Worsley, M., 3(2), 213-219, , 2016

30. Cooperative learning principles enhance online interaction, Seow, P., 4(1), 28-38, , 2015

31. Analytics for learning design A layered framework and tools, Pardo, A., Rodríguez‐Triana, M. J., Martinez-Maldonado, R., Muñoz-Cristóbal, J. A., 50(1), 139-152, , 2019

32. Mapping the study of learning analytics in higher education, Sun, Y., Ordóñez, D. E., Pablos, P., Jiang, S., Zhang, X., 37(10-11), 1142-1155, , 2018

33. Predicting course outcomes with digital textbook usage data, Clem, C., The Internet and Higher Education, 27, 54-63, , 2015

34. Special issue on the current trends in e-learning assessment, Aljawarneh, S., Pamplona, S., 32, 1-8, , 2020

35. The role of learning theory in multimodal learning analytics, Cukurova, M., 54(5), 1246-1267, , 2023

36. Participation in online courses from the students perspective, Cakir, H., Interactive Learning Environments, 26(7), 924-942, , 2018

37. Using mobile devices to support online collaborative learning, Xhafa, F., Barolli, L., Mobile Information Systems, 6(1), 27-47, , 2010

38. Collaboration analytics —current state and potential futures, Dowell, N., Thompson, K., 8(1), 1-12, , 2021

39. Visions of CSCL Eight provocations for the future of the field, Schwarz, B. B., International Journal of Computer-Supported Collaborative Learning, 12, 423-467, , 2017

40. Supporting awareness in communities of learning design practice, Hernández-Leo, D., 85, 255-270, , 2018

41. The current landscape of learning analytics in higher education, Hatakka, M., Bälter, O., Mavroudi, A., 89, 98-110, , 2018

42. The privacy paradox and its implications for learning analytics, Gašević, D., Whitelock-Wainwright, A., In Proceedings of the tenth international conference on learning analytics knowledge (pp. 230-239, , 2020

43. Multimodal data fusion in learning analytics A systematic review, Cui, M., Huang, X., 20(23), 6856, , 2020

44. A comparison of learning analytics frameworks A systematic review, Slade, S., pp. 152-163, , 2022

45. From cognitive load theory to collaborative cognitive load theory, Kirschner, F., Zambrano R, J., Sweller, J., 13, 213-233, , 2018

46. Cooperative learning-applications for children from primary school, Pop, M., Plus Education, 22(1), 78-87, , 2019

47. Data analytics in higher education Key concerns and open questions, Jones, K., 11(1), 25-43, , 2017

48. Welcome to Jacksonville and the 2014 AECT International Convention, Dousay, T. A., Sat, 10, 9–15, , 2015

49. Cognitive automation as part of Deakin universitys digital strategy, Scheepers, R., Willcocks, L. P., MIS Quarterly Executive, 17(2), 89-107, , 2018

50. Design principles Literature review, analysis, and future directions, Wood, K. L., Yang, M. C., 138(10), 101103, , 2016

51. A conceptual framework linking learning design with learning analytics, Corrin, L., Gašević, D., De Barba, P., Lockyer, L., Kennedy, G., Mulder, R. …, pp. 329-338, , 2016

52. Challenges of implementing E-learning in developing countries A review, Khaing, S. S., pp. 405-411), , 2016

53. Covid-19 pandemic and online learning the challenges and opportunities, Soykan, E., 31(2), 863-875, , 2023

54. Far apart, yet close together Cooperative learning in online education, Jacobs, G. M., Renandya, W. A., Studies in English Language and Education, 7(2), 271-289, , 2020

55. Teachers' reflections on cooperative learning Issues of implementation, Boyle, M., Teaching and Teacher Education, 26(4), 933-940, , 2010

56. Collaborative learning in small groups in an online course-a case study, Rosenberg, I., Aasekjær, K., BMC Medical Education, 22(165), 1-10, , 2022

57. Role of systematic reviews and meta-analysis in evidence-based medicine, Seiler, C. M., 29, 582-587, , 2005

58. An empirical study on the effect of group awareness in CSCL environments, Fan, C., 31(1), 38-53, , 2023

59. Towards a method for design principle development in information systems, Otto, B., Guggenberger, T. M., In Proceeding 15th International Conference on Design Science Research in Information Systems and Technology, Kristiansand, Norway, , 2020

60. Using learning analytics to scale the provision of personalised feedback, Gašević, D., Mirriahi, N., Jovanovic, J., Dawson, S., 50(1), 128-138, , 2019

61. Changing course Ten years of tracking online education in the United States, Seaman, J., Babson Park, MA Babson Survey Research Group and Quahog Research Group, , 2013

62. Effects of role assignment in concept mapping mediated small group learning, Mercer, N., Wang, M., 23, 27-38, , 2014

63. Blended learning Uncovering its transformative potential in higher education, Kanuka, H., 7(2), 95-105, , 2004

64. Group awareness increases student engagement in online collaborative writing, Liu, L., Liu, L., 38, 1-8, , 2018

65. Translating learning into numbers A generic framework for learning analytics, Drachsler, H., 15(3), 42-57, , 2012

66. Informing pedagogical action Aligning learning analytics with learning design, Heathcote, E., Dawson, S., 57(10), 1439-1459, , 2013

67. Scaffolding and supporting use of information for ambitious learning practices, Hmelo-Silver, C. E., 120(1/2), 39-58, , 2019

68. An integrated framework for course adapted student learning analytics dashboard, Aslam, M. A., Daud, A., Alowibdi, J. S., Abbasi, R. A., Basheri, M., 92, 679-690, , 2019

69. Learning and teaching with technology in higher education—A systematic review, Børte, K., Nesje, K., Ruud, E., Oslo Knowledge Centre for Education, 2, 40-64, , 2018

70. Predictive learning analytics using deep learning model in MOOCs courses videos, Ahmed, S. A., Cao, H., Education and Information Technologies, 26(1), 371-392, , 2021

71. Supporting collaborative design activity in a multi-user digital design ecology, Thompson, K., Dimitriadis, Y., Hernández-Leo, D., Goodyear, P., Prieto, L., Wardak, D., Carvalho, L., 71, 327-342, , 2017

72. Design principles for learning analytics information systems in higher education, Tuunanen, T., Gardner, L., Sheridan, D., 30(5), 541-568, , 2021

73. Novel CSCL design processes prompted by exploration of role-playing and emotions, Rose, C. P., International Journal of Computer-Supported Collaborative Learning, 17(4), 457-461, , 2022

74. Designing the online learning experience Evidence-based principles and strategies, Howles, L., New York Taylor Francis, , 2023

75. Online cooperative learning for communication and team working skills enhancement, Aris, B, Yusof, F. M., 69(1), 137-143, , 2014

76. Values Cultural and individual Fundamental questions in cross-cultural psychology, In F. J. R. van de Vijver, A. Chasiotis, & S. M. Breugelmans (Eds., pp. 463-493, , 2011

77. Attention-based recurrent convolutional neural network for automatic essay scoring, Zhang, Y., Yang, J., In Proceedings of the 21st conference on computational natural language learning (CoNLL 2017pp. 153-162, , 2017

78. BiERU Bidirectional emotional recurrent unit for conversational sentiment analysis, Ji, S., Cambria, E., Shao, W., 467, 73-82, , 2022

79. Collaborative learning Improving public deliberation in ecosystem-based management, Walker, G. B., 16(2), 71-102, , 1996

80. Knowledge-based design analytics for authoring courses with smart learning content, Barria-Pineda, J., Hernández-Leo, D., Brusilovsky, P., 32(1), 4-27, , 2022

81. Collaborative group engagement in a computer-supported inquiry learning environment, Adams-Wiggins, K. R., Rogat, T., Hmelo-Silver, C. E., International Journal of Computer-Supported Collaborative Learning, 10, 273–307, , 2015

82. Evaluation of service-learning in project WeCan under COVID-19 in a chinese context, Dou, D., Shek, D., 18(7), 3596, , 2021

83. Motivation in Collaborative Groups International handbook of collaborative learning, In C. Hmelo-Silver, C. Chinn, C. Chan, & A. O’Donnell (Eds, pp. 250-267, , 2013

84. Development and evaluation of a virtual campus on second life The case of second DMI, Francese, R., Passero, I., Tortora, G., 52(1), 220-233, , 2009

85. License to evaluate Preparing learning analytics dashboards for educational practice, Scheffel, M., Drachsler, H., Specht, M., pp. 31-40, , 2018

86. Systematic reviews in educational research Methodology, perspectives and application, Bedenlier, S., Kerres, M., Bond, M., Buntins, K., Germany Springer Nature, , 2020

87. Consulting the oracle Ten lessons from using the Delphi technique in nursing research, McKenna, H., Hasson, F., 53(2), 205-212, , 2006

88. Effects of instructional conditions and experience on the adoption of a learning tool, Joksimović, S., Dawson, S., Mirriahi, N., 67, 207-220, , 2017

89. Critical inquiry in a text-based environment Computer conferencing in higher education, Archer, W., Anderson, T., 2, 87-105, , 2000

90. The effects of the regulated learning-supported flipped classroom on student performance, Park, J., Kim, H. S., Park, S. W., 34(1), 132-153, , 2022

91. Analysis of potential educational areas of Pokémon Go! game use based on mini FGI method, Kot, M., 15(3), 33-50, , 2023

92. Speech acts and discourse pattern in information sharing practice in Malaysian web-forums, Thayalan, X., Suppiah, P., In Proceedings The 2nd Annual International Conference on Language and Literature (AICLL 2019) (pp. 888-901) Medan, Indonesia KnE Social Sciences, , 2019

93. Transportability from multiple environments with limited experiments Completeness results, Pearl, J., In Proceedings Advances in Neural Information Processing Systems (NIPS 2014). Montréal, Canada, , 2014

94. Multimodal data capabilities for learning What can multimodal data tell us about learning?, Giannakos, M., 51(5), 1450-1484, , 2020

95. Digital transformation in higher education-The role of enterprise architectures and portals, Lehmann, H., In Proceedings Digital Enterprise Computing 2017 (DEC 2017)pp. 49-60, , 2017

96. From posts to patterns A metric to characterize discussion board activity in online courses, Lawrence, B, 13(2), 15-32, , 2009

97. Exploring embedded guidance and self-efficacy in educational multi-user virtual environments, Ketelhut, D. J., 3, 413-427, , 2008

98. Predicting success, preventing failure Utilizing learning analytics to support study success, In D. Ifenthaler, D. K. Mah, & J. Y. K. Yau (Eds., Warschauer, M., pp. 249-273, , 2019

99. Bridging the gap from knowledge to action Putting analytics in the hands of academic advisors, Teasley, S., Krumm, A., Waddington, R., In Proceedings of the Second International Conference on Learning Analytics and knowledge Conference (pp. 184-187)., , 2012

100. Is there a right way? Productive patterns of interaction during collaborative problem solving, Burnheimer, E., Franke, M. L., Zimmerman, J., Johnson, N. C., Ing, M., 11(5), 214, , 2021

101. Robots and agents to support collaborative learning In U. Cress C. Rosé A. Wise J. OshimaEds, Clarke, S. N., International Handbook of Computer-Supported Learning (pp. 407-424) Cham, Switzerland Springer, , 2021

102. Utilising learning analytics to support study success in higher education A systematic review, Yau, J. Y. K., Educational Technology Research and Development, 68(4), 1961-1990, , 2020

103. Youth, technology, and DIY Developing participatory competencies in creative media production, Peppler, K. A., 35(1), 89-119, , 2011

104. Collaborative serious game as a sociable computer-supported collaborative learning environment, Hämäläinen, R., In Proceedings of the 6th European conference on game based learning (pp. 364-371). UK Academic Publishing International, , 2012

105. Piecing the learning analytics puzzle A consolidated model of a field of research and practice, Kovanović, V., Joksimović, S., Learning Research and Practice, 3(1), 63-78, , 2017

106. Sharks are not the threat humans are Argument component segmentation in school student essays, Ghosh, D., arXiv preprint arXiv, Retrieved April 3, 2024 from https//arxiv. org/abs/2103.04518, , 2021

107. The core features of CSCL Social situation, collaborative knowledge processes and their design, Ludvigsen, S., Stahl, G., Law, N., 10, 109-116, , 2015

108. The role of multi-attributional student diversity in computer-supported collaborative learning, Raimann, J., Reich-Stiebert, N., Stürmer, S., 55, 100868, , 2022

109. Tools and resources for setting up collaborative spaces In U. C. Rosé A. F. Wise J. OshimaEds, Dimitriadis, Y. Cress, International handbook of computer-supported learning (pp. 445-460 Cham, Switzerland Springer, , 2021

110. Distance dynamics based overlapping semantic community detection for node‐attributed networks, Huang, R., Huang, J., Jia, X., Wang, P., Wang, C., 37(2), 678-700, , 2021

111. Nine elements for robust collaborative learning analytics A constructive collaborative critique, Järvelä, S., Rosé, C., 18(1), 1-9, , 2023

112. Scaling up learning design impact of learning design activities on LMS behavior and performance, Toetenel, L., Bryan, A., In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (pp. 315-319, , 2015

113. Peer feedback to support collaborative knowledge improvement What kind of feedback feed-forward?, Chen, W., Computers Education, 187, 104467, , 2022

114. Social presence, constructs of sociability and social space in computer-mediated online learning, Ekalestari, S., Suhendi, A., Shanthi, A., Mustapha, Z. W., In European Proceedings of Educational Sciences EpES (pp. 167-177) European Publisher, , 2023

115. The NISPI framework Analysing collaborative problem-solving from students' physical interactions, Millán, E., Mavrikis, M., Luckin, R., 116, 93-109, , 2018

116. Visual learning analytics of educational data A systematic literature review and research agenda, Parsons, P., Byrd, V., Computers Education, 122, 119-135, , 2018

117. Deep learning model for home automation and energy reduction in a smart home environment platform, Castiglione, A., Pop, F., Serbanescu, C., Neural Computing and Applications, 31, 1317-1337, , 2019

118. Design and development research Handbook of research on educational communications and technology, In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds, pp. 141-150, , 2014

119. Sustainability and Scalability of CSCL Innovations In U. Cress C. Rosé A. F. Wise J J. OshimaEds, Peppler, K., Zhang, J., International handbook of computer-supported seriesVol. 19 Cham Springer, , 2021

120. CC–LR Providing interactive, challenging and attractive collaborative complex learning resources, Gañán, D., Prieto, J., Mora, N., Feidakis, M., Daradoumis, T., Conesa, J., 30(1), 51-67, , 2014

121. Predictive learning analytics at scale Guidelines to successful implementation in higher education, Boroowa, A., Verdin, B., Rienties, B., 6(1), 85-95, , 2019

122. Strategic visualization the (real) usefulness of cognitive mapping in smart city conceptualization, Ferreira, F. A., Pereira, L. F., Banaitis, A., Correia, R. J., Management Decision, 60(4), 916-939, , 2022

123. MORF A framework for predictive modeling and replication at scale with privacy-restricted MOOC data, Baker, R. S., Brooks, C., Andres, J. M., In Proceedings 2018 IEEE International Conference on Big Data (Big Data) (pp. 3235-3244)., , 2018

124. Multimodal learning analytics of collaborative patterns during pair programming in higher education, Wu, Y., Ouyang, F., 20(8), 1-20, , 2023

125. What information should CSCL teacher dashboards provide to help teachers interpret CSCL situations?, Rummel, N., van-Gog, T., 14(3), 261-289, , 2019

126. Developmental research methods creating knowledge from instructional design and development practice, Klein, J. D., 16(2), 23-38, , 2005

127. Effects of online cooperative learning on students problem-solving ability and learning satisfaction, Wu, T. J., Frontiers in Psychology, 13, 817968, , 2022

128. Multimodal teaching analytics Automated extraction of orchestration graphs from wearable sensor data, Kidzinski, Ł., Sharma, K., Dillenbourg, P., Rodríguez‐Triana, M. J., 34(2), 193-203, , 2018

129. The effect of aptitude treatment interaction learning model on mathematical creative thinking skills, Miranti, I. P., Kusuma, J. W., Gunawan, G., Setyaningsih, E., 2(4), 532-541, , 2023

130. Visualization of agreement and discussion processes during computer-supported collaborative learning, Kanselaar, G., Erkens, G., 23(3), 1105-1125, , 2007

131. Applying evidence-centered design for the development of game-based assessments in physics playground, Shute, V. J., Almond, R. G., 16(2), 142-163, , 2016

132. Innovative ways for using study to orchestrate and research social aspects of self-regulated learning, Winne, P. H., Gress, C. L., Oshige, M., 26(5), 794-805, , 2010

133. Effects of group awareness support in CSCL on students learning performance A three-level meta-analysis, Zhu, Z., Luo, H., Lin, Y., Zhang, Y., Ma, J., 19(1), 97-129, , 2024

134. Design for networked learning Framing relations between participants activities and the physical setting, Carvalho, L., Dohn, N. B., In Proceedings of the 9th international conference on networked learning. Edinburgh Networked Learning Conference Organiser, , 2014

135. Student attitudes toward learning analytics in higher education The fitbit version of the learning world, Gibson, D. C., Seaman, K., Howell, J. A., Frontiers in Psychology, 7, 228974, , 2016

136. Cooperative team learning and the development of social skills in higher education The variables involved, Polo-del-Río, M. I., Felipe-Castaño, E., León-del-Barco, B., Iglesias-Gallego, D., 9, 374188, , 2018

137. In search for the most informative data for feedback generation Learning analytics in a data-rich context, Giesbers, B, Rienties, B., 47, 157-167, , 2015

138. Facilitating students motivation in the basic communication course A self-determination theory perspective, Frey, T. K., Kaufmann, R., Basic Communication Course Annual, 32(9), 124-147, , 2020

139. The relationships between social participatory roles and cognitive engagement levels in online discussions, Chang, Y. H., 50(3), 1396-1414, , 2019

140. A collaborative game-based learning approach to improving students' learning performance in science courses, Hwang, G. J., Computers Education, 63, 43-51, , 2013

141. Connecting the dots An exploratory study on learning analytics adoption factors, experience, and priorities, Kovanović, V., Gašević, D., 50, 100794, , 2021

142. A pedagogical perspective on big data and learning analytics A conceptual model for digital learning support, Rietsche, R., Meier, C., Soellner, M., 24, 599-619, , 2019

143. Digital transformation in German higher education Student and teacher perceptions and usage of digital media, Dolch, C., Marín, V. I., Bedenlier, S., Zawacki-Richter, O., 15(1), 1-20, , 2018

144. Improving learning and writing outcomes Influence of cognitive and behavioral group awareness tools in Wikis, Heimbuch, S., Bodemer, D., International Journal of Computer-Supported Collaborative Learning, 16(2), 225-259, , 2021

145. Knowledge convergence in computer-supported collaborative learning The role of external representation tools, Mandl, H., 14(3), 405-441, , 2005

146. Motivation and cognitive engagement in learning environments The Cambridge handbook of the learning sciences, In R. K. Sawyer (Eds., pp. 475-488, , 2006

147. Perspectives on the challenges of generalizability, transparency and ethics in predictive learning analytics, Ramaswami, G., Susnjak, T., Barczak, A., Computers and Education Open, 2, 100060, , 2021

148. Computer-supported collaborative learning. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences, Koschmann T., Suthers D. D., 2nd Ed., pp. 479-500 New York Cambridge University Press, , 2014

149. Individual and group-based learning from complex cognitive tasks Effects on retention and transfer efficiency, Kirschner, P. A., Paas, F., 25(2), 306-314, , 2009

150. Supporting classroom orchestration with real-time feedback A role for teacher dashboards and real-time agents, Slotta, J., 14(3), 325-351, , 2019

151. What makes an online problem-based group successful? A learning analytics study using social network analysis, Nouri, J., Malmberg, J., Vartiainen, H., BMC Medical Education, 20, 1-11, , 2020

152. A review of empirical studies on MOOC adoption Applying the unified theory of acceptance and use of technology, Jung, I., Kobayashi, S., 11(1), 15-24, , 2017

153. The effects of a group awareness tool on knowledge construction in computer‐supported collaborative learning, Zhang, Y., Li, X., Li, X., 52(3), 1178-1196, , 2021

154. The impact of two types of peer assessment on students' performance and satisfaction within a Wiki environment, Lucking, R., 11(3-4), 186-193, , 2008

155. A systematic review of empirical studies on learning analytics dashboards A self-regulated learning perspective, Gašević, D., Pardo, A., IEEE Transactions on Learning Technologies, 13(2), 226-245, , 2019

156. Clustering blended learning courses by online behavior data A case study in a Korean higher education institute, Jo, I. H., Yu, J. H., 29, 1-11, , 2016

157. Computer-supported collaborative learning An historical perspective Cambridge handbook of the learning sciences, Suthers, D. D., In R. K. Sawyer (Ed, Koschmann, T., pp. 409-426 Cambridge Cambridge University Press, , 2006

158. Explaining authors contribution to pivotal artifacts during mass collaboration in the Wikipedias knowledge base, Cress, U., Moskaliuk, J., Kimmerle, J., 9, 97-115, , 2014

159. Making sense of change management A complete guide to the models, tools and techniques of organizational change, Green, M., New York Kogan Page Publishers, , 2019

160. The effect of online cooperative learning on students' writing skills and attitudes through telegram application, Adloo, M., 11(3), 433-448, , 2018

161. The role of collaboration, computer use, learning environments, and supporting strategies in CSCL A meta-analysis, Tsai, C. C., Wang, M., Kirschner, P. A., 88(6), 799-843, , 2018

162. The COVID-19 pandemic and E-learning Challenges and opportunities from the perspective of students and instructors, Aljawarneh, S., Elberkawi, E. K., Alharbi, H., Rashaideh, H., 34(1), 21-38, , 2022

163. A large-scale implementation of predictive learning analytics in higher education The teachers role and perspective, Boroowa, A., Hlosta, M., Rienties, B., Zdrahal, Z., 67, 1273-1306, , 2019

164. Learning analytics for learning design A systematic literature review of analytics-driven design to enhance earning, Giannakos, M., 12(4), 516-534, , 2018

165. Supporting computer experts' adaptation to the client's knowledge in asynchronous communication The assessment tool, Wittwer, J., Renkl, A, In Proceedings of Eurocogsci 03 (pp. 247-252, , 2019

166. Productivity in online CSCL groups A Rasch analysis approach to the preliminary validation of the Productivity Scale, Weidlich, J, Henderikx, M. A., In 16th International Technology, Education and Development Conference (pp. 2265-2274), , 2022

167. Technology supports in CSCL In Proceedings of the 10th International Conference of the Learning Sciences (ICLS 2012), In J. van Aalst, K. Thompson, M. J. Jacobson & P. Reimann (Eds, pp. 339-346, , 2012

168. User experience (UX) design elements for design of smart phone learning experience based on embodied cognition theory, Kim, S. Y., 23(2), 281-313, , 2017

169. User-centered/user experience UC/UX design thinking approach for designing a university information management system, Adebayo, A. O., Kuyoro, S. O., Priscilla, E. A., Amanze, R. C., 27(4), 577-590, , 2022

170. Video conference program enhances social support, loneliness, and depressive status of elderly nursing home residents, Chang, Y. C., Tsai, Y. F., Wang, H. H., Chu, H. H., 14(8), 947-954, , 2010

171. Expanding horizons for the future with an open educational model for complex thinking External and internal validation, Portuguez-Castro, M., On the Horizon The International Journal of Learning Futures, , 2024

172. Analytics from an institutional lens Definition, theory, design and impact Learning analytics From Research to practice, In J. A. Larusson & B. White (Eds, pp. 79-102, , 2014

173. Learner interface interaction in distance education An extension of contemporary models and strategies for practitioners, Willis, D. J., Gunawardena, C. N., 8(2), 30-42, , 1994

174. What do you recommend? Implementation and analyses of collaborative information filtering of web resources for education, Walker, A., Lawless, K., 31, 299-316, , 2003

175. Multimodal learning analytics and education data mining Using computational technologies to measure complex learning tasks, Worsley, M., 3(2), 220-238, , 2016

176. Self-regulated learning strategies academic achievement in online higher education learning environments A systematic review, Poon, W. L., 27, 1-13, , 2015

177. The technical framework of interactive functions for course‑management systems Students perceptions, uses, and evaluations, Peng, H., Chang, C. Y., Computers Education, 55(3), 1004-1017, , 2010

178. Supporting adaptive learning pathways through the use of learning analytics Developments, challenges and future opportunities, Giannakos, M., Krogstie, J., 26(2), 206-220, , 2018

179. Mass collaboration as an emerging paradigm for education? Theories, cases, and research methods Mass collaboration and education, In U. Cress, J. Moskaliuk, & H. Jeong (Eds, pp. 3-27, , 2016

180. Seven affordances of computer-supported collaborative learning How to support collaborative learning? How can technologies help?, Hmelo-Silver, C. E., 51(2), 247-265, , 2016

181. Identifying the pitfalls for social interaction in computer-supported collaborative learning environments A review of the research, Kirschner, P. A., Jochems, W., 19(3), 335-353, , 2003

182. The impact of learning design on student behaviour, satisfaction and performance A cross-institutional comparison across 151 modules, Toetenel, L., 60, 333-341, , 2016

183. Theories of CSCL In U. Cress, C. Rose, A. F. Wise J. Oshima (Eds. International handbook of computer-supported collaborative learning, Hakkarainen, K., pp. 23-43 Cham, Switzerland Springer, , 2021

184. Overcoming challenges to the adoption of learning analytics at the practitioner level A critical analysis of 18 learning analytics frameworks, Mørch, A. I., Kluge, A., 66(3), 367-381, , 2022

185. An investigation of learners collaborative knowledge construction performances and behavior patterns in an augmented reality simulation system, Li, N., Wang, H. Y., Tsai, C. C., Duh, H. B. L., Computers Education, 68, 314-321, , 2013

186. Flipping the flipped classroom A study of the effectiveness of video lectures versus constructivist exploration using tangible user interfaces, Blikstein, P., IEEE Transactions on Learning Technologies, 9(1), 5-17, , 2015

187. Comparison of mathematics learning results with the use of WhatsApp group platform and zoom webinars in distance learning during Covid 19 pandemic, Hamidah, H., 5(1), 97-106, , 2020

188. Assessing students use of evidence and organization in response-to-text writing Using natural language processing for rubric-based automated scoring, Correnti, R., Wang, E., Matsumura, L. C., Litman, D., 27(4), 694-728, , 2017

189. Linking learning behavior analytics and learning science concepts Designing a learning analytics dashboard for feedback to support learning regulation, Järvelä, S., Malmberg, J., Kirschner, P. A., Verbert, K., 107, 105512, , 2020

190. Applying multimodal learning analytics to examine the immediate and delayed effects of instructor scaffoldings on small groups collaborative programming, Dai, X., Chen, S., 9(45), 1-21, , 2022

191. Knowledge building and the quantity, content and quality of the interaction and participation of students in an online collaborative learning environment, Usluel, Y. K., Computers Education, 97, 31-48, , 2016

192. Design and implementation of a cooperative learning system for digital content design curriculum Investigation on learning effectiveness and social presence, Hu, M. H., Chang, T. S., Hsiao, W. H., 11(4), 94-107, , 2012

193. Supporting the instructional videos with chatbot and peer feedback mechanisms in online learning The effects on learning performance and intrinsic motivation, Gencel, N., 60(7), 1716-1741, , 2022

194. Historical, contemporary, and future perspectives on self-regulated learning and performance J. A. Greene (Eds Handbook of self-regulation of learning and performance, In D. H. Schunk, Greene, J. A, pp. 1-15, , 2018

195. A comparison of the cooperative learning and traditional learning methods in theory classes on nursing students' communication skill with patients at clinical settings, Rezaei, K., Koohestani, H. R., 31(8), 877-882, , 2011

196. A brief history of mass collaboration How innovations over time have enabled people to work together more effectively J. Moskaliuk, H. Jeong (Eds. Mass collaboration and education, In U. Cress, pp. 31-42, , 2016

197. Using chatGPT for research report design A collaborative learning experience with students and professors in Honduras In Interactive Mobile Communication, Technologies and Learning, Rivera-García, M., pp. 224-231 Cham Springer Nature Switzerland, , 2023

198. Facilitating individual construction of knowledge in an online community of learning and inquiry through concept maps. In R. Marriott P. Torres (Eds.), Handbook of research on collaborative learning using concept mapping, Desnoyers, C. A, pp. 100-119, , 2009

199. Systematic reviews in educational research Methodology, perspectives and application In O. Zawacki-Richter M. Kerres S. Bedenlier M. Bond, K. BuntinsEds Systematic reviews in educational research Methodology, perspectives and application, Gough, D., pp. 3-22 Germany Springer Nature, , 2020