Automated assessment system based on function execution log for physical computing education : 피지컬 컴퓨팅 교육을 위한 함수 실행 로그 기반 자동 채점 시스템

다국어 초록 (Multilingual Abstract)

Recently, automated assessment systems have been widely used in various computer science lectures, and these systems are mainly developed for languages with a console-based practice environment. On the other hand, embedded systems such as physical computing are executed in hardware-based environment and their verification is conducted by manually checking the operation of the hardware. Therefore, it is difficult to automatically assess the physical computing system developed by students. In the embedded system, hardware and software have to be developed at the same time, and it increases more efforts and time for the instructors to teach students, especially in exercise based learning. This thesis proposes an physical computing exercise assessment system based on the function execution log in virtual execution environment. It performs both Fritzing based hardware configuration checking and source code testing based on virtual execution environment, where hardware operations are replaced by mock-up functions. In addition, more diverse exercises are possible by providing an experimental environment where students can freely enter input data and check the results. A case study shows that the developed system can be applied to the lectures and help reduce instructors’ work and increase students’ understanding through the system.

국문 초록 (Abstract)

최근 각종 컴퓨터 공학 분야 강의에 자동 채점 시스템이 널리 활용되고 있으며, 이러한
시스템은 주로 콘솔 기반의 실습 환경을 갖춘 언어 위주로 개발된다. 반면, 피지컬 컴퓨팅
과 같은 임베디드 시스템은 하드웨어 기반 환경에서 실행되며 하드웨어의 동작을 수동으로
확인하는 방식으로 검증이 이루어진다. 따라서 학생들이 개발한 피지컬 컴퓨팅 시스템을
자동으로 채점하는 것은 쉽지 않다. 임베디드 시스템에서는 하드웨어와 소프트웨어가 동시
에 개발되어야 하며, 특히 실습 기반 강의에서는 강사가 학생들을 가르치는 데 더 많은 노
력과 시간이 요구된다. 본 논문에서는 가상 실행 환경에서 함수 실행 로그를 기반으로 한 피지컬 컴퓨팅 실습 채
점 시스템을 제안한다. 제안하는 시스템에서는 Fritzing 기반의 하드웨어 구성 확인과 하
드웨어 동작을 Mock-up 기능으로 대체하는 가상 실행 환경 기반의 소스 코드 테스트를
수행한다. 또한, 학생들이 자유롭게 입력 데이터를 입력하고 결과를 확인할 수 있는 실험
환경을 제공함으로써 보다 다양한 실습이 가능하다. 실제 강의에 개발된 시스템을 적용하
여, 시스템을 통해 강사의 업무를 줄이고 학생들의 이해도를 높일 수 있음을 보였다.

목차 (Table of Contents)

• Abstract 1
• Chapter 1. Introduction 2
• Chapter 2. Related Works and Background 7
• 2.1 Physical computing 7
• 2.2 Automated assessment system 10
• 2.3 Automated assessment system for physical computing 14
• 2.4 TTCN-3 19
• 2.5 Fritzing 23
• Chapter 3. Automated assessment system for physical computing 25
• 3.1 Overview of automated assessment system for physical computing
• 26
• 3.2 Function execution log 28
• 3.2.1. Definition of function execution log · 28
• 3.2.2. Virtual execution environment 29
• 3.2.3. Example of function execution log 34
• 3.3 Assessment of physical computing exercise 37
• 3.3.1. Assessment of physical computing exercise(instructor) 37
• 3.3.2. Assessment of physical computing exercise(student) · 39
• 3.4 Exercise execution environment 42
• 3.5 Fritzing based hardware configuration · 45
• Chapter 4. Assessment method 48
• 4.1 Assessment procedure · 48
• 4.2 Hardware check condition 50
• 4.3 Test case description 54
• 4.4 SW assessment procedure 60
• 4.4.1. Log abstraction 61
• 4.4.2. Events ordering assessment · 62
• 4.4.3. Constraints assessment 64
• 4.5 Plagiarism · 65
• 4.6 Assessment result and feedback · 67
• Chapter 5. Evaluation · 69
• 5.1 Experiment composition and methodology · 71
• 5.1.1. I/O implementation in automated assessment systeme 72
• 5.1.2. Test the accuracy of the assessment function · 75
• 5.2 Common features of automated assessment system for physical
• computing · 78
• 5.2.1. Configuration of test case 78
• 5.2.2. Automated assessment 81
• 5.2.3. Check the assessment result and submission history 83
• 5.2.4. Case study 85
• 5.3 Analysis of submission data 87
• Chapter 6. Conclusion 93
• References 96
• 초록 105

참고문헌 (Reference)

1. 2, DOMjudge, https://www. domjudge. org in, , 2023

2. 4, Aizu Online Judge, https://judge. u-aizu. ac. jp/onlinejudge/Accessed in, , 2023

3. 54, Tinkdercad, https://www. tinkercad. com/Accessed in, , 2023

4. 55, Wowki, https://wokwi. com/Accessed in, , 2023

5. 21 Firitzing website, Fritzing, https://fritzing. org/Accessed in, , 2023

6. Automatic grading software for 2D CAD files, Bryan JA, Computer Applications in Engineering Education 28(1): 51–61, , 2020

7. Automated assessment of UML activity diagrams In, GOEDICKE, Michael, STRIEWE, Michael, Proceedings of the 2014 conference on Innovation & technology in computer science education p. 336-336, , 2014

8. Testing embedded real time systems with TTCN-3 In, GROSSMANN, Juergen, SERBANESCU, Diana, SCHIEFERDECKER, Ina, 2009 International Conference on Software Testing Verification and Validation. IEEE p. 81-90, , 2009

9. CloudCoder: A web-based programming exercise system, Spacco J., Hovemeyer D, Journal of Computing Sciences in Colleges 28(3): 30, , 2013

10. Continuous TTCN-3: testing of embedded control systems, BRINGMANN, Eckard, SCHIEFERDECKER, Ina, GROßMANN, Jürgen, In Proceedings of the 2006 international workshop on Software engineering for automotive systems p. 29-36, , 2006

11. A survey on online judge systems and their applications, Badura J, Antczak M, MichWasik S, Laskowski A, Sternal T., ACM Computing Surveys (CSUR) 51(1): 1–34, , 2018

12. SQL tester: an online SQL assessment tool and its impact, KLEEREKOPER, Anthony, SCHOFIELD, Andrew, In Proceedings of the 23rd annual ACM conference on innovation and technology in computer science education p. 87-92, , 2018

13. Data Analysis of Online Judge System-Based Teaching Model, ZHANG, Yuting, et al, International Conference on Computer Science and Education. Singapore: Springer Nature Singapore p. 531-543, , 2022

14. V-REP A versatile and scalable robot simulation framework, ROHMER, Eric, FREESE, Marc, SINGH, Surya PN, In 2013 IEEE/RSJ international conference on intelligent robots and systems. IEEE p. 1321-1326, , 2013

15. Web-CAT: automatically grading programming assignments In, EDWARDS, Stephen H., PEREZ-QUINONES, Manuel A., Proceedings of the 13th annual conference on Innovation and technology in computer science education p. 328-328, , 2008

16. Codeflex: A web-based platform for competitive programming, Brito M, Goncalves C., In IEEE 1–6, , 2019

17. Open-source multipurpose remote laboratory for IoT education, Pirrone D, Fornaro C, Assante D., In IEEE 1462–1468, , 2021

18. An evaluation of model-based testing in embedded applications, Schlingloff H., Weißleder S, In IEEE 223–232, , 2014

19. Who Judges the Judge: An Empirical Study on Online Judge Tests, LIU, Kaibo, et al, In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 2023, , 2023

20. Building a comprehensive automated programming assessment system, MEKTEROVIĆ, Igor, et al, IEEE Access 8: 81154-81172, , 2020

21. The DOMJudge based online judge system with plagiarism detection, PHAM, Minh Tuan, NGUYEN, Tan Bao, In 2019 IEEE-RIVF International Conference on Computing and Communication Technologies (RIVF). IEEE p. 1-6, , 2019

22. An introduction to the testing and test control notation (TTCN-3), GRABOWSKI, Jens, et al, Computer Networks 42.3: 375-403, , 2003

23. Cloud-based linux kernel practice environment and judgment system, Kim Y. Clik, Park H, Computer Applications in Engineering Education 28(5): 1137– 1153, , 2020

24. Towards a workflow for model-based testing of embedded systems In, AFZAL, Wasif, ZAFAR, Muhammad Nouman, ENOIU, Eduard, Proceedings of the 12th International Workshop on Automating TEST Case Design, Selection, and Evaluation. 2021. p. 33-40, , 2021

25. Conceptual/functional Co-simulation technique for embedded systems, AYED, Mossaad Ben, ABID, Mohamed, SALAH, Yosri Ben, In 2019 International Conference on Computer and Information Sciences (ICCIS). IEEE p. 1-5, , 2019

26. Design of Broadband PLC Conformance Testing System Based on TTCN-3, LIU, Xuan, ZHENG, Ke., ZHANG, Hailong, In 2018 International Conference on Information Systems and Computer Aided Education (ICISCAE). IEEE p. 35-41, , 2018

27. Model-based testing for software safety: a systematic mapping study, TEKINERDOGAN, Bedir, GURBUZ, Havva Gulay, 26: 1327-1372, , 2018

28. Online judge MySQL for learning process of database practice course, Sholihah P., Saputra P, Puspitasari D, Himawan H, Arhandi P, Syaifudin Y, In 523 IOP Publishing. 012046, , 2019

29. A survey of open-source UAV flight controllers and flight simulators, Ebeid , Emad , al e., Microprocessors and Microsystems 61: 11–20, , 2018

30. A proposed architecture for automated assessment of use case diagrams, Vachharajani V, Pareek J, 108(4): 35–40, , 2014

31. ARDUINO Tutor: An Intelligent Tutoring System for Training on ARDUINO, Mosa MJ, Albatish I, Abu-Naser SS, International Journal of Engineering and Information Systems(IJEAIS) 2(1): 236–245, , 2018

32. Automated feedback generation for introductory programming assignments, SOLAR-LEZAMA, Armando, GULWANI, Sumit, SINGH, Rishabh, In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation p. 15-26, , 2013

33. Exploring a Comprehensive Approach for the Automated Assessment of UML, CHEERS, Hayden, et al, In 2019 8th International Congress on Advanced Applied Informatics (IIAI-AAI). IEEE p. 133-139, , 2019

34. Model tracing–A diagnostic technique in intelligent tutoring systems, STANKOV, Slavomir, ROSIĆ, Marko, AMIŽIĆ, Ani, PhD Thesis. diploma thesis, Faculty of Natural Sciences, Mathematics and Education, University of Split, , 2001

35. A simple lightweight framework for testing RESTful services with TTCN-3, VASSILIOU-GIOLES, Theofanis, In 2020 IEEE 20th International Conference on Software Quality, Reliability and Security Companion (QRS-C). IEEE p. 498-505, , 2020

36. Automated assessment of Android exercises with cloudnative technologies, Bruzual D, Montoya Freire ML, Di Francesco M., 40–46, , 2020

37. Heyteddy: Conversational test-driven development for physical computing, KIM, Yoonji, et al, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3.4: 1-21, , 2019

38. Example-tracing tutors: Intelligent tutor development for non-programmers, ALEVEN, Vincent, et al, 26: 224-269, , 2016

39. Model-based testing of automotive software: Some challenges and solutions, Petrenko A, Ramesh S., Timo ON, In ACM 1–6, , 2015

40. Online judging platform utilizing dynamic plagiarism detection facilities, IFFATH, Fariha, et al, Computers 10.4: 47, , 2021

41. Tiny TTCN-inspired testing tools for experimenting with hybrid IoT systems, BRZEZIŃSKI, Krzysztof M., In 2018 11th International Conference on Human System Interaction (HSI). IEEE p. 261-267, , 2018

42. Automated grading and tutoring of SQL statements to improve student learning, HEINE, Felix, TEBBE, Christopher, KLEINER, Carsten, In Proceedings of the 13th Koli Calling International Conference on Computing Education Research p. 161-168, , 2013

43. Automated testing environment and assessment of assignments for Android MOOC, MADEJA, Matej, PORUBÄN, Jaroslav, Open Computer Science, 8.1: 80-92, , 2018

44. Automated assessment in computer science education: A state-of-the-art review, Paiva , Carlos J, Figueira, Leal JP, ACM Transactions on Computing Education (TOCE) 22(3): 1–40, , 2022

45. Function Execution Log Based Judgment System for Arduino Learning Practice In, Woojin L., Kangbok S, International Conference on Intelligent Tutoring Systems. Cham: Springer International Publishing p. 17-23, , 2022

46. Applications of Hardware-in-the-Loop Simulation in Automotive Embedded Systems, MENG, Aidong, AHMAD, Nadeem, SULTAN, Myrna, SAE Technical Paper, , 2020

47. Design and implementation of a remote lab for teaching programming and robotics, Lima dJPC, Simao JPS, Silva dJB, Mafra PM, Carlos LM, Pereira J, IFACPapersOnLine 49(30): 86–91, , 2016

48. Software testing in introductory programming courses: A systematic mapping study, Scatalon , Passos L al e., In ACM 421–427, , 2019

49. HoRoSim, a holistic robot simulator: arduino code, electronic circuits and physics, FAINA, Andres, In Robotics in Education: RiE 2021 12. Springer International Publishing p. 256-267, , 2022

50. Automated assessment of the visual design of android apps developed with app inventor, SOLECKI, Igor, et al, In Proceedings of the 51st ACM technical symposium on computer science education p. 51-57, , 2020

51. Three Approaches for Detecting Direct Output Cheating in Program Online Judge Systems, LV, Yuehua, QIU, Jing, SHI, Chunmei, 2023, 33.04: 461-486, , 2023

52. Teaching Programming and Microcontrollers with an Arduino Remote Laboratory Application, Maravić U, Pleše E, Petrović P, Bukovac A, Jagušt T., In IEEE 2023: 1738–1741, , 2023