Classification and digital restoration of Korean coins using deep learning and reflectance transformation imaging

국문 초록 (Abstract)

동전 표면의 부식과 물리적인 손상은 제작연도, 주전소 그리고 그 동전이 가지고 있는 특징과 같은 중요한 정보를 파악하는데 어려움을 줄 수 있다. 이 연구는 이미지 분석만으로 동전의 유형과 연도를 분류하는 합성곱 신경망(CNN) 모델의 가능성에 대하여 살펴보고자 한다. 사용되는 이미지는 일반적으로 촬영된 RGB 이미지와 더불어 반사율 변환 이미징을 사용하여 획득한 이미지로 구성된다. 반사율 변환 이미징은 필터를 통해 가시적인 특징을 향상시켜 육안으로 확인이 어려운 표면의 세부 정보를 파악할 수 있다. 이 연구에서는 현행 화폐 중 10원 동전을 대상으로 하며, 조선시대의 상평통보에도 적용하였다. 상평통보는 1678년 숙종 4년부터 발행하기 시작한 조선시대의 대표적인 화폐로, 압인화가 도입되기 전까지 약 200년 동안 전국적으로 통용되어 한국의 역사에 있어서 빼 놓을 수 없는 중요한 화폐이다. 데이터세트는 10원 동전 1960개와 상평통보 320점으로 구성되었으며, 부식과 오염 그리고 훼손의 정도에 따라 10원 동전은 세 개의 테스트 세트로, 상평통보는 두 개의 테스트 세트로 나누었다. 기울기 가중치 클래스 활성화 매핑(Grad-CAM)과 Occlusion 결과에서 모델은 주로 동전 뒷면에 주조되어 있는 발행연도, 주조 관청, 주전소 등과 같은 정보에 초점을 맞추었다. 예측 정확도를 분석한 결과 오염 및 훼손의 정도가 증가함에 따라 감소하는 것으로 나타났다. RGB 이미지의 경우 앞면과 뒷면 모두 정확도가 가장 많이 감소했으며, RTI image는 RGB 이미지보다 적은 감소량을 보였다. 특히 RTI enhancement의 경우 부식 및 오염이 심해져도 가장 높은 정확도를 보였다. 이러한 결과는 RTI 이미지가 동전을 정확하게 분류하는데 우수한 성능을 보인다는 것을 시사한다. 이 연구는 특히 표면 부식과 물리적 손상이 기존의 분석 방법을 저해하는 상황에서 합성곱 신경망 모델과 반사율 변환 이미징을 활용하여 동전 분류 방법을 제안하는데 의의가 있다. 또한, 분류된 정보를 기반으로 디퓨전 모델을 사용하여 오염 및 훼손된 동전의 이미지를 디지털 복원하였다. Retrieval 알고리즘을 활용하여 입력되는 이미지와 가장 유사한 이미지를 추천받으며, 해당 이미지를 참고로 하여 훼손된 이미지를 완형의 형태로 복원시켰다. 디지털 복원한 결과를 살펴보면, 입력된 이미지는 주위의 정보와 어우러지게 복원부가 생성되는 것이 확인되므로, 인공지능을 이용하여 동전 이미지의 복원이 가능하다.

다국어 초록 (Multilingual Abstract)

Corrosion and physical damage make it difficult to obtain important information such as the year of production, mint, and characteristics of the coin on the surface. This study explores the potential of a Convolutional Neural Networks (CNN) model to classify coins based on their type and year using image analysis alone. The dataset employed includes RGB images, typically captured, and images acquired using Reflectance Transformation Imaging (RTI), which enhances visible characteristics through filters, revealing more details about the coin's surface compared to RGB images. The experimental subject chosen for this study was the 10-won coins from Korea and Sangpyeongtongbo from the Joseon dynasty. Sangpyeongtongbo, a representative currency of the Joseon Dynasty issued in the fourth year of King Sukjong's reign in 1678, was widely circulated nationwide for about 200 years until the introduction of stamp manufacturing technology. It is an indispensable and important currency in Korean history. The dataset consists of 1,960 10-won coins and 320 Sangpyeongtongbo, divided into three test sets based on corrosion, contamination, and damage levels for the 10-won coins, and two test sets for the Sangpyeongtongbo. The gradient-weighted class activation mapping and Occlusion results indicated that the model primarily focused on the symbol on the obverse side and the year of issue on the reverse side. Analysis of the prediction accuracy revealed a decline as contamination levels increased. Specifically, for RGB images, the accuracy decreased the most for both the obverse and reverse sides. In contrast, RTI images exhibited a smaller decrease compared to RGB images, with RTI enhancement showing the highest accuracy even under severe corrosion and contamination. These findings demonstrate that RTI images exhibit superior performance in accurately classifying coins. By employing CNN models and utilizing the advantages of RTI, this research contributes to the development of coin classification techniques, particularly in scenarios where surface corrosion and physical damage hinder traditional analysis methods. Based on the classified information, a diffusion model was used to digitally restore images of contaminated and damaged coins. Utilizing the retrieval algorithm, the image most closely resembling the input image was obtained. The recommended image was then employed as a reference to reconstruct the damaged image to its pristine state. Upon examining the digitally restored results, it was confirmed that the input image generates restoration parts in harmony with the surrounding information. It was also verified that coin image restoration is possible using artificial intelligence.

목차 (Table of Contents)

• 1. Introduction 1
• 2. Literature Review 5
• 2.1 Reflectance Transformation Imaging (RTI) in conservation science 5
• 2.2 Coin classification with artificial intelligence 7
• 2.3 Image generation using artificial intelligence 11
• 3. Materials 15
• 3.1 10-won coins 17
• 3.2 Sangpyeongtongbo 20
• 4. Methodology 23
• 4.1 RTI Image Acquisition 24
• 4.2 CNN models for 10-won coins classification 30
• 4.2.1 Implementation details 32
• 4.2.2 Prediction of overfitting degree 33
• 4.3 CNN models for Sangpyeongtongbo classification 36
• 4.3.1 Implementation details 38
• 4.4 Generation models for digital restoration 39
• 4.4.1 Implementation details 41
• 4.4.2 Evaluation metrics 42
• 5. Results and discussion 44
• 5.1 Dataset Acquisition 44
• 5.2 10-won coins classification 51
• 5.2.1 Comparisons of other coin classification 51
• 5.2.2 Training Results 54
• 5.2.3 Overfitting Estimation and Test Performance 68
• 5.3 Sangpyeongtongbo classification 75
• 5.4 Generation 10-won coins 83
• 5.5 Generation Sangpyeongtongbo 89
• 6. Conclusion 94
• References 99
• Appendix A 107
• Appendix B 134
• 국문초록 206

참고문헌 (Reference)

1. Deep learning, Courville, A., Goodfellow, I., Bengio, Y., MIT press, , 2016

2. 41 Guide to rtiviewer, Cultural Heritage Imaging, v 1.1 http://culturalheritageimaging. org/What_We_Offer/Downloads/rtiviewer/ RTIViewer_Guide_v1_1. pdf (accessed 15.12. 14, , 2015

3. Image inpainting: A review, Akbari, Y, Almaadeed, N., Elharrouss, O., Al-Maadeed, S, 51, 2007-2028, , 2020

4. Polynomial texture maps in, Malzbender, T., Gelb, D., Wolters, H., Proceedings of the 28th annual conference on Computer graphics and interactive techniques. Los Angeles, California, USA pp. 519–528, , 2001

5. A note on the inception score, Sharma. R., Barratt, S. and, Machine Learning Article ASAP. DOI: arXiv:1801.01973 (accessed 2018- 06-21), , 2018

6. 40 Changes in current currency, Department in charge Money Museum, The Bank of Korea, , 2005

7. Decoupled weight decay regularization, Loshchilov, I., Hutter, F., Machine Learning Article ASAP. DOI: arXiv:1711.05101 (accessed 2019- 01-04), , 2017

8. Focal loss for dense object detection, He, K., Lin, T. Y., Dollár, P, Girshick, R., Goyal, P., In Proceedings of the IEEE international conference on computer vision. Venice, Italy pp. 2980-2988, , 2017

9. Denoising diffusion probabilistic models, Abbeel, P., Ho, J., Jain, A., Advances in neural information processing systems 33, 6840-6851, , 2020

10. Why Why was the Sangpyeong Notice created, Lee, C. S., 9, 133-144, , 2002

11. A Computational Approach to Edge Detection, Canny, J., 6, 679–698, , 1986

12. Graph transduction as a noncooperative game, Erdem, A., Pelillo, M., 24 (3), 700–723, , 2012

13. Deep residual learning for image recognition, He, K., Zhang, X., Sun, J., Ren, S., pp. 770-778, , 2016

14. Anydoor: Zeroshot object-level image customization, Huang, L., Zhao, H., Chen, X., Liu, Y., Zhao, D., Shen, Y., 2023, Article ASAP. DOI: arXiv:2307.09481 (accessed 2024-05-08), , 2023

15. On the foundations of relaxation labeling processes, Zucker, S. W., Hummel, R. A., IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-5 (3), 267–287, , 1983

16. Automatic attribution of ancient roman imperial coins, Arandjelović, O., 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Cape Town, South Africa pp. 1728–1734, , 2010

17. A study of historical developments and design of money, Nam, S., Jo, S., In Book of Abstracts, Korea Contents Association 2017 Spring Conference, Daejeon, Korea; 163-164, , 2017

18. Learning Deep Features for Discriminative Localization, Torralba, A., Khosla, A., Lapedriza, A., Oliva, A., Zhou, B., in Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas, NV, USA pp. 2921–2929, , 2016

19. Mobilenetv2: Inverted residuals and linear bottlenecks in, Zhu, M., Howard, A., Zhmoginov, A., Sandler, M., Chen, L. C., Proceedings of the IEEE conference on computer vision and pattern recognition. Salt Lake City, UT, USA pp. 4510–4520, , 2018

20. Multiscale structural similarity for image quality assessment, Wang, Z., Bovik, A. C., Simoncelli, E. P., The Thrity-Seventh Asilomar Conference on Signals, Systems Computers. Pacific Grove, CA, USA 2, pp. 1398-1402, , 2003

21. 39 Korean currency from ancient times to the Korean Empire era, Department in charge Money Museum, The Bank of Korea, , 2006

22. Structure-guided image inpainting using homography transformation, Fang, Y., Yang, S., Liu, J., Guo, Z., 20 (12), 3252-3265, , 2018

23. Repaint: Inpainting using denoising diffusion probabilistic models, Van Gool, L., Yu, F., Danelljan, M., Lugmayr, A., Timofte, R., Romero, A., In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, New Orleans, LA, USA pp. 11, , 2022

24. Retrieval-augmented generation for large language models: A survey, Pan, J., Wang, M., Jia, K., Sun, J., Gao, X., Dai, Y., Xiong, Y., Bi, Y., Gao, Y., Wang, H., Computation and Language. 2023, Article ASAP. DOI: arXiv:2312.10997 (accessed 2024-03-27)., , 2023

25. Very deep convolutional networks for large-scale image recognition, Zisserman, A., Simonyan, K., Article ASAP. DOI: arXiv:1409.1556 (accessed 2015-04-10), , 2014

26. Ancient coin classification based on recent trends of deep learning, Manzoor, S., Ayub, A. A., Ali, N., Raees, M., Khan, K. A., Usama, M., VIPERC2022: 1st International Virtual Conference on Visual Pattern Extraction and Recognition for Cultural Heritage Understanding, Pescara, Italy, , 2022

27. A novel patch matching algorithm for exemplar-based image inpainting, Zhang, L., Fan, Q., 77 (9), 10807-10821, , 2018

28. Paint by example: Exemplar-based image editing with diffusion models, Zhang, B., Chen, D., Zhang, T., Yang, B., Gu, S., Sun, X., Chen, X., Wen, F., In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Rome, Italy 2023, pp. 18381-18391, , 2023

29. On-the go reflectance transformation imaging with ordinary smartphones, Pistellato, M., Bergamasco, F., in European Conferenceon Computer Vision, Springer pp. 251–267, , 2022

30. Money its soul and truth: In search of the essence and history of money, Lietaer, B. A., Kang, N., Chamsol, , 2004

31. Image inpainting for high-resolution textures using CNN texture synthesis, Franz, M. O., Umlauf, G., Grunwald, M., Laube, P., Article ASAP. DOI: arXiv:1712.03111 (accessed 2018-02-12), , 2017

32. Score-based generative modeling through stochastic differential equations, Jascha, S. D., Song, Y., Ermon, S., Diederik, P. K., Poole, B., Kumar, A., Machine Learning Article ASAP. DOI: arXiv:2011.13456 (accessed 2021-02-10), , 2020

33. Context-aware patch-based image inpainting using markov random field modeling, Pižurica, A., Ružić, T, 24 (1), 444-456, , 2014

34. Grad-camVisual explanations from deep networks via gradient-based localization, Das A, Parikh D, Batra D., Vedantam R, Cogswell M, Selvaraju RR, in Proceedings of the IEEE international conference on computer vision. Venice, Italy pp. 618–626, , 2017

35. A novel convolutional neural network for classifying indian coins by denomination, Chauhan, Y., Singh, P., Engineering Archive. Article ASAP. DOI: https://doi. org/10.31224/osf. io/znxrg (accessed 2021- 12-27), , 2021

36. Deep ancient roman republican coin classification via feature fusion and attention, Zambanini, S., Anwar, H., Anwar, S., Porikli, F., 114, 107871, , 2021

37. Generalized cross entropy loss for training deep neural networks with noisy labels, Zhang, Z., Sabuncu, M., Advances in neural information processing systems 31, 1-11, , 2018

38. Mobilenets: efficient convolutional neural networks for mobile vision applications, Kalenichenko, D., Zhu, M., Weyand, T., Howard, A. G., Adam, H., Chen, B., Andreetto, M., Wang, W., Article ASAP. DOI: arXiv:1704.04861 (accessed 2017-04-17)., , 2017

39. Semantic image inpainting vsing self-learning encoder-decoder and adversarial loss, Salem, N. M., Abbas, H., Mahdi, H. M., in 2018 13th International Conference on Computer Engineering and Systems(ICCES), Kuala Lumpur, Malaysia pp. 103-108, , 2018

40. IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representation, Xiong, W., Song, Y., Cohen, S., Zhang, Z., Aliaga, D., Zhang, J., Zhang, H., Lin, Z., Price, B., Kim, S. Y., Computer Vision and Pattern Recognition. 2024, Article ASAP. DOI: arXiv:2403.10701 (accessed 2024-03-15)., , 2024

41. Python Excel deep learning: Understanding the algorithmic principles of deep learning, Hong, J., Yoon, D., Lee, S., Information Culture Company, 2023, , 2023

42. Semantic image inpainting through improved wasserstein generative adversarial networks, Vitoria, P., Ballester, C., Sintes, J., Article ASAP. DOI: arXiv:1812.01071 (accessed 2018-12-03), , 2018

43. Guide to non-destructive penetration investigation of cultural properties using radiation, National Research Institute of Cultural Heritage, Cultural Heritage Conservation Science Center, , 2020

44. Reference-based painterly inpainting via diffusion: Crossing the wild reference domain gap, Xu, D., Cong, W., Shi, H., Wang, Z., Xu, X., 2023, Article ASAP. DOI: arXiv:2307.10584 (accessed 2024-05-08), , 2023

45. Coarse-grained ancient coin classification using image-based reverse side motif recognition, Zambanini, S., Kampel, M., Anwar, H., 26, 295–304, , 2015

46. Two sides of the same coin: Improved ancient coin classification using graph transduction games, Vascon, S., Pelillo, M., Aslan, S., 131, 158–165, , 2020

47. Reflectance transformation imaging for documenting changes through treatment of joseon dynasty coins, Ahn, S., Jeong, S., Lee, D., Choi, Y., Park, K., Ahn, J., Min, J., Har, D., 9 (1) 1–12, , 2021