Sound Classification Model on a Resource-Constrained Edge Device for Fruit Theft Prevention = 과일 도난 방지를 위한 저자원 에지 디바이스용 음향 분류 모델

다국어 초록 (Multilingual Abstract)

In recent years, fruit theft has become a serious social problem in Japan, where
conventional surveillance measures are limited due to the unique challenges of agricul-
tural environments, such as restricted power supply and insufficient lighting. To address
this issue, the prior study developed a battery-powered Microphone-based Surveillance
System (MSS) that operates on a resource-constrained microcontroller, the Seeeduino
XIAO nRF52840 (1 MB Flash, 256 kB RAM). However, achieving high classification
accuracy under such severe computational and memory constraints remains a major
challenge. This study presents a design and training methodology for a compact and
high-accuracy sound classification model for fruit theft prevention. The proposed ap-
proach integrates Depthwise Separable Convolution (DSC) to substantially reduce model
parameters and computation, and employs Ensemble Distillation (EnD), which trans-
fers complementary knowledge from two heterogeneous teacher models—a CNN-based
model and a Transformer-based model—to mitigate the accuracy degradation typically
caused by aggressive model compression.

Experiments on a three-class sound classification task for suspicious sound detection
(environmental sounds, speech, and footsteps) demonstrated that the proposed model
achieved an F1-score of 88.61%, an improvement of 12.00 percentage points over the
baseline. The inference memory usage was limited to 49 kB, and when including all
non-inference system components, the total memory usage remained within 160 kB,
well below the 256 kB RAM capacity of the target hardware. Moreover, the model
achieved an inference time of approximately 0.45 seconds per 1-second audio segment
on a Seeeduino XIAO nRF52840 microcontroller.

Taken together, these results show that the proposed approach satisfactorily fulfills
the three key requirements of accuracy, lightweight design, and real-time inference under
strict device constraints, thereby enhancing the robustness and practical deployability
of the MSS for fruit theft prevention in orchard environments.

국문 초록 (Abstract)

최근 일본에서는 제한된전력 공급과부족한 조도와같은농업 환경의특수성으로 인해 기존의감시수단이
충분히 기능하지못하고있으며, 이로 인해 과수절도가심각한 사회적문제로 대두되고있다. 이러한 문제를
해결하기위해, 선행 연구에서는 마이크로컨트롤러 Seeeduino XIAO nRF52840(1 MB Flash, 256 kB RAM)
을기반으로 하는 배터리 구동형Microphone-based Surveillance System(MSS)을개발하였다. 그러나 이러한
극도로 제한된연산 및메모리 환경에서높은분류 정확도를 달성하는 것은여전히 큰 도전과제이다.

본연구는 과수절도방지를 위한 소형· 고정확도음향 분류 모델의설계및학습방법을제안한다. 제안
모델은Depthwise Separable Convolution(DSC)을이용하여 매개변수와연산량을크게줄이고, CNN 기반모
델과Transformer 기반모델이라는 서로 다른 구조의두교사 모델로부터 보완적인지식을전달받는 Ensemble
Distillation(EnD)을적용하여, 경량화로 인해 발생하기쉬운정확도저하를 완화한다.

환경음· 음성· 발걸음소리의3개클래스를 대상으로 한 실험 결과, 제안 모델은F1-score 88.61%를 달성
하여 베이스라인대비 12.00%포인트 향상된성능을보였다. 추론 시필요한 메모리는 49 kB에 불과하였으며,
비추론 시스템 처리를 포함한 전체 메모리 사용량은160 kB 이내로 유지되어 목표 하드웨어의256 kB RAM
한계를 크게하회하였다. 또한 제안 모델은Seeeduino XIAO nRF52840에서1초길이의음향 신호당 약 0.45
초의추론 시간을기록하였다.

이러한 결과는 제안 접근법이정확도, 경량성, 그리고실시간추론이라는 세가지핵심요구사항을충족하
며, 실제과수원환경의엄격한 자원제약 하에서도과수절도방지를 위한 MSS의신뢰성과실용적배치 가능성을
크게향상시킴을보여준다.

목차 (Table of Contents)

• I. Introduction 1
• 1. Background and Objectives 1
• 2. Related Work 4
• 2.1. Security Systems for Fruit Orchards 4
• 2.2. AI Models for Acoustic Classification 4
• 2.3. Acoustic Classification on Microcontrollers 5
• 2.4. Review of Related Work and Novelty of This Study 6
• 3. Overview of This Study 6
• 4. Organization of This Thesis 8
• II. Deep Learning 10
• 1. Overview of Deep Learning 10
• 1.1. Deep Neural Network 10
• 1.2. Activation Functions 11
• 1.3. Loss Functions 13
• 1.4. Optimization Methods 14
• 1.5. Training of DNNs 16
• 1.6. Training Techniques 18
• 2. Deep Learning Models 19
• 2.1. Convolutional Neural Network 19
• 2.2. Audio Spectrogram Transformer(AST) 25
• 3. Model Compression Techniques 30
• 3.1. Knowledge Distillation(KD) 30
• 3.2. Quantization 32
• 3.3. Depthwise Separable Convolution(DSC) 34
• III. Preprocessing of Acoustic Signals 37
• 1. Mel Spectrogram Transformation 38
• IV. Deployment to the Device 42
• 1. Hardware Configuration and Available Memory 42
• 1.1. Hardware Configuration of the MSS 43
• 1.2. Specifications of the XIAO nRF52840 44
• 1.3. Available Memory in the MSS 45
• 2. Model Design Under TFLM Operator Constraints 46
• 3. Multi-stage Conversion of Deep Learning Models 46
• 4. Practical Deployment Procedures 48
• 4.1. Conversion of the Model into a C Array 48
• 4.2. Memory Management with Tensor Arena 48
• V. Proposed Method 50
• 1. Model Design 50
• 1.1. Limitations of the Baseline Model 51
• 1.2. Design Policy of the Proposed Model and Comparison Models 52
• 2. Ensemble Distillation(EnD) 55
• 2.1. Selection of Teacher Models 55
• 2.2. Key Components of the EnD Framework 57
• VI. Experiments 60
• 1. Experimental Objectives 60
• 2. Dataset 62
• 3. Experimental Settings 65
• 4. Experimental Results 67
• 4.1. Classification Accuracy 67
• 4.2. Model Size and Memory Usage 71
• 4.3. Inference Speed 73
• VII.Conclusion 75
• 1. Summary of This Thesis 75
• 2. Future Work 79
• A. Model Structure 82
• Bibliography 86
• Acknowledgement 93