Hybrid Algorithmic and Explainable Models Integrating Temporal Features for Multi-horizon Cereal Price Forecasting

다국어 초록 (Multilingual Abstract)

The global cereal market is influenced by a complex interplay of factors, including economic conditions, climatic changes, and socio-political dynamics. Accurately forecasting cereal prices over multiple horizons requires advanced predictive models capable of capturing these intricate, non-linear relationships. In recent years, Artificial Intelligence (AI), particularly Machine Learning (ML) and Deep Learning (DL), has shown remarkable effectiveness in various domains such as image classification, speech recognition, and time-series prediction. However, despite their high accuracy, these models often lack interpretability, making it challenging for stakeholders to fully understand or trust their predictions. This need for transparency has driven the growth of Explainable Artificial Intelligence (XAI), a field focused on enhancing the interpretability of AI models.
While much XAI research has concentrated on image data, there has been limited exploration of its application to time-series data, which is essential for economic and agricultural forecasting. This thesis addresses this gap by employing a model-agnostic technique called the Contrastive Explanation Method (CEM) to interpret predictions of a hybrid deep learning model designed for multi-horizon cereal price forecasting. The proposed hybrid framework combines state-of-the-art models, including TabNet, Neural Oblivious Decision Ensembles (NODE), and the Temporal Fusion Transformer (TFT), to improve both predictive accuracy and interpretability.
The key innovation in this research lies in creating a hybrid model that not only forecasts cereal price fluctuations across different time horizons with enhanced precision but also provides stakeholders with interpretable insights through contrastive explanations. By leveraging temporal attention mechanisms and explainability techniques, the model offers transparency, allowing users to understand the key drivers behind its multi-horizon predictions. Integrating temporal, economic, climatic, and socio-political data, the model employs sophisticated data harmonization and preprocessing methods to ensure robustness and reliability across various forecasting periods. This approach empowers stakeholders with actionable insights, supporting informed decision-making at local, national, and international levels.
The results demonstrate that the hybrid model significantly outperforms traditional models like XGBoost and NODE with an accuracy of 98.5% while these traditional models lie between 94.07% and 98.4%. Moreover, the application of CEM provides clear, contrastive explanations, ensuring that decision-makers can trust and act upon the insights generated. For instance, in scenarios where the Hybrid model predicted a significant price increase, CEM could pinpoint whether this was due to a sudden change in climatic variables or an abrupt socio-political event. This research contributes to the advancement of both predictive modeling and explainability in AI, offering a comprehensive approach to addressing complex forecasting problems in the global agricultural sector.

목차 (Table of Contents)

• I. INTRODUCTION 1
• 1. RESEARCH BACKGROUND 3
• 1.1. Economic Fluctuations and Market Dynamics 3
• 1.2. Climate Dynamics 4
• 1.3. Hybrid Computational Model 4
• 2. PROBLEM STATEMENT 5
• 3. OBJECTIVES AND GOAL 7
• 4. SIGNIFICANCE OF STUDY 8
• 5. RESEARCH QUESTIONS 9
• II. LITERATURE REVIEW 9
• INTRODUCTION 9
• i- Time series Analysis 11
• ii- Existing forecasting models for crop price prediction 19
• iii- Regression analysis 28
• iv- Machine learning 37
• v- Deep learning 44
• vi- TabNet 50
• vii- Neural Oblivious Decision Ensembles (NODE) 64
• viii- Hybrid model 67
• ix- Temporal Fusion Transformer (TFT) 77
• x- Explainable AI (XAI) 86
• xi- Gap Analysis 96
• CONCLUSION 97
• III. METHODOLOGY 97
• 1. DATA ACQUISITION AND SYNTHESIS 98
• 1.1. Economic Factors 101
• 1.2. Climatic Factors 101
• 1.3. Socio-Political Factors 101
• 1.4. Data Merging and harmonization 101
• 1.5. Data harmonization 102
• 1.6. Data joining techniques 102
• 1.7. Validation of merged data 102
• 2. DATA PREPROCESSING AND EDA 103
• 2.1. Data Cleaning 103
• 2.2. Exploratory Data Analysis (EDA) 104
• 2.3. METHODS 107
• 2.3.1 TabNet 108
• 2.3.2 NODE (Neural Oblivious Decision Ensembles) 108
• 2.3.3 Hybrid Model Architechture 109
• 2.3.4 Explanable AI (XAI) technique 111
• 2.4 MODEL DEVELOPMENT 111
• 2.5 MODEL TRAINING AND EVALUATION 112
• 2.6 EXPLAINABILITY 113
• 2.7 TOOLKITS AND SOFTWARE 113
• 2.8 EXPERIMENT SETTINGS AND ALGORITHM 116
• IV. RESULT AND DISCUSSION 116
• RESEARCH QUESTION 1: 117
• RESEARCH QUESTION 2: 120
• V. TECHNOLOGY COMPARISON 122
• 1. PRE-TECHNOLOGY COMPARISON 122
• 1.1. Defining the Problem 122
• 1.2. Establishing Criteria for Evaluation 123
• 1.3. Establishing the Hierarchy 124
• 1.4. Pairwise Comparison Matrix 127
• 1.5 AHP Mathematics Calculation 131
• 1.6 Aggregation and Decision 136
• 2. POST-COMPARISON ANALYSIS 137
• VI. ACTION PLAN 144
• INTRODUCTION 144
• II. SYSTEM DEVELOPMENT AND DEPLOYMENT 146
• III. TRAINING AND CAPACITY BUILDING 147
• IV. POLICY INTEGRATION AND GOVERNMENT PARTNERSHIPS 149
• V. MONITORING AND IMPACT ASSESSMENT 150
• VI. EXPANSION AND SCALABILITY 153
• VII. REPORTING AND KNOWLEDGE SHARING 156
• CONCLUSION AND FUTURE DIRECTIONS 159
• CONCLUSION AND FUTURE WORK 163
• VII. REFERENCES 165