Semi-supervised Representation Learning with Decomposition-based Data Augmentation for Time Series Analysis = 시계열분석을 위한 분해 기반의 데이터 증강 기법과 준지도 표현 학습 기법 연구

다국어 초록 (Multilingual Abstract)

The rapid advancements in data collection methods and storage technologies have dramatically increased the variety and volume of time series data available. Although this data is pivotal for numerous industrial applications and decision-making processes, a significant challenge arises due to the labor-intensive and time-consuming nature of data labeling. This challenge is compounded by the continuous accumulation of data, which leads to a scenario where unlabeled data far outnumbers the labeled data.
In response to these challenges, this thesis employs deep learning-based representation learning techniques within a semi-supervised framework for time series analysis. These techniques facilitate automated labeling for subsets of data, aiding decision-making processes in scenarios with limited labeled data. By integrating deep learning into representation learning, our method effectively addresses the imbalance between labeled and unlabeled data, extracting valuable insights even from sparsely labeled datasets.
This thesis initially proposes a deep learning-based representation model, termed NNCLR-TS, designed to extract features from univariate time series data using a novel single-step, semi-supervised contrastive learning approach. This model comprises an encoder for representation extraction, and a memory structure known as the support set, which aids in pseudo-labeling and facilitates nearest neighbor operations. Within the encoder, two convolutional networks analyze the data from both temporal and frequency perspectives, allowing the model to learn a diverse range of features. Furthermore, the Support set, a dedicated memory structure, stores representations extracted by the encoder in a latent space. This arrangement aids in pseudo-labeling and the selection of training pairs via nearest-neighbor operations.
Appropriate augmentation techniques are essential for contrastive learning. We introduce a novel time series decomposition-based data augmentation technique based on STL decomposition. Unlike jittering and scaling, which may compromise intrinsic time series characteristics such as periodicity, our proposed augmentation technique preserves these features, resulting in more natural augmented data.
We also propose new loss functions that utilize label information, enhancing the learning performance beyond traditional contrastive learning loss functions. These include loss functions considering the similarity within a batch and between the nearest neighbors of given data. This novel approach not only improves the model's accuracy but also ensures its applicability in various real-world scenarios.
The proposed model is applied to various time series classification datasets to validate its performance in univariate time series classification. We investigate performance improvements achieved by utilizing label information, even in scenarios with minimal labeled data.
Finally, we adapt the proposed model for anomaly detection tasks within a self-supervised framework, applying it to various anomaly detection datasets. We assess the model's performance using metrics like precision and recall and explore the potential for performance enhancement through transfer learning.
Our experimental results demonstrate that, in both time series classification and anomaly detection tasks, the proposed model outperforms existing semi-supervised and self-supervised representation learning models.

국문 초록 (Abstract)

데이터 수집 수단 및 저장 기술의 발전에 따라 활용할 수 있는 시계열 데이터의 종류 및 양이 증가하고 있다. 이러한 데이터는 다양한 산업 현장에서 필수적인 역할을 하며, 그 데이터가 갖고 있는 의미를 파악함으로써 의사결정에 도움을 받을 수 있게 된다. 이를 위해 일반적으로 전문가의 레이블링 작업이 필수적으로 요구된다. 그러나, 지속적으로 수집되는 대량의 데이터를 전문가가 일일이 레이블하는 것은 비효율적이며 시간과 비용이 많이 든다는 문제가 있다.
이에, 본 논문은 준지도 학습 기법을 이용하여 시계열 분석을 진행한다. 이 기법은 일부 데이터만 레이블링이 되어 있는 상황에서 나머지 데이터에 대한 자동화된 레이블링을 가능하게 하여, 사용자의 의사결정에 도움을 줄 수 있다.
본 논문은 먼저 단변량 시계열 데이터로부터 대조적 학습을 통해 표현을 추출하는 모델을 제안한다. 인코더 내부의 두 개의 합성곱 연산 기반의 네트워크는, 데이터를 시간적 관점 뿐만 아니라 주파수적 관점에서도 접근하여 다양한 특징을 모델이 학습할 수 있도록 한다. 또한 메모리 구조의 차용을 통해 인코더로부터 추출된 표현을 잠재 공간 내에 저장해두고 이를 수도 레이블링 및 최근접이웃 연산을 통한 학습쌍 선정을 할 수 잇도록 한다.
대조적 학습에는 적절한 증강 기법이 필수적으로 요구된다. 본 논문에서는 STL기법을 기반으로 하는 새로운 시계열 분해 기반의 데이터 증강 기법을 제안한다. 이 때 분해된 각 요소 중 일부 요소에 대해서 샘플링 기반의 변형을 가함으로써 기반 데이터의 분포를 따르는 증강된 데이터를 생성할 수 있도록 한다. 지터링 및 스케일링 등은 시계열의 주기성 등의 특징을 해칠 수 있는 위험성이 존재하는 반면, 제안된 증강 기법은 시계열의 특징을 따르도록 하여 보다 더 자연스러운 증강된 데이터를 생성할 수 있도록 한다.
본 논문에서는 레이블 정보를 활용할 수 있는 새로운 손실함수를 제안한다. 배치내 데이터간의 유사도를 고려하는 손실함수와, 주어진 데이터의 최근접이웃간의 유사도를 고려하는 손실함수를 새롭게 제안함으로써 기존 대조적 학습 손실함수에서 활용할 수 없었던 레이블 정보를 활용하여 학습 성능을 높일 수 있도록 한다.
제안된 모델을 다양한 시계열 분류 데이터셋에 적용하여 단변량 시계열 데이터 분류 문제에서의 성능을 검증한다. 레이블이 극히 일부만 존재하는 상황에서 레이블 정보를 활용했을 때 성능이 향상될 수 있는지 탐구한다.
마지막으로 레이블 정보를 활용할 수 없는 이상 탐지 문제를 위해 제안 모델을 자기지도 학습 상황에 맞춰 모델을 수정한다. 수정된 모델을 여러 이상 탐지 데이터셋에 적용하여 정밀도 및 재현율 등의 지표를 통해 모델의 성능을 검증한다. 또한 전이학습을 통한 모델의 성능 향상 가능성을 탐구한다.
실험결과 분석을 통해 시계열 분류 문제 및 이상 탐지 문제에서 제안모델이 기존 준지도 및 자가지도 표현 학습 모델에 비해 더 뛰어난 성능을 보임을 확인한다.

목차 (Table of Contents)

• Abstract i
• Contents vii
• List of Tables x
• List of Figures xiv
• Chapter 1 Introduction 1
• 1.1 Background and Motivation 1
• 1.2 Objectives 4
• 1.3 Thesis Outline 6
• Chapter 2 Literature Review 7
• 2.1 Semi-supervised Learning 7
• 2.2 Representation Learning in Time Series 11
• 2.2.1 Traditional Methods 11
• 2.2.2 Deep Learning Approaches 12
• 2.2.3 Contrastive Learning 13
• 2.3 Data Augmentation for Time Series Data 17
• 2.3.1 Random Transformation 17
• 2.3.2 Data-driven Approaches 18
• 2.4 Time Series Classification 20
• 2.5 Anomaly Detection 23
• Chapter 3 Proposed Method 26
• 3.1 Preliminaries 26
• 3.2 Nearest Neighbor Contrastive Learning for Time Series 28
• 3.2.1 Model Architecture 28
• 3.2.2 Representation Encoder 32
• 3.2.3 Support Set & Pseudo-labeling 34
• 3.2.4 Nearest Neighbor 37
• 3.3 Decomposition-based Data Augmentation 39
• 3.3.1 Time Series Decomposition 39
• 3.3.2 Augmentation Method 41
• 3.4 Contrastive & Similarity Loss 44
• 3.4.1 Normalized Temperature-scaled Cross-entropy Loss 44
• 3.4.2 Instance-wise Cross-entropy Loss 47
• 3.4.3 Intra-batch Similarity Loss 50
• 3.4.4 Triplet Loss for Representation Learning 52
• Chapter 4 Time Series Classification 54
• 4.1 Problem Definition 54
• 4.2 Experimental Settings 56
• 4.2.1 Datasets 56
• 4.2.2 Implementations Details 58
• 4.2.3 Baseline Models 59
• 4.3 Results 61
• 4.3.1 Performance Comparison with Baseline Models 61
• 4.3.2 Analyzing the Effect of the Ratio of Labeled Data 64
• 4.3.3 Evaluating the Robustness of NNCLR-TS’s Pseudo-Labeling to Dataset Biasness 67
• 4.3.4 Model Training Time Analysis 74
• 4.4 Model Analysis 77
• 4.4.1 Loss Coefficients 77
• 4.4.2 Support Set Size Capacity 79
• 4.4.3 Augmentation Methods 80
• 4.4.4 Ablations on Model Architecture 83
• 4.5 Visualized Explanations 86
• Chapter 5 Anomaly Detection 88
• 5.1 Problem Definition 88
• 5.2 Self-supervised Adaptation of NNCLR-TS 90
• 5.3 Experimental Settings 91
• 5.3.1 Datasets 91
• 5.3.2 Implementation Details 93
• 5.3.3 Baseline Models 95
• 5.4 Results 97
• 5.5 Model Analysis 101
• 5.5.1 Representation Dimensions 101
• 5.5.2 Effect on the Augmented Part of STLDDA 103
• 5.5.3 Evaluating Cross-domain Adaptability with Transfer Learning 105
• 5.6 Visualized Explanations 110
• Chapter 6 Conclusion 114
• 6.1 Summary and Contributions 114
• 6.2 Limitations and Future Research 117
• Bibliography 119
• 국문초록 144
• 감사의 글 146

참고문헌 (Reference)

1. Deep learning, G. E. Hinton, Y. LeCun, Y. Bengio and, 521(7553):436– 444, , 2015

2. Time series analysis, James Douglas Hamilton, Princeton university press, , 2020

3. Long short-term memory, S. Hochreiter and, J. Schmidhuber, Neural computation, 9(8):1735–1780, , 1997

4. Long short-term memory, J¨urgen Schmidhuber, Sepp Hochreiter and, Neural computation, 9(8):1735–1780, , 1997

5. Attention is all you need, Ashish Vaswani, Lukasz Kaiser and, Aidan N Gomez, Llion Jones, Niki Parmar, Jakob Uszkoreit, Illia Polosukhin, Noam Shazeer, Advances in neural information processing systems, 30, , 2017

6. Attention is all you need, A. Vaswani, N. Parmar, L. Jones, A. N. Gomez, J. Uszkoreit, L. Kaiser and, I. Polosukhin, N. Shazeer, In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), pages 6000–6010, , 2017

7. The ucr time series archive, Kaveh Kamgar, Chin-Chia Michael Yeh, Shaghayegh Gharghabi, Hoang Anh Dau, Eamonn Keogh, Yan Zhu, Anthony Bagnall, Chotirat Ann Ratanamahatana and, 6(6):1293–1305, , 2019

8. Visualizing data using t-SNE, Laurens Van der Maaten and, Geoffrey Hinton, 9(11), , 2008

9. Auto-encoding variational bayes, Max Welling, Diederik P Kingma and, arXiv preprint arXiv:1312.6114, , 2013

10. Contrastive multiview coding In, Yonglong Tian, Dilip Krishnan and, Phillip Isola, Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, Proceedings, Part XI 16, pages 776–794. Springer, , 2020

11. Supervised contrastive learning, Ce Liu and, Prannay Khosla, Phillip Isola, Dilip Krishnan, Aaron Sarna, Piotr Teterwak, Aaron Maschinot, Yonglong Tian, Chen Wang, Advances in neural information processing systems, 33:18661–18673, , 2020

12. Multiadversarial domain adaptation, Jianmin Wang, Zhongyi Pei, Zhangjie Cao, Mingsheng Long and, In Proceedings of the AAAI conference on artificial intelligence, volume 32, , 2018

13. A survey on concept drift adaptation, Albert Bifet, Mykola Pechenizkiy and, Abdelhamid Bouchachia, Jo˜ao Gama, Indr˙e ˇZliobait˙e, ACM computing surveys (CSUR), 46(4):1–37, , 2014

14. Forecasting: principles and practice, Rob J Hyndman and, George Athanasopoulos, OTexts, , 2018

15. Transformers in time series: A survey, Weiqi Chen, Liang Sun, Junchi Yan and, Tian Zhou, Chaoli Zhang, Ziqing Ma, Qingsong Wen, arXiv preprint arXiv:2202.07125, , 2022

16. mixupBeyond empirical risk minimization, David Lopez-Paz, Yann N. Dauphin and, Hongyi Zhang, Moustapha Cisse, International Conference on Learning Representations, , 2018

17. Introduction to semi-supervised learning, Xiaojin Zhu and, Andrew B Goldberg, Synthesis lectures on artificial intelligence and machine learning, 3(1):1– 130, , 2009

18. Large-scale unusual time series detection, Earo Wang and, Nikolay Laptev, Rob J Hyndman, In 2015 IEEE international conference on data mining workshop (ICDMW), pages 1616–1619. IEEE, , 2015

19. Induction of selective bayesian classifiers, Stephanie Sage, Pat Langley and, In Uncertainty Proceedings 1994, pages 399–406. Elsevier, , 1994

20. Time-series generative adversarial networks, Daniel Jarrett and, Jinsung Yoon, Mihaela Van der Schaar, Advances in neural information processing systems, 32, , 2019

21. Cnn approaches for time series classification, Lamyaa Sadouk, 5, , 2019

22. S4l: Self-supervised semi-supervised learning, Alexander Kolesnikov and, Lucas Beyer, Xiaohua Zhai, Avital Oliver, In Proceedings of the IEEE/CVF international conference on computer vision, pages 1476–1485, , 2019

23. A survey on contrastive self-supervised learning, Fillia Makedon, Ashish Jaiswal, Debapriya Banerjee and, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, 9(1):2, , 2020

24. Efficient sampling from the bingham distribution, Rong Ge, Andrej Risteski, Jianfeng Lu and, Holden Lee, In Proceedings of the 32nd International Conference on Algorithmic Learning Theory, pages 673–685, , 2021

25. Gpvae: Deep probabilistic time series imputation, Gunnar R¨atsch and, Stephan Mandt, Dmitry Baranchuk, Vincent Fortuin, In International conference on artificial intelligence and statistics, pages 1651–1661. PMLR, , 2020

26. Temporal ensembling for semi-supervised learning, Timo Aila, Samuli Laine and, In International Conference on Learning Representations, , 2016

27. An introduction to variable and feature selection, Andr´e Elisseeff, Isabelle Guyon and, 3(Mar):1157–1182, , 2003

28. Deep active learning for named entity recognition, Hyokun Yun, Yakov Kronrod and, Yanyao Shen, Animashree Anandkumar, Zachary C. Lipton, In International Conference on Learning Representations(ICLR), , 2018

29. Deep echo state network (deepesn): A brief survey, Claudio Gallicchio and, Alessio Micheli, arXiv preprint arXiv:1712.04323, , 2017

30. Time-series anomaly detection service at microsoft, Congrui Huang, Tony Xing, Bixiong Xu, Xiaoyu Kou, Mao Yang, Jie Tong and, Yujing Wang, Hansheng Ren, Chao Yi, Qi Zhang, In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3009– 3017, , 2019

31. Regularization of neural networks using dropconnect, Matthew Zeiler, Rob Fergus, Sixin Zhang, Yann Le Cun and, Li Wan, In International conference on machine learning, pages 1058–1066. PMLR, , 2013

32. Characteristic-based clustering for time series data, Rob Hyndman, Kate Smith and, Xiaozhe Wang, Data mining and knowledge Discovery, 13:335–364, , 2006

33. Semi-supervised learning with deep generative models, MaxWelling, Danilo Jimenez Rezende and, Shakir Mohamed, Durk P Kingma, 27, , 2014

34. Spice: Semantic pseudo-labeling for image clustering, Chuang Niu, Hongming Shan and, Ge Wang, IEEE Transactions on Image Processing, 31:7264–7278, , 2022

35. Time-series data augmentation based on interpolation, Seungmin Han and, Cheolhwan Oh, Jongpil Jeong, Procedia Computer Science, 175:64–71, , 2020

36. A survey on image data augmentation for deep learning, Taghi M Khoshgoftaar, Connor Shorten and, 6(1):1–48, , 2019

37. Deep cotraining for semi-supervised image recognition, Bo Wang and, Siyuan Qiao, Zhishuai Zhang, Wei Shen, Alan Yuille, In Proceedings of the european conference on computer vision (eccv), pages 135–152, , 2018

38. Improved baselines with momentum contrastive learning, Kaiming He, Ross Girshick and, Haoqi Fan, Xinlei Chen, arXiv preprint arXiv:2003.04297, , 2020

39. Multivariate lstm-fcns for time series classification, Samuel Harford, Fazle Karim, Somshubra Majumdar, Houshang Darabi and, 116:237– 245, , 2019

40. Anomaly detection in streams with extreme value theory, Alban Siffer, Alexandre Termier and, Pierre-Alain Fouque, Christine Largouet, In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1067–1075, , 2017

41. Deep learning for time series classification: a review, Hassan Ismail Fawaz, Germain Forestier, Lhassane Idoumghar and, Jonathan Weber, Pierre-Alain Muller, 33(4):917–963, , 2019

42. Gradientbased learning applied to document recognition, Yann LeCun, Yoshua Bengio and, Patrick Haffner, L´eon Bottou, 86(11):2278–2324, , 1998

43. Contrastive regularization for semi-supervised learning, Yeongjae Cheon, Minsu Cho and, Ildoo Kim, Wook-Shin Han, Sungwoong Kim, Doyup Lee, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3911–3920, , 2022

44. Multi-scale context aggregation by dilated convolutions, Fisher Yu and, Vladlen Koltun, arXiv preprint arXiv:1511.07122, , 2015

45. Ts2vec: Towards universal representation of time series, Zhihan Yue, Yujing Wang, Tianmeng Yang, Juanyong Duan, Congrui Huang, Yunhai Tong and, Bixiong Xu, In Proceedings of the AAAI Conference on Artificial Intelligence volume 36, pages 8980–8987, , 2022

46. Deep contrastive one-class time series anomaly detection, Xiaohui Guo, Rui Wang, Kai Gao, Tianyu Wo and, Chongwei Liu, Xudong Mou, Xudong Liu, Pin Liu, In Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pages 694–702. SIAM, 2023, , 2023

47. Gradient-based learning applied to document recognition., Y. Bengio and, P. Haffner, Y. Lecun, L. Bottou, 86(11):2278–2324, , 1998

48. Learning to diagnose with lstm recurrent neural networks, Randall Wetzel, Zachary C Lipton, David C Kale, Charles Elkan and, arXiv preprint arXiv:1511.03677, , 2015

49. Reducing the dimensionality of data with neural networks, Ruslan R Salakhutdinov, Geoffrey E Hinton and, 313(5786):504–507, , 2006

50. Soft-dtw: a differentiable loss function for time-series, Mathieu Blondel, Marco Cuturi and, In International conference on machine learning, pages 894–903. PMLR, , 2017

51. A cnn–lstm model for gold price time-series forecasting, Emmanuel Pintelas and, Panagiotis Pintelas, Ioannis E Livieris, 32:17351–17360, , 2020

52. A review on outlier/anomaly detection in time series data, Jose A Lozano, Usue Mori and, Angel Conde, Ane Bl´azquez-Garc´ıa, ACM Computing Surveys (CSUR), 54(3):1–33, , 2021

53. Mixmatch: A holistic approach to semi-supervised learning, Colin A Raffel, David Berthelot, Nicholas Carlini, Ian Goodfellow, Avital Oliver and, Nicolas Papernot, 32:5050–5060, , 2019

54. Time series data augmentation for deep learning: A survey, Jingkun Gao, Huan Xu, Xue Wang and, Fan Yang, Xiaomin Song, Qingsong Wen, Liang Sun, In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, volume 8, pages 4653–4660, , 2021

55. Highly comparative feature-based timeseries classification, Ben D Fulcher and, Nick S Jones, IEEE Transactions on Knowledge and Data Engineering, 26(12):3026–3037, , 2014

56. Machine learning on big data: Opportunities and challenges, Lina Zhou, Jianwu Wang and, Athanasios V Vasilakos, Shimei Pan, 237:350– 361, , 2017

57. Representation Learning with Contrastive Predictive Coding, Oriol Vinyals, Yazhe Li and, Aaron van den Oord, arXiv preprint arXiv:1807.03748, , 2018

58. Towards a universal neural network encoder for time series, Santiago Pascual and, Joan Serra, Alexandros Karatzoglou, In CCIA, pages 120–129, , 2018

59. Generating synthetic time series to augment sparse datasets, Fran¸cois Petitjean, Geoffrey I Webb and, Hoang Anh Dau, Eamonn Keogh, Germain Forestier, In 2017 IEEE international conference on data mining (ICDM), pages 865– 870. IEEE, , 2017

60. Outlier detection in weight time series of connected scales, Misha Pavel, Elina Helander, Angela Chieh and, Saeed Mehrang, Ilkka Korhonen, In 2015 IEEE international conference on bioinformatics and biomedicine (BIBM), pages 1489–1496. IEEE, , 2015

61. Unsupervised real-time anomaly detection for streaming data, Subutai Ahmad, Scott Purdy and, Alexander Lavin, Zuha Agha, 262:134–147, , 2017

62. Imaging time-series to improve classification and imputation, Tim Oates, Zhiguang Wang and, arXiv preprint arXiv:1506.00327, , 2015

63. Self-supervised learning of pretextinvariant representations, Ishan Misra and, Laurens van der Maaten, Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6707–6717, , 2020

64. Time series anomaly discovery with grammar-based compression, Crystal Chen and, Arnold P Boedihardjo, Tim Oates, Xing Wang, Susan Frankenstein, Pavel Senin, Sunil Gandhi, Jessica Lin, In Edbt, pages 481–492, , 2015

65. Time series feature learning with labeled and unlabeled data, Jia Wu, Shirui Pan and, Haishuai Wang, Qin Zhang, Yixin Chen, 89:55– 66, , 2019

66. Weighted dynamic time warping for time series classification, Myong K Jeong and, Olufemi A Omitaomu, Young-Seon Jeong, 44(9):2231–2240, , 2011

67. Anomaly detection for time series using vae-lstm hybrid model, Shuyu Lin, Niki Trigoni and, Stephen Roberts, Sandro Sch¨onborn, Robert Birke, Ronald Clark, In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4322–4326. Ieee, , 2020

68. Contrastive learning of general-purpose audio representations, Neil Zeghidour, David Grangier and, Aaqib Saeed, In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3875–3879. IEEE, , 2021

69. The performance of lstm and bilstm in forecasting time series, Akbar Siami Namin, Sima Siami-Namini, Neda Tavakoli and, In 2019 IEEE International conference on big data (Big Data), pages 3285–3292. IEEE, , 2019

70. Big self-supervised models are strong semi-supervised learners, Kevin Swersky, Ting Chen, Mohammad Norouzi and, Simon Kornblith, Geoffrey E Hinton, Advances in neural information processing systems, 33:22243–22255, , 2020

71. Singular-value decomposition approach to time series modelling, James A Cadzow, Behshad Baseghi and, Tony Hsu, In IEE Proceedings F (Communications, Radar and Signal Processing), volume 130, pages 202–210. IET, , 1983

72. Wcp: Worst-case perturbations for semisupervised deep learning, Guo-Jun Qi, Liheng Zhang and, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3912–3921, , 2020

73. Barlow twins: Self-supervised learning via redundancy reduction, Yann LeCun and, St´ephane Deny, Li Jing, Jure Zbontar, Ishan Misra, In International Conference on Machine Learning, pages 12310–12320. PMLR, , 2021

74. Dropouta simple way to prevent neural networks from overfitting, Ilya Sutskever and, Ruslan Salakhutdinov, Geoffrey Hinton, Alex Krizhevsky, Nitish Srivastava, 15(1):1929–1958, , 2014

75. Gcrnn: Group-constrained convolutional recurrent neural network, Sangdi Lin and, George C Runger, IEEE transactions on neural networks and learning systems 29(10):4709–4718, , 2017

76. Imagenet classification with deep convolutional neural networks, Geoffrey E Hinton, Ilya Sutskever and, Alex Krizhevsky, Advances in neural information processing systems 25, , 2012

77. Statistics-based outlier detection for wireless sensor networks, Paul JM Havinga, Nirvana Meratnia, Nicholas AS Hamm, Marlies Van de Voort and, Alfred Stein, Yang Zhang, 26(8):1373–1392, , 2012

78. Realistic evaluation of deep semi-supervised learning algorithms, Ian Goodfellow, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk and, Avital Oliver, 31, , 2018

79. Averaging weights leads to wider optima and better generalization, Dmitry Vetrov and, Timur Garipov, Dmitrii Podoprikhin, Andrew Gordon Wilson, Pavel Izmailov, In Converence on Uncertatinty in Ariticial Intelligence(UAI), pages 876–885. AUAI Press for Association for Uncertainty in Artificial Intelligence, , 2018

80. Momentum Contrast for Unsupervised Visual Representation Learning, Saining Xie and, Haoqi Fan, Yuxin Wu, Kaiming He, Ross Girshick, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, , 2020

81. Unsupervised visual representation learning by context prediction, Abhinav Gupta and, Alexei A Efros, Carl Doersch, In Proceedings of the IEEE international conference on computer vision, pages 1422–1430, , 2015

82. An algorithm for the machine calculation of complex fourier series, JohnWTukey, JamesWCooley and, 19(90):297–301, , 1965

83. Self-supervised ecg representation learning for emotion recognition, Pritam Sarkar and, Ali Etemad, IEEE Transactions on Affective Computing, 13(3):1541– 1554, , 2020

84. Semi-supervised contrastive learning for human activity recognition, Tarek Abdelzaher, Dongxin Liu and, In 2021 17th International Conference on Distributed Computing in Sensor Systems (DCOSS), pages 45–53. IEEE, , 2021

85. Adversarial training methods for semi-supervised text classification, Ian Goodfellow, Takeru Miyato, Andrew M Dai and, In International Conference on Learning Representations, , 2016

86. Efficient search of the best warping window for dynamic time warping, Germain Forestier, Matthieu Herrmann, ChangWei Tan, Geoffrey IWebb and, Fran¸cois Petitjean, In Proceedings of the 2018 SIAM International Conference on Data Mining, pages 225–233. SIAM, , 2018

87. Long short term memory networks for anomaly detection in time series, Gautam Shroff, Pankaj Malhotra, Lovekesh Vig, Puneet Agarwal et al, In Esann, volume 2015, page 89, , 2015

88. A Simple Framework for Contrastive Learning of Visual Representations, Ting Chen, Mohammad Norouzi and, Geoffrey Hinton, Simon Kornblith, In International conference on machine learning, pages 1597–1607. PMLR, , 2020

89. Active Learning Methods for Low Resource End-to-End Speech Recognition, Shubham Bansal and, Sriram Ganapathy, Karan Malhotra, In Proc. Interspeech 2019, pages 2215–2219, , 2019

90. Pseudo-labeling and confirmation bias in deep semi-supervised learning, Kevin McGuinness, Eric Arazo, Paul Albert, Diego Ortego, Noel E O’Connor and, In International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, , 2020

91. The wavelet transform, time-frequency localization and signal analysis, Ingrid Daubechies, 36(5):961–1005, , 1990

92. Time-series classification using mixed-state dynamic bayesian networks, Brendan J Frey and, Vladimir Pavlovic, Thomas S Huang, In Proceedings. 1999 IEEE computer society conference on computer vision and pattern recognition (Cat. No PR00149), volume 2, pages 609–615. IEEE, , 1999

93. Self-supervised learning for semi-supervised time series classification, Lars Schmidt-Thieme, Shayan Jawed, Josif Grabocka and, In Advances in Knowledge Discovery and Data Mining: 24th Pacific-Asia Conference, PAKDD 2020, Singapore, Proceedings, Part I 24, pages 499–511. Springer, , 2020

94. Data augmentation with suboptimal warping for time-series classification, Mariusz Oszust, Krzysztof Kamycki, Tomasz Kapuscinski and, 20(1):98, , 2019

95. Unsupervised feature learning via non-parametric instance discrimination, Zhirong Wu, Dahua Lin, Yuanjun Xiong, Stella X Yu and, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733– 3742, , 2018

96. Adaptive consistency regularization for semi-supervised transfer learning, Abulikemu Abuduweili, Xingjian Li, Cheng-Zhong Xu and, Humphrey Shi, Dejing Dou, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6923–6932, , 2021

97. Data augmentation techniques in time series domain: a survey and taxonomy, Alberto Mozo and, Sandra G´omez-Canaval, ´Angel Gonz´alez-Prieto, Guillermo Iglesias, Edgar Talavera, Neural Computing and Applications, pages 1–23, 2023, , 2023

98. Eeg signal classification using pca, ica, lda and support vector machines, Abdulhamit Subasi and, M Ismail Gursoy, 37(12):8659–8666, , 2010

99. Trajectory outlier detection approach based on common slices sub-sequence, Qingying Yu, Yonglong Luo, Chuanming Chen and, XiaohanWang, 48:2661–2680, , 2018

100. Automatic outlier detection for time series: an application to sensor data, Sabyasachi Basu and, Martin Meckesheimer, Knowledge and Information Systems, 11:137–154, , 2007

101. Ms-tcn: Multi-stage temporal convolutional network for action segmentation, Jurgen Gall, Yazan Abu Farha and, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3575–3584, , 2019

102. Recurrent neural networks for multivariate time series with missing values, Kyunghyun Cho, Zhengping Che, David Sontag and, Sanjay Purushotham, Yan Liu, 8(1):6085, , 2018

103. Semi-supervised time series classification by temporal relation prediction, Fengbin Zhang, Xunhua Huang and, Haoyi Fan, Zuoyong Li, Ruidong Wang, In 46th International Conference on Acoustics, Speech, and Signal Processing. IEEE, , 2021

104. Unsupervised scalable representation learning for multivariate time series, Aymeric Dieuleveut and, Jean-Yves Franceschi, Martin Jaggi, 32, , 2019

105. Time-series representation learning via temporal and contextual contrasting, Chee Keong Kwoh, Mohamed Ragab, Emadeldeen Eldele, Xiaoli Li and, Cuntai Guan, Min Wu, Zhenghua Chen, In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2352–2359, , 2021

106. Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning, Paola Cascante-Bonilla, Yanjun Qi and, Fuwen Tan, Vicente Ordonez, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 6912–6920, , 2021

107. Simple: Similar pseudo label exploitation for semi-supervised classification, Ram Nevatia, Zijian Hu, Zhengyu Yang, Xuefeng Hu and, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15099–15108, , 2021

108. Fastdtw is approximate and generally slower than the algorithm it approximates, Eamonn J Keogh, Renjie Wu and, IEEE Transactions on Knowledge and Data Engineering, 34(8):3779–3785, , 2020

109. Fixmatch: Simplifying semi-supervised learning with consistency and confidence, Chun-Liang Li, Colin A Raffel, Ekin Dogus Cubuk, Zizhao Zhang, Han Zhang, Alexey Kurakin and, Nicholas Carlini, Kihyuk Sohn, David Berthelot, Advances in neural information processing systems, 33:596–608, , 2020

110. Electroencephalogram (eeg) time series classification: Applications in epilepsy, Wanpracha Art Chaovalitwongse, Panos M Pardalos, Oleg A Prokopyev and, 148(1):227–250, , 2006

111. Informer: Beyond efficient transformer for long sequence time-series forecasting, Haoyi Zhou, Shanghang Zhang, Jianxin Li, Jieqi Peng, Hui Xiong and, Shuai Zhang, Wancai Zhang, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106–11115,, , 2021

112. Outliers detection in multivariate time series by independent component analysis, Roberto Baragona and, Francesco Battaglia, 19(7):1962–1984, , 2007

113. Transfer learning for short-term wind speed prediction with deep neural networks, Rujia Zhang and, Qinghua Hu, Yucan Zhou, 85:83–95, , 2016

114. Adaptive split-and-merge segmentation based on piecewise leastsquare approximation, Xiaolin Wu, 15(8):808–815, , 1993

115. Time series forecasting of petroleum production using deep lstm recurrent networks, Alaa Sagheer and, Mostafa Kotb, 323:203–213, , 2019

116. Deepant: A deep learning approach for unsupervised anomaly detection in time series, Mohsin Munir, Shoaib Ahmed Siddiqui, Andreas Dengel and, Sheraz Ahmed, Ieee Access, 7:1991–2005, , 2018

117. A review of unsupervised feature learning and deep learning for time-series modeling, Lars Karlsson and, Martin L¨angkvist, Amy Loutfi, 42:11–24, , 2014

118. Data augmentation for time series classification using convolutional neural networks, Romain Tavenard, Arthur Le Guennec, Simon Malinowski and, In ECML/PKDD workshop on advanced analytics and learning on temporal data, , 2016

119. Feasibility of an ai-based measure of the hand motions of expert and novice surgeons, Morimasa Tomikawa, Tomohiko Akahoshi, Makoto Hashizume et al, Ryota Souzaki, Satoshi Ieiri, Tiejun Miao, Munenori Uemura, Alan K Lefor, Computational and mathematical methods in medicine 2018, , 2018

120. Learning the difference that makes a difference with counterfactually-augmented data, Divyansh Kaushik, Eduard Hovy and, Zachary C Lipton, In ICLR, , 2020

121. Time series classification from scratch with deep neural networks: A strong baseline, Tim Oates, ZhiguangWang, Weizhong Yan and, In 2017 International joint conference on neural networks (IJCNN), pages 1578–1585. IEEE, , 2017

122. Semi-supervised end-to-end automatic sleep stage classification based on pseudo-label, Lu Guanze, Bai Haoran and, In 2021 IEEE International Conference on Power Electronics, Computer Applications (ICPECA), pages 83–87. IEEE, , 2021

123. Selfsupervised contrastive pre-training for time series via time-frequency consistency, Marinka Zitnik, Theodoros Tsiligkaridis and, Xiang Zhang, Ziyuan Zhao, volume 35, pages 3988–4003, , 2022

124. A deep transfer learning approach for improved post-traumatic stress disorder diagnosis, Cai Lei, Roger Xu, Guangfan Zhang, Shuiwang Ji and, Debrup Banerjee, Lemin Xiao, Gang Mei, Kazi Islam, Jiang Li, Keyi Xue, 60:1693–1724, , 2019

125. Mixing up contrastive learning: Self-supervised representation learning for time series, Kristoffer Wickstrøm, Karl Øyvind Mikalsen and, Michael Kampffmeyer, Robert Jenssen, 155:54–61, , 2022

126. Micos: Mixed supervised contrastive learning for multivariate time series classification, Wei Zhang, Afanasiev D. Alexander, Shilei Hao, Jidong Yuan and, Zhihai Wang, 260:110158, 2023, , 2023

127. Unsupervised domain adaptation via structured prediction based selective pseudo-labeling, Toby Breckon, Qian Wang and, volume 34, pages 6243–6250, , 2020

128. Anomaly detection in streaming environmental sensor data: A data-driven modeling approach, David J Hill and, Barbara S Minsker, 25(9):1014–1022, , 2010

129. Cost-sensitive convolution based neural networks for imbalanced time-series classification, Xinyu Luo, Yue Geng and, arXiv preprint arXiv:1801.04396, , 2018

130. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring, Alex Kurakin, Kihyuk Sohn and, Han Zhang, Ekin Dogus Cubuk, David Berthelot, Nicholas Carlini, Colin Raffel, In ICLR, , 2020

131. An empirical survey of data augmentation for time series classification with neural networks, Brian Kenji Iwana and, Seiichi Uchida, 16(7):e0254841, , 2021

132. Comparison of linear, nonlinear, and feature selection methods for eeg signal classification, Deon Garrett, Michael H Thaut, Charles W Anderson and, David A Peterson, IEEE Transactions on neural systems and rehabilitation engineering, 11(2):141–144, , 2003

133. Analysis and forecasting of financial time series using cnn and lstm-based deep learning models, Jaydip Sen, Sidra Mehtab and, In Advances in Distributed Computing and Machine Learning: Proceedings of ICADCML 2021, pages 405– 423. Springer, , 2022

134. Nonparametric outliers detection in multiple time series a case study: Power grid data analysis, Yuxun Zhou, Reza Arghandeh, Han Zou, Weixi Gu and, Costas Spanos, volume 32, , 2018

135. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks, Dong-Hyun Lee et al, In Workshop on challenges in representation learning, ICML, volume 3, page 896, , 2013

136. Time series data augmentation for neural networks by time warping with a discriminative teacher, Seiichi Uchida, Brian Kenji Iwana and, In 2020 25th International Conference on Pattern Recognition (ICPR), pages 3558–3565. IEEE, , 2021

137. An improved piecewise aggregate approximation based on statistical features for time series mining, Chonghui Guo, Donghua Pan, Hailin Li and, In Knowledge Science, Engineering and Management: 4th International Conference, KSEM 2010, Belfast, Northern Ireland, UK Proceedings 4, pages 234–244. Springer, , 2010

138. Regularization with stochastic transformations and perturbations for deep semi-supervised learning, Mehran Javanmardi and, Mehdi Sajjadi, Tolga Tasdizen, Advances in neural information processing systems 29, , 2016

139. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations, Pierre Sermanet and, Debidatta Dwibedi, Andrew Zisserman, Yusuf Aytar, Jonathan Tompson, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588–9597,, , 2021

140. Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress, Eamonn Keogh, Renjie Wu and, IEEE Transactions on Knowledge and Data Engineering, , 2021

141. Data augmentation of wearable sensor data for parkinsons disease monitoring using convolutional neural networks, Muriel Lang, Daniel Pichler, Sandra Hirche, Terry T Um, Urban Fietzek and, Franz MJ Pfister, Dana Kuli´c, Satoshi Endo, In Proceedings of the 19th ACM international conference on multimodal interaction, pages 216–220, , 2017

142. A consistency regularization based semi-supervised learning approach for intelligent fault diagnosis of rolling bearing, Kun Yu, Xiang Li, Hui Ma, Tian Ran Lin and, 165:107987, , 2020

143. Semi-supervised learning quantization algorithm with deep features for motor imagery eeg recognition in smart healthcare application, Mingming Zhou, Naixue Xiong, Minjie Liu, Tao Zhang and, 89:106071, , 2020

144. Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state, Christoph Rieke, Ralph G Andrzejak, Christian E Elger, Klaus Lehnertz, Florian Mormann, Peter David and, 64(6):061907, , 2001