Self-supervised learning for time-series data holds potential similar to that recently unleashed in Natural Language Processing and Computer Vision. While most existing works in this area focus on contrastive learning, we propose a conceptually simple yet powerful non-contrastive approach, based on the data2vec self-distillation framework. The core of our method is a student-teacher scheme that predicts the latent representation of an input time series from masked views of the same time series. This strategy avoids strong modality-specific assumptions and biases typically introduced by the design of contrastive sample pairs. We demonstrate the competitiveness of our approach for classification and forecasting as downstream tasks, comparing with state-of-the-art self-supervised learning methods on the UCR and UEA archives as well as the ETT and Electricity datasets.
@article{arxiv.2311.11335,
title = {Self-Distilled Representation Learning for Time Series},
author = {Felix Pieper and Konstantin Ditschuneit and Martin Genzel and Alexandra Lindt and Johannes Otterbach},
journal= {arXiv preprint arXiv:2311.11335},
year = {2023}
}
Comments
Presented at the NeurIPS 2023 Workshop: Self-Supervised Learning - Theory and Practice