【Paper】PYRAFORMER: low-complexity pyramidal attention

Abstract

In practice, the challenge is to build a flexible but parsimonious model that can capture a wide range of temporal dependencies.

parsimonious
英
/ˌpɑːsɪˈməʊniəs/
美
/ˌpɑːrsɪˈmoʊniəs/
adj.
吝啬的;小气的;悭吝的
派生词： parsimoniously adv.
在实际中，挑战在于建立灵活且简约的模型用于捕获长时依赖

In this paper, we propose Pyraformer by exploring the mult-resolution representation of the time series. Specifically, we introduce the pyramidal attention module(PAM) in which the inter-scale tree struct summarizes features at different resolutions and the intra-scale neighboring connections model the temporal dependencies of different ranges.

Under mild conditions, the maximum length of the signal traversing path in Pyraformer is a constant with regard to the sequence length $L$, while its time and space complexity scale linearly with $L$.

Introduction

Time series forecasting is the cornerstone for downstream tasks such as decision making and risk management. As an example, reliable prediction of the online traffic for micro-services can yield early warning of the potential risk in cloud systems.

The major challenge of time series forecasting lies in constructing a powerful but parismonious model that can compactly capture temporal dependencies of different ranges.

Time series often exhibit both short-term and long-term repeating pattens, and taking them into account is the key to accurate prediction. Of particular note is the more difficult task of handing long-range dependencies, which is characterized by the length of the longest signal travesing path between any two positions in the time series. The shorter the path, the better the dependencies are captured. Addtionally, to allow the models to learn these long-term patterns, the historical input to the model should also be along, To this end, low time and space complexity is a priority.