EA-LSTM: Evolutionary attention-based LSTM for time series prediction

原文链接

1. Introduction

做法： Based on the idea of evolutionary computation[21],we propose a competitive random search (CRS) instead of the gradient-based method to solve the attention layer weights.

Genetic Algorithms and the Optimal Allocation of Trials

为什么要引入CRS：change the search direction to avoid falling into local optimum.

[22] X. Zhang, J. Clune, K.O. Stanley, On the relationship between the OpenAI evolution strategy and stochastic gradient descent, 2017, arXiv:1712. 06564.
[23] E. Conti, V. Madhavan, F. Petroski Such, J. Lehman, K.O. Stanley, J. Clune, Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents, 2017, arXiv:1712. 06560.
[24] Joel Lehman, Jay Chen, Jeff Clune, Kenneth O. Stanley, Safe mutations for deep and recurrent neural multi-order through output gradients, 2017, arXiv:1712.06563.

对GA的操作：

1. In particular, theimproved crossoveroperator has integrated more stochastic mechanisms to maintain the differences between the progeny individuals,
   目的：avoiding premature convergence of the algorithm and being trapped in local optimum.

2. use the basic bit mutation operator to specifically perform the mutation operation by randomly inverting one or several gene values at the locus according to the mutation rate on a single encoded string.

introduction的知识点：

1. 遗传算法

---

2. Preliminaries(预先准备)

时序数据用于分类和回归。

数据集为$X=(X_1,X_2,...X_T)$，其中每一个$X_t=(x_t^1,x_t^2,...x_t^L)$代表$L$个timestamps。其中每个时刻对应的输出值则记为$y$。

离散or 回归则取决于Y的数据是连续的还是discrete的。

目标：根据历史时刻的输入数据和输出数据找到映射函数$y_T$，数学表达式为：
$$y_T=f(X,y)$$

3. Methodology(方法论)

3.1 Overview

本章结构：

1. ,we first give the overview of the model we proposed
2. we will detail the evolutionary attention-based LSTM.
3. we present the competitive random search and a collaborative(协作训练) training mechanism.

工作流程如下：

3.2 整体算法流程

1. 定义注意力层的权重为：
   $$W=(W^1,W^2,...,W^L)$$
   这里的L是timestamps的个数。根据注意力层的权重对LSTM层的输出进行采样。
   $$X_t=(x_t^1{W}^1,x_t^1{W}^2,...,x_t^1{W}^L)$$
2. 然后把$X_t$喂到LSTM层中，LSTM的计算公式：
   
3. 作者把$h^{t-1}$作为输出$y_t$，然后拼成一个矩阵。$y_T=(y^1,y^2,...,y^T)$

3.2 Competitive random search

1. 把part a中的权重进行二进制编码， 每一个个体 $W_i$ 对应的权重传递到 part b，利用遗传算法筛选出最合适的 权重组合

这里并未使用所有的权重，而是挑选出了最合适的权重，umm，跟原来想的不太一样，原本以为是通过遗传算法训练attention的weight，现在只是通过遗传算法找到那些weight合适，其实做了一个筛选操作。送到LSTM神经网络中根据误差进行训练。

2. 然后重复步骤 c。
3. 最后构建新的种群。