Incremental Implementation

Stationary Problem

In stationary problems, the probability of reward do not change over time.
As step goes larger, we would like to compute the value of an action efficiently, in particular, with constant memory and constant per-time-step computation.
To simplify notation, we concentrate on a single action.

1. Denote $R_i$ the reward received after the $i^{th}$ selection of this action, and let $Q_n$ denote the estimate of its action value after it has been selected $n-1$ times, which we can now write simply as: $Q_n\doteq \frac{R_1+R_2+...+R_{n-1}}{n-1}$ .
2. Given $Q_n$ and the $n^{th}$ reward, $R_n$, the new average of all $n$ rewards can be write as: $Q_{n+1}=\frac{1}{n}{\sum }_{i=1}^{n-1}{R}_i=Q_n+\frac{1}{n}[R_n-Q_n]$.

The general form of this update rule is:
$NewEstimate\leftarrow OldEstimate+StepSize[Target-OldEstimate]$
StepSize is always denoted as ${\alpha }_t(a)$

Nonstationary Problem

In nonstationary problems, the probability of reward change over time. In this cases, it makes sense to give more weights to recent rewards than to long-term rewards.

1. The incremental update rule is modified to be:
   $Q_{n+1}\doteq Q_n+\alpha [R_n-Q_n]$, where $\alpha \in (0,1]$
2. Expotential recency-weighted average:This results in $Q_{n+1}$ being a weighted average of past rewards and the initial estimate $Q_1$:
   $Q_{n+1}=(1-\alpha {)}^n{Q}_1+{\sum }_{i=1}^n\alpha (1-\alpha {)}^{n-i}{R}_i$