PREDICTING MARKET DATA WITH A KALMAN
FILTER
by Rick Martinelli & Neil Rhoads
Haiku Laboratories, 2009
Note: This
article appears in Technical
Analysis of Stocks and Commodities in two parts as Predicting Market Data Using The Kalman Filter, January 2010 and Part 2, February
2010.
Available Profit and the Fortune Chart
Introduction
The chart below shows daily opens
for one year (252 days) of Ford Motor Co. (F).
Figure 1. Daily opens
for Ford Motor Co. (F) ending 07/30/09.
According
to modern financial engineering principals, market data such as this is
supposed to be a Brownian motion, which means that the daily price changes form
a whitenoise process. A whitenoise is
a random process in which consecutive values are independent of each other
(among other things), meaning a price increase is just as likely as a decrease
each day. However, in reality, it is not
uncommon for a particular market item to have several consecutive down days, or
up days, over a short time span. During
such spans the prices are said to be correlated. The objective is to harness these
correlations with a Kalman filter for prediction.
In a
previous article, a simple linear extrapolation was employed to predict
tomorrow’s pricechange; the prediction was then used to calculate the Alpha
statistic which compares the predicted pricechange to a recent average of
pricechanges. (See[1]) Relatively
large, positive values of Alpha indicate a long position, and relatively large,
negative values indicate a short position.
To test the effectiveness of the procedure as a trading scheme, it was
backtested on a random selection of stocks and indexes. Indicated positions were taken, then closed
out the next trading day, and all profits and losses were accumulated in a
chart called the Fortune. Surprisingly,
of the 28 items tested, 20 produced greater profits than a simple buy&hold
position for the same time period.
The
current article expands on the previous work, replacing the simple oneday
predictor with a Kalman Filter. The
Kalman Filter, as applied here, is a twostage algorithm which assumes there is
a smooth trendline within the data that represents the 'true' value of the market
item before being perturbed by 'market noise'.
In the first stage, a few previous trendline values are fit to a
suitable model, which is then extrapolated to the next time value to generate a
prediction and its error variance. In
the second stage, the corresponding data value is read and a new trend value is
computed as a compromise between the prediction and actual data value. The compromise is based on the relative
amounts of noise in the data and predictions.
The filter then repeats the cycle of prediction and correction as each
new data value is read.
The predicted price change and its standard deviation from the filter’s firststage are combined to produce the Alpha statistic which is then used to determine buysell signals. A simulated trading scheme executes those signals and, as before, profits and/or losses are accumulated in the Fortune. Items are selected for simulation based on a new dataproperty called the available profit. The ratio of the Fortune to available profit defines a characteristic of the filter called its efficiency. The next sections discuss the Kalman filter and details of the simulation method. The last sections discuss results of simulations and offer some conclusions.
The
Kalman filter is a recursive algorithm invented in the 1960’s to track a moving
target from noisy measurements of its position, and predict its future position
(See [2] for details). Applying this
technology to financial market data, the noisy measurements become the sequence
of prices
y_{1}, y_{2},…,y_{N}
where y_{k}
is the price of a market item on day k and N is the total number of days; for
the Ford data in Figure 1, N=252. The
simple equation
y_{k} = x_{k} + e_{k}
then
says that the each data value y_{k} is composed of a smooth trend x_{k}
and a random noise component e_{k} which is assumed to have zero
mean. The objective here is to estimate
the trend and use it to predict future data values. To do this, the Kalman filter employs an
auxiliary equation to predict a future trend value from previous trend
values. We chose
x_{kk1} = 3 (x_{k1}  x_{k2}) + x_{k3},
where x_{kk1}
is the predicted trend value on day
k, given data through day k1, and the x_{k}’s are the previous
filtered trend values. This is the model, or process, by which trend values are predicted. The resulting filter is called the quadratic filter, because it assumes
that every four consecutive trend values fit a quadratic curve. Associated with the model is a processnoise sequence q_{k},
which in this case represents a correction term in the quadratic assumption.
A final trend estimate is made after the current
data value y_{k} is input to the system:
x_{kk} = x_{kk1}
+ G_{k}(y_{k} – x_{kk1})
is
called the filtered trend value on
day k. The difference between actual and filtered values on day k is called the
residual: r_{k} = y_{k}
– x_{kk}. The innovation v_{k} = y_{k}
– x_{kk1 }(above in parentheses) represents the new information in y_{k}
that is not available in x_{kk1}.
The filter gain G_{k} determines
the contribution made by the innovation in the final estimate and thus controls
the tradeoff between adherence to the model and fidelity to the data. The gain, in turn, is determined by the
relative amounts of noise in the data and in the model, which are maintained in
the filter.
The
amount of noise in the data is denoted by R and is the variance of the residual
sequence r_{k} defined above.
The amount of noise in the model, or process, is denoted by Q and is the
variance of the process noise q_{k}.
While R may be calculated from the residuals, Q is completely
unknown. To estimate Q, we define the tracking parameter T by the equation
T = log(Q/R).
T
will be positive whenever Q < R and negative when Q > R. A negative T indicates the model has more
noise than the original data and may need to be changed. In the simulations below, values of T ranging
between 5 and 5 are tested and the value that minimizes the variance of the
innovation sequence v_{k} is taken as the optimal T value.
The
data in Figure 1 was tracked with the quadratic filter and an optimal T
= 1.86 was determined. This corresponds
to measurement noise that is about 72 times greater than model noise. Figure 2 shows how the innovation variance
behaves as T varies and shows the minimum value at 1.86. The data was tracked again using optimal T and
the resulting trend (filtered version of Figure 1) is shown in Figure 3. The gain for the track is shown in Figure 4;
the fact that it approaches a constant value (about 0.37) is typical of a
wellmodeled system.
Figure 2.
Innovation variance versus Tracking Parameter for the Ford data in
Figure 1. Optimal T is 1.86.
Figure 3.
Data from Figure 1 (green dots) and the Kalman estimated trend line
(red). Tracking Parameter T=1.86
Figure 4. Kalman gain corresponding to the
track in Figure 3.
How
well the filter and its intrinsic model perform depends on the filter’s
intended application. The current
application is oneday prediction of financial market pricechanges. To evaluate the filter in this context, the
following section describes an idealized trading scheme where the predictions x_{kk1}
and their standard deviation are used to calculate buy/sell signals.
The
Kalman filter provides predictions for each day in the data range (except the
first few startup points). Figure 5
shows predictions for a short portion of the data in Figure 1 (green
triangles).
Figure 5.
Kalman predictions for a portion of the data from 11/18/08 to 12/09/08
(green) together with the data.
The
filter also provides standard deviations of these predictions. Letting Δy_{k} = x_{kk1}
– y_{k1} denote the predicted price change on day k, and σ_{k}
the standard deviation of that prediction, the Alpha indicator for day k is defined as
a_{k}= Δy_{k} / σ_{k} .
Figure
6 shows the Alpha sequence for the data in Figure 1, where the horizontal lines
indicate the two Alpha cutoff values.
The irregular cyclic component visible in Alpha is typical of Kalman
tracks of market data, likely due to inherent correlations in the data.
Figure 6.
Alpha values for the Ford data in Figure 1. The horizontal lines indicate the Alpha
cutoff values 0.38 and 0.38.
If
Alpha is positive then the predicted movement is upward, and if Alpha is also
greater than one, the predicted change exceeds the average. Conversely, if Alpha is negative, the
predicted movement is downward, and if Alpha is less than negative one, the
predicted change once again exceeds the average. In the previous article, a_{k}
> 1 indicated a long position and a_{k} < 1 a short position. The present article generalizes the situation
somewhat. Here, a critical value C, called the Alpha cutoff, is determined for each track, where a_{k}
> C indicates a long position, a_{k}
< C indicates a short position,
and no position is taken otherwise.
Positions are cancelled the following day and the associated profit/loss
is accumulated in the Fortune. The cutoff C is determined by testing values between 0 and 3 in small
increments, and choosing the value that either yields the largest lastdayfortune, or yields the Fortune
chart closer to the available profit line than any other value. These
techniques are described next.
AVAILABLE PROFIT AND THE FORTUNE CHART
The
profit/loss on day k may be written
P_{k} = A W_{k} (y_{k}/y_{k1}
– 1),
where
the quantity in parentheses is the relative pricechange, or return, on day k, A is the trade amount
in dollars, and W_{k} = 1 if a_{k} > C,
W_{k} = 1 if a_{k}
< C and W_{k} = 0
otherwise (W for wager). Note that W_{k} = 0
corresponds to no trade on day k and so P_{k} = 0 as well. Note further that A is the same for every
trade. For the Ford data, C was found to be 0.38, and Figure 7
shows its P_{k} values, where the trade amount A was set to one dollar
(zero values are not shown). The red
point at 11/28/08 represents the trade having the largest profit. The reason for the large profit can be seen
by recalling Figure 5 above, where the red point in Figure 7 corresponds to
point 8 in Figure 5. The prediction of 1.84 was nowhere near the actual at
2.47, but it was in the correct direction,
up from 1.72, resulting in a 0.436 profit.
Figure 7.
P_{k} values for the Ford data in Figure 1. T=1.86, C=0.38, A=1.
The Fortune sequence is the accumulation of the P_{k}’s:
_{}
Its
last value F_{N} is called the lastday
fortune (LDF) and represents the amount of profit/loss realized at the end
of the simulation. A graph of F_{n}
versus day n, corresponding to the P_{k} data in Fig 7, is shown in
Figure 8. The LDF is 1.37 on 124 trades,
or about 1.1% average profit per trade.
This is, of course, an idealized fortune in which there are no trade
commissions, and trades can be transacted at the prescribed buy/sell prices (no
slippage). It is used here primarily to
evaluate the Kalman filter’s ability to predict the direction a stock price
will take.
Figure 8.
Fortune chart for a static Kalman track of the Ford data in Figure 1.
Quadratic Model.
T=1.86, C=0.38, A=1,
LDF=1.37.
The
Fortune plot is one indicator of that ability, and so is the profit ratio, defined as the ratio of
number of profitable trades to total trades.
In the Ford simulation it was 0.59.
In Figure 7, the profit ratio can be roughly visualized as the ratio of
number of points above the zero line to total points. A third way to evaluate the filter is by its efficiency. Suppose the scheme was able to capture every price change in the data, that is,
correctly predict direction each trading day. Assuming A=1, the sum
_{}
of
the absolute values of the returns represents the maximum amount of profit that
can be realized by day n (using this scheme), and is called the available profit (AP) on day n. The last value, S_{N}, is the total
AP in the data, and the ratio F_{N}/S_{N} is taken as the
filter’s efficiency. For the Ford
example, S_{N} is 13.15 making the efficiency 0.104, or about 10%. Figure 9 plots F_{n}, the evolution
of the Fortune (green line) compared with S_{n}, the evolution of the
AP, providing a view of the filter’s efficiency over time.
Figure 9. A comparison of the Fortune chart (green)
with the AP line for the Ford data.
The
AP line has a desirable property, called zero
downside volatility which we would like to see in the Fortune. Hence we should choose the Fortune line that
is ‘nearest’ the AP line, where the distance
D between the two lines is calculated as
_{}.
The
value of C that minimizes the distance
D can be taken as optimal C, which is
the method used with the Ford data. The
other method, maximizing the LDF, usually yields the same C value, but not always.
Given a choice, we take the largest C
value because, although the LDF may be smaller, the Fortune line is smoother.
The
filter was tested on one year of daily opens for a large group of selected
stocks (all data obtained from Yahoo
Finance). The simulations involved two
optimizations. The first optimization
determines the best Kalman tracking parameter and the second finds the best
Alpha cutoff. The simulation proceeds in
five steps as follows:
While
the Ford data was chosen moreorless at random, and for being wellknown, the
items used in the simulations were carefully selected to have the largest
AP’s. Opening prices for approximately
five thousand stocks were scanned for their oneyear available profit and many
of the stocks having AP’s greater than 7 were tracked with the quadratic
filter. One reason for choosing opens is
that they usually have the largest AP of the four daily prices. A representative sample of the results is
shown in the table below sorted by AP. Results for the Ford data are included and
have been highlighted. The last column
is the average profit/loss in dollars, or dollar
return, based on a trade amount A=$1000. To get a better feel for the nature of these
results, key features are plotted separately, starting with the sorted AP’s in
Figure 10.
Figure 10. Sorted AP’s for the 36 items in the
Table.
Figure 11. LDF’s corresponding to the AP’s in
Figure 10.
Figure 12. Filter efficiencies corresponding to the
AP’s in Figure 10.
Figures
10 and 11 show that filter efficiency is essentially uncorrelated with AP
(actual correlation is 0.07). The
obvious similarity between Figures 11 and 12 is artificial, because LDF values
are the product of AP and efficiency.
The average LDF is 2.2±1.9, a fairly wide range, and the average
efficiency is 12.0%±9.5%, a slightly narrower range than the LDF’s, but both
are still clearly dependent on the data.
Available profit is a property of the data only, and is independent of
the filter. Only the filter’s profit
ratio (Figure 13) seems to be (nearly) independent of the data being tracked,
its average being 0.51±0.07.
Figure 13.
Profit Ratios for the 36 test items and their average line (red).
The
highest efficiency is for number 18 in the table, BASI, at an incredible 52%,
with an accompanying LDF of 8.9. Data
and Fortune for BASI are shown in Figure 14.
BASI has the highest optimal T
at 4.8, implying the quadratic model is wellsuited to this data. It also has the smallest optimal C at 0.0, meaning every possible trade
is executed. The least efficient is PWR
at 1.1% with an LDF of 0.105, shown in Figure 15. Optimal T
and C values, AP and profit ratio
values are all near average, suggesting there is something inherent in this
data responsible for the filter’s relatively poor performance in this
case.
The
stock having the fewest trades and best dollar return is HEB at about $88 per
trade on 18 trades, shown in Figure 16. Its Fortune
chart differs from the others in that there are relatively long time frames
where the chart is flat, i.e., no trades occur.
This is due to the large optimal C value at 2.38, the largest in the
table. Results like HEB are attractive
for their minimal downward movement. TCIC has the smallest tracking parameter
at T=0.66, meaning model noise is about
double data noise suggesting another model should possibly be considered.
Figure 14.
Data (blue) and Fortune for BASI ending 07/10/09. Optimal T=4.80,
optimal C=0.0, AP=17.0, LDF=8.9,
profit ratio 0.47, 247 trades and about $36 per trade.
Figure 15.
Data and Fortune for PWR ending 06/10/09. Optimal T=1.80,
optimal C=0.12, AP=9.7, LDF=0.105,
profit ratio 0.51, 206 trades and $0.51 per trade.
Figure 15.
Data and Fortune for HEB ending 07/29/09. Optimal T=3.94,
optimal C=2.38, AP=18.9, LDF=1.59,
profit ratio 0.61, 18 trades and $88.28 per trade.
The goal here was to determine if a Kalman filter
could be exploited to predict the direction of stock price movements. Based on the LDF’s of the selected stocks in
the table, the implementation described here is remarkably effective in some
cases (e.g. BASI). The effectiveness of
the filter is traced to two primary factors: the nature of the data, namely,
its available profit and amount of correlation, and the efficiency of the
filter, which is determined by the filter’s model in combination with a
particular data set. Only the profit
ratio appears to be (nearly) independent of the data. It is apparent that selecting only the top
performers from the table would lead to a highly profitable portfolio. Unfortunately, the simulation presented here
could never be implemented in the real world because it requires all of the data on day one to determine
the optimal C and T values responsible for the filter’s
performance. However, application of
this scheme to a shorter time period, say the previous quarter or two, should
yield the best values for tomorrow’s
prediction! A simulation of this scheme
must await another article.
[1] Martinelli, Rick, "Harnessing
the (mis)Behavior of Markets", Technical Analysis of Stocks & Commodities,
June 2006
[2] Martinelli, Rick and
Rhoads, Neil, "Linear
Estimation and the Kalman Filter", Haiku Laboratories, 2008.

Symbol 
End
Date 
Last
Price 
AP 
T 
Optimal
C 
LDF 
Efficiency 
Profit
Ratio 
#
Trades 
Dollar
Return 
1 
MECA 
10/30/08 
1.62 
39.15 
4.720 
0.08 
1.901 
4.9% 
0.489 
221 
8.60 
2 
BUTL 
07/13/09 
0.02 
32.92 
3.380 
0.06 
5.926 
18.0% 
0.354 
206 
28.77 
3 
CTIC 
08/18/09 
1.54 
31.30 
0.660 
0.02 
1.610 
5.1% 
0.390 
187 
8.61 
4 
ANPI 
07/10/09 
1.41 
27.59 
4.000 
0.12 
6.441 
23.3% 
0.521 
192 
33.55 
5 
BJCT 
07/13/09 
0.20 
27.02 
3.280 
0.04 
5.048 
18.7% 
0.358 
232 
21.76 
6 
BPUR 
07/10/09 
0.16 
25.12 
3.380 
0.26 
2.452 
9.8% 
0.493 
150 
16.35 
7 
TKO 
07/29/09 
0.14 
23.87 
3.540 
0.36 
1.653 
6.9% 
0.443 
115 
14.38 
8 
BBGI 
07/10/09 
2.23 
22.03 
2.020 
0.34 
3.176 
14.4% 
0.509 
116 
27.38 
9 
CENX 
08/18/09 
11.00 
20.40 
1.460 
0.64 
1.003 
4.9% 
0.632 
38 
26.39 
10 
RAE 
07/29/09 
1.75 
20.34 
3.140 
0.36 
1.836 
9.0% 
0.515 
130 
14.12 
11 
MDH 
07/29/09 
1.26 
19.54 
4.120 
0.18 
5.536 
28.3% 
0.555 
182 
30.42 
12 
ACAS 
07/29/09 
3.35 
19.47 
0.640 
0.06 
1.787 
9.2% 
0.500 
192 
9.31 
13 
HEB 
07/29/09 
2.30 
18.86 
3.940 
2.38 
1.589 
8.4% 
0.611 
18 
88.28 
14 
CLZR 
08/18/09 
1.38 
18.64 
1.940 
0.24 
1.798 
9.6% 
0.467 
120 
14.98 
15 
NG 
07/29/09 
3.94 
18.22 
1.480 
0.06 
4.907 
26.9% 
0.563 
222 
22.10 
16 
GAN 
07/29/09 
14.24 
18.16 
3.300 
0.40 
1.456 
8.0% 
0.451 
122 
11.93 
17 
PSTI 
08/20/09 
1.63 
17.04 
2.000 
0.22 
1.343 
7.9% 
0.425 
160 
8.39 
18 
BASI 
07/10/09 
0.85 
16.97 
4.800 
0.00 
8.898 
52.4% 
0.470 
247 
36.02 
19 
PAL 
07/29/09 
2.54 
16.93 
1.500 
0.08 
1.327 
7.8% 
0.507 
217 
6.11 
20 
BCRX 
07/13/09 
3.86 
16.68 
1.600 
0.40 
0.463 
2.8% 
0.538 
106 
4.37 
21 
TIV 
07/29/09 
0.99 
16.53 
1.840 
0.00 
1.306 
7.9% 
0.498 
247 
5.29 
22 
BANR 
07/10/09 
2.93 
16.31 
1.960 
0.32 
1.522 
9.3% 
0.579 
95 
16.02 
23 
ABG 
07/29/09 
11.99 
16.27 
1.720 
1.04 
1.116 
6.9% 
0.611 
36 
30.99 
24 
FRG 
07/29/09 
3.68 
15.70 
1.540 
0.16 
1.316 
8.4% 
0.585 
200 
6.58 
25 
CNXT 
08/18/09 
2.35 
14.61 
0.860 
0.16 
2.201 
15.1% 
0.525 
160 
13.76 
26 
BPFH 
07/13/09 
4.48 
13.92 
1.280 
0.74 
0.122 
0.9% 
0.514 
37 
3.29 
27 
FAC 
06/10/09 
2.11 
13.72 
2.540 
0.04 
1.368 
10.0% 
0.465 
228 
6.00 
28 
BPOP 
07/13/09 
1.40 
13.16 
1.560 
0.10 
1.829 
13.9% 
0.543 
173 
10.57 
29 
F 
07/29/09 
7.08 
13.15 
1.860 
0.38 
1.367 
10.4% 
0.589 
124 
11.02 
30 
BELFA 
07/10/09 
13.39 
13.12 
2.920 
0.28 
3.131 
23.9% 
0.546 
152 
20.60 
31 
CACH 
08/18/09 
4.43 
12.82 
1.820 
0.18 
1.160 
9.1% 
0.522 
113 
10.26 
32 
PWR 
06/10/09 
25.66 
9.76 
1.800 
0.12 
0.105 
1.1% 
0.505 
206 
0.51 
33 
CMCO 
08/18/09 
14.03 
9.60 
1.580 
0.12 
1.019 
10.6% 
0.539 
206 
4.95 
34 
CAKE 
08/18/09 
17.91 
9.18 
2.600 
0.94 
0.489 
5.3% 
0.545 
33 
14.82 
35 
GE 
06/11/09 
13.43 
8.38 
2.740 
0.66 
0.717 
8.6% 
0.614 
57 
12.58 
36 
CALM 
08/18/09 
28.72 
7.50 
1.180 
0.28 
0.941 
12.5% 
0.532 
126 
7.47 
Results of simulations of 36 large AP stocks.