PREDICTING MARKET DATA WITH A KALMAN FILTER

by Rick Martinelli & Neil Rhoads

Haiku Laboratories, 2009

Note: This article appears in Technical Analysis of Stocks and Commodities in two parts as Predicting Market Data Using The Kalman Filter, January 2010 and Part 2, February 2010.

Introduction

The chart below shows daily opens for one year (252 days) of Ford Motor Co. (F). Figure 1. Daily opens for Ford Motor Co. (F) ending 07/30/09.

According to modern financial engineering principals, market data such as this is supposed to be a Brownian motion, which means that the daily price changes form a white-noise process.  A white-noise is a random process in which consecutive values are independent of each other (among other things), meaning a price increase is just as likely as a decrease each day.  However, in reality, it is not uncommon for a particular market item to have several consecutive down days, or up days, over a short time span.  During such spans the prices are said to be correlated.  The objective is to harness these correlations with a Kalman filter for prediction.

In a previous article, a simple linear extrapolation was employed to predict tomorrow’s price-change; the prediction was then used to calculate the Alpha statistic which compares the predicted price-change to a recent average of price-changes. (See)  Relatively large, positive values of Alpha indicate a long position, and relatively large, negative values indicate a short position.  To test the effectiveness of the procedure as a trading scheme, it was back-tested on a random selection of stocks and indexes.  Indicated positions were taken, then closed out the next trading day, and all profits and losses were accumulated in a chart called the Fortune.  Surprisingly, of the 28 items tested, 20 produced greater profits than a simple buy&hold position for the same time period.

The current article expands on the previous work, replacing the simple one-day predictor with a Kalman Filter.  The Kalman Filter, as applied here, is a two-stage algorithm which assumes there is a smooth trend-line within the data that represents the 'true' value of the market item before being perturbed by 'market noise'.  In the first stage, a few previous trend-line values are fit to a suitable model, which is then extrapolated to the next time value to generate a prediction and its error variance.  In the second stage, the corresponding data value is read and a new trend value is computed as a compromise between the prediction and actual data value.  The compromise is based on the relative amounts of noise in the data and predictions.  The filter then repeats the cycle of prediction and correction as each new data value is read.

The predicted price change and its standard deviation from the filter’s first-stage are combined to produce the Alpha statistic which is then used to determine buy-sell signals. A simulated trading scheme executes those signals and, as before, profits and/or losses are accumulated in the Fortune. Items are selected for simulation based on a new data-property called the available profit. The ratio of the Fortune to available profit defines a characteristic of the filter called its efficiency. The next sections discuss the Kalman filter and details of the simulation method. The last sections discuss results of simulations and offer some conclusions.

THE KALMAN FILTER

The Kalman filter is a recursive algorithm invented in the 1960’s to track a moving target from noisy measurements of its position, and predict its future position (See  for details).  Applying this technology to financial market data, the noisy measurements become the sequence of prices

y1, y2,…,yN

where yk is the price of a market item on day k and N is the total number of days; for the Ford data in Figure 1, N=252.  The simple equation

yk = xk + ek

then says that the each data value yk is composed of a smooth trend xk and a random noise component ek which is assumed to have zero mean.  The objective here is to estimate the trend and use it to predict future data values.  To do this, the Kalman filter employs an auxiliary equation to predict a future trend value from previous trend values.  We chose

xk|k-1 = 3 (xk-1 - xk-2) +  xk-3,

where xk|k-1 is the predicted trend value on day k, given data through day k-1, and the xk’s are the previous filtered trend values.  This is the model, or process, by which trend values are predicted.  The resulting filter is called the quadratic filter, because it assumes that every four consecutive trend values fit a quadratic curve.  Associated with the model is a process-noise sequence qk, which in this case represents a correction term in the quadratic assumption.

A final trend estimate is made after the current data value yk is input to the system:

xk|k = xk|k-1 + Gk(yk – xk|k-1)

is called the filtered trend value on day k. The difference between actual and filtered values on day k is called the residual: rk = yk – xk|k.  The innovation vk = yk – xk|k-1 (above in parentheses) represents the new information in yk that is not available in xk|k-1.  The filter gain Gk determines the contribution made by the innovation in the final estimate and thus controls the tradeoff between adherence to the model and fidelity to the data.  The gain, in turn, is determined by the relative amounts of noise in the data and in the model, which are maintained in the filter.

The amount of noise in the data is denoted by R and is the variance of the residual sequence rk defined above.  The amount of noise in the model, or process, is denoted by Q and is the variance of the process noise qk.  While R may be calculated from the residuals, Q is completely unknown.  To estimate Q, we define the tracking parameter T by the equation

T = -log(Q/R).

T will be positive whenever Q < R and negative when Q > R.  A negative T indicates the model has more noise than the original data and may need to be changed.  In the simulations below, values of T ranging between -5 and 5 are tested and the value that minimizes the variance of the innovation sequence vk is taken as the optimal T value.

The data in Figure 1 was tracked with the quadratic filter and an optimal T = 1.86 was determined.  This corresponds to measurement noise that is about 72 times greater than model noise.   Figure 2 shows how the innovation variance behaves as T varies and shows the minimum value at 1.86.  The data was tracked again using optimal T and the resulting trend (filtered version of Figure 1) is shown in Figure 3.  The gain for the track is shown in Figure 4; the fact that it approaches a constant value (about 0.37) is typical of a well-modeled system. Figure 2.  Innovation variance versus Tracking Parameter for the Ford data in Figure 1.  Optimal T is 1.86. Figure 3.  Data from Figure 1 (green dots) and the Kalman estimated trend line (red). Tracking Parameter T=1.86 Figure 4. Kalman gain corresponding to the track in Figure 3.

How well the filter and its intrinsic model perform depends on the filter’s intended application.  The current application is one-day prediction of financial market price-changes.  To evaluate the filter in this context, the following section describes an idealized trading scheme where the predictions xk|k-1 and their standard deviation are used to calculate buy/sell signals.

The ALPHA Indicator

The Kalman filter provides predictions for each day in the data range (except the first few startup points).  Figure 5 shows predictions for a short portion of the data in Figure 1 (green triangles). Figure 5.  Kalman predictions for a portion of the data from 11/18/08 to 12/09/08 (green) together with the data.

The filter also provides standard deviations of these predictions.  Letting Δyk = xk|k-1 – yk-1 denote the predicted price change on day k, and σk the standard deviation of that prediction, the Alpha indicator for day k is defined as

ak= Δyk / σk .

Figure 6 shows the Alpha sequence for the data in Figure 1, where the horizontal lines indicate the two Alpha cutoff values.  The irregular cyclic component visible in Alpha is typical of Kalman tracks of market data, likely due to inherent correlations in the data. Figure 6.  Alpha values for the Ford data in Figure 1.  The horizontal lines indicate the Alpha cutoff values 0.38 and -0.38.

If Alpha is positive then the predicted movement is upward, and if Alpha is also greater than one, the predicted change exceeds the average.  Conversely, if Alpha is negative, the predicted movement is downward, and if Alpha is less than negative one, the predicted change once again exceeds the average.  In the previous article, ak > 1 indicated a long position and ak < -1 a short position.  The present article generalizes the situation somewhat.  Here, a critical value C, called the Alpha cutoff, is determined for each track, where ak > C indicates a long position, ak < -C indicates a short position, and no position is taken otherwise.  Positions are cancelled the following day and the associated profit/loss is accumulated in the Fortune.  The cutoff C is determined by testing values between 0 and 3 in small increments, and choosing the value that either yields the largest last-day-fortune, or yields the Fortune chart closer to the available profit line than any other value. These techniques are described next.

AVAILABLE PROFIT AND THE FORTUNE CHART

The profit/loss on day k may be written

Pk = A Wk  (yk/yk-1 – 1),

where the quantity in parentheses is the relative price-change, or return, on day k, A is the trade amount in dollars, and Wk = 1 if ak > C, Wk = -1 if ak < -C and Wk = 0 otherwise (W for wager).  Note that Wk = 0 corresponds to no trade on day k and so Pk = 0 as well.  Note further that A is the same for every trade.  For the Ford data, C was found to be 0.38, and Figure 7 shows its Pk values, where the trade amount A was set to one dollar (zero values are not shown).  The red point at 11/28/08 represents the trade having the largest profit.  The reason for the large profit can be seen by recalling Figure 5 above, where the red point in Figure 7 corresponds to point 8 in Figure 5. The prediction of 1.84 was nowhere near the actual at 2.47, but it was in the correct direction, up from 1.72, resulting in a 0.436 profit. Figure 7.  Pk values for the Ford data in Figure 1. T=1.86, C=0.38, A=1.

The Fortune sequence is the accumulation of the Pk’s: Its last value FN is called the last-day fortune (LDF) and represents the amount of profit/loss realized at the end of the simulation.  A graph of Fn versus day n, corresponding to the Pk data in Fig 7, is shown in Figure 8.  The LDF is 1.37 on 124 trades, or about 1.1% average profit per trade.  This is, of course, an idealized fortune in which there are no trade commissions, and trades can be transacted at the prescribed buy/sell prices (no slippage).  It is used here primarily to evaluate the Kalman filter’s ability to predict the direction a stock price will take. Figure 8.  Fortune chart for a static Kalman track of the Ford data in Figure 1.

Quadratic Model.  T=1.86, C=0.38, A=1, LDF=1.37.

The Fortune plot is one indicator of that ability, and so is the profit ratio, defined as the ratio of number of profitable trades to total trades.  In the Ford simulation it was 0.59.  In Figure 7, the profit ratio can be roughly visualized as the ratio of number of points above the zero line to total points.  A third way to evaluate the filter is by its efficiency.  Suppose the scheme was able to capture every price change in the data, that is, correctly predict direction each trading day. Assuming A=1, the sum of the absolute values of the returns represents the maximum amount of profit that can be realized by day n (using this scheme), and is called the available profit (AP) on day n.  The last value, SN, is the total AP in the data, and the ratio FN/SN is taken as the filter’s efficiency.  For the Ford example, SN is 13.15 making the efficiency 0.104, or about 10%.  Figure 9 plots Fn, the evolution of the Fortune (green line) compared with Sn, the evolution of the AP, providing a view of the filter’s efficiency over time. Figure 9. A comparison of the Fortune chart (green) with the AP line for the Ford data.

The AP line has a desirable property, called zero downside volatility which we would like to see in the Fortune.  Hence we should choose the Fortune line that is ‘nearest’ the AP line, where the distance D between the two lines is calculated as .

The value of C that minimizes the distance D can be taken as optimal C, which is the method used with the Ford data.  The other method, maximizing the LDF, usually yields the same C value, but not always.  Given a choice, we take the largest C value because, although the LDF may be smaller, the Fortune line is smoother.

BACK-TESTING THE FILTER

The filter was tested on one year of daily opens for a large group of selected stocks (all data obtained from Yahoo Finance).  The simulations involved two optimizations.  The first optimization determines the best Kalman tracking parameter and the second finds the best Alpha cutoff.  The simulation proceeds in five steps as follows:

1. Track one year of data multiple times to find the optimal tracking parameter T
2. Track the data once more using optimal T
3. Use the resulting Kalman predictions and their standard deviations to find the Alpha sequence ak
4. Use that Alpha sequence to find the optimal cutoff C
5. Use the optimal cutoff C to calculate the Fortune chart

While the Ford data was chosen more-or-less at random, and for being well-known, the items used in the simulations were carefully selected to have the largest AP’s.   Opening prices for approximately five thousand stocks were scanned for their one-year available profit and many of the stocks having AP’s greater than 7 were tracked with the quadratic filter.  One reason for choosing opens is that they usually have the largest AP of the four daily prices.  A representative sample of the results is shown in the table below sorted by AP.  Results for the Ford data are included and have been highlighted.  The last column is the average profit/loss in dollars, or dollar return, based on a trade amount A=\$1000.  To get a better feel for the nature of these results, key features are plotted separately, starting with the sorted AP’s in Figure 10. Figure 10. Sorted AP’s for the 36 items in the Table. Figure 11. LDF’s corresponding to the AP’s in Figure 10. Figure 12. Filter efficiencies corresponding to the AP’s in Figure 10.

Figures 10 and 11 show that filter efficiency is essentially uncorrelated with AP (actual correlation is 0.07).  The obvious similarity between Figures 11 and 12 is artificial, because LDF values are the product of AP and efficiency.  The average LDF is 2.2±1.9, a fairly wide range, and the average efficiency is 12.0%±9.5%, a slightly narrower range than the LDF’s, but both are still clearly dependent on the data.  Available profit is a property of the data only, and is independent of the filter.  Only the filter’s profit ratio (Figure 13) seems to be (nearly) independent of the data being tracked, its average being 0.51±0.07. Figure 13.  Profit Ratios for the 36 test items and their average line (red).

The highest efficiency is for number 18 in the table, BASI, at an incredible 52%, with an accompanying LDF of 8.9.  Data and Fortune for BASI are shown in Figure 14.  BASI has the highest optimal T at 4.8, implying the quadratic model is well-suited to this data.  It also has the smallest optimal C at 0.0, meaning every possible trade is executed.  The least efficient is PWR at 1.1% with an LDF of 0.105, shown in Figure 15.  Optimal T and C values, AP and profit ratio values are all near average, suggesting there is something inherent in this data responsible for the filter’s relatively poor performance in this case.

The stock having the fewest trades and best dollar return is HEB at about \$88 per trade on 18 trades, shown in Figure 16. Its Fortune chart differs from the others in that there are relatively long time frames where the chart is flat, i.e., no trades occur.  This is due to the large optimal C value at 2.38, the largest in the table.  Results like HEB are attractive for their minimal downward movement. TCIC has the smallest tracking parameter at T=-0.66,  meaning model noise is about double data noise suggesting another model should possibly be considered.  Figure 14.  Data (blue) and Fortune for BASI ending 07/10/09.  Optimal T=4.80, optimal C=0.0, AP=17.0, LDF=8.9, profit ratio 0.47, 247 trades and about \$36 per trade.  Figure 15.  Data and Fortune for PWR ending 06/10/09.  Optimal T=1.80, optimal C=0.12, AP=9.7, LDF=0.105, profit ratio 0.51, 206 trades and \$0.51 per trade.  Figure 15.  Data and Fortune for HEB ending 07/29/09.  Optimal T=3.94, optimal C=2.38, AP=18.9, LDF=1.59, profit ratio 0.61, 18 trades and \$88.28 per trade.

CONCLUSIONS & QUESTIONS

The goal here was to determine if a Kalman filter could be exploited to predict the direction of stock price movements.   Based on the LDF’s of the selected stocks in the table, the implementation described here is remarkably effective in some cases (e.g. BASI).  The effectiveness of the filter is traced to two primary factors: the nature of the data, namely, its available profit and amount of correlation, and the efficiency of the filter, which is determined by the filter’s model in combination with a particular data set.  Only the profit ratio appears to be (nearly) independent of the data.   It is apparent that selecting only the top performers from the table would lead to a highly profitable portfolio.  Unfortunately, the simulation presented here could never be implemented in the real world because it requires all of the data on day one to determine the optimal C and T values responsible for the filter’s performance.  However, application of this scheme to a shorter time period, say the previous quarter or two, should yield the best values for tomorrow’s prediction!  A simulation of this scheme must await another article.

References

 Martinelli, Rick, "Harnessing the (mis)Behavior of Markets", Technical Analysis of Stocks & Commodities, June 2006

 Martinelli, Rick and Rhoads, Neil, "Linear Estimation and the Kalman Filter", Haiku Laboratories, 2008.

 Symbol End Date Last Price AP T Optimal C LDF Efficiency Profit Ratio # Trades Dollar Return 1 MECA 10/30/08 1.62 39.15 4.720 0.08 1.901 4.9% 0.489 221 8.60 2 BUTL 07/13/09 0.02 32.92 3.380 0.06 5.926 18.0% 0.354 206 28.77 3 CTIC 08/18/09 1.54 31.30 -0.660 0.02 1.610 5.1% 0.390 187 8.61 4 ANPI 07/10/09 1.41 27.59 4.000 0.12 6.441 23.3% 0.521 192 33.55 5 BJCT 07/13/09 0.20 27.02 3.280 0.04 5.048 18.7% 0.358 232 21.76 6 BPUR 07/10/09 0.16 25.12 3.380 0.26 2.452 9.8% 0.493 150 16.35 7 TKO 07/29/09 0.14 23.87 3.540 0.36 1.653 6.9% 0.443 115 14.38 8 BBGI 07/10/09 2.23 22.03 2.020 0.34 3.176 14.4% 0.509 116 27.38 9 CENX 08/18/09 11.00 20.40 1.460 0.64 1.003 4.9% 0.632 38 26.39 10 RAE 07/29/09 1.75 20.34 3.140 0.36 1.836 9.0% 0.515 130 14.12 11 MDH 07/29/09 1.26 19.54 4.120 0.18 5.536 28.3% 0.555 182 30.42 12 ACAS 07/29/09 3.35 19.47 0.640 0.06 1.787 9.2% 0.500 192 9.31 13 HEB 07/29/09 2.30 18.86 3.940 2.38 1.589 8.4% 0.611 18 88.28 14 CLZR 08/18/09 1.38 18.64 1.940 0.24 1.798 9.6% 0.467 120 14.98 15 NG 07/29/09 3.94 18.22 1.480 0.06 4.907 26.9% 0.563 222 22.10 16 GAN 07/29/09 14.24 18.16 3.300 0.40 1.456 8.0% 0.451 122 11.93 17 PSTI 08/20/09 1.63 17.04 2.000 0.22 1.343 7.9% 0.425 160 8.39 18 BASI 07/10/09 0.85 16.97 4.800 0.00 8.898 52.4% 0.470 247 36.02 19 PAL 07/29/09 2.54 16.93 1.500 0.08 1.327 7.8% 0.507 217 6.11 20 BCRX 07/13/09 3.86 16.68 1.600 0.40 0.463 2.8% 0.538 106 4.37 21 TIV 07/29/09 0.99 16.53 1.840 0.00 1.306 7.9% 0.498 247 5.29 22 BANR 07/10/09 2.93 16.31 1.960 0.32 1.522 9.3% 0.579 95 16.02 23 ABG 07/29/09 11.99 16.27 1.720 1.04 1.116 6.9% 0.611 36 30.99 24 FRG 07/29/09 3.68 15.70 1.540 0.16 1.316 8.4% 0.585 200 6.58 25 CNXT 08/18/09 2.35 14.61 0.860 0.16 2.201 15.1% 0.525 160 13.76 26 BPFH 07/13/09 4.48 13.92 1.280 0.74 0.122 0.9% 0.514 37 3.29 27 FAC 06/10/09 2.11 13.72 2.540 0.04 1.368 10.0% 0.465 228 6.00 28 BPOP 07/13/09 1.40 13.16 1.560 0.10 1.829 13.9% 0.543 173 10.57 29 F 07/29/09 7.08 13.15 1.860 0.38 1.367 10.4% 0.589 124 11.02 30 BELFA 07/10/09 13.39 13.12 2.920 0.28 3.131 23.9% 0.546 152 20.60 31 CACH 08/18/09 4.43 12.82 1.820 0.18 1.160 9.1% 0.522 113 10.26 32 PWR 06/10/09 25.66 9.76 1.800 0.12 0.105 1.1% 0.505 206 0.51 33 CMCO 08/18/09 14.03 9.60 1.580 0.12 1.019 10.6% 0.539 206 4.95 34 CAKE 08/18/09 17.91 9.18 2.600 0.94 0.489 5.3% 0.545 33 14.82 35 GE 06/11/09 13.43 8.38 2.740 0.66 0.717 8.6% 0.614 57 12.58 36 CALM 08/18/09 28.72 7.50 1.180 0.28 0.941 12.5% 0.532 126 7.47

Results of simulations of 36 large AP stocks.