pitfalls of backtesting

"I only believe in the backtests that I doctored myself." -Winston Churchill

Backtest a simulation that seeks to estimate the future performance of an investment strategy by testing its performance in the past.

Because who wouldn't want to test out their new option strategy over the last 100 years to see if it makes money before putting up actual cash? Backtesting can be a great way to test your strategies to see how they perform during various types of markets and conditions. How does your strategy perform when the market drops 10% in 5 days, or how does it hold up when a company reports earnings? These questions can be answered by building your strategy into a backtest and running it over various conditions.

If only it were that easy. When backtesting a strategy, traders will typically fall into a few common pitfalls that will skew their performance from actual results. Such mistakes can lead to fake profits but real losses.

Have you ever backtested an option strategy only to have it fail when you tried to trade it for real?

Don't worry, you are not alone. We will show you what the common pitfalls are and how to avoid them when building your backtest. If you don’t plan on building your own backtest for your strategies, we want you to at least take this away: never blindly trust a backtest.

Pitfall 1: Survivorship Bias

After you have built your strategy, the first thing you will want to do is test it on a certain time period in the past. Most backtesting software will fail you at this point due to survivorship bias. Survivorship bias means you are excluding companies that no longer exist from your test. Typically, a test will only include current companies, and will ignore the companies that are no longer around.

If you were to run a backtest on all stocks available in the S&P 500 over the last 20 years, there would be a lot of significant companies missing from that test. The S&P 500 is periodically rebalanced, and companies that have gone bankrupt no longer exist in the index.

source: http://www.cboe.com/rmc/2015/olivier-pdf-Backtesting-Full.pdf

(source: http://www.cboe.com/rmc/2015/olivier-pdf-Backtesting-Full.pdf)

By not including these companies, your results will be skewed higher than the actual results, because all of the bankrupt companies will not be included in your calculations. Ernest Chan ran a backtest with and without survivorship bias. He found, "if we use a database of stocks that has survivorship bias: we will find this portfolio returns 388% in 2001. But what if we use a good, survivorship-bias-free database? The portfolio will return -42%, because it will contain picks such as ETYS, INTW, and FDHG that have all gone out of business by the end of 2001."

To solve this pitfall, you want to find a backtesting program that is survivorship bias free. This will make sure your test includes all of the stocks that were there in the past. Unfortunately, the only way this can be solved is with the use of a program. You can no longer go back and see charts of Enron, WorldComm, or the many companies that failed in the Nasdaq Bubble.

Pitfall 2: Lookahead Bias

The lookahead bias is a very popular trap in which to fall into. It is very easy to peer into the future when you have all the data right in front of your eyes and can make trades according to that data.

If you are calculating information, or making trades on the day's closing price, you won't be able to make those trades until the next day. Think of any indicators you have setup to trade off of, and figure out which ones require the end-of-day data. You can't actually trade on the close, which is a common mistake of backtesting.

When setting up your backtest, make sure you are not taking the trades until morning, when you can actually make the trades.

Pitfall 3: In-Sample Bias

When you are building your strategy, it is common to focus on one time period. You will use that time period to alter the rules and parameters of your strategy so you can hone it. The problem arises when you use that time period as your only period for testing. If you test in a bull market only, your strategy is only bound to work when we are in a bull market. If you are thinking, that is fine, because you will only run your strategy in bull markets remember that would fall under the lookahead bias. In reality, you would need to identify a bull market as it’s beginning, a feat, as you know, that can be quite difficult.

It is best to refine your strategy in one period, but test it over many periods. Try to find times when the market has levels of extreme volatility, low volatility, bull markets, bear markets, and when we are stuck in a range.

Pitfall 4: Data-Mining

You know the saying; it is better to be lucky than good. Unfortunately, this is not the case when it comes to trade performance. What happens is that you select a set of data points on which to hone your strategy. You then make small variations to your parameters until one hits and produces positive performance.

The problem is that these results are more likely to be luck, rather than you discovering an actual winning strategy. These types of strategies lack any type of predictive abilities, and won’t produce your tested results in real time.

For example, you want to test a trading strategy that buys a stock based on moving averages. You want to know if there will be positive performance when you long the stock when the 1 week moving average is greater than the 2 week moving average. If that doesn't produce good enough results, you could test the 2 week moving average being greater than the 3 week moving average. This could go on forever until you hit that one lucky time the strategy produced a gain.

The best way to avoid this pitfall is to make sure you paper trade your strategy once you have developed it. If you can prove significant gains going forward; that will show the strategy was not sheer luck, but it is actually a profitable system.

Pitfall 5: Forgetting The Little Things

Backtesting is good because you can setup the strategy you want to test, click a couple of buttons and calculate your results. If only it were that easy in real life. When you are actually trading, you have to account for liquidity, feasibility and transaction cost.

It is hard to get proper position fills if you are trying to trade a lot of size in an illiquid name, or if you are buying too much of a stock that doesn’t have a lot of volume. Backtesting won't typically pick up on which names are illiquid or which stocks won't be able to fill your orders. You will need to set up your backtest to ignore these names or at least take into consideration how much size you are playing with in the market.

If you are running a strategy that deals with shorting stocks, you need to be aware that not all stocks can be shorted. A lot of the time small capitalization stocks can be placed on a hard-to-borrow list making it more difficult or even impossible to initiate a short position. Shorting stocks can also incur additional fees and margin requirements.

Transaction cost will typically make up a small percentage of a trade, but it does add up over time. This is especially true if your strategy aims to enter and exit a lot of positions quickly. Fred Piard notes "even with a low-cost broker, trading fees can have a heavy drag, especially on daily strategies." Adding in transaction cost, such as commissions and regulatory fees, can turn a profitable strategy into a losing strategy.

Conclusion

Backtesting can be a successful way to test your option strategies without risking capital. Unfortunately, you cannot simply setup your strategy and click a couple of buttons to calculate the results. Before you trust any backtest blindly, you need to run it through a couple of tests to make sure it will hold up to the real deal. The most common pitfalls you need to avoid are survivorship bias, lookahead bias, in-sample bias, data mining, and forgetting the little things that make the trades work.

When developing your trading strategies, you want to build them on the past data, test them over a wide range of market conditions, and then test them going forward using paper money. Once your strategy has proven itself, you can finally take it to live trading where you risk actual capital.

How are you using backtesting in your trading? Tell us in the comments...

round testimonial

"Last year The Option Prophet absolutely transformed my trading account from one where monthly gains were often uncertain and unpredictable into one where a steady monthly income stream through OTM credit spreads bought amazingly steady and consistent growth."

-Andrew B. 

growmyportfoliobig