Backtesting Is Finance’s Version of Retrospective Research

November 29th, 2017
5

A lot of effort and money is spent looking for ways to make money in the stock market. While you can get perfectly good investment returns through index fund investing, everyone is looking for that magic trading strategy that will help them get a little bit of extra return. After all, if you can consistently earn just 1% more in returns each year, you might end up with millions more in retirement.

What better way is there to find ways to beat the market than to study the past? If history repeats itself, then maybe the stock market does as well.

Enter backtesting.

How Backtesting Works

Backtesting is when you test a trading strategy on past stock market prices.

Here’s a simple example. Let’s say that you have a theory that the stock market tends to go up in the week after Thanksgiving. Everyone is in a really great mood after the holiday. People shop online and visit malls to buy gifts for their family and friends, which puts them in a buying mood in the stock market as well.

So you might look at the last 50 Black Fridays and Cyber Mondays and see how the Dow Jones Industrial Average or the S&P 500 has performed. If the market has gone up more often than not in the week after Thanksgiving, maybe your theory is correct and you should actually buy the S&P 500 this year.

Many technical analysis strategies are evaluated with backtesting. For example, you might think that you could make money if you bought a stock whenever it crossed above its 200-day moving average and sold whenever it crossed below its 200-day moving average. To test this theory, you’d look back at past charts to see if this trading strategy would have worked.

Traders have advanced systems to formally backtest their trading strategies. Even Fidelity (Wealth-Lab Pro) and Schwab (Strategy Tester) have backtesting software available to active traders.

Backtested Trading Strategies Always Look Good On Paper

Many new hedge funds or trading newsletters have centered their sales pitches around backtested data. Because they have no track record of making money for their clients, they rely on beautiful charts or graphs showing how their trading strategies would have worked in the past.

Does Backtesting Sound Familiar, Residents and Academic Physicians?

Backtesting occurs in medicine as well. It’s called retrospective research.

A medical student or resident is asked to look at a bunch of charts and determine whether clinical or treatment factors are associated with improved patient outcomes. For example, you might look at the charts of 100 pancreatic cancer patients treated at a single hospital and see whether patients who received radiation performed better than those who did not receive radiation.

The Weaknesses of Retrospective Research (And Backtesting)

But any physician who has done even a little bit of medical research can identify the weaknesses of retrospective research.

One issue is that many hypotheses are often tested, but only the positive results are reported. Applying the standard p < 0.05 threshold for statistical significance will lead to many false-positive results if you do not formally account for testing multiple hypotheses.

This has led to a term called p-hacking. The data science blog FiveThirtyEight has explored the problem of p-hacking in academic research, with a cool interactive graphic that allows users to quickly generate “publishable” results.

There is also significant selection bias in publicized backtested results (whether it be in a hedge fund prospectus, academic financial journal, or message board). Because the financial literature, like the medical literature, prefers positive results rather than negative results because they are more likely to be high impact, people are motivated to publicize positive results, even if they end up being spurious conclusions.

Is There Such A Thing As Prospective Research In Finance?

In medicine, retrospective research is generally considered hypothesis-generating, and it typically does not change routine clinical practice until a prospective trial can be performed. For many clinical questions, a randomized controlled clinical trial is done to confirm retrospective research, but some diseases in medicine are too rare to run a clinical trial.

Whenever a finance researcher discovers a trading strategy based on historical data, is there any way to confirm whether these results are real or spurious?

Unfortunately in finance, there’s no way to run a randomized controlled clinical trial. Ideally, you would try to validate the trading strategy prospectively (i.e. start trading with real money). Unfortunately, it can take years or even decades to truly know whether a trading strategy works, because there is so much variance in stock market returns.

That’s not to say that there is nothing that can be gleaned from past data in medicine or in finance. Some clinical questions occur too infrequently to run a randomized clinical trial, and treatment decisions are guided by retrospective research alone. Similarly, there probably have been finance strategies that have been identified through backtesting and made their discoverers very wealthy (although they probably won’t share it with you or me).

Conclusion

Be wary of backtested results that you see on the Internet and elsewhere. Understand that the usual limitations of “hypothesis-generating” retrospective research applies to backtested trading strategies in finance as well.

What do you think? Have you ever tested a trading strategy on historical data? What do you think of backtesting in finance?

5 COMMENTS

  1. Really nice post!I love the parallel with medical hypothesis testing,that seems very natural to me.
    I am schizophrenic on back testing. In my job I despise the investment banks and hedge funds that come to me with their beautiful looking back tested results. But in my blog I use it a lot!
    I’ve become very jaded with Monte Carlo simulations (that are forward looking) so back testing for my own work has some appeal.
    But I recognize that I can talk out of both sides of my mouth.

    • I lost my job few months back and there was no way to get income for my family, things was so tough and I couldn’t get anything for my children, not until a met a recommendation on a page writing how Mr Bernie Wilfred helped a lady in getting a huge amount of profit every 6 working days on trading with his management on the cryptocurrency Market, to be honest I never believe it but I took the risk to take a loan of $2000. and I contacted him unbelievable and I was so happy I earn $22,500 in 6 working days, the most joy is that I can now take care of my family I don’t know how to appreciate your good work Mr. Bernie Doran God will continue to bless you for being a life saver I have no way to appreciate you than to tell people about your good services.
For a perfect investment and good strategies contact Mr Bernie Doran via Telegram : @Bernie_fx or Email : Bernie.doranfx01@gmail.com

  2. I think to some extent we can work to gain extra support for a strategy that looks profitable from past data. We can look to see if the strategy has worked in other time periods, other geographic markets, even other asset classes. Granted these are all retrospective, but supporting data nonetheless. For example, Fama-French first displayed value based on US equity data about 1962-1989. Subsequently, they showed the data from 1926-1963. Subsequent data from international equity markets has further supported the finding. And value has even been shown to work in bonds, currencies, commodities.
    I strongly recommend Larry Swedroe’s book Your Complete Guide To Factor Based Investing. In it he lays out 5 criteria for evaluating a potential driver of returns: persistent, pervasive, robust, intuitive, investable. The first three are backward looking but nonetheless add strength to the arguments. Intuitive risk based or behavioral based explanations for a premium to exist in the future are absolutely necessary to maintain conviction during inevitable expected periods of underperformance.

    Dave

  3. My name is Tom Peacock from USA, I want to say thank you to Dr Emu for the good thing he has done for me, Though am not sure if this is the best forum to show my joy and happiness for what he has done for me but i can’t hide my happiness and my joy so i have to share it with people, my marriage got crashed about two years ago and i tried all i could within my power but to no avail. I saw a post and testimonial about the good things Dr Emu has been doing so I decided to give it a try. though he is always a busy man but when he responded back to my email, he gave me 48 hours for my marriage to be restored really just like he said my marriage was restored since then I am happy and i am living happily i am so grateful to Dr Emu you can always email him here: {emutemple@gmail. com} or WhatsApp: {+2347012841542}

  4. HOW I FINALLY RECOVERED MY LOST CRYPTO: I lost all my crypto to a fake investment scam to someone I met online. I started searching for help legally to recover my funds, and I came across a lot of Testimonies about HACKER STEVE. I contacted him, providing the necessary information and it took him and his team of experts about 36 hours to locate and help recover my stolen funds. I am so relieved and the best part was, the scammer was located and arrested by local authorities in his region. I hope this helps as many out there who are victims. I strongly recommend Steve professional services for assistance with swift and efficient recovery (Crypto, Credit card, Forex, NFT, etc) on Gmail: Hackersteve911@gmail.com | https://hackersteve.great-site.net/

LEAVE A REPLY

Please enter your comment!
Please enter your name here