A recent paper (“Evaluating Trading Strategies”) on the hazards of backtesting in The Journal of Portfolio Management has been receiving a fair amount of attention lately, inspiring some folks to go off the deep end and argue that econometric analysis of investment strategies is always a bad idea. That's a bit much for the simple reason that the only alternative to backtesting is blindly throwing money at markets without the benefit of perspective. Yes, backtesting can be dangerous when designed without best practices in mind. But in the grand scheme of investing, we're all relying on one form of backtesting or another.

Some investors think they're immune, but they're fooling themselves. For instance, take the buy-and-hold investor who claims to invest for the long run in a simple stock/bond mix that's periodically rebalanced and thereby shuns the evils of short-term trading and, by extension, backtesting. Sounds good, but there's a good chance that the reason this investor embraces the strategy is because he's aware of the encouraging track record in balanced portfolios over the long haul. A cursory glance at the Ibbotson data, for instance, has been known to motivate some to argue that you can do quite well by holding a stock/bond portfolio through time.

Perhaps, but the fact that you peeked at the historical record and decided that you'd like to jump on board this investment train is a conclusion based on… drumroll… a backtest! Okay, it's an informal one if you're reviewing someone else's handiwork. But whether you're diving into the nitty gritty details and writing your own code or relying on someone else to do the heavy lifting it's all part of the same game: using history to make a decision on how to invest for the future.

That's a broad brush, of course, and there are as many ways to backtest as there are stars in the sky. That leaves plenty of room for error. For an amusing review of how things can go horribly wrong, take a look at Jason Zweig's column from last June in The Wall Street Journal (“Huge Returns at Low Risk? Not So Fast”).