Backtesting Pitfalls: 7 Mistakes That Make Strategies Look Better Than They Are
Every tactical strategy begins with a backtest. The results look impressive — strong returns, shallow drawdowns, consistent performance through multiple crises. They have to look impressive, because strategies with mediocre backtests never get published. The thousands of tested parameter combinations that produced ordinary results are quietly discarded, never seen by anyone.
This selection process means that the backtested strategies you encounter have already survived a filter that biases toward overstatement. Understanding how backtests go wrong — and developing the critical eye to spot the problems — is the difference between investing in a genuinely robust strategy and investing in a statistical mirage.
Pitfall 1: Overfitting
Overfitting is the most dangerous backtesting error because it is the hardest to detect and produces the most convincing-looking results.
A strategy is overfit when its parameters have been tuned to match historical data so precisely that it captures noise — random patterns specific to the test period — rather than signal — genuine, persistent relationships that will continue into the future. The overfit strategy has effectively "memorized" past markets rather than learning their structure.
How to spot it: Count the parameters. A strategy with 2-3 parameters (lookback period, number of holdings, defensive threshold) is likely capturing genuine effects. A strategy with 10+ parameters is almost certainly overfit — the additional parameters are not adding structural insight but rather contorting the model to fit specific historical events.
Test parameter sensitivity. Change the lookback period from 10 months to 9 or 11. If performance remains similar, the strategy is capturing a robust pattern. If performance degrades dramatically, the strategy depends on one specific parameter value — a hallmark of overfitting.
Apply the "too good to be true" test. \1s above 2.0 or CAGR above 20% with drawdowns under 5% are not realistic for any liquid, implementable strategy. They indicate a model optimized to perfection on historical data — perfection that will not repeat.
The most durable tactical strategies use simple rules. The 10-month \1 works not because 10 is a magic number but because the underlying concept — medium-term trend detection — is sound across nearby parameter values. Simplicity resists overfitting; complexity invites it.
Pitfall 2: Survivorship Bias
Survivorship bias enters a backtest when the asset universe includes only securities that survived to the present day, excluding those that were delisted, merged, or went bankrupt.
For ETF-based tactical strategies, this is less severe than for individual stock strategies — major ETFs rarely disappear. But it appears in subtler forms. When a backtest extends before an ETF's inception date using reconstructed index data, the index may have been built using only surviving securities, biasing the early portion upward. And at the strategy level, the publication process itself is a form of survivorship bias: the strategy you are reading about is the survivor of dozens or hundreds of tested alternatives that the creator discarded.
Protection: Check whether the backtest uses actual traded ETF prices (which are immune to survivorship bias) or reconstructed data. For the period with real ETF data, results are reliable. For extended backtests, apply a mental discount of 0.5-1.0% to the reported CAGR.
Pitfall 3: Look-Ahead Bias
Look-ahead bias occurs when the backtest uses information that would not have been available at the time the trading decision was made.
The most common form: using month-end closing prices for same-day trades. In reality, you see the closing price after the market closes and execute trades the following morning — potentially at a different price. A backtest that assumes execution at the closing price captures a small but systematic advantage that is impossible to replicate.
A subtler form: using revised economic data. GDP, employment, and inflation figures are frequently revised weeks or months after initial publication. A macro-based strategy backtested with final revised data benefits from information that simply did not exist when the trading decision would have been made.
Protection: Confirm that the backtest uses next-day execution (signals at month-end, trades at the next trading day's open). For economic data strategies, verify the use of initially reported data, not revised figures.
Pitfall 4: Ignoring Transaction Costs
Many backtests assume frictionless execution — zero commissions, zero bid-ask spreads, zero market impact. Commission-free trading has eliminated the commission component for most investors, but spreads and impact are real costs.
For major ETFs (SPY, TLT, GLD), bid-ask spreads are typically 0.01-0.03% — negligible even with monthly trading. But for less liquid ETFs (some commodity, international, or sector funds), spreads can reach 0.05-0.20%. A strategy trading four positions monthly with 0.10% round-trip costs incurs approximately 0.40% per month — nearly 5% annually. That is enough to erase the excess return of many tactical strategies.
Protection: Use only high-volume, large-AUM ETFs with tight spreads. Apply a cost estimate of 0.05-0.10% per trade to backtest results and verify the strategy's edge survives after costs.
Pitfall 5: Insufficient Sample Size
A backtest covering 2010-2020 tells you how the strategy performs during a sustained bull market with low volatility and near-zero interest rates. It tells you nothing about financial crises (2008), dot-com busts (2000-2002), rate shocks (2022), or stagflation (1970s).
A strategy optimized on a single market environment may fail catastrophically in a different one — and it will fail precisely when the investor needs it most, because the crisis it was never tested against is the one that produces the damage.
Protection: Require backtests covering at least two major bear markets. The gold standard is a period including 2000-2002, 2008-2009, 2020, and 2022 — four distinct stress events with different characteristics (bursting bubble, credit crisis, sudden shock, rate shock). A strategy that performs well across all four is testing its logic against genuinely diverse conditions.
Pitfall 6: Cherry-Picked Start and End Dates
Starting a backtest at a market trough (where the strategy is fully invested and captures the recovery) or ending at a peak (before a drawdown that would reduce reported returns) can dramatically inflate performance numbers.
A strategy claiming 15% CAGR "since March 2009" is starting from the single most favorable entry point in modern market history — the exact bottom of the financial crisis. The same strategy measured from January 2007 (including the crisis) would show very different results. The data is not wrong, but the framing is misleading.
Protection: Evaluate strategies using the longest available period with reliable data, starting from the earliest available date rather than a convenient one. If comparing strategies, ensure identical measurement periods. Be skeptical of performance claims that begin at obvious market troughs or end before recent drawdowns.
Pitfall 7: Ignoring Taxes
Most backtests report pre-tax returns. For strategies held in taxable accounts, after-tax performance can be meaningfully lower — especially for high-turnover strategies that generate short-term capital gains taxed at ordinary income rates (up to 37% for higher earners).
A strategy with 10% pre-tax CAGR and monthly turnover might deliver 7-8% after taxes for a high-income investor in a taxable account. The same strategy in an IRA keeps the full 10%. The tax drag is not a flaw of the strategy — it is a consequence of implementation that many backtests fail to disclose.
Protection: Hold higher-turnover strategies in tax-advantaged accounts. For taxable implementations, prefer lower-turnover strategies and mentally reduce reported CAGR by 1-3% depending on turnover frequency and tax bracket.
A Practical Evaluation Checklist
Before committing capital to any backtested strategy, verify:
| Criterion | Green Flag | Red Flag |
|---|---|---|
| Parameter count | 2-4 parameters | 8+ parameters |
| Parameter sensitivity | Similar results for nearby values | Performance depends on exact values |
| Time period | 20+ years including 2+ bear markets | Single bull market only |
| Start/end dates | Full available history | Begin at market trough |
| Transaction costs | Realistic costs included | Zero-cost assumption |
| Execution timing | Next-day open | Same-day closing price |
| Data source | Actual traded ETF prices | Reconstructed index data only |
| Reported Sharpe ratio | 0.7-1.3 | Above 2.0 |
A strategy that passes all eight criteria is not guaranteed to succeed in the future — no backtest can provide that guarantee. But it has survived the scrutiny that eliminates the majority of strategies that look good only because of methodological flaws.
The Right Expectation
A reasonable rule of thumb: expect live performance to deliver approximately 70-80% of backtested performance. If a strategy shows 12% CAGR and −15% max drawdown in the backtest, plan for approximately 9-10% CAGR and −18% to −22% max drawdown in live trading. This discount accounts for execution slippage, real-world costs, the general tendency of backtests to overstate, and market environments that differ from the test period.
This is not a reason to distrust backtesting — it is a reason to calibrate expectations. A strategy that delivers 80% of its backtested risk-adjusted performance is still dramatically better than no strategy at all. The goal is not to find a perfect backtest but to find a robust one — and then implement it with discipline.
On PortfolioWiser, all strategy backtests use actual ETF price data (with proxy periods clearly identified), next-day execution assumptions, and consistent time periods encompassing all major market events. The methodology is documented transparently so that each result can be evaluated against the criteria above.