What is lookahead bias in a backtest?

Lookahead bias is using information the strategy could not have had at the moment it acted. The most common version is an entry price timed slightly too early, like buying at the opening price on the day a piece of news was released when the news actually came out that afternoon. It is easy to write and hard to spot, because nothing errors out. The fix is to be strict about entering only after the information was genuinely available, which usually means the next trading session.

How do I know if my backtest benchmark is fair?

Run the same strategy with the same entry and exit dates but substitute the benchmark for whatever you were buying. If your strategy is only invested during certain stretches, comparing it to a buy-and-hold benchmark measures your deployment schedule as much as your picks. Swapping the asset while holding the timing fixed isolates selection, and it is the single most useful check we know.

What is survivorship bias and how does it inflate results?

Survivorship bias is when failed assets quietly drop out of your data. If your price source has no history for a company that got delisted or acquired, those trades often get skipped without any warning, and delistings skew toward bad outcomes. The result is a backtest built only on the names that made it. Count how many trades you dropped for missing data and report it, because a silent exclusion is a thumb on the scale.

Why do backtested strategies stop working live?

Usually because the backtest was measuring something other than what its author thought. Lookahead entries, unfair benchmarks, survivorship, selecting the asset after seeing its results, and outright coding errors all push results the same direction, which is up. Live trading removes all of those advantages at once. If a strategy looks good on paper and dies in practice, the paper version almost always had a leak.

How do you catch a bug in your own backtest code?

Feed it something with a known answer. We caught a serious bug in our own portfolio simulator by running it on nothing but S&P 500 returns and checking whether it reproduced the S&P 500. It did not, it reported more than double the real return, and that told us the simulator was broken before any strategy conclusion depended on it. Any simulator should reproduce a benchmark when fed the benchmark.

Does a statistically significant backtest mean the strategy works?

Not on its own, and significance is often overstated because trades are treated as independent when they are not. Trades placed on the same day share that day's market move, so a standard test counts correlated observations as separate evidence and produces confidence intervals that are too narrow. Resampling whole days rather than individual trades gives a more honest picture, and it frequently turns a clear result into an ambiguous one.

Five Backtest Mistakes That Fake an Edge

Backtest mistakes turn losing strategies into winners on paper. Here are five we hit testing a real strategy.

We spent a few days testing whether copying congressional stock trades beats the index. The strategy question turned out to be less interesting than what happened along the way, which is that our own analysis tried to lie to us five separate times.

Every one of those errors pushed the result in the same direction. Not one made the strategy look worse. That’s the thing worth knowing about backtesting: the mistakes are not random, they’re biased toward good news, which is exactly why bad strategies survive testing and then fall apart with real money behind them.

What are the most common backtest mistakes?

The five that cost us the most were entering at a price we couldn’t have gotten, silently dropping trades that ran past the end of our data, comparing against a benchmark that wasn’t matched on timing, choosing what to test after seeing which choice looked good, and a plain coding bug in the portfolio math. Each one made a losing strategy look like a winner, and each one needed a specific check to catch.

1. The entry price you couldn’t actually get

Our first version bought at the opening price on the day a congressional filing was published.

That looks reasonable until you ask when the filings actually post. They go up at all hours, and plenty land after the market has closed for the day. Buying at that morning’s open means buying before the document existed.

The fix is boring. Enter at the open of the next trading session, strictly after the information became public. The cost of getting this wrong is not huge on any single trade. Across thousands of trades it’s a steady free lunch that live trading will never serve you.

The general form of this mistake: any time your entry timestamp and your information timestamp are on the same day, check which came first.

2. Trades that quietly disappear when they don’t finish

This one is nastier because nothing looks wrong.

Our code asked for the closing price 90 days after entry. When that date ran past the end of the available price history, which happens for every recent trade, it fell back to the most recent price it had. So a trade opened three weeks ago was recorded as a completed 90-day hold, marked at whatever the price happened to be.

No error. No warning. Just a pile of unfinished trades reported as finished ones, marked at a date chosen by the edge of the dataset rather than by the strategy.

The fix is to require a real exit and exclude anything that doesn’t have one, then count the exclusions and print them. In our final run 31 trades got dropped this way. Small, but we only know it’s small because we counted it.

Any time your code has a fallback that silently substitutes a value, treat it as a place where results go to get invented.

3. The benchmark that wasn’t measuring what we thought

This is the one that fooled us longest, and it produced a genuinely exciting wrong answer.

Our portfolio of copied trades compounded at 19.29% a year. The S&P 500 returned 15.27% over the same calendar period. Four points a year of outperformance, from a strategy we’d already half decided didn’t work.

The problem is that the strategy isn’t invested all the time. It buys when filings appear and holds cash otherwise. And filings don’t arrive evenly. When we checked, the market stretches our trades occupied averaged out to roughly 20.6% annualized, well above the 15.27% the index delivered across the whole period. We weren’t measuring good stock selection. We were measuring the fact that the strategy happened to be deployed during good months.

The test that settles it takes about ten lines of code. Run the identical portfolio, same trades, same entry and exit dates, same everything, but buy the benchmark each time instead of the thing you were testing.

identical trades and dates	annual return
the strategy’s picks	19.29%
the benchmark, same schedule	19.68%

The schedule was doing all the work and the picks were a slight drag. We’d recommend this check to anyone testing anything, because it’s cheap and it converts a flattering comparison into an honest one.

4. Choosing what to test after seeing the answer

The plan we started from named one specific person as the pick worth copying, cited their strong recent return as the reason, and then proposed backtesting that person over five years.

You can probably see the problem. Selecting someone because their past results look good, then testing those same past results, is not a test. The answer was determined by the selection.

With enough candidates, someone is always going to look brilliant. We ranked all 64 members who had enough trades to rank, then checked what pure chance produces: shuffle all the returns randomly across members, keep each member’s trade count fixed, and see how far apart the best and worst end up.

Chance alone produced a best-to-worst spread of 13.48% at the median. Our observed spread was 22.90%, which does exceed randomness, so there’s something real in the dispersion. But if we’d simply looked at a leaderboard and crowned the top name, most of what we’d have been admiring was luck with a 13-point range.

The discipline here is to define the rule first, apply it to every candidate, and report the whole distribution instead of the winner.

5. The simulator that was broken the whole time

The last one was our own bug, and we only found it because of a habit rather than a suspicion.

Our portfolio simulator credited each closing trade a share of current equity times its return, but never actually deducted capital when a position opened. So money stayed available even while it was supposedly invested, and returns compounded off capital that was already committed. A slow leak that scaled with how often positions turned over.

At a 90-day holding period it produced numbers that looked completely plausible. We’d have shipped them.

What exposed it was running the same simulator at a 30-day hold, on nothing but S&P 500 returns, as a control. If you feed a portfolio simulator pure index returns, it should give you roughly the index.

	annual return
our simulator, fed only S&P 500 returns	35.74%
what the S&P 500 actually did	15.21%

A simulator that turns the index into a 35% compounder isn’t measuring a strategy. It’s measuring its own defect. We rebuilt it with real cash accounting, and the corrected version says the strategy trailed the same-schedule benchmark by about 7.8 points a year.

Feed your test something with a known answer. If it can’t reproduce the known answer, nothing else it tells you counts.

The one statistical check we’d add

There’s a sixth item that isn’t a mistake exactly, more a habit worth building.

Standard significance tests assume your observations are independent. Trades aren’t. A dozen filings often land the same day, and every one of them shares that day’s market move. Treating them as a dozen separate pieces of evidence produces confidence intervals that are too narrow and turns noise into findings.

Resampling whole days instead of individual trades fixes it. On our own numbers the difference was stark:

treating 8,329 trades as	95% confidence interval	p
independent	−1.12% to −0.32%	0.0003
785 correlated filing days	−1.43% to +0.07%	about 0.07

Same data, same average. The naive version says the effect is definitely there. The honest version says we can’t separate it from zero. Note that this cut against a conclusion we’d have enjoyed, since the effect in question was the strategy underperforming. Narrow intervals don’t only manufacture good news, they manufacture confidence, and confidence in either direction is the thing you have to earn.

Why this matters more than the strategy did

We went in expecting to publish a verdict on copying Congress. What we got was a reminder that the testing apparatus needs more skepticism than the idea being tested.

Five errors. All of them flattering. Any one, left in place, would have produced a confident write-up recommending a strategy that our corrected numbers say trailed the index. The strategy conclusion is in our congress copy-trade backtest if you want it, along with the disclosure lag analysis and what the member-by-member rankings actually show.

But the transferable part is the checklist. Enter after the information, not with it. Never let a fallback invent a number. Match your benchmark on timing, not just on period. Pick your candidates before you see their results. And feed your machinery a known answer before you trust an unknown one.

If you want to compare any strategy against the version that requires none of this work, our compound interest calculator and fee drag calculator cover most of what actually determines long-run outcomes. In our experience the gap between a real edge and a testing artifact shows up there faster than anywhere else.