A single bad price can turn a 5% system into a 30% one

A backtest is judged by its result, and the result is computed on a table of prices. That table is almost never audited: it is taken as correct. This note shows, with a test anyone can repeat, how much false performance one single wrong price produces among tens of thousands, and what it takes to avoid it.

The short answer: twenty-six points of annual return, and one line of code.

The error that invents a gain

Intuition says the dangerous price is the one that is too high. It is the other way around.

A false high price produces a large gain followed by an equivalent loss when the data returns to normal: the system buys high and sells low, and the loss offsets the gain. Annoying, but it does not inflate the result.

What inflates is the low price. A stock worth 80 dollars stored as 0.80 — a misplaced decimal, a split nobody adjusted — does not look like an error: it looks like a collapse. And a collapse is, for many strategies, a buy signal. The system buys at 0.80, next month's price is the real one, and the backtest records a gain of almost ten thousand percent that never happened.

That trade did not exist. But it stays in the record, and it drags the average of every year with it.

The test, so that you can repeat it

The strategy is deliberately simple and well known, so that nothing has to be taken on faith:

• Universe: the stocks of the S&P 500, with the composition the index had in each year, not today's.
• Rule: at each month's close, buy the ten stocks that fell the most that month, in equal parts.
• Hold one month. Repeat.
• Period: January 2015 to December 2025. No transaction costs.

The strategy is about twenty lines of code. The result on clean data is 5.12% a year, which is what one would expect: buying declines was a poor business in that decade.

Before measuring, two series with known corrupt data were removed from the price table: MYL from 17 November 2020 (Mylan merged into Viatris and the series continues for two days at 0.079, coming from 15.855) and WIN from 27 April 2015 (a step from 193.62 to 43.79 with no recovery). They are named on purpose: measuring this effect with the effect inside would make no sense, and an unnamed exclusion cannot be verified.

Then one single price is corrupted: any stock, any month, its closing price divided by one hundred. One data point, among the 64,270 month-end prices the calculation uses.

The result

It was repeated 191 times, each with a different stock and a different month:

The median inflation is 25.93 points, and 191 out of 191 cases inflated by more than 20 points. This is not an extreme case picked on purpose: it is what happens every time.

One data point among sixty-four thousand multiplies a system's return by six. And nothing looks odd in the spreadsheet: what one sees is an excellent system.

One line of code switches it off

The defence does not require auditing the whole database. It is enough to discard the impossible: no stock in a large index falls eighty percent in a month without it being world news. If the calculation finds such a decline, the right move is to suspect the data, not to buy it.

The filter avoids twenty-six points of invented return and costs a quarter of a point on clean data. In 179 of the 191 cases the corrupted backtest returns exactly the same number as the clean one: the dirty data point ceases to exist for the calculation.

A quarter of a point of cost for twenty-six points of protection. It is one of the best deals a trading system can buy.

What the filter does not protect

It is worth looking at the twelve cases where the filter does not return exactly the clean number, because they are not all the same thing:

The last two are the limit of the remedy. The filter applies to the purchase, not to the open position. If the dirty data point appears in a stock the system already holds, the false ninety-nine-percent decline hits the result anyway, and discarding the purchase does not prevent it, because the purchase already happened.

Those two cases are also the largest deviations of the twelve.

The complete defence, then, is not one line but two ideas: discard the impossible before buying, and review impossible moves on what is already held. The first costs a quarter of a point; the second costs a periodic review of the series, which is work and not code.

Why this is not caught by looking at the result

A result inflated by dirty data does not look like an error. It looks like a discovery.

The return goes up, the curve looks better, and whoever ran the backtest has every incentive not to look for flaws in the one version that worked. An error that makes the result worse gets investigated until it is found; one that makes it better gets published.

That is why data should be reviewed before celebrating, not after.

How to spot it in someone else's result

When somebody presents a backtest, three questions separate those who looked at their data from those who did not:

1. What happens if the largest trades are removed? If taking out the best five collapses the result, the system does not live on a mechanism: it lives on five data points. It is worth looking at which ones.
2. What was the single best trade, and what caused it? A gain of several thousand percent on a stock from a large index deserves a concrete explanation. If there is none, it is dirty data.
3. Is the universe composition the one of each year, or today's? That is a different problem — survivorship bias — but it is detected with the same question: where exactly did each number come from.

None of the three requires seeing the code. All three are answered with the list of trades, which is precisely what is worth asking for.