Prospective Tracking vs. Backtesting

A backtest runs a model over past games as if each were being predicted in turn, then scores the results. It is fast, it can cover many seasons, and it is genuinely useful for development: it catches bugs, rules out ideas that clearly do not work, and shows roughly how large a model’s typical error is.

FDI uses historical testing in exactly that role. What a backtest cannot do is prove that a model will perform the same way on games that have not happened yet.

• Look-ahead: information that only existed after a game — a final injury report, a closing line, a season-long average — slips into the inputs for that game.
• Choice after the fact: many versions of a model are tried and the one that scored best on the same history is kept, so part of its score is luck that will not repeat.
• Tuning and testing on the same games: settings are adjusted until the history looks good, then the history is used to judge the result.
• Quiet revisions: an old prediction is recalculated with improved code, and the new number replaces the one that would actually have been shown.

None of these requires bad intent. They are the ordinary ways a careful analysis drifts toward the answer it hoped for, and they are why a strong backtest, on its own, is weak evidence.

Prospective tracking fixes the order of events. The model runs before the game, on information that existed at that time, and the output is recorded and frozen. After the game the result is joined back to the original record. The prediction is never rewritten, so the record shows exactly what the model said when it said it.

That order removes the problems above by construction. There is no look-ahead, because the future had not happened. There is no choosing the best version afterward, because only the version that was running made a prediction. And there is no quiet revision, because the record is written once.

Prospective tracking, step by step

The same discipline applies when FDI decides whether a model is good enough to show members. The test, the comparison it must beat and the standard it must meet are written down before the test data is examined. If the model falls short, it is not promoted — and the standard is not relaxed afterward to let it through.

Some models are also run prospectively while they are still being evaluated, so they can be measured on the current season rather than argued for on old data. Running a model that way is not the same as presenting it to members as supported.

The same distinction is a useful lens for any claim about how a model has performed, from FDI or anyone else. A few questions separate a record that can be trusted from one that only looks good.

• Were the predictions recorded before the games, with a time you can see — or reconstructed afterward?
• Is every prediction included, or only the ones that were eventually shown or highlighted?
• Was the standard for success fixed before the results were known, or chosen once they were?
• Is the comparison fair — against the market price available at the time, not a later or better one?
• Are games where the underlying assumption failed, such as a player not taking part, handled by a stated rule rather than case by case?
• Is the sample large enough that a lucky stretch could not explain the result on its own?

A record that answers all of these well is rare and slow to build. That is exactly why it is worth more than a backtest that answers none of them.

Prospective evidence is slow. A season has a limited number of games, and it takes time before a record is large enough to say much. It is also humbling: early results can be noisy in either direction, and a model has to be judged on the record it actually produces, not on the one its builders expected.

That slowness is the point. A record written before the outcomes cannot be tuned to them. When FDI describes how a model has done, the claim rests on predictions that were fixed in advance — the only kind of history a reader can take at face value.

How First Down Intel reads the NFL