Five Backtest Invariants Worth Enforcing In Code

A backtest has one job: answer what would have happened if you had run this strategy, knowing only what you knew at the time. Every bug in a backtest is a leak in that second clause, and leaks are asymmetric. A backtest that accidentally sees the future does not produce a random error. It produces a better number.

That asymmetry is why I now write backtest invariants as tests rather than as care. Here are five from keeks-elote, which couples a rating system to a bankroll strategy, in the order I’d add them.

1. Periods are processed in chronological order

If your periods live in a dictionary keyed by week or date, the iteration order is insertion order, and insertion order is whatever your data loader happened to do. Load a CSV that isn’t sorted, or merge two sources, and you are now training on week 10 before week 3.

The fix is one sorted() call, and it belongs in the backtest driver rather than in every caller:

for period in sorted(periods):
    ...

The test that pins it inserts periods out of order deliberately:

def test_periods_run_in_chronological_order():
    data = {}
    for p in [3, 1, 2]:            # inserted out of order on purpose
        data[p] = games_for(p)
    seen = []
    run_backtest(data, on_period=seen.append)
    assert seen == [1, 2, 3]

Worth handling sparse keys too. If you have weeks 1, 2, and 5, “the next period” is 5, not 3, and code that increments an index rather than walking the sorted keys will silently look at nothing.

2. Ratings update before the bets they inform are priced

This one is the classic lookahead and it hides well, because the code reads fine in either order.

The correct sequence within a period is: price the bets using ratings as of the end of the previous period, then settle the games, then update the ratings. If you update ratings first and then price, you are betting on games using a rating that already incorporates their outcomes. Your model will look extraordinary.

for period in sorted(periods):
    bets = price_bets(arena, periods[period])   # ratings are pre-period
    results = settle(bets, periods[period])
    arena.update(periods[period])               # now learn from it

The test asserts the call order rather than the returns, because returns are exactly what the bug improves:

def test_bets_are_priced_before_ratings_update():
    calls = []
    run_backtest(data,
                 on_price=lambda *_: calls.append("price"),
                 on_update=lambda *_: calls.append("update"))
    assert calls == ["price", "update", "price", "update"]

3. Same-period bets are sized from one bankroll snapshot

This is the subtlest of the five and the one I’d never have thought to test.

Within a single period, you place several bets. If you size each one against the current bankroll, and you settle each bet as you go, then bet two is sized using the proceeds of bet one. That is a claim about the world: that you knew the outcome of one game before placing a wager on another simultaneous game.

The fix is a snapshot taken once at the top of the period, with every stake in that period sized from it:

for period in sorted(periods):
    opening_bankroll = bankroll.total          # snapshot once
    for game in periods[period]:
        stake = strategy.evaluate(prob, opening_bankroll)
        place(stake)
    settle_all(period)                          # sequential settlement is fine

Settlement can still be sequential. It’s sizing that has to use the opening balance. The test uses two bets on the same period and checks a hand-computed ending balance:

def test_same_period_bets_use_the_opening_bankroll():
    balances = run_two_bet_period(starting=100.0)
    assert balances == [100.0, 100.0]     # both sized off 100

When the stake base is computed inside the loop instead, that assertion comes back [100.0, 115.0], which is the whole bug in one list.

4. No lookahead is asserted, not assumed

Invariants 2 and 3 are specific instances of a general property, and the general property deserves its own test. The pattern I like is a data-level trap: build a fixture where a future period contains a value that would be irresistible if visible, and assert that the strategy’s output is unchanged when that value is altered.

def test_future_periods_cannot_affect_earlier_decisions():
    base = run_backtest(data)
    tampered = deepcopy(data)
    tampered[5] = wildly_different_games()    # change only the future
    after = run_backtest(tampered)
    assert base.decisions_through(4) == after.decisions_through(4)

If a later period’s contents can change an earlier period’s decisions, you have a leak, and you did not have to know where it was to find it.

5. Bets are priced against the odds you could actually have taken

The last one is less about code structure and more about data. If your prices come from a closing line and your bets are notionally placed before the game, you are backtesting against odds that already absorbed information you didn’t have. Similarly, a moneyline needs converting to an implied probability the same way every time, with the vig handled explicitly rather than left in.

The invariant I hold: the edge is a comparison between two numbers, my probability and the market’s, and both must be as-of the same instant, computed by the same code path used in production. That last clause is the one that catches people. When the backtest has its own odds parser, it will drift from the live one, and the drift will be in your favor because that’s the version you tuned until the returns looked good.

What a backtest is actually simulating

A backtest is a simulation of what you knew and when. Every one of these invariants is a statement about time, and every one of them fails silently and profitably.

So write them down as tests. Not comments, not care, tests, because the code reads correctly in the broken ordering too. And if you make a change and your returns get better, treat that as a bug report until you can explain why.