Twelve Rating Systems, Walked Forward Through Eight Seasons

elote v1.3.0 added four rating systems: Massey, Keener, Pythagorean expectation, and Whole-History Rating. Three of those were invented specifically to rank college football teams. The library already had Elo, Glicko, Glicko-2, DWZ and ECF, which were invented to rank chess players.

That sets up a question worth answering with a measurement rather than an opinion. Do the football systems win at football?

Predict the week, then learn it

The way to evaluate this is the way you would actually use it. Every Saturday, you know the results of every game played before today and nothing about today. So: predict this week’s games from everything before them, then fold the week’s results in, then step forward. Six thousand games, one week at a time.

for week in weeks:
    for home, away, outcome, date, attrs in week:
        if home in arena.competitors and away in arena.competitors:
            record(arena.expected_score(home, away), outcome)
    train_arena_with_dataset(arena, week, score_keys=("home_score", "away_score"))

That is the entire protocol. Ratings are always current, never stale, and never contaminated by a game that hadn’t happened yet. Eight seasons, 2015 through 2022, with 2015 used as warm-up and 6,385 games scored from 2016 on. No threshold tuning: a prediction above 0.5 is a pick for the home team, which is what you would get out of the box.

The number to beat is home field. Home teams won 62.1% of those games, and that costs nothing to compute.

The results

Twelve rating systems evaluated week by week over eight seasons of college football

SystemBuilt forAccuracyLog lossCompute
Glicko-2chess0.71530.5531<1s
DWZchess0.70750.5525<1s
Glickochess0.70730.5990<1s
Masseycollege football0.70580.55719s
WHRgeneral0.70550.5892608s
ECFchess0.69970.5623<1s
TrueSkillgeneral0.69930.5774<1s
Bradley-Terrygeneral0.69920.5881298s
Colleycollege football0.69480.569111s
Elochess0.69230.5802<1s
Pythagoreancollege football0.69170.5785<1s
Keenercollege football0.67790.6147130s

Glicko-2 wins, by nearly a point over the next system, in under a second. The top three are all chess systems. The best college-football system, Massey, is fourth.

By family, chess averages 0.7044 and the football systems 0.6925.

So the answer to “surely the sport-specific methods win at their own sport” is no, and the reason turns out to be more interesting than the question.

What the chess systems have that the football systems don’t

Colley, Massey, Keener and Pythagorean have no concept of time. They fit a season, or a whole history, as one static system of equations. A game from week one and a game from week twelve carry identical weight, and if you ask them to rate a team, they answer with one number that describes the entire period they were given.

Elo, Glicko, Glicko-2 and DWZ are all incremental: each result nudges a rating, so recent games matter more simply because they came last. Glicko-2 goes further and models volatility explicitly, and DWZ carries a development coefficient that changes how fast a competitor’s rating moves.

College football is a sport where a team in November is measurably not the team from September. Freshmen develop, quarterbacks get hurt, schemes get figured out. Tracking that turns out to be worth more than modelling strength of schedule, which is the thing the football-specific methods were built to do and do genuinely well.

Massey being fourth is not a knock on Massey. It is the best static system here by a good margin, and it does it in nine seconds using margins, which is exactly the job it was designed for. It just isn’t the job that pays best in this sport.

What actually moved the needle

Two things, both about the same size as the entire spread across the top eight systems.

Feeding margins instead of outcomes. v1.3.0 added an optional score channel to the result API. For the systems that can use it:

Massey       outcomes only 0.6964   with scores 0.7058   (+0.0094)
Pythagorean  outcomes only 0.6772   with scores 0.6917   (+0.0144)
Keener       outcomes only 0.6622   with scores 0.6779   (+0.0157)
Colley       outcomes only 0.6948   with scores 0.6948   (+0.0000)

One to one and a half points. The top eight systems in the table are separated by 1.6 points in total, so choosing to feed margins is about as consequential as choosing the algorithm. Colley is the control: it is a pure win-loss method, it ignores the channel, and it moves not at all. That is what tells you the ablation is measuring the thing it claims to.

Getting competitor identity right. The feed exposes both a team name and a display name, and the name field holds only the mascot. Across these seasons there are 295 programs and 154 distinct mascots. Fourteen teams are “Tigers”: Auburn, Clemson, LSU, Grambling State, Jackson State and nine more. Keyed on the mascot, all fourteen are one competitor with one rating, and every Auburn-LSU game is a team playing itself. No rating system recovers from that, and no accuracy metric tells you it happened.

What the extra information is worth

Two of these systems are separated by less than the gain from telling them more about the same games, which is worth keeping in proportion when choosing between them.

Time is the clearest case. The four systems that model it are four of the top five, and the one that models it most heavily, Whole-History Rating, is the one whose placement depends most on having real dates in the data. Feeding a time-aware system a season with no timestamps in it does not produce an error; it produces a system that answers a different question, so it is worth confirming the dates actually reach the model before comparing anything.

The same holds for margins. The score channel is optional, and a system that can use it and isn’t given it simply behaves like a win-loss system, which is the ablation above.

What I’d use

Glicko-2. It wins outright, it needs nothing but outcomes and dates, it finishes in under a second, and the uncertainty it tracks is useful for other things. There is no argument for anything else as a default here.

DWZ if you care about calibrated probabilities more than picks. It has the best log loss in the table by a hair, and log loss is the metric that matters if you are going to do anything with the number other than read off a favourite.

Massey if your data has margins and no useful timestamps. It is the best static system here, it is denominated in points, and the strength-of-schedule fit is real work.

Not WHR, unless you need the curve. 608 seconds to land fifth is a bad trade for prediction. What WHR gives you that nothing else here does is a queryable rating history: how good was this team in October, not just how good are they now. If you need that, it is the only option in the library.

And before tuning anything, spend the time on your inputs instead. Competitor identity, the score channel and real timestamps were each worth more on this data than the choice between the top eight systems.

Reproducing this

The walk-forward protocol used here shipped in elote 1.3.2, because I got tired of writing it by hand:

from elote import CollegeFootballDataset, group_by_period, walk_forward, Glicko2Competitor

ds = CollegeFootballDataset(start_year=2015, end_year=2022)
ds.download()

periods = group_by_period(ds.load())          # ISO week by default
report = walk_forward(
    Glicko2Competitor,
    periods,
    warmup=23,                                 # 2015 fits, and is not scored
    score_keys=("home_score", "away_score"),
)
print(report.accuracy, report.log_loss, report.brier)

walk_forward predicts every bout in a period before learning any of it, skips competitors it has not seen rather than inventing them, and reports calibration alongside accuracy. There is a tune() beside it for grid-searching competitor parameters, which defaults to log loss for reasons the Pythagorean post explains.

pip install elote==1.3.2, and take 1.3.2 rather than 1.3.1 if you are evaluating any time-aware system against a dataset.