Twelve Rating Systems, Walked Forward Through Eight Seasons
elote v1.3.0 added four rating systems: Massey, Keener, Pythagorean expectation, and Whole-History Rating. Three of those were invented specifically to rank college football teams. The library already had Elo, Glicko, Glicko-2, DWZ and ECF, which were invented to rank chess players.
That sets up a question worth answering with a measurement rather than an opinion. Do the football systems win at football?
Predict the week, then learn it
The way to evaluate this is the way you would actually use it. Every Saturday, you know the results of every game played before today and nothing about today. So: predict this week’s games from everything before them, then fold the week’s results in, then step forward. Six thousand games, one week at a time.
for week in weeks:
for home, away, outcome, date, attrs in week:
if home in arena.competitors and away in arena.competitors:
record(arena.expected_score(home, away), outcome)
train_arena_with_dataset(arena, week, score_keys=("home_score", "away_score"))
That is the entire protocol. Ratings are always current, never stale, and never contaminated by a game that hadn’t happened yet. Eight seasons, 2015 through 2022, with 2015 used as warm-up and 6,385 games scored from 2016 on. No threshold tuning: a prediction above 0.5 is a pick for the home team, which is what you would get out of the box.
The number to beat is home field. Home teams won 62.1% of those games, and that costs nothing to compute.
The results

| System | Built for | Accuracy | Log loss | Compute |
|---|---|---|---|---|
| Glicko-2 | chess | 0.7153 | 0.5531 | <1s |
| DWZ | chess | 0.7075 | 0.5525 | <1s |
| Glicko | chess | 0.7073 | 0.5990 | <1s |
| Massey | college football | 0.7058 | 0.5571 | 9s |
| WHR | general | 0.7055 | 0.5892 | 608s |
| ECF | chess | 0.6997 | 0.5623 | <1s |
| TrueSkill | general | 0.6993 | 0.5774 | <1s |
| Bradley-Terry | general | 0.6992 | 0.5881 | 298s |
| Colley | college football | 0.6948 | 0.5691 | 11s |
| Elo | chess | 0.6923 | 0.5802 | <1s |
| Pythagorean | college football | 0.6917 | 0.5785 | <1s |
| Keener | college football | 0.6779 | 0.6147 | 130s |
Glicko-2 wins, by nearly a point over the next system, in under a second. The top three are all chess systems. The best college-football system, Massey, is fourth.
By family, chess averages 0.7044 and the football systems 0.6925.
So the answer to “surely the sport-specific methods win at their own sport” is no, and the reason turns out to be more interesting than the question.
What the chess systems have that the football systems don’t
Colley, Massey, Keener and Pythagorean have no concept of time. They fit a season, or a whole history, as one static system of equations. A game from week one and a game from week twelve carry identical weight, and if you ask them to rate a team, they answer with one number that describes the entire period they were given.
Elo, Glicko, Glicko-2 and DWZ are all incremental: each result nudges a rating, so recent games matter more simply because they came last. Glicko-2 goes further and models volatility explicitly, and DWZ carries a development coefficient that changes how fast a competitor’s rating moves.
College football is a sport where a team in November is measurably not the team from September. Freshmen develop, quarterbacks get hurt, schemes get figured out. Tracking that turns out to be worth more than modelling strength of schedule, which is the thing the football-specific methods were built to do and do genuinely well.
Massey being fourth is not a knock on Massey. It is the best static system here by a good margin, and it does it in nine seconds using margins, which is exactly the job it was designed for. It just isn’t the job that pays best in this sport.
What actually moved the needle
Two things, both about the same size as the entire spread across the top eight systems.
Feeding margins instead of outcomes. v1.3.0 added an optional score channel to the result API. For the systems that can use it:
Massey outcomes only 0.6964 with scores 0.7058 (+0.0094)
Pythagorean outcomes only 0.6772 with scores 0.6917 (+0.0144)
Keener outcomes only 0.6622 with scores 0.6779 (+0.0157)
Colley outcomes only 0.6948 with scores 0.6948 (+0.0000)
One to one and a half points. The top eight systems in the table are separated by 1.6 points in total, so choosing to feed margins is about as consequential as choosing the algorithm. Colley is the control: it is a pure win-loss method, it ignores the channel, and it moves not at all. That is what tells you the ablation is measuring the thing it claims to.
Getting competitor identity right. The feed exposes both a team name and a display name, and the name field holds only the mascot. Across these seasons there are 295 programs and 154 distinct mascots. Fourteen teams are “Tigers”: Auburn, Clemson, LSU, Grambling State, Jackson State and nine more. Keyed on the mascot, all fourteen are one competitor with one rating, and every Auburn-LSU game is a team playing itself. No rating system recovers from that, and no accuracy metric tells you it happened.
What the extra information is worth
Two of these systems are separated by less than the gain from telling them more about the same games, which is worth keeping in proportion when choosing between them.
Time is the clearest case. The four systems that model it are four of the top five, and the one that models it most heavily, Whole-History Rating, is the one whose placement depends most on having real dates in the data. Feeding a time-aware system a season with no timestamps in it does not produce an error; it produces a system that answers a different question, so it is worth confirming the dates actually reach the model before comparing anything.
The same holds for margins. The score channel is optional, and a system that can use it and isn’t given it simply behaves like a win-loss system, which is the ablation above.
What I’d use
Glicko-2. It wins outright, it needs nothing but outcomes and dates, it finishes in under a second, and the uncertainty it tracks is useful for other things. There is no argument for anything else as a default here.
DWZ if you care about calibrated probabilities more than picks. It has the best log loss in the table by a hair, and log loss is the metric that matters if you are going to do anything with the number other than read off a favourite.
Massey if your data has margins and no useful timestamps. It is the best static system here, it is denominated in points, and the strength-of-schedule fit is real work.
Not WHR, unless you need the curve. 608 seconds to land fifth is a bad trade for prediction. What WHR gives you that nothing else here does is a queryable rating history: how good was this team in October, not just how good are they now. If you need that, it is the only option in the library.
And before tuning anything, spend the time on your inputs instead. Competitor identity, the score channel and real timestamps were each worth more on this data than the choice between the top eight systems.
Reproducing this
The walk-forward protocol used here shipped in elote 1.3.2, because I got tired of writing it by hand:
from elote import CollegeFootballDataset, group_by_period, walk_forward, Glicko2Competitor
ds = CollegeFootballDataset(start_year=2015, end_year=2022)
ds.download()
periods = group_by_period(ds.load()) # ISO week by default
report = walk_forward(
Glicko2Competitor,
periods,
warmup=23, # 2015 fits, and is not scored
score_keys=("home_score", "away_score"),
)
print(report.accuracy, report.log_loss, report.brier)
walk_forward predicts every bout in a period before learning any of it, skips competitors it
has not seen rather than inventing them, and reports calibration alongside accuracy. There is
a tune() beside it for grid-searching competitor parameters, which defaults to log loss for
reasons the Pythagorean post
explains.
pip install elote==1.3.2, and take 1.3.2 rather than 1.3.1 if you are evaluating any
time-aware system against a dataset.
Linked from
- A College Football Season, Backtested Best rating system, every bankroll strategy, one season of real moneylines. …
- Whole-History Rating: Ratings With a Past Tense Every other rating system tells you how good a competitor is now. WHR tells you …
- Margin of Victory: Massey and Pythagorean Two rating systems that read the scoreboard instead of just the result, and …
Stay in the loop
Get notified when I publish new posts. No spam, unsubscribe anytime.