Scoring a Three-Outcome Predictor

Every rating system in elote hands back one number: the probability that the first competitor beats the second. In chess, football and most of the sports people rate, there is a third thing that can happen.

One number cannot express three outcomes, which means every metric you compute from it has to decide what to do with the draws. The default decision, in most code I have read including my own, is to drop them without saying so.

Here is what that costs, and whether the alternative is worth it.

The setup

40,000 Lichess games from January 2013, rated with Glicko, walking forward a day at a time: predict each day’s games from everything before, then learn the day. 27,232 of them are scored, once both players have a history.

The draw rate is 3.24%, which is low because these are online blitz games. Classical chess at the top level runs above 50%. Hold that range in mind, because it turns out to be the whole story.

What dropping them costs

Two ways to report binary accuracy on this history:

draws dropped entirely   0.6181   (over 26,349 games)
draws counted as misses  0.5980   (over 27,232 games)

Two full points of accuracy, from a category that is 3.24% of the data.

The arithmetic is not subtle. Drop 883 games you were always going to get wrong and your denominator shrinks while your numerator does not. What makes it worth naming is that the first number is what you get by default, from code that never mentions draws at all, and it is not obviously wrong when you read it. It looks like an accuracy. It is an accuracy conditional on the game not being drawn, which is a different and less useful quantity, and nothing in the output says so.

At 3% the gap is two points. On classical chess at a 50% draw rate, the same silent default reports the accuracy of a model on half its data.

So predict the draw

The obvious fix is to let the predictor say “draw”. A rating system gives you one number, so the natural way is a band: if the prediction is close enough to a coin flip, call it drawn.

That is only reasonable if draws actually concentrate near parity. They do:

The left panel is real signal. Games the model reads as near coin flips are drawn 3.8% of the time; games it reads as lopsided are drawn 1.9% of the time and, at the extreme, never. Evenly matched players draw more often, the ratings know which players are evenly matched, and so the ratings carry information about draws.

Then the right panel says never to use it.

Every band is worse than no band, and it degrades monotonically. Even the narrowest one calls 961 games drawn to catch 38, and those 961 games were ones a binary prediction would have won about half of. You trade roughly 480 correct picks for 38.

Why no band can work here

The band was never going to help, and you can see it without running anything. Take the games nearest a coin flip and look at all three outcomes:

|p - 0.5| < 0.05, 4,894 games
  first player wins   49.5%
  second player wins  46.7%
  draw                 3.8%

A draw is the least likely of the three, right where draws are most likely. An accuracy-maximizing predictor names the most probable outcome, so it never names the draw, anywhere.

That generalizes into a number worth carrying around. In a band near parity the two win outcomes split what the draw leaves, so predicting a draw beats predicting the favourite when

p_draw > (1 - p_draw) / 2      i.e.      p_draw > 1/3

A third outcome has to clear roughly 33% before an accuracy-maximizing model should ever predict it. These games are at 3.8%. Classical chess is above 50% and comfortably clears it, which is why draw prediction is a real subject there and a waste of time here. The dashed line in that first panel is the 33% mark, and the gap to the blue line is the whole answer.

What to do instead

Report the denominator. If you drop draws, say so and say how many, because a reader cannot tell 0.6181 from 0.5980 by looking. elote’s calculate_metrics and calculate_metrics_with_draws are separate calls for this reason, and the confusion matrix reports skipped bouts rather than swallowing them.

Use the probability, not the pick. Accuracy asks one question: was the most likely outcome the one that happened. It cannot see that the model correctly raised its draw estimate from 1.9% to 3.8%, because 3.8% still is not the argmax. Log loss and Brier can, they use the whole distribution, and on a three-outcome problem they are the metrics that reward getting the rare outcome’s probability right.

Ask what a draw costs you. Accuracy assumes every mistake is worth the same. If you are betting, a draw priced at 25-1 that hits 3.8% of the time is a good bet, and the accuracy-optimal decision of never predicting it is the wrong decision. The 33% threshold is the break-even for accuracy specifically; the break-even for money is wherever the price says it is.

That last one is worth stating plainly, because it is the general form: the outcome you should predict is the one that maximizes your objective, and accuracy is rarely anyone’s actual objective. It is just the one that is easiest to compute.

Running it

from elote import ChessDataset, LambdaArena, GlickoCompetitor

ds = ChessDataset(max_games=40000, year=2013, month=1)
ds.download()

arena = LambdaArena(lambda a, b, attributes=None: True, base_competitor=GlickoCompetitor)
# ... walk forward, then:
arena.history.calculate_metrics()                       # binary
arena.history.calculate_metrics_with_draws(0.33, 0.66)  # three-outcome, with a draw band
arena.history.confusion_matrix()

pip install 'elote[datasets]' for the chess loader.