How Should You Interpret Our Projected Win Totals?

Last week, we published our playoff odds for the 2023 season. Those odds have a lot of interesting bells and whistles, from win distributions to the odds of getting bye to play. At their core, however, they are based on one number: win totals. Win totals determine who makes the playoffs, so our predictions are, at their core, a machine for sparking win totals and assigning playoff spots from there.

We’ve been making these predictions since 2014, so I thought it would be interesting to see how our overall winning predictions stack up against reality. After all, winning predictions are only useful if they do an acceptable job of predicting what will happen during the season. If we projected 113 wins for the Royals each year, to select a random sample, the model would not be very useful. The Royals won anywhere from 58 to 95 games in that span.

I’m not sure exactly what data is most useful for our predictions, so I decided to run a bunch of different tests. That way, whichever description best helps you understand their volatility, you can listen to that one and ignore everything else I’ve presented. Or, you know, consider a bunch of them. It is your brain, after all.

Before I get into these, I’d like to point out that I’ve already given a similar test to our playoff odds estimates in these two articles. If you’re looking for a tl;dr summary, I’d go with this: our odds are pretty good, mainly because they come together on the teams that are very likely or unlikely to make the playoffs quickly . Odds on teams that have playoff odds of 5-10% of the distribution are probably overly optimistic, although that is more observational than data can prove. For the most part, what you see is what you get: projections do a good job of separating the wheat from the chaff.

With that out of the way, let’s get back to projected win totals. Here’s the bottom line: the mean error of our total win predictions is 7.5 wins, and the median error is 6.5 wins. In other words, if we say we think your team is going to win 85.5 games, that means they will have won between 79 and 92 games at half time. Past performance is no guarantee of future results, but for what it’s worth, that error is consistent over time. In terms of standard deviation, that’s about 9.5 wins.

It’s a surprising error, because baseball is a game of uncertainty and probability. This is not basketball, where the results feel predetermined; sometimes the bad hit hits a homer, and sometimes your sharpness gets chipped. Look at the distribution of our own contracts to illustrate this. In these simulations, the level of talent is fixed, and yet the teams’ success rates vary greatly from one run to the next. The median absolute error from one run to the next is about four beats; in other words, even if you knew exactly how good each team was and played at exactly that level, there is a limit to how accurate your predictions could get.

One useful takeaway from this: the number we’re reporting is more of a probability cluster than a point estimate. When we go for a team for 85 wins, we’re saying they’re an 85-ish win type team. You probably have a rough idea of ​​what that means in your head. You probably also have a rough feeling that teams that look like 85 wins sometimes have 90 games, or 80 games. Heck, sometimes they win 95 or 75 games. Our odds even have some extra variation around that feeling, because they’re estimating how good a team will be before the fact, but the general concept applies.

Specifically, here’s a look at the distribution of losses:

The difference between our projected win totals and actual team win totals is slightly skewed to the right; the most frequent result is that our model predicts two to four more wins than a team actually achieves, and the median is ever so slightly negative (-.25, to be exact). I don’t see a clear explanation for that, but it’s not a huge effect anyway. For the most part, our projections tend to distribute the missions overall.

With that out of the way, let’s get into the details. I broke down projections into win buckets to see if they have any clear trends based on the talent level of the team. I started out with two win buckets and looked for our average miss in each bucket:

As far as I can tell, there isn’t much of a pattern here. Many buckets when we lost low sit right next to buckets when we lost high. Teams with 80-82 wins performed almost two wins per year worse than that, and teams we expected to win 82-84 performed more than their projections. It’s all a big zig-zag.

To zoom out a bit, I bucketed out in tens instead of twos. A clearer pattern then emerges, and it is a logical one:

There are teams that we think are going to be terrible, and they are even a little worse than we think they are going to be. Likewise, very good teams are, in our view, very good – better than we predicted on average. There is an obvious reason for this: trades. Bad teams tend to trade their good players. Good teams usually trade for good players. We can’t account for those trades in the season projections, so I mean the natural flow. In fact, I’d be surprised if it wasn’t there.

That’s the extent of my serious look at our data, but I parsed the data up one more way for fun. You know how FanGraphs always hates your team, no matter what team that is? Well, if you root for the Astros, you might just be right. We’ve come up with a lot on our Astros win projections: a lowly 6.4 win average, to be exact. We have also been low on the Brewers, Dodgers, and Cardinals by about five wins each. A lot of that comes down to what I was talking about above: good teams tend to rise during the season, and the four teams we were down on were good for most of the window. we had projections for it.

You might think we’re always low on the Rays, what with their front office made up of 2/3rds brain surgeons, 1/3 rocket scientists, and a bonus 1/3 former FanGraphs employees. It wasn’t: we averaged just under two wins, which is in the middle of the pack in terms of true error. The A’s are another team that people often point to as being smarter than projections – but they’re the team we predicted better in our data set, they averaged 79.95 wins, and won an average of 80 games per season.

On the other side of the coin, Tigers fans may be angry with FanGraphs for giving them too much hope. We have lost 6.6 wins per season, and not in a good way; they averaged just 71.2 wins over the eight seasons I looked at, and we projected them for 77.8. We’re way too high on the Padres, Nationals, and Reds (and yes, there are some midseason trade deals here, too).

So what do FanGraphs’ projected win totals mean? I would treat them as a rough gauge of the major league franchise’s prospects in the coming year. Angry about your team being projected for 86 wins instead of 88? I don’t think our projections are great at doing that kind of fine parsing, and I think architects would agree with the projections that contribute to our model. Angry that we projected your team for 72 wins when you think they are a playoff contender? Well, that’s not the kind of thing we really miss.

Specifically, 88% of our projections get within 15 wins of the team’s actual total. Only 7% of the teams in our total sample were better than 15 wins or more. That’s not to say it’s impossible – 7% is more than 0%, of course – but it’s a reminder of gravity. If our playoff confidence and projected win total think your team is bad, it doesn’t mean they are 100%. But it means that most of the teams that do a project like them were bad.