SITE NEWS: We are moving all of our site and company news into a single blog for Sports-Reference.com. We'll tag all PFR content, so you can quickly and easily find the content you want.
Also, our existing PFR blog rss feed will be redirected to the new site's feed.
Pro-Football-Reference.com » Sports Reference
For more from Chase and Jason, check out their work at Football Perspective and The Big Lead.
ShareThisTen thousand seasons with no standards
In case you just stumbled in today, this post is the latest in a long string. Read these first: I, II, III, IV, V.
I got into a conversation yesterday with one of the two readers of this blog who I actually see in person on a regular basis. I conjectured, based on yesterday's results, that, assuming we keep the eight four-team divisions and demand that the winners of those divisions get seeds one through four in the playoffs, the current system (two wildcards) is the one that maximizes the chances of the best team in football ending up with the Lombardi Trophy. My reasoning: if you eliminate the wildcard, you will too often shut the best team out of the playoffs altogether (we saw this in yesterday's post). And if you have more than two wildcards, you will too often make the best team navigate an extra round of playoffs.
My friend then made this bold claim:
I'll bet that if you let all 16 teams from each conference into the tournament, then the best team's chances of winning it all would be greater than they are with the current system.
At first I thought this was ridiculous, but it didn't take too much thought to realize that he might be right. For one thing, letting everyone in would guarantee that the best team actually makes the playoffs. And in the usual case, where they win their division and post a good record, all it really does is add a game against the 16th seed and a game against the 8th or 9th seed. Not too much different from a couple of byes. Sure, there is a slim chance of an upset. But there is also a chance of an upset that knocks off the best team's toughest competition.
Only one way to find out.
I'm going to apologize in advance for the lack of decent formatting. I just don't have time to get it done the way it ought to be done. So it's going to be long and unwieldy. I will look at four different playoff formats, two of which will be review. For each one, I'll show the number of times (out of 10,000) that the true #1 team in the NFL won the Super Bowl, the number of times the true #2 won it, and so on. Then, I'll show how often the Super Bowl winner had each given number of regular season wins. I'll add some brief thoughts at the end.
The current system: two wildcards
Tm# SBwins Cumulative
=====================
1 2399 2399
2 1441 3840
3 1064 4904
4 826 5730
5 652 6382
6 559 6941
7 492 7433
8 386 7819
9 312 8131
10 293 8424
11 231 8655
12 210 8865
13 176 9041
14 162 9203
15 138 9341
16 120 9461
17 100 9561
18 73 9634
19 83 9717
20 58 9775
21 44 9819
22 50 9869
23 38 9907
24 24 9931
25 21 9952
26 14 9966
27 9 9975
28 12 9987
29 4 9991
30 8 9999
31 1 10000
32 0 10000
Wins Times Cumulative
======================
6 2 2
7 7 9
8 150 159
9 651 810
10 1528 2338
11 2413 4751
12 2502 7253
13 1674 8927
14 822 9749
15 214 9963
16 37 10000
If you can see that the 4th-best team in football won the Super Bowl 8.26% of the time, that a 9-7 team won the Super Bowl 6.51% of the time, and that one of the top four teams in football won it 57.3% of the time, then you're reading the tables right. You'll note that I've added a cumulative column to make things a bit easier to summarize. You'll also note that the numbers don't match those shown in the original posts. That's due to random variation, of course. Although, amazingly, the top team won exactly 2399 out of each run of 10,000.
No wildcards - four division winners play a standard tournament
Tm# SBwins Cumulative
=====================
1 2315 2315
2 1448 3763
3 1039 4802
4 853 5655
5 625 6280
6 533 6813
7 499 7312
8 406 7718
9 308 8026
10 288 8314
11 249 8563
12 228 8791
13 191 8982
14 162 9144
15 167 9311
16 96 9407
17 104 9511
18 88 9599
19 80 9679
20 55 9734
21 54 9788
22 53 9841
23 32 9873
24 32 9905
25 27 9932
26 15 9947
27 18 9965
28 15 9980
29 9 9989
30 7 9996
31 4 10000
32 0 10000
Wins Times Cumulative
======================
6 1 1
7 17 18
8 158 176
9 711 887
10 1600 2487
11 2368 4855
12 2462 7317
13 1607 8924
14 797 9721
15 242 9963
16 37 10000
Four wildcards - division winners get seeds 1 through 4, four next-best teams regardless of division get seeds 5--8. Straight 8-team tournament with no re-seeding between rounds.
Tm# SBwins Cumulative
=====================
1 2285 2285
2 1411 3696
3 1006 4702
4 795 5497
5 689 6186
6 572 6758
7 488 7246
8 400 7646
9 362 8008
10 311 8319
11 260 8579
12 217 8796
13 194 8990
14 156 9146
15 130 9276
16 136 9412
17 100 9512
18 74 9586
19 84 9670
20 72 9742
21 55 9797
22 52 9849
23 31 9880
24 34 9914
25 18 9932
26 26 9958
27 19 9977
28 9 9986
29 7 9993
30 5 9998
31 1 9999
32 1 10000
Wins Times Cumulative
======================
7 37 37
8 507 544
9 1239 1783
10 1874 3657
11 2099 5756
12 2032 7788
13 1335 9123
14 623 9746
15 223 9969
16 31 10000
Twelve wildcards (i.e. all teams make playoffs) - division winners get seeds 1 through 4. Straight 16-team tournament with no re-seeding between rounds.
Tm# SBwins Cumulative
=====================
1 2111 2111
2 1318 3429
3 999 4428
4 804 5232
5 635 5867
6 559 6426
7 489 6915
8 372 7287
9 344 7631
10 322 7953
11 268 8221
12 249 8470
13 189 8659
14 200 8859
15 159 9018
16 150 9168
17 147 9315
18 132 9447
19 106 9553
20 74 9627
21 79 9706
22 64 9770
23 56 9826
24 36 9862
25 41 9903
26 22 9925
27 27 9952
28 17 9969
29 13 9982
30 8 9990
31 5 9995
32 5 10000
Wins Times Cumulative
======================
2 2 2
3 4 6
4 23 29
5 83 112
6 151 263
7 358 621
8 728 1349
9 1296 2645
10 1686 4331
11 1889 6220
12 1807 8027
13 1209 9236
14 552 9788
15 176 9964
16 36 10000
Thoughts:
- I am floored by how little the playoff format seems to matter.
- While it doesn't matter much "morally," the 16-team free-for-all would lead to some embarrassment, as a .500-or-worse team would be a near-lock to win the Super Bowl every fifteen years or so. You probably noticed some 2-14 and 3-13 teams winning Super Bowls in that format. Those were really strange seasons, but a 6-10 Super Bowl winner would be a real possibility.
- If we re-seeded between rounds of the 16-team free-for-all, I bet the best team would win more than 24% of the time.
This entry was posted on Thursday, June 29th, 2006 at 4:10 am and is filed under Statgeekery. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.
Good Stuff, Doug. Here are some fantasy playoff thoughts that the NFL would probably never implement for a variety of reasons, but might make sense given some of what you've shown here.
1) Have maximum of 8 playoff teams per conf, minimum of 4 (division winners), with the remainder having to meet some sort of wins standard (for example, 9+ wins) to be eligible to qualify as a wildcard. Thus, the good teams have an incentive to try to keep playing all year, and keep other teams from getting above .500 and increasing playoff field. You also know that if you get to 10 wins, you are virtually a lock to be in (regardless of tiebreakers), and if you get to 9, you have a good chance. Would it really be diluting the field to let teams in with good records? And they may be the "true best team" or one of them that played in a much tougher division. Interestingly, though, San Diego still does not make it last year because of the loss to Miami. And this would never happen because of TV schedules and multiple networks.
2) Get rid of automatic home field advantage for the division winners. Fine, they get in if they win their group of 4, but they actually have to show excellence against the rest of the conference to guarantee home games. In my opinion, this would have two effects that would be positive. First, the better team would win more often. Now, if you play in a tough division, and barely miss out on winning it, you will have to play 3 road games to make the SB. Prior to 2002, these teams (like Denver '97, Tennessee '99, Baltimore '00) at least had the possibility of getting one home game. We're going to see some 8-8 division winners get to host 13-3 wildcards in the future, which I don't see as being fair.
Second, it would decrease the number of laugher games in the final two weeks. Prior to 2002, I believe it was rare that teams rested during the final week, and that was usually only the #1 seed if they clinched. Now, it seems as though everyone is resting starters. I think this is because too many playoff positions are set before hand. NE had no chance of getting a bye, but also no chance of losing out on a home game. JAX had no chance of missing the playoffs, but no posssibility of getting a home game. CIN had no chance of losing a home game. Make this change, and I guarantee NE has something to play for in week 17, because a win moves them in front of Cincinnati and Pittsburgh for a home game, while a loss makes them a 6th seed, not a 4th seed.
It would be interesting to study JKL's contention that the current system increases the chances of meaningless late-season games involving playoff teams who are already locked into their seed. I wonder if the NFL considers this a problem (reduced revenues?) and whether a change in the playoff format would help.
I also find it fascinating, but entirely believable, that the exact playoff format really doesn't matter much. One could probably make similar conclusions about the NCAA hoops tourney despite the widespread belief that the 64-team single-elimination format makes for an "anything can happen" atmosphere.
Excellent point, JKL, and one with much relevance for fantasy football players. I did an article back in 2003 in which I looked at week 17 data and did not find a strong pattern of stars resting in week 17. Ever since then, resting does seem to be way up, and not just in week 17. I never considered that it might be the new divisional alignment that was the culprit, but that may be it.
I think an appropriate analysis of that situation would require me to program the tiebreakers, and that just ain't gonna happen.
Maybe cdcox can use his simulator to shed some light on that subject.
I think the current structure contributes to more meaningless games for two reasons.
1) The difference between a four team division and a five team division (more teams increases, at least to some extent, the chances the difference between the first and second place team in the division will be smaller); and
2) The fact that all home games in the wildcard round automatically go to division winners, where in the past a wildcard could be competing against teams with similar records from other divisions to host a game.
Here is some data:
Average win difference between 1st and 2nd in division (thus, teams getting to host games):
2002-2005 (8 4-team divisions): 2.47
1995-2001 (6 5-team divisions, with 6 in AFC Central from 1999-2001): 1.93
1995-2001 (average diff between wildcard 1 and wildcard 2): 0.93
1995-2001 (combined division winners and wildcards): 1.67
1976-1994 (4 5-team divisions): 1.67
1976-1994 (2 4-team divisions): 2.01
1976-1994 (diff between WC1 & 2): 0.76
In 10 of 32 divisions since 2002, the second place team in the division finished within 1 game of the winner. In comparison, between 1995-2001, 31 of 56 second place teams (either in division or in wildcard race for home game) were within 1 game of the winner.
To check this with a model, I dont know that you would have to have tiebreakers, you could just look at the number of teams within 1 game or tied for a certain position in different formats.
Speaking of tiebreakers, I just finished some analysis of them. Although not directly on topic, this seems as good of place as any to post it. This study was motivated by the first post in this series in which Doug contended that the tiebreakers don't matter for his analysis. That got my curiosity up.
So I basically simulated the 2005 season 100,000 times; and noted whether each tiebreaker decision favored the team with the better or worse power rating. I only considered two team tiebreakers. My software breaks all ties (even between 3 rd and 4th place teams in a division) so I only considered tiebreaers between teams with 0.600 and better records. Listed below are the tiebreaker at which the tie was broken and the fraction of time that the best team won the tiebreaker.
Head to head: 0.48
Division: 0.505
Common Opponent: 0.485
Conference: 0.51
Strength of Victory: 0.45
Strength of Schedule: 0.47
These numbers are accurate to the number of signficant figures shown as they hold up to repeated simulations with different random seeds.
Commentary:
1) For the most part tiebreakers are no better than a coin flip for determining which of two teams is better. They do serve to emphasize certain games as more important than others. List this one under "things the NFL doesn't want you to know."
2) Certain tiebreakers seem to favor the weaker teams by small, but statistically significant amounts. This could be an artifact of my 2005 power ratings and the 2004 finishes (which determines the 2005 schedule). I haven't yet found an obvious explanation. I'll do this analysis again at the end of 2006 and see if the trends are real.
c) as far as resting players durng week 17 goes, the preliminary analysis done by JKL seems convincing. I don't think simulation would shed more light unless you considered tiebreakers. Some teams within 1 game could rest their starters anyway because they hold a TB clinch even if they don't have an WLT% clinch. This proposal would give an additional penalty to teams in very competitive divisions that might be intrinsically strong, but lose out on a homefield playoff game because their WLT% is lower due to a tougher schedule. I'd hate to do anything else to emphasize division alignment more than it already is.
cdcox, wouldn't your analysis be stronger if you used Doug's original system of analyzing a model rather than using real statistics? Then you wouldn't be relying on the accuracy and predictive power of the statistics. Also, it avoids pollution from the real life issues (e.g. is Seattle really the fourth best team in football or just lucky to be in a weak division; should Pittsburgh with a healthy Roethlisberger be ranked based on data with Maddox or a recovering Roethlisberger).
That said, your observation that tiebreakers actually favor the weaker team does not surprise me. Actually, what does surprise me is that division and conference play favor the stronger team. I wonder if you'd get the same results in a model, where the stats are derived from the ranking rather than deriving the ranking from stats.
Head-to-head is especially odious to me in that context. Take two 11-5 teams that faced each other once. If you take away that game, then you have a 11-4 team and a 10-5 team. No tie breaker necessary. On average in that situation, the stronger team (the one with the better record otherwise) lost the game (because if they'd won, there wouldn't have been a tiebreaker). Therefore, the only time the tiebreaker rewards the team that really is better is if their record in other games is wrong. I.e. if the 10-5 team is really better than the 11-4 team. If the standings have any meaning at all, that should usually be untrue.
By contrast, your strength of schedule results surprise me. I wonder if that might be model pollution. I.e. strength of schedule is probably already used to generate your power rankings. I wonder if that might be interfering with the model.
Hi Matt, thanks for the comments; I agree with many of them. The problem is that it is difficult to unwrap the issue of tiebreakers from the issue of schedule. All teams in a division share the exact same schedule except 4 games: the two games where they play another division foe instead of themselves and the two games in which they play teams from another division based on last year's finish (1st vs. 1st, 2nd vs. second, etc.). In reality, a team's strength has some correlation with its previous year's strength. Now, consider the strength of schedule tiebreaker under 3 different models.
1) Doug's model: he did not consider this correlation. In this case, we expect the SOS tiebreaker to favor the stronger team exactly half the time.
2) A model in which a team rankings are perfectly correlated with the season before. Here, the SOS tiebreaker would favor the stronger team more than 50% of the time.
3) The model I ran. Here the SOS tiebreaker favored the weaker team slightly more than half the time. This was most likely due to the fact that some of the first place NFC teams from 2004 faded in strength (Phil, GB, and to a lesser extent Atlanta), giving stronger teams a weaker schedule.
Obviously, the schedule from one season is too little data to draw any conclusions. We would need a model where the team strength had moderate correlation from one season to the next. However, if we consider Doug's model, where the correlation is zero and it favors the stronger team exactly 50% of the time, and the second model where the correlation is 1.0 and would favor the stronger team more than half the time, it would seem reasonable that correlations between 0 and 1 would also slightly favor the stronger team. Thus, the result that the SOS tiebreaker for the 2005 season slightly favored the weaker team would seem to be an anomoly.
The effect of schedule differences stemming from year-to-year correlations in team strength are much less clear to me for the other tiebreakers.
I think the current structure contributes to more meaningless games for two reasons.
1) The difference between a four team division and a five team division (more teams increases, at least to some extent, the chances the difference between the first and second place team in the division will be smaller); and
2) The fact that all home games in the wildcard round automatically go to division winners, where in the past a wildcard could be competing against teams with similar records from other divisions to host a game.
Here is some data:
Average win difference between 1st and 2nd in division (thus, teams getting to host games):
2002-2005 (8 4-team divisions): 2.47
1995-2001 (6 5-team divisions, with 6 in AFC Central from 1999-2001): 1.93
1995-2001 (average diff between wildcard 1 and wildcard 2): 0.93
1995-2001 (combined division winners and wildcards): 1.67
1976-1994 (4 5-team divisions): 1.67
1976-1994 (2 4-team divisions): 2.01
1976-1994 (diff between WC1 & 2): 0.76
In 10 of 32 divisions since 2002, the second place team in the division finished within 1 game of the winner. In comparison, between 1995-2001, 31 of 56 second place teams (either in division or in wildcard race for home game) were within 1 game of the winner.
To check this with a model, I dont know that you would have to have tiebreakers, you could just look at the number of teams within 1 game or tied for a certain position in different formats.