I wrote this post for the sabernomics blog back in January of 2005 --- two weeks after the Patriots beat Colts in the divisional round and a week before the Patriots beat the Eagles in the Super Bowl. Despite its being out of date, I decided to leave the entire article unedited.

A year later --- just before the Steelers beat Seattle in Super Bowl XL --- I updated it with another year's worth of data and added a few extra thoughts. That article --- also unedited --- follows.

Last week, much was made of the fact that the Colts are 3-5 in playoff games started by Peyton Manning. Is Peyton a choker? I don’t think we’ve got sufficient evidence to make that claim, especially in light of the fact that the Colts have been the higher seed in only 3 of those 8 playoff games. In other words, the Colts’ postseason record in the Manning era is exactly what one would expect using an admittedly crude but very reasonable predictor. I thought that was interesting, so I decided to refine it just a bit and run that query for all the great past and present QBs.

First the refining.

I looked at all postseason games since 1978 and ran a logit regression (there’s the economics content in this post) with a win dummy as the output variable and the team records and game location as the inputs. For those curious, the formula is

Probability of winning = (1 + exp(-.43(windiff)-.24(homefield)))^(-1)

where windiff = the given team’s regular season wins minus its opponents’ regular season wins and homefied = 1 if home, -1 if road, 0 if neutral site. So, for example, the 14-2 Patriots taking on the 12-4 Colts in Foxboro would have a windiff of 2 and a homefield of 1, which translates to an expected win probability of .748. Now all that’s left to do is tally up every quarterback’s expected wins (which is the sum of the win probabilities for each game) and his actual wins, and sort the list:

                     Expected  Actual    Diff

Tom Brady                 4.5       8      +3.5

Joe Montana              13.7      16      +2.3

Trent Dilfer              2.8       5      +2.2

John Elway               12.2      14      +1.8

Troy Aikman               9.2      11      +1.8

Mark Rypien               3.4       5      +1.6

Jeff Hostetler            2.5       4      +1.5

Wade Wilson               1.7       3      +1.3

Brett Favre              10.2      11      +0.8

Drew Bledsoe              3.3       4      +0.7

Phil Simms                5.4       6      +0.6

Doug Williams             3.4       4      +0.6

Jay Schroeder             2.4       3      +0.6

Brad Johnson              3.5       4      +0.5

Jim Everett               1.6       2      +0.4

Donovan McNabb            6.6       7      +0.4

Steve McNair              4.6       5      +0.4

Jim Harbaugh              1.7       2      +0.3

Kurt Warner               4.9       5      +0.1

Rich Gannon               3.9       4      +0.1

Stan Humphries            3.0       3      +0.0

Mark Brunell              3.0       3      +0.0

Jim Kelly                 9.3       9      -0.3

Kerry Collins             3.3       3      -0.3

Vinny Testaverde          2.4       2      -0.4

Dave Krieg                3.4       3      -0.4

Bernie Kosar              3.5       3      -0.5

Mike Tomczak              3.6       3      -0.6

Peyton Manning            3.8       3      -0.8

Neil O'Donnell            3.9       3      -0.9

Kordell Stewart           3.0       2      -1.0

Steve Young               9.1       8      -1.1

Jim McMahon               4.2       3      -1.2

Randall Cunningham        4.2       3      -1.2

Dan Marino                9.4       8      -1.4

Warren Moon               4.9       3      -1.9

Fine print: the list includes all quarterbacks whose careers began in 1978 or later (hence no Terry Bradshaw or Snake Stabler) and played in at least five postseason games. A QB was credited with a game played if he attempted 10 or more passes in the game.

Just to be clear, I believe that teams — not quarterbacks — win football games, so I’m not claiming this is the One True Measure Of Clutchness. Whether I like it or not though, wins are credited to quarterbacks in virtually every discussion about quarterback greatness. This is merely a way of putting a quarterback’s win-loss record into perspective.

I hate to admit it, but the deification of Tom Brady is getting tougher and tougher to argue with. This metric overvalues him just a tad by giving him credit for the 2001 victory at Pittsburgh (Bledsoe was probably more responsible for that win), but still. The probability of going 8-for-8 in the specific collection of postseason games Brady has played in is .004.

The following was written just prior to the Seahawks/Steelers Super Bowl:

Last year in this space, I observed that Peyton Manning’s teams had won exactly as many playoff games as they had been the higher seed in. This fact, to my mind, ran contrary to the popular wisdom that Manning is a choker. So I came up with something I called the Manning Index, which essentially measures how many playoff games a quarterback has won compared to how many he “should have” won. Click the link above for more discussion; I won’t re-hash much of it here, but I will give a quick summary of the specifics.

Based on a logit regression of all playoff games during the past 30 years, I arrived at this formula for determining the probability of a given team winning a given playoff game:

Probability of winning = (1 + exp(-.43(windiff)-.24(homefield)))^(-1)

where windiff is the team’s wins minus the opponent’s wins and homefield is 1 if it’s a home game, -1 if it’s a road game, and 0 if it’s at a neutral site (i.e. a Super Bowl). For an example, let’s look at the Steelers’ and Seahawks’ 2005 playoff runs:

                                           prob of

Game                  windiff   homefield  winning

Steelers vs. Bengals      0        -1        44.1%

Steelers vs. Colts       -3        -1        18.1%

Steelers vs. Broncos     -2        -1        25.3%

Seahawks vs. Redskins    +3        +1        81.9%

Seahawks vs. Panthers    +2        +1        74.7%

Roethlisberger’s team won three games this postseason, when it should have been expected to win only about .88 games, so I’ll give Big Ben credit for 3 - .88 = 2.12 wins worth of clutchness. As I said last year, I think that awarding wins to quarterbacks is a suspect practice, but people are going to do it anyway. My only goal here is to put a quarterback’s postseason win-loss record into the proper perspective.

The main point of this post is to refresh the rankings with another year’s worth of data now in the books. So here they are. Some discussion follows.

                      Expected  Actual  Marginal

Quarterback            record   record    wins

Tom Brady               6- 5    10- 1     +4.4

Trent Dilfer            3- 3     5- 1     +2.2

Jake Delhomme           3- 4     5- 2     +2.3

Ben Roethlisberger      2- 3     4- 1     +1.6

Jeff Hostetler          2- 2     4- 1     +1.5

Mark Rypien             3- 5     5- 3     +1.6

Wade Wilson             2- 4     3- 3     +1.3

Joe Montana            14- 9    16- 7     +2.3

Troy Aikman             9- 7    11- 5     +1.8

John Elway             12-10    14- 8     +1.8

Jay Schroeder           2- 3     3- 2     +0.6

Drew Bledsoe            3- 4     4- 3     +0.7

Doug Williams           3- 4     4- 3     +0.6

Phil Simms              5- 5     6- 4     +0.6

Jim Everett             2- 3     2- 3     +0.4

Brad Johnson            3- 4     4- 3     +0.5

Brett Favre            10-10    11- 9     +0.8

Mark Brunell            4- 6     4- 6     +0.5

Jim Harbaugh            2- 3     2- 3     +0.3

Steve McNair            5- 4     5- 4     +0.4

Kurt Warner             5- 2     5- 2     +0.1

Rich Gannon             4- 4     4- 4     +0.1

Stan Humphries          3- 3     3- 3     +0.0

Donovan McNabb          7- 5     7- 5     +0.0

Jim Kelly               9- 7     9- 7     -0.3

Dave Krieg              3- 6     3- 6     -0.4

Kerry Collins           3- 3     3- 3     -0.3

Vinny Testaverde        2- 3     2- 3     -0.4

Bernie Kosar            3- 4     3- 4     -0.5

Jake Plummer            2- 4     2- 4     -0.5

Mike Tomczak            4- 2     3- 3     -0.6

Steve Young             9- 5     8- 6     -1.1

Dan Marino              9- 9     8-10     -1.4

Randall Cunningham      4- 6     3- 7     -1.2

Kordell Stewart         3- 2     2- 3     -1.0

Peyton Manning          5- 4     3- 6     -1.6

Jim McMahon             4- 2     3- 3     -1.2

Warren Moon             5- 5     3- 7     -1.9

First note that the records are all rounded to the nearest integer — records just don’t look right if they’re not integers — but the Marginal Wins column is not rounded (well, less rounded). Also, note that the list is sorted not by the Marginal Wins column, but by an approximation of the probability that an average quarterback would achieve the given record or a better one by sheer chance. For example, Joe Montana is +2.3 wins and Jeff Hostetler is +1.5, but Hoss rates higher than Joe because it’s less likely that random chance would produce a 4-1 record in the games Hostetler played in than that it would produce a 16-7 record in the games Montana played in. Incidentally, only one of the 38 guys on the list appears to be significantly better than chance, and none are significantly worse than chance. Make of that what you will.

Bonus fun fact I uncovered while running these numbers: according to the formula given at the top of this post, this season’s Steeler team is the most improbable Super Bowl team in history. Their estimated win probabilities were .441, .181, and .253, which means that their probability of winning all three (making all the usual incorrect assumptions about independence) was about .02, which is the lowest figure of any team to ever make a Super Bowl. Now that’s not too surprising, since they played three games and most Super Bowl teams only play two. But if you throw out the Cincinnati game, their probability would be .045, which would still be the lowest in history.

Most Improbable Super Bowl Teams

Team    Probability

pit 2005    2.0

nwe 1985    5.1

dal 1975    5.3

car 2003    7.6

ram 1979    8.6

bal 2000    9.9

oak 1980   10.9

ten 1999   11.2

sfo 1988   11.5

den 1997   12.3

buf 1992   12.8

This entry was posted on Thursday, August 3rd, 2006 at 5:36 am and is filed under General, History, Statgeekery. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.