SITE NEWS: We are moving all of our site and company news into a single blog for Sports-Reference.com. We'll tag all PFR content, so you can quickly and easily find the content you want.

Also, our existing PFR blog rss feed will be redirected to the new site's feed.

Pro-Football-Reference.com ยป Sports Reference

For more from Chase and Jason, check out their work at Football Perspective and The Big Lead.

Quarterback post-season records and Simpson’s Paradox

Posted by Chase Stuart on January 31, 2011

I almost deleted this post before I hit "Publish." There are so many caveats I'm urged to proclaim, and so many nits at which any reader could pick, that I'm still not sure if this is worth posting. Further, on some level, I fundamentally disagree with the not-so-subtle argument this post implicitly endorses. Allow me to cut you off, by noting that yes, this post is stupid, yes I forgot about X, Y and Z, yes, this doesn't even make sense once you realize M, N and Q, yes I've never watched a football game before, and yes I'm biased against Player A and Player B. And, of course, I am Player C's mother. Note that I've categorized this post under both Rant and Insane ideas.

The comments to Neil's post on The Rivers Index raised some interesting questions. Commenter Sean played the role of Marino backer and noted how Miami was always being outrushed in those playoff losses. He pointed out, correctly, that Dan Marino is the only victor of the 52 quarterbacks to start a playoff game in the last 60 years when his team was outrushed by 150+ yards .

I started wondering how to break down each playoff game based on the level of support each quarterback received, from both the running game and the defense. Game-ending stats are deceiving -- just one of the many caveats in my head as I wrote paragraph 1 -- but I figured there was little harm in doing some back of the envelope calculations. If nothing else, this post can just add some layers to the typical discussion of post-season records. Here's what I did:

1) Noted the rushing yardage differential for each team in the 440 playoff games from 1950 to 2009.

2) Calculated the standard deviation of the rushing margins in those games. The average margin, by definition, was zero, with a standard deviation of 93.1 yards. So if Dan Marino's Dolphins were outrushed by 186 yards, Marino would be credited with -2.0 standard deviations of rushing support for that game. When Roger Staubach's Cowboys outrushed the Bears by 152 yards in 1977, Staubach received 1.6 standard deviations of support from the ground game that day.

3) Noted how many points the team allowed in each game, after subtracting seven points for interception return touchdowns by the opponent, i.e., pick-sixes thrown by the team, and all defensive points scored by the team. This shows how many points that team's offense needed to score, in a vacuum, to win the game. For example, in Super Bowl XXXVII, the Bucs beat the Raiders 48-21. But Rich Gannon threw three interceptions that were returned for touchdowns, and the Raiders scored a touchdown on special teams. As a result, Gannon's game will be coded as one where his team allowed only 20 adjusted points, because that's how many points the Raiders offense would have needed to score to tie the game. On the other side, Brad Johnson received a ton of support that day -- his team allowed zero adjusted points. The goal in this step three was to adjust the points allowed number to show what the quarterback really needed to exceed after factoring out his own pick-sixes and points scored by his own defense and special teams.

4) Calculate the standard deviation for adjusted points allowed in these games. Each team allowed, on average, 18.9 adjusted points, with a standard deviation of 11.4. When Brett Favre's Packers allowed 31 points to the Broncos in Super Bowl XXXII, this system says he received -1.1 standard deviations of support from his defense. If the quarterback's defense pitches a shutout, the quarterback received 1.7 standard deviations of support at a minimum (and more, of course, if the defense scores, too).

5) Add the standard deviations of support received from both the defense and the running game to come up with a total number of support in the game.

By this admittedly flawed metric, what was the easiest game any quarterback has been asked to win? John Elway against the Dolphins in 1998. Elway gets credit for a victory in a game where his defense allowed 3 points, scored a touchdown, and limited the Dolphins to just 14 rushing yards, while his running game produced 250 yards of its own. Elway won a lot of games for the Broncos, but on a day where his defense outscored Miami's entire team and Denver outrushed the Dolphins by 236 yards, the 2010 version of Elway could have won that game.

The toughest game any quarterback had to win? Miami the next season against Jacksonville, when Dan Marino needed to win a game where the opponent scored 62 points and his team was (again) outrushed by 236 yards.

The toughest game any quarterback's been asked to win that he actually did? Over the course of his career, Terry Bradshaw may have received more support than any quarterback in history; still, you can't take credit away from Bradshaw for winning Super Bowl XIII. Dallas scored 31 points and outrushed the Steelers by 88 yards, but Bradshaw threw for 318 yards and 4 TDs on only 30 passes that day, and was rightfully named the game's most valuable player. Jeff Garcia's performance against the Giants when he rallied San Francisco back from a 38-14 deficit matched Bradshaw on the difficulty index. No team has ever won a playoff game where they allowed more adjusted points. Dan Marino's 1985 game against the Browns, where Cleveland outgained by Miami by 159 rushing yards, comes in third.

After every game was coded and each quarterback received a grade based on how many standard deviations of support he received, I decided to group the games into buckets. I placed all 880 playoff games into one of five categories for the quarterback based on the likelihood of success: Impossible, Difficult, Neutral, Easy and Gimmes. A game was labeled Impossible if the quarterback received fewer than -2.0 standard deviations of support (rushing and defense combined); quarterbacks were 0-111 when placed in Impossible situations. Gimmes were the games where the quarterback received 2.0 or more standard deviations of support. Quarterbacks went 117-0 in those games.

Difficult games were ones where the QB was faced with a standard deviation between -1.0 and -1.99; in those situations, quarterbacks were 13-129, winning 9% of those games. On the flip side, quarterbacks won 92% of the Easy games, when they got to play with a defense/running game that provided a standard deviation of between 1.0 and 1.99. Finally, all games between -0.99 and 0.99 were defined as neutral, and quarterbacks won 48% of those games. (Because not all Neutral games for one team are Neutral for the other, this does not need to be 50% by definition.)

We can then come up with expected winning percentages for each quarterback based on the number of games he played in those situations. As it turns out, Jim Plunkett would have been expected to win about 80% of his 10 playoff games based on the support he received from his running game and defense. He played in one "Gimme" game, which based on league averages is an automatic win. He had six "Easy" games, and since quarterbacks win 92% of those games, we would have expected him to win 5.52 of those games. His other three games were neutral, so we credit him with 1.44 expected wins, for a total of 7.96 wins. In reality, Plunkett was 8 of his 10 playoff games, and is credited with being a "winner."

On the flip side? Dan Marino, the genesis behind this post. Eight of his 18 games received Impossible grades compared to just one "Gimme." So Marino went 1-8 in those games where every other quarterback would be expected to go 1-8. The other half of his playoff games? Five Neutral, one Difficult and one Easy. Marino won both the Easy and the Difficult game, and went 5-2 in the seven Neutral games. So his 8-10 record looks pretty good compared to the 30% winning percentage we would expect from these games. He would have been expected to win just 5.4 of his 18 games, or be 3.6 games below .500. Here are the QBs with the best and worst expected records, relative to .500:

QB Record Games Exp. W Exp. W% Exp. O.500
Dan Marino 8-10 18 5.4 30.0% -3.6
Warren Moon 3-7 10 2.9 29.2% -2.1
Peyton Manning 9-9 18 7.2 39.8% -1.8
Dave Krieg 3-6 9 3.1 34.1% -1.4
Ken Anderson 2-4 6 1.6 26.4% -1.4
Jake Plummer 2-4 6 1.6 26.4% -1.4
Jeff Garcia 2-4 6 1.7 28.8% -1.3
Vinny Testaverde 2-3 5 1.4 28.0% -1.1
Wade Wilson 2-3 5 1.5 29.0% -1.0
Randall Cunningham 3-6 9 3.5 38.5% -1.0
Steve Grogan 0-3 3 0.5 16.1% -1.0
Bert Jones 0-3 3 0.5 16.1% -1.0
Marc Bulger 1-2 3 0.5 16.1% -1.0
Pete Beathard 0-2 2 0.0 0.0% -1.0
Dan Fouts 3-4 7 2.5 35.8% -1.0
...
Don Heinrich 2-1 3 2.5 82.8% 1.0
Boomer Esiason 3-2 5 3.5 69.8% 1.0
Jim McMahon 3-3 6 4.1 67.6% 1.1
Len Dawson 5-3 8 5.1 63.4% 1.1
Brad Johnson 4-3 7 4.8 69.2% 1.3
Tom Brady 14-4 18 10.6 58.9% 1.6
Steve McNair 5-5 10 6.6 66.1% 1.6
Joe Montana 16-7 23 13.1 57.1% 1.6
Troy Aikman 11-4 15 9.2 61.0% 1.7
Joe Theismann 6-2 8 5.9 73.6% 1.9
Trent Dilfer 5-1 6 5.3 88.6% 2.3
Bart Starr 9-1 10 7.6 76.4% 2.6
Terry Bradshaw 14-5 19 12.3 64.6% 2.8
Jim Plunkett 8-2 10 8.0 79.5% 3.0
Roger Staubach 11-6 17 11.8 69.1% 3.3

Click here for the full list of quarterbacks.

Dan Marino and Troy Aikman can be used to illustrate the phenomenon known as Simpson's Paradox. Despite Aikman's three rings and sparkling 11-4 playoff record, he doesn't have a better winning percentage than Marino in a single subset of games. Neither man won an Impossible game. In two Difficult games, Aikman went 0-2; Marino won the only Difficult game in which he played. Each quarterback played 7 Neutral games and both went 5-2. Neither lost an Easy game, with Marino going 1-0 and Aikman winning all five in which he played. Finally, both won their one Gimme playoff game. It's tempting to wonder how Marino's 8-10 playoff record and Aikman's 11-4 mark would look if their situations had been reversed.

How about Brady and Manning? Without getting to this year's results, they look pretty similar. Brady went 4-0 and 2-0 in Easy and Gimme games, while Manning went 2-0 and 1-0 in those spots. Brady was 0-1 in Impossible games (the Ravens disaster last year) while Manning lost two such games. Brady won his only Difficult game, against the Chargers when San Diego scored 21 points and outrushed New England by 96 yards when they weren't busy giving the game away. Manning went 1-4 in Difficult games, beating Kansas City when neither team punted. In neutral games, Brady has been 7-3 while Manning went 5-3. [2010 playoff addendum: The Jets scored 17 points and outrushed the Colts by 76 yards; he would get credit for -0.6 standard deviations of support, making this a Neutral loss for Manning. The Jets scored 28 points and outrushed the Patriots by 7 yards in the playoffs this year; Brady would get credit for -0.9 standard deviations of supporting, making this a Neutral loss for him.]

Click here for the full breakdown for each quarterback.

Finally, here's a page that lists every playoff game with the statistics used from this study.

Once again, I don't know what any of this means. I don't have an overarching conclusion, but I wanted to add to an interesting discussion in the comments by providing some data. I've provided the large above table in the hopes that you, the reader, can build off this as part of whatever research you plan on performing (perhaps by breaking down the Neutral games into Neutral-Easy and Neutral-Hard, or coming up with your own formula to rank the games).

This entry was posted on Monday, January 31st, 2011 at 1:19 pm and is filed under Insane ideas, Quarterbacks, Rant. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.