This is our old blog. It hasn't been active since 2011. Please see the link above for our current blog or click the logo above to see all of the great data and content on this site.

Ranking the quarterbacks: schedule and weather adjustments

Posted by Doug on August 12, 2009

At least in principle, the schedule part is easy.

For several years we have often used a mathematical procedure called the Simple Rating System to adjust team performances for the schedules they faced. This post has the details. In a couple of later posts we discussed how to apply the same method to statistics other than points --- like passing yards per attempt for instance --- and how to decompose it into offensive and defensive components.

I'll do a quick recap here.

First we compute every quarterback's adjusted yards per pass attempt and compare it to league average. We likewise compute each defense's adjusted yards per attempt allowed compared to league average. For example, Tony Romo in 2008 was a +0.97, which means he was .97 adjusted yards per attempt above average. Tyler Thigpen was .62 adjusted yards per attempt below average, so he goes down as a -0.62. The Eagles' AY/A was 0.99 lower (i.e. better) than league average, so we write down a +0.99 for them. The Jaguars, on the other hand, were a well below average team at -1.53. Now every quarterback and every defense has an initial rating.

The next step is to adjust each quarterback's rating by adding to it the (attempt-weighted) average of the ratings of the defenses he faced. A quarterback who faced tough defenses will have a positive number added to his rating --- i.e. his rating will be increased. A quarterback who faced an easy schedule will see his rating go down at this stage. Then we do the same thing from the other end. That is, we adjust each defense's ratings to account for the fact that some of them had to play against a lot of good quarterbacks and others had the good fortune of facing a weak collection of passers.

Now every quarterback and every defense will have a new rating. But those ratings were based on strength-of-schedule calculations derived from the old ratings, which have now been superseded by the new ratings. We can't have that. So we adjust the quarterbacks once again, based on the new and improved defense ratings, and we adjust the defenses again based on the new and improved quarterback ratings. Now every player and team has a new new rating. When we repeat this process a few hundred times, the numbers stop changing. Those numbers are the final ratings.

Romo's raw numbers say he was .97 adjusted yards per attempt above average. His final rating turns out to be +1.48, which means that he faced a pretty tough slate of defenses, which included Pittsburgh and Baltimore in addition to the fairly solid NFC East slate. The main component of Chase's Greatest QB of All-Time formula is a yards-per-attempt above average figure. So we'll just slot in the 1.48 in place of the .97 for Romo. That's the process by which we adjust for schedules, and we have done so for each season back to 1960, which is the beginning of the we-have-game-by-game-stats era here at pro-football-reference.

Adjusting for weather is a bit more complicated.

First let's be clear: we do not have weather data for each individual game. Believe it or not, it's likely that within a year or two we will. But for now we don't. So we're forced to deal with generalities. In a perfect world, we'd be applying the "23 degrees but clear" adjustment to all games played when it was 23 degrees but clear, and the "55 degrees and heavy rain" adjustment to games played in those conditions. Instead, we'll have to settle for applying things like a "December game in a cold weather city" adjustment, and other general categorizations along those lines, even though December games in cold weather cities can vary a lot in their weather characteristics. And that does make these adjustments a bit fishy for dealing with small samples of games. Over the course of a long career, I'm confident that the adjustments we're applying are fairly accurate. For a given single season, they may not be.

So how do we come up with these adjustments in the first place? The first step is to steal a decades-old idea from Bill James, who used to compute ballpark factors for baseball parks. The way he did it, for Wrigley Field in 1982 let's say, was to count up all the runs scored by the Cubs and their opponents in Wrigley Field in 1982. And then count up all the runs scored by the Cubs and their opponents in Cubs' road games in 1982. Compare those numbers and you've got an idea of the magnitude of Wrigley Field's effect on run production. Because of the way James set this up, with the same groups of Cubs and non-Cubs being counted the same number of times on each side of the ledger, the difference should in theory isolate the effect of the park independent of the players who happened to call it home.

Baseball seasons have 162 games and football seasons have only 16, so we'll have to aggregate across seasons to get a decent sample size. And we'll aggregate cities with similar weather patterns as well. But the main idea will be the same: passing numbers here compared to passing numbers of the same group of players in locations other than here.

Let's look at domes first. Since 1996, when the Rams moved into their dome, the Rams, Colts, Falcons, Saints, Lions, and Vikings have played all their home games indoors. So I looked at games played by those teams from 1996--2008. Just to be completely sure I was comparing the same set of teams in both situations, I limited the sample to intra-division games (and I ignored intra-division games between two dome teams, like Lions-Vikings). So the 2008 Titans, for instance, will be included in this data set precisely twice: once in the dome and once in Tennessee. And that's the whole point: every single team that's included will be included exactly twice, once on each side of the ledger. The 2008 Colts, of course, will be included four times, but precisely the same number of times and against precisely the same defenses, on each side of the ledger. Here are the results:

In domes: 7.15 yards per attempt
Same teams / same opponents playing outdoors: 6.76 yards per attempt
Difference: .39 yards per attempt

So we declare that a dome game inflates a quarterback's Y/A by about .4 compared to playing outdoors. We'll write down +.4 to symbolize that.

Now we do the same for our other broad categories of games. It was a long and somewhat winding road to get to this point, and I won't go through the details, but these are the categories I ended up with, and their corresponding adjustments:

Dome: +.4
December / January game in a cold weather city: -.6
November game in a cold weather city: -.3
December / January game in a moderately cold city: -.3
Everything else: 0

Cold weather cities are New England, Philadelphia, Cleveland, Green Bay, Cincinnati, and Minnesota and Detroit in the pre-dome years. Moderately cold weather cities are Washington, Baltimore, Carolina, Kansas City, Chicago, Pittsburgh, Buffalo, and New York. These designations might seem a bit odd; Buffalo is surely quite a bit colder than Cincinnati. As I alluded to above, this is as much art as science. I used a mixture of data and common sense. Some of these cities, like Chicago and Buffalo for instance, don't show any real tendency at all to depress passing numbers in winter time. But on general principles, they just can't be in the Everything Else category. So the compromise is to call them moderately cold weather cities.

A few quick notes:

1. Dan Marino and Dan Fouts are often grouped in with Peyton Manning in the "all time great QBs who played in favorable conditions" category, but I see no evidence to support the notion that passing is easier in Florida or Southern California than in any other locations that fall into the Everything Else category.

2. Seattle and Denver seem to be interesting special cases. It's tempting to lump Denver into the cold weather cities, but the data don't back that up. Even in December games, the passing numbers in Bronco games have been almost identical at home and on the road. That, combined with Denver's geographical uniqueness caused me to classify Denver as Everything Else. Since the Seahawks moved outdoors, passing numbers have been substantially higher (about .6) in Seahawk road games than in Seahawk home games, indicating that Seattle is a very tough place to pass. This is worth keeping an eye on, but for now I just decided to declare Small Sample Size and throw Seattle into the Everything Else pile.

At this point, we have weather adjustments for certain categories of games. We need to bootstrap up to obtaining weather adjustments for particular quarterbacks. In order to do this, we need to remember one more thing: conditions that make it more difficult for a quarterback to produce passing numbers necessarily make it easier for the defense to prevent passing numbers. This is part of the equation that rarely gets mentioned, but if Peyton Manning's numbers are inflated because he plays half his games in a dome, it must also be the case that the Colts' defense is better than its raw numbers show, for the very same reason. Likewise, those Northeast and Midwest teams' quarterbacks' numbers are slightly deflated by the late-season weather, but their defenses' numbers' against the pass are inflated by the same amount. So the weather adjustment has an impact on the strength-of-schedule adjustment that we described at the top of this post.

The plan to account for all this is to fold the weather adjustments into the same iterative process that produces the schedule adjustment, turning it into a single "overall context adjustment" that accounts for both weather and schedule. It's a bit technical, so I'll just leave it at that, but the main idea is exactly the same as the one described at the beginning of this post.

After we've done the Overall Context Adjustment, we can then break it back down into a schedule component and a weather component, in case Chase ever wants to include one but not the other.

I'll close with a list of all quarterbacks with 2000 or more career attempts, first sorted by schedule adjustment, then by weather adjustment, then by total combined adjustment. A score with a + in front indicates tough conditions, hence we should add to that players yards per attempt. A negative score indicates easy conditions.

Schedule Adjustment

John Brodie           +0.45
Greg Landry           +0.22
Bart Starr            +0.22
Milt Plum             +0.20
Dan Fouts             +0.19
Daunte Culpepper      +0.17
Dave Krieg            +0.16
Roman Gabriel         +0.16
Peyton Manning        +0.16
Fran Tarkenton        +0.15
Carson Palmer         +0.14
Doug Flutie           +0.12
Scott Mitchell        +0.12
Vinny Testaverde      +0.12
Tom Brady             +0.12
Jim Zorn              +0.11
Dan Pastorini         +0.11
Tommy Kramer          +0.10
Johnny Unitas         +0.10
Kerry Collins         +0.10
Jon Kitna             +0.10
Brian Griese          +0.09
Mark Rypien           +0.09
Trent Green           +0.09
Erik Kramer           +0.08
Drew Bledsoe          +0.08
Joey Harrington       +0.07
Chad Pennington       +0.07
Jim Everett           +0.07
Jeff Hostetler        +0.07
Jim Hart              +0.06
Warren Moon           +0.06
Ken Anderson          +0.06
John Elway            +0.05
Eli Manning           +0.05
Billy Wade            +0.05
Danny White           +0.05
Roger Staubach        +0.03
Joe Theismann         +0.03
Rick Mirer            +0.02
Archie Manning        +0.02
Randall Cunningham    +0.02
Donovan McNabb        +0.02
Boomer Esiason        +0.01
Drew Brees            +0.01
Jim McMahon           +0.01
Chris Miller          +0.01
Babe Parilli          +0.01
Jim Plunkett          +0.01
Mike Tomczak          +0.01
Bobby Hebert          +0.00
Steve Young           +0.00
Jake Plummer          +0.00
Troy Aikman           +0.00
John Hadl             -0.00
Neil Lomax            -0.00
Steve Bartkowski      -0.01
Jay Schroeder         -0.01
Wade Wilson           -0.01
Brian Sipe            -0.01
Rodney Peete          -0.01
Charlie Conerly       -0.01
David Carr            -0.02
Daryle Lamonica       -0.02
Steve DeBerg          -0.02
Len Dawson            -0.02
Brett Favre           -0.02
Craig Morton          -0.02
Jack Kemp             -0.02
Tobin Rote            -0.03
Jeff George           -0.04
Gus Frerotte          -0.04
Elvis Grbac           -0.04
Joe Ferguson          -0.05
Dan Marino            -0.05
Chris Chandler        -0.05
George Blanda         -0.05
Aaron Brooks          -0.05
Rich Gannon           -0.06
Jim Harbaugh          -0.06
Phil Simms            -0.06
Brad Johnson          -0.06
Bernie Kosar          -0.06
Bobby Layne           -0.06
Terry Bradshaw        -0.06
Richard Todd          -0.07
Jeff Blake            -0.07
Joe Montana           -0.07
Tony Banks            -0.08
Ron Jaworski          -0.08
Ken O'Brien           -0.08
Bert Jones            -0.09
Bubby Brister         -0.09
Jeff Garcia           -0.09
Bob Griese            -0.09
Doug Williams         -0.09
Earl Morrall          -0.10
Lynn Dickey           -0.10
Joe Namath            -0.10
Steve Beuerlein       -0.10
Mark Brunell          -0.11
Trent Dilfer          -0.11
Steve McNair          -0.12
Stan Humphries        -0.12
Bill Kenney           -0.13
Ken Stabler           -0.13
Marc Wilson           -0.14
Y.A. Tittle           -0.16
Kordell Stewart       -0.17
Neil O'Donnell        -0.18
Charley Johnson       -0.19
Jim Kelly             -0.19
Billy Kilmer          -0.22
Matt Hasselbeck       -0.22
Steve Grogan          -0.23
Jake Delhomme         -0.24
Norm Snead            -0.25
Frank Ryan            -0.25
Kurt Warner           -0.25
Marc Bulger           -0.29
Sonny Jurgensen       -0.29
Don Meredith          -0.43

Weather adjustment

Boomer Esiason        +0.14
Tom Brady             +0.13
Brett Favre           +0.12
Donovan McNabb        +0.12
Drew Bledsoe          +0.11
Bernie Kosar          +0.11
Jeff Blake            +0.10
Brian Sipe            +0.09
Ron Jaworski          +0.09
Carson Palmer         +0.09
Ken Anderson          +0.08
Steve Grogan          +0.08
Eli Manning           +0.08
Mike Tomczak          +0.08
Lynn Dickey           +0.07
Randall Cunningham    +0.07
Mark Rypien           +0.06
Babe Parilli          +0.06
Kordell Stewart       +0.06
Bubby Brister         +0.06
Fran Tarkenton        +0.06
Neil O'Donnell        +0.05
Jeff Hostetler        +0.05
Chad Pennington       +0.05
Ken O'Brien           +0.04
Vinny Testaverde      +0.04
Milt Plum             +0.04
Jim Kelly             +0.04
Jay Schroeder         +0.04
Phil Simms            +0.04
Marc Wilson           +0.04
Elvis Grbac           +0.03
Jim Plunkett          +0.03
Bill Kenney           +0.03
Doug Flutie           +0.03
Frank Ryan            +0.03
Jack Kemp             +0.03
Neil Lomax            +0.03
Joe Theismann         +0.03
Joe Ferguson          +0.03
Norm Snead            +0.03
Steve Young           +0.02
Kerry Collins         +0.02
Sonny Jurgensen       +0.02
Len Dawson            +0.02
Jake Plummer          +0.02
Steve DeBerg          +0.02
Jim McMahon           +0.02
Bert Jones            +0.02
Erik Kramer           +0.02
Joe Namath            +0.02
Jake Delhomme         +0.02
Trent Dilfer          +0.01
Dan Marino            +0.01
David Carr            +0.01
Jeff Garcia           +0.01
Earl Morrall          +0.01
Stan Humphries        +0.01
Steve McNair          +0.01
Greg Landry           +0.01
John Elway            +0.01
Joe Montana           +0.01
Terry Bradshaw        +0.01
Doug Williams         +0.01
Mark Brunell          +0.01
Bart Starr            +0.01
Steve Beuerlein       +0.01
Troy Aikman           +0.01
Danny White           +0.00
Matt Hasselbeck       +0.00
Charlie Conerly       +0.00
Bobby Layne           -0.00
Craig Morton          -0.00
Brian Griese          -0.00
Steve Bartkowski      -0.00
Roman Gabriel         -0.00
Daryle Lamonica       -0.01
Tobin Rote            -0.01
Y.A. Tittle           -0.01
John Hadl             -0.01
Dan Fouts             -0.01
Billy Kilmer          -0.01
Richard Todd          -0.01
Chris Miller          -0.01
Trent Green           -0.01
Bob Griese            -0.01
Jim Hart              -0.02
Johnny Unitas         -0.02
Billy Wade            -0.02
Jim Harbaugh          -0.02
Roger Staubach        -0.02
Gus Frerotte          -0.03
Jon Kitna             -0.03
Tony Banks            -0.04
John Brodie           -0.04
Jim Everett           -0.04
George Blanda         -0.04
Don Meredith          -0.05
Tommy Kramer          -0.05
Rich Gannon           -0.06
Brad Johnson          -0.06
Ken Stabler           -0.06
Rodney Peete          -0.07
Charley Johnson       -0.07
Kurt Warner           -0.07
Rick Mirer            -0.09
Drew Brees            -0.10
Wade Wilson           -0.11
Dave Krieg            -0.11
Scott Mitchell        -0.12
Jeff George           -0.12
Aaron Brooks          -0.13
Archie Manning        -0.13
Chris Chandler        -0.13
Jim Zorn              -0.16
Bobby Hebert          -0.16
Joey Harrington       -0.18
Daunte Culpepper      -0.18
Warren Moon           -0.18
Peyton Manning        -0.19
Dan Pastorini         -0.20
Marc Bulger           -0.21

Total combined adjustment

John Brodie           +0.41
Tom Brady             +0.25
Milt Plum             +0.24
Carson Palmer         +0.23
Greg Landry           +0.23
Bart Starr            +0.23
Fran Tarkenton        +0.20
Drew Bledsoe          +0.19
Dan Fouts             +0.18
Vinny Testaverde      +0.16
Roman Gabriel         +0.16
Mark Rypien           +0.15
Boomer Esiason        +0.15
Doug Flutie           +0.15
Ken Anderson          +0.14
Donovan McNabb        +0.14
Eli Manning           +0.13
Kerry Collins         +0.12
Jeff Hostetler        +0.12
Chad Pennington       +0.12
Erik Kramer           +0.10
Brett Favre           +0.09
Brian Griese          +0.09
Randall Cunningham    +0.09
Mike Tomczak          +0.09
Johnny Unitas         +0.08
Brian Sipe            +0.08
Trent Green           +0.08
Babe Parilli          +0.07
Jon Kitna             +0.06
John Elway            +0.06
Joe Theismann         +0.05
Danny White           +0.05
Tommy Kramer          +0.05
Dave Krieg            +0.05
Jim Plunkett          +0.04
Bernie Kosar          +0.04
Jim Hart              +0.04
Jay Schroeder         +0.03
Billy Wade            +0.03
Jim McMahon           +0.03
Jeff Blake            +0.03
Steve Young           +0.03
Neil Lomax            +0.02
Jim Everett           +0.02
Jake Plummer          +0.02
Ron Jaworski          +0.01
Roger Staubach        +0.01
Troy Aikman           +0.01
Jack Kemp             +0.01
Chris Miller          +0.00
Scott Mitchell        +0.00
Steve DeBerg          -0.00
Len Dawson            -0.00
David Carr            -0.00
Steve Bartkowski      -0.01
John Hadl             -0.01
Elvis Grbac           -0.01
Charlie Conerly       -0.01
Daunte Culpepper      -0.01
Joe Ferguson          -0.02
Daryle Lamonica       -0.02
Phil Simms            -0.02
Craig Morton          -0.02
Lynn Dickey           -0.03
Peyton Manning        -0.03
Bubby Brister         -0.03
Tobin Rote            -0.04
Dan Marino            -0.04
Ken O'Brien           -0.04
Jim Zorn              -0.04
Terry Bradshaw        -0.06
Bobby Layne           -0.06
Gus Frerotte          -0.06
Joe Montana           -0.06
Rick Mirer            -0.07
Bert Jones            -0.07
Richard Todd          -0.07
Jeff Garcia           -0.08
Rodney Peete          -0.08
Jim Harbaugh          -0.08
Earl Morrall          -0.08
Doug Williams         -0.09
Joe Namath            -0.09
Drew Brees            -0.09
Dan Pastorini         -0.09
George Blanda         -0.09
Trent Dilfer          -0.10
Steve Beuerlein       -0.10
Mark Brunell          -0.10
Bill Kenney           -0.10
Marc Wilson           -0.11
Bob Griese            -0.11
Joey Harrington       -0.11
Steve McNair          -0.11
Stan Humphries        -0.11
Tony Banks            -0.11
Archie Manning        -0.11
Kordell Stewart       -0.11
Rich Gannon           -0.12
Wade Wilson           -0.12
Brad Johnson          -0.12
Warren Moon           -0.13
Neil O'Donnell        -0.13
Steve Grogan          -0.15
Jim Kelly             -0.15
Jeff George           -0.16
Y.A. Tittle           -0.16
Bobby Hebert          -0.16
Aaron Brooks          -0.18
Chris Chandler        -0.19
Ken Stabler           -0.20
Matt Hasselbeck       -0.22
Norm Snead            -0.22
Frank Ryan            -0.22
Jake Delhomme         -0.23
Billy Kilmer          -0.23
Charley Johnson       -0.25
Sonny Jurgensen       -0.27
Kurt Warner           -0.33
Don Meredith          -0.48
Marc Bulger           -0.50