SITE NEWS: We are moving all of our site and company news into a single blog for Sports-Reference.com. We'll tag all PFR content, so you can quickly and easily find the content you want.

Also, our existing PFR blog rss feed will be redirected to the new site's feed.

Pro-Football-Reference.com » Sports Reference

For more from Chase and Jason, check out their work at Football Perspective and The Big Lead.

Ranking the QBs — Methodology Discussion

Posted by Chase Stuart on June 20, 2008

A couple of years ago, I ran a post here titled "The Best QB of All Time." It was a pretty popular and controversial post, so that's reason enough to run it again every couple of years. All next week, I'll be running posts on the best and worst QBs of all time. Fortunately for me, I've got a whole lot more data (and brains) at my disposal now than I did two years ago. The biggest change, by far, is the addition of every QB in NFL history, as opposed to just QBs since 1970.

For those that don't remember, here's the basic summary. We start with each QB's adjusted yards per attempt average. AY/A is simply passing yards, adjusted by adding 10 yards for every passing TD and subtracting 45 yards for every interception. It's a neat, relatively simple stat that does a nice job of approximating how good a QB was in a given season. Obviously this isn't going to be perfect, as QBs on bad teams will be undervalued, while those that play with great WRs and offensive lines will be overvalued. For now, I'm just going to have to throw up my hands and so, 'oh well.' Additionally, of course, there's no adjustment whatsoever for playoff performance.

To rank QBs in a single season, I simply took each QB's AY/A ratio, subtracted the league average AY/A ratio, and then multiplied that difference by the QB's number of attempts. This does a nice job of balancing the trade-off between compilers and guys that excelled for a short period of time.

For example...

the league average QB passed for 5.85 AY/A in 2007. So a QB that averaged 8.00 AY/A last year would have been terrific; but if he had only 300 attempts last year, he'd be ranked equally with a guy that had 500 attempts and a 7.14 AY/A ratio. If we used raw numbers, the guy with 500 attempts would almost always win; if we used averages, the guy with an 8.00 AY/A metric would always win. I think this strikes a nice and acceptable balance.

So what changes have I made? For starters, I've removed each QB's stats from the league average, when comparing the QB to the league average. So more precisely, each QB's stats is compared to the stats of every other QB, and not the league average. This has little practical effect for recent QBs, but it does bump up some great old QB seasons, which I think is always a good thing.

Also, I chose to handle the AFL and NFL as separate leagues, so Joe Namath is compared to other AFL QBs and Johnny Unitas is compared to other NFL QBs. This seemed fair to me. AAFC stats, much to the dismay of Otto Graham's heirs, have been left out. Perhaps just as importantly, I've made a small adjustment for the non-16 game season leagues (everything before 1978, 1982 and 1987). It's easier to excel for 12 games (i.e., Otto Graham's 1953 season) than it is for 16 games, so I thought it would be unfair to pro-rate Graham's seasons to 16 games. On the other hand, it's not his fault that the league only played 12 games back then, and it's certainly easier to accumulate a ton of yards above league average when you play four more games. So I split the baby, and pro-rated all non-16 game seasons to the difference between 16 games and the number of games in the season. Graham's numbers are pro-rated to a 14 game season, Dan Fouts' great 1982 season is pro-rated as if it was a 12.5 game season, and so on. This just felt right to me, even if it's sort of a cop out.

I then took a page out of Doug's Approximate Value book and weighted each season in a QB's career. So when calculating a quarterback's career value, I gave him 100% of his rating in his best year, plus 95% of his rating in his second best season, plus 90% of his rating in his third best year, and so on. This strikes a nice balance between rewarding the guys who played really well for a long time, but without killing guys with really bad rookie years or seasons late in their career.

With all that out of the way, there are three variables I chose to focus on when ranking the QBs. I think you could make a decent argument for including or not including any of these stats. Let me go into a bit of detail here.

First, we could choose to include rushing yards or not. I've never really seen a study that properly incorporated rushing data into a QB's overall statistical bio. What I did in my History of the Black QB: Part II post was use "Rushing Yards Over 4.0." So a QB that runs 100 times for 600 yards will get a bonus of 200 yards added to his score. I like that system, and I'm going to also add 10 yards for every rushing TD, too. After all, there's no reason why a QB shouldn't be rewarded when he runs for a score. Now, a hypothetical QB with 500 attempts and 7.14 AY/A (and say, 50 rushes for 75 yards) would be equivalent to a QB with 450 attempts, 100 rushes, 500 rushing yards, 5 rushing TDs, and 6.95 adjusted yards per pass. Both would come in at 645 yards above average. Once again, the 4.0 number is pretty arbitrary, but I feel okay about it. I want to reward good runners, but I also don't want to penalize QBs that have a lot of kneel downs, either.

Second, we could choose to lower the baseline from "league average" to three-quarters of league average. The main reason to do this is you can get some pretty quirky results the other way. For example, is Drew Bledsoe not one of the top 100 QBs of all time? Is Trent Dilfer one of the fifteen worst QBs ever? (Well, maybe, maybe not. We'll table that until next week). The bottom line is, QBs that are exactly average have some value. They just do. If the Minnesota Vikings had an average QB last year, they would have made the playoffs. On the other hand, there are good reasons for preferring to keep the league average baseline -- more on this later.

Third, we could choose to include sacks allowed and sack yards lost data into our system. I didn't have this data last time around, and while it's controversial, I think it should be used. After all, my current system would penalize a QB for an incompletion, but not for a 10-yard sack. QBs that throw the ball away and avoid sacks are actually pretty valuable. While sacks certainly aren't solely the fault of the QB, I think including the data do more good than bad. So we can use adjusted net yards per attempt, instead of regular old AY/A (net means we subtract sack yards lost from our numerator, and add sacks to our denominator).

So far, I've been talking mostly in the abstract. Let me give you some examples.

Including Rushing Stats: Not surprisingly, including rushing stats helps Randall Cunningham, Kordell Stewart and Mike Vick the most. It also gives a big boost to Bobby Douglass, Vince Evans, Archie Manning, Steve Young, Steve McNair, Fran Tarkenton, Donovan McNabb, and Jake Plummer.

No QB really drops once you include rushing stats. That's because I don't assign any negative points for being a bad rusher. I'm pretty happy with how this turned out. Including rushing stats hurts Marino and Montana, as they may get jumped by some running QBs. But mostly, excluding running stats doesn't help anyone, and really undervalues some good runners.

1.00 or 0.75: Lowering the baseline from league average to three-fourths of league average is obviously going to help compilers, but maybe not as much as you'd expect. Using the 100%/95%/90%/85%/etc. formula, the value of compilers is automatically lowered. Vinny Testaverde is helped by lowering the baseline, but it only bumps him from the 50s or 60s to the 30s. Jon Kitna is the QB that's helped the most by using the league average baseline. Trent Dilfer, Jake Plummer, and Drew Bledsoe are, too. It doesn't necessarily helps compilers, but it helps guys who had seasons with a lot of attempts where they were just under the league average.

Brett Favre only ranks incredibly high on these lists if you use 75% of league average, and not league average. Warren Moon and John Elway also get big bumps by dropping the baseline. Do we think of them as compilers? I'm not so sure.

On the other hand, some old timers are helped a bunch by using the league average baseline. Otto Graham shoots up the list. Sid Luckman, Sammy Baugh and Norm Van Brocklin do, too. Guys that were great for short periods -- Steve Young, Bart Starr, Roger Staubach, Kurt Warner, Bert Jones and Len Dawson -- all receive nice boosts by using the league average baseline. On the whole, I like rewarding the Otto Grahams of the world and not the Jon Kitnas of the world. I'm sure Favre fans would prefer we use three-fourths of league average, but I slightly prefer using league average. Additionally, Favre gets only a small penalty for his awful 2005 season because of the weighted average scale, so his fans can't complain too much.

Including Sacks -- net data: Who moves up if we switch from adjusted yards per attempt to adjusted net yards per attempt? Well, Dan Marino gets a huge boost. So do Dan Fouts and Jim Hart. Jim Everett and Favre didn't get sacked often, so they look better, too. Deberg, Aikman, and Montana get sizable bumps, as well.

Conversely, some guys got sacked a lot. Steve Young and Randall Cunningham fall a bit once you penalize for sacks. So do Jeff George, Ken O'Brien, Phil Simms, Mike Vick and Fran Tarkenton. It's clear that a sack isn't entirely the fault of the QB, but guys like Marino should get credit for avoiding the sack. Young, especially since he played in the WCO, really should be penalized for having quite a few more sacks than I remember. And I think we're all happy (Jason Whitlock doesn't read this blog) that Jeff George ranks as the 93rd best QB of all time, and not the 60th best, as he would if sack data were excluded.

Final thoughts

There is some overlap here, of course. Using the weighted scale tends to actually help some compilers, because a lot of them had some really bad years. Therefore, I feel better about using league average as the baseline, as compilers would probably get too much credit any other way. And because of the way using 1.00 of league average helps the old time QBs, and because accounting for era is really, really important, I think using league average is slightly preferable.

On the other side, the decision to use rushing yards is connected with the decision to use net yards. Cunningham and Tarkenton had a ton of rushing yards...but also, because they were scramblers, they were sacked a lot. If you're going to give them credit for making something out of nothing and gaining 16 yards on a scramble, you should penalize them when they try to make something out of nothing and lose 8 yards on a sack (instead of throwing the ball away). Because rushing flat out matters -- it's silly to pretend that Dan Marino and Steve Young are equal -- I'm going to use rushing and also use net adjusted yards. I think that's a fair compromise.

With these three binary variables -- Rushing Included (or not), Sack Data Included (or not), and League Average Baseline (or 75% of league average baseline) -- we can get 8 different lists. The one I'm going to use is rushing included, sack data included and the league average baseline. From time to time next week, I'll mention where some QBs rank on other lists. And while I'll focus mostly on just one of those lists, I'm not sure there's such a thing as a "best" list. They're merely different tools, just like a hammer, a screwdriver and a saw.

Using no rushing bonus and 100% of league average is probably best for 'Best Passer Ever' lists. Using 0.75 of league average, and a rushing bonus, is probably best if you're judging career value -- like whether or not a player was a good pick. A guy who was exactly average for 15 years was a pretty good draft pick, so you need to lower the baseline there. Dan Pastorini, Trent Dilfer and Kerry Collins deserve a bit of credit when deciding if they were good draft picks, but they don't need to get much credit when we're ranking the greatest QBs ever. Using 100% of league average, no rushing bonus, and including sacks data is probably best for a unique look at the worst passers of all time. So really, you have to decide what you want to look at, and all of the eight lists serve some value in some circumstances. You just have to decide what you want to study.

I'd love to hear your thoughts. And be sure to check next week for lists of the best and worst QBs of all time.

This entry was posted on Friday, June 20th, 2008 at 8:05 am and is filed under History, Statgeekery. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.