SITE NEWS: We are moving all of our site and company news into a single blog for Sports-Reference.com. We'll tag all PFR content, so you can quickly and easily find the content you want.

Also, our existing PFR blog rss feed will be redirected to the new site's feed.

Pro-Football-Reference.com » Sports Reference

For more from Chase and Jason, check out their work at Football Perspective and The Big Lead.

Greatest QB of All-Time, Version III (Methodology)

Posted by Chase Stuart on August 10, 2009

In 2006, I devised a system to rank every quarterback in NFL history. Not surprisingly, I found that post hopelessly out of date and imprecise after just two years. I created a new formula in June 2008 that was a big improvement, but still left a bit to be desired. I have some controversial thoughts that I'm considering implementing to improve the formula, but I'm not ready to make those changes right now and the season is fast approaching. On the other hand, there are some relatively noncontroversial tweaks I can make to the '08 system that would improve the results, and there is no reason to wait a full year to make those changes. I like the idea of updating the series every two years (methodology, worst QBs ever, best QBs ever, playoff analysis, best overall QBs ever), so I won't be doing a full update this year. Just a methodology discussion today and a look at the best QBs ever (by career) tomorrow. On Wednesday, Doug is going to break out some exciting new data, and I'll show you how those data affect the top QBs on Thursday. Next summer I'll have a full update, but we'll visit some of the issues relating to grading QBs in the coming months (and we'll need feedback from you guys).

For now, I've made three key updates to the formula that are clear improvements to the '08 version. For those that don't remember, here's a quick summary of that method: We begin by calculating adjusted net yards, which is done by starting with passing yards, adding a 10-yard bonus for all passing touchdowns, subtracting 45 yards for all interceptions, and subtracting out the number of sack yards lost. That number (ANY) is divided by adjusted attempts, calculated as pass attempts plus sacks. Then we compare each QB to the league average (excluding the QB in question) to see how many ANY/A each player was above or below league average. That difference is then multiplied by each QB's adjusted attempts to determine how many adjusted net yards over average he added. Finally, to give credit to rushing quarterbacks (but not too much credit), the last step in the old formula was to add adjusted rushing yards (10*rushing touchdowns plus rushing yards) over 4.0 yards per carry; so 400 yards (and zero touchdowns) on 50 carries would be worth +200.

Here are the three changes I'm making:

  • 1) Increasing the value of a touchdown from 10 to 20 yards. This was thoroughly derived last October. In addition to being more precise from a theoretical standpoint, it also conforms to common perception better than the smaller bonus. From now on, all passing touchdowns are worth 20 yards.
  • I have not done any rigorous analysis on the value of an interception, but it's on the to-do list. For now, I'm sticking with the 45-yard penalty as derived by the writers of The Hidden Game of Football.

  • 2) Combining rushing and passing touchdowns. While a passer rating would not combine the two, our goal is to measure quarterbacks as players. There's no reason to separate rushing and passing touchdowns from a value standpoint, so all passing and rushing touchdowns are worth 20 yards and are calculated in the numerator of adjusted yards per attempt. To be consistent (a TD pass is worth 20 additional yards to the length of the pass and counts as an attempt), I've added TD runs to the denominator. The big reason for this combination was that a QB with 40 attempts for 60 yards and four scores would get zero rushing bonus and zero credit for four rushing scores. That was Just Plain Wrong. I understand this may ruffle some feathers -- who wants to say QB X was better than QB Y because he ran for more TDs -- but I think from a theoretical standpoint, it's the correct way to do things.
  • 3) Fumble data. The Hidden Game of Football values a fumble at -50 yards, making them slightly more costly than interceptions. We don't have "fumbles lost" data at PFR but we do have "fumbles" and "fumbles recovered" data, and the difference between those two could be characterized as net fumbles. However, just because a quarterback fumbles and does not recover the ball does not mean his team loses possession (and remember, he is already penalized for the sack and the sack yards lost). I wasn't sure what percentage of the time a QB would fumble, not recover, and his own team would recover, so I arbitrarily guessed at 30%. If someone has some actual evidence on answering this question, please post it in the comments so I can use a more precise ratio. Until then, I'll stick with the assumption that the other team would recover 70% of all quarterback net fumbles, and therefore the value of "net fumbles" is -35 yards in the new formula.
  • Note: Doug, Jason and I have spent a lot of time debating whether or not to include a completion bonus. We've thought through the issue and only concluded that we need some more time to think. People have suggested including one, but at this point, none of us are convinced that it is appropriate. One day soon, we'll devote a whole post or two to explaining this. As always, there is a trade-off between getting something perfect and getting something published, and I wanted to get this update in before the season started.

If you're still with me, here is the new, updated formula:

[(PYD + 20*(PTD + RTD) - 45*INT - SKYDLST - 35*(FUM-FumRec)) / (ATT + SK +RTD)]

That calculates each QB's value per play; we then compare that number to the league average, and multiply the difference by his (ATT + SK + RTD) to get each QB's value added over average. Finally, we will still add (if positive) all QB rushing yards over 4.0 yards per carry. I'm going to call this metric converted yards per play (CYP); I want to keep around ANY/A for when we need to use that formula. ANY/A ((PYD + PTD*20 - INT*45 - SKYDLST)/(ATT + SK)) still has some value to it, but it just won't be my primary measure to grade historical performances by quarterbacks. Finally, for pre-1969 quarterbacks, sack data are not included because we have only partial individual sack data for those players (although we have complete sack data on the team level -- creating estimated sacked numbers is on the to-do list).

At this point it's probably good to remind you that this formula is designed to be retrodictive and not predictive. The goal is to see which QBs were the most valuable in which seasons, not which QBs project to be the best the next season. Rushing touchdowns, interceptions and fumbles recovered are very inconsistent from season to season; passing touchdowns aren't much easier to predict. But retrodictive formulas have their place when ranking past performances.

Let's run through Aaron Rodgers' 2008 season as an example of how to use the new formula.

Att	Yds	TD	Int	Sk	SkYds	Rsh	RshYd	RshTD	Fum	FR
536	4038	28	13	34	231	56	207	4	10	5

Let's start with the numerator to determine his gross converted yards. We start with his 4038 passing yards, add 20 yards for each of his 28 passing scores and four rushing scores, subtract 45 yards for each of his 13 interceptions, remove the 231 yards he lost from sacks, and finally subtract 35 yards because of his five net fumbles. That gives him 3687 converted yards on 574 plays (536 attempts, 34 sacks, 4 rushing scores), or 6.423 converted yards per play. The league average was 5.31 including Rodgers and 5.277 for all other QBs, giving him 658 converted yards of value over the average quarterback for the season. For all QBs, I always exclude their performances from the league average when calculating their final grade. Rodgers' +658 score was good enough to make him the 7th most valuable QB in the NFL. Here's the full list of QBs with at least 100 pass attempts last year:

att     pyd    ptd/int	 sk/yd	   rsh-yd-td	fum/fr	CY/A	Value	Team	Name
635	5069	34/17	 13/92	   22- -1-0	 6/2	7.33	1358	NOR	Drew Brees
478	4009	34/11	 25/151	   31- 84-0	 8/2	7.62	1194	SDG	Philip Rivers
476	3653	19/ 7	 24/121	   30- 62-1	 3/1	7.08	 910	MIA	Chad Pennington
555	4002	27/12	 14/ 86	   20- 21-1	 1/0	6.84	 901	IND	Peyton Manning
598	4583	30/14	 26/182	   18- -2-0	11/2	6.50	 767	ARI	Kurt Warner
616	4526	25/18	 11/ 69	   57-200-2	 5/0	6.38	 694	DEN	Jay Cutler
536	4038	28/13	 34/231	   56-207-4	10/5	6.42	 658	GNB	Aaron Rodgers
434	3440	16/11	 17/104	   55-104-1	 6/2	6.73	 656	ATL	Matt Ryan
414	3288	15/12	 20/130	   20- 21-2	 5/1	6.46	 514	CAR	Jake Delhomme
380	3043	15/10	 23/149	   31- 68-2	10/4	6.36	 432	HOU	Matt Schaub
376	2712	12/ 6	 23/100	   35-148-1	 7/4	6.24	 388	TAM	Jeff Garcia
450	3448	26/14	 20/123	   28- 41-0	13/3	6.10	 377	DAL	Tony Romo
571	3916	23/11	 23/149	   39-147-2	 7/0	5.92	 372	PHI	Donovan McNabb
516	3693	21/11	 47/219	   73-270-2	 7/2	5.78	 270	NWE	Matt Cassel
415	2676	12/ 7	  8/ 60	   25- 49-0	 5/2	5.76	 193	TEN	Kerry Collins
479	3238	21/10	 27/174	   20- 10-1	 5/0	5.68	 190	NYG	Eli Manning
149	1056	 9/ 2	 14/ 97	   26-145-0	 5/1	5.58 	  84	MIN	Tarvaris Jackson
242	1532	11/ 3	 14/ 76	   16- 78-0	 4/0	5.47	  55	SEA	Seneca Wallace
288	2046	13/ 8	 23/148	   24-115-2	 8/3	5.31	  19	SFO	Shaun Hill
374	2699	11/10	 23/143	   36-117-3	 9/1	5.27	- 20	BUF	Trent Edwards
420	2608	18/12	 26/162	   62-386-3	 6/3	4.95	- 31	KAN	Tyler Thigpen
506	3245	13/ 6	 38/266	   47-258-1	 7/1	5.10	- 51	WAS	Jason Campbell
174	1431	 6/10	  9/ 58	   11- 37-0	 4/0	4.93	- 70	HOU	Sage Rosenfels
535	3620	15/13	 42/288	   73-322-2	 7/3	5.09	-104	JAX	David Garrard
465	2972	18/12	 27/160	   24- 49-3	 6/1	5.08	-117	CHI	Kyle Orton
255	1616	 8/ 8	 14/ 95	    7- 29-0	 3/2	4.78	-145	DET	Dan Orlovsky
129	 731	 3/ 4	 11/ 67	    6- 38-0	 2/1	3.64	-223	CIN	Carson Palmer
120	 758	 5/ 5	 15/ 89	    6- 34-0	 3/1	3.51	-235	DET	Jon Kitna
428	2971	14/12	 32/276	   52-180-2	11/3	4.75	-267	BAL	Joe Flacco
184	1073	 5/ 7	  9/ 69	    5- -1-0	 1/0	3.91	-275	TAM	Brian Griese
301	2157	12/15	 29/164	   19-  7-1	 4/1	4.45	-292	MIN	Gus Frerotte
368	2423	13/ 8	 31/210	   17-127-1	12/2	4.46	-292	OAK	JaMarcus Russell
115	 786	 4/ 6	 14/ 95	   12- 25-1	 5/1	2.93	-312	DET	Daunte Culpepper
469	3301	17/15	 46/284	   34-100-2	14/3	4.52	-423	PIT	Ben Roethlisberger
104	 584	 2/ 5	 15/119	   12- 70-2	 8/3	1.20	-479	BUF	J.P. Losman
209	1216	 5/10	 19/119	   11- 69-0	 1/0	3.12	-481	SEA	Matt Hasselbeck
283	1615	 9/ 8	 14/ 87	   25- 55-0	 9/2	3.71	-484	CLE	Derek Anderson
522	3472	22/22	 30/213	   21- 43-1	10/2	4.43	-506	NYJ	Brett Favre
220	1678	 8/11	 32/197	   30-145-0	11/1	3.16	-526	SFO	J.T. O'Sullivan
440	2720	11/13	 38/263	   14- 41-0	 5/1	4.08	-605	STL	Marc Bulger
372	1905	 8/ 9	 38/193	   60-304-2	11/8	3.40	-742	CIN	Ryan Fitzpatrick

Let's just go ahead and get this out of the way. Someone please cut-and-paste this into the comments:

Ben Roethlisberger won the Super Bowl last year, so this formula is obviously garbage. I'm pretty sure Ben would rather have TWO Super Bowl rings than be at the top of this list, and by the way, none of the top three guys have ever even MADE a Super Bowl.

Moving on, it's it's important to remember that this is just a measure of each team's passing game, assigned to the quarterback on the field for those plays. Obviously the quality of the offensive line, the ability of the receivers, the versatility of the tight ends and running backs, the philosophy of the coaches, the strength of the schedule, and good old randomness have a significant impact on the above numbers. The reason for these posts is to accurately measure quarterback statistics, and nothing else. Once we have strong measures of QB performance, we can then judge QBs based on how much of their success (or lack thereof) we want to assign to the QB and how much to other people/factors.

Check back tomorrow for career regular season rankings, career rankings (including playoffs), and year-by-year leaders.

This entry was posted on Monday, August 10th, 2009 at 6:52 am and is filed under Best/Worst Ever. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.