Pro Football Reference Blog

Everyone knows that quarterback rating is flawed. Everyone has known this for a long time. But because of the drawbacks to other statistics -- touchdowns, wins, interceptions, yards -- QB rating has persevered as the most mainstream singular statistic for grading quarterbacks. PFR has used Adjusted Net Yards per Attempt; Advanced NFL Stats using Expected Points Added; Football Outsiders uses DYAR. Now, ESPN takes its turn at measuring quarterback play.

The actual formula behind quarterback rating is complicated, but it can be reduced to a simple formula. That's what Bob Carroll, Pete Palmer and John Thorn discovered in their groundbreaking book, The Hidden Game of Football. Essentially, QB rating is equivalent to yards per attempt, but with a 20-yard-bonus for each completion, an 80-yard-bonus for each touchdown, and a 100-yard-penalty for an interception. Such adjustments should seem ridiculous to every reader, which is why everyone finds quarterback rating ridiculous. By way of comparison, PFR's ANY/A formula -- in addition to including relevant data on sacks -- gives no bonus for completions, a 20-yard bonus for touchdowns and a 45-yard penalty for interceptions.

But on Thursday, ESPN released the methodology behind its new QB Rating. And last night, ESPN aired an hour-long segment at 8 PM to discuss the new formula. So how does ESPN's formula look?

There's some good and some bad, which means it has exceeded my expectation. As Jason Lisk said, ESPN will promote it ad nauseum but it should have value. It's not perfect, but it's almost certainly better than the traditional passer rating and possibly the best single statistic out there. Here's my take.

ESPN has essentially limitless resources, which means it can: (1) hire some very good thinkers; (2) hire some very good computer programmers; (3) gain access to thousands of hours of high quality film; (4) hire lots of people to log key data for each play; and (5) discuss ways to tweak the formulas with current and former NFL QBs. On the surface, ESPN would be in perfect position to really knock this one out of the park. They claim to have used these abundant resources to develop this fantastic formula. It improves on QB rating by including data on rushing, sacks, fumbles, drops, and by separating yards gained through the air from yards gained after the catch. The formula also gives more weight to third down- and other high leverage-plays, especially later in close games.

Unfortunately, ESPN is keeping its formula a secret from the public. There are lots of legitimate reasons for doing this, but that decision makes it impossible to fully criticize the ESPN QBR. It's possible that there are serious bugs in the formula, but we have no way of knowing or discovering them. It's hard to get excited about a rating that says "Matt Hasselbeck is better than Sam Bradford because Hasselbeck has a 42.4 rating and Bradford a 41.0." Well, why is Hasselbeck's rating because than Bradford's? Because ESPN says so, of course. That's just not very convincing. For all the flaws in traditional passer rating or any of the formulas I've come up with over the years, you at least know what the flaws are. You can recreate the rankings because you have access to the formula. You can catch errors. You understand why rankings appear the way they do. And you can catch a simple programming mistake that throws off the computer ratings because of human error.

Still, I think the formula will have some utility. ESPN is building on the great work Brian Burke has done on expected points, and that makes sense. The QBR seems lot like what Football Outsiders has done with their method of ranking quarterbacks. I don't think ESPN is just being ESPN here; I think the powers that be have actually spent a bit of time kicking the tires and deciding what works. But too much is left to having faith in ESPN. For example:

The ESPN video tracking has been useful in helping to separate credit in plays like these. We track over-throws, under-throws, dropped passes, defended passes and yards after the catch. The big part was taking this information and analyzing how much of it was related to the QB, the receivers and the blockers. Not surprisingly, pass protection is related mostly to the QB and the offensive line, but yards after the catch is more about what the receiver does. Statistical analysis was able to show this and we divided credit based on those things.

My initial response is: Maybe. I'm certainly not satisfied by ESPN's mandate that YAC belongs mostly to the receiver. For example, what is worth more points in this system: an 8-yard pass to a WR that goes for 15 yards or a screen pass to the WR that goes for 30 yards? Who knows. On a screen pass, we know the QB doesn't receive credit for all the yards gained, but is the QB given credit for zero yards, some percentage of total yards gained, or the amount of yards the average receiver gains on a screen pass? How many yards does a quarterback get for a perfect pass that's dropped? Again, we're in the dark on what ESPN is doing, which makes it difficult to put complete faith in this system.

When I go to a restaurant, I don't need the chef to give me her recipe for me to decide whether or not I like my entree. On the other hand, just because my soda tastes sweet doesn't mean it has sugar. There might be an artificial sweetener in it, instead. If we don't know the formula, how can we understand its flaws? How can we say that it makes sense?

The amount of faith you put in this formula comes down to how much faith you have in ESPN. They're saying all the right things, but the devil is in the details. And we have little to no idea how the details work. I've generally found that when people hide things, they do so for a good reason. I suspect if ESPN released the formula, we would all scratch our heads at some of the ways ESPN calculates its QBR.

Aaron Schatz' biggest complain is the failure to adjust for strength of schedule. My biggest complaint is that this looks to be very retrodictive or explantory statistic but will be used in a predictive manner.

The formula will give a lot of weight to clutch plays; that is helpful to explain what happened. That's why ESPN notes that from '08 to '10, quarterbacks with the higher QBR won 85.7% of games. I'm sure if ESPN threw in "QB kneels at the end of the 4th quarter" in there, and gave it a large enough weight, they could get that number close to 100%. But that just makes it a stat that correlates with winning; it says nothing about it causing winning or how it will help you predict what will happen in the future. While explanatory stats are useful, I like predictive stats a bit more.

No one has ever shown that if QB A plays like an 80 normally, but has played like a 90 late in close games, that QB A is more likely to play like a 90 than an 80 the next time he's playing late in a close game. It hasn't been proven that average quarterbacks who played above average on third downs play above average on future third downs. These high-leverage situations will have a huge impact on the ESPN QBR, but they won't have any value in showing us which quarterbacks are the best. ESPN acknowledged that Matt Ryan looked fantastic in ESPN QBR -- much better than in traditional statistics -- and patted itself on the back for making the QB of the team with the best record in the NFC look like a top-three QB in the league. But guess what: Ryan wasn't that good last year. He was lucky in a lot of ways. And then he played terribly in the Falcons' only playoff game. ESPN's QBR wouldn't have predicted that, because ESPN's QBR was in love with Matty Ice's clutch play.

It's this predictive/explanatory dichotomy that causes Brian Burke to use both Expected Points Added and Win Probability Added to grade QBs. WPA is the explanatory statistic, and Ryan ranked 1st in WPA in 2010. But Ryan ranked 7th in EPA last year, and ranked behind Roethlisberger and Vick in EPA/game. He ranked below league average in Net Yards per Attempt, PFR's preferred predictive stat (whereas ANY/A is more of an explanatory stat). A lower ranking helps to explain why Ryan bombed in the playoffs, and also means I don't expect the Falcons to come close to matching the success they had last year.

I like all the factors that go into the ESPN QBR, as long as we understand that it's explanatory, not predictive. But I'm still disappointed that we have to go on blind faith. On the bright side, ESPN's QBR tastes sweet, but that's as far as we can go. I can't tell you if it's full of natural sugar or something much different.