Posted by Doug on July 24, 2006
Here is Part I. In that post, I promised a set of quarterback rankings, and I promised that they'd be meaningless. I'm going to deliver on both promises.
As I talk my way through this, I'll continue with the college football analogy. A Quarterback is analogous to a team. A comparison of the stats of two quarterbacks on the same team in the same or consecutive years is analagous to a game. So I'll say, for example, that Donovan McNabb beat Mike McMahon in 2005. He beat him by 2.2 adjusted yards per pass, which would roughly be the equivalent of a 20-point win in a college football game.
Now that the stage is set, let's talk about Bill Kenney. Kenney has a record of 10-1. Sounds good, right? It even works out nicely with the college football analogy. He's the West Virginia Mountaineers. But here is where the analogy breaks down. The Mountaineers compiled their record against 11 different teams of varying quality levels. Bill Kenney did not. He was 9-0 against Todd Blackledge, 1-0 against Steve DeBerg, and 0-1 against Steve Fuller.
But it gets worse. Blackledge played no other games. Kenney is the only quarterback that directly compares to him. So what does the computer think of Blackledge? Well, it has a good idea that he's quite a bit worse than Kenney. But will it conclude that Kenney is good and Blackledge is average? Or will it conclude that Kenney is average and Blackledge is bad? That hinges on Fuller and Deberg. Fuller only played Kenney and Mike Livingston, and Livingston only played Fuller.
So what we have here is a conference --- let's call it the Big Chief Conference --- consisting of Kenney, Fuller, Livingston, and Blackledge. They've played a bunch of intraconference games, so the computer has some confidence in its ability to rank order those teams. But where do they rank nationally? That depends on how strong the conference is.
They played exactly one out of conference game. It was against Steve Deberg. Fortunately for the Big Chief Conference, Bill Kenney represented well in that game, beating Deberg by a fair margin. Deberg was pretty darn good, compiling a 12-6 record against what may have been the toughest schedule in the nation, including games against John Elway, Dan Marino, Joe Montana, Steve Young, Vinny Testaverde, and Dave Krieg.
Because of that, Kenney ranks #3 and Blackledge ranks #74 out of 127 quarterbacks who played at least five games. In 1988, Deberg threw for 2935 yards with 16 touchdowns and 16 interceptions. Had he thrown instead for 3400 yards and only 13 interceptions, then he would have beaten Kenney, and Kenney would have be ranked #60 and Blackledge #124.
So for Bill Kenney, a few hundred yards and a handful of interceptions in one Steve Deberg season are the difference between #3 and #60. That's a problem. That's not the only problem with this method, but it's the most serious one. The quarterbacks simply aren't sufficiently well connected.
If the NFL had a rule stating that no quarterback is allowed to appear in more than eight games in a season, then I think this method would produce a meaningful set of rankings. But there is no such rule, and the rankings produced by this system are worthless. Only because I promised to, I will post them here. Tomorrow I'll wrap this up by looking at some of the more interesting rankings, and I'll post a chart that details every quarterback's full "schedule." For now, here are the rankings.
Fine print: only seasons since 1978 were included, and only seasons with more than 150 pass attempts. Comparisons were based on adjusted yards per attempt, which was further adjusted for the quarterback's age at the time.