Ranking the QBs — Methodology Discussion
Posted by Chase Stuart on Friday, June 20, 2008
A couple of years ago, I ran a post here titled "The Best QB of All Time." It was a pretty popular and controversial post, so that's reason enough to run it again every couple of years. All next week, I'll be running posts on the best and worst QBs of all time. Fortunately for me, I've got a whole lot more data (and brains) at my disposal now than I did two years ago. The biggest change, by far, is the addition of every QB in NFL history, as opposed to just QBs since 1970.
For those that don't remember, here's the basic summary. We start with each QB's adjusted yards per attempt average. AY/A is simply passing yards, adjusted by adding 10 yards for every passing TD and subtracting 45 yards for every interception. It's a neat, relatively simple stat that does a nice job of approximating how good a QB was in a given season. Obviously this isn't going to be perfect, as QBs on bad teams will be undervalued, while those that play with great WRs and offensive lines will be overvalued. For now, I'm just going to have to throw up my hands and so, 'oh well.' Additionally, of course, there's no adjustment whatsoever for playoff performance.
To rank QBs in a single season, I simply took each QB's AY/A ratio, subtracted the league average AY/A ratio, and then multiplied that difference by the QB's number of attempts. This does a nice job of balancing the trade-off between compilers and guys that excelled for a short period of time.
For example...
the league average QB passed for 5.85 AY/A in 2007. So a QB that averaged 8.00 AY/A last year would have been terrific; but if he had only 300 attempts last year, he'd be ranked equally with a guy that had 500 attempts and a 7.14 AY/A ratio. If we used raw numbers, the guy with 500 attempts would almost always win; if we used averages, the guy with an 8.00 AY/A metric would always win. I think this strikes a nice and acceptable balance.
So what changes have I made? For starters, I've removed each QB's stats from the league average, when comparing the QB to the league average. So more precisely, each QB's stats is compared to the stats of every other QB, and not the league average. This has little practical effect for recent QBs, but it does bump up some great old QB seasons, which I think is always a good thing.
Also, I chose to handle the AFL and NFL as separate leagues, so Joe Namath is compared to other AFL QBs and Johnny Unitas is compared to other NFL QBs. This seemed fair to me. AAFC stats, much to the dismay of Otto Graham's heirs, have been left out. Perhaps just as importantly, I've made a small adjustment for the non-16 game season leagues (everything before 1978, 1982 and 1987). It's easier to excel for 12 games (i.e., Otto Graham's 1953 season) than it is for 16 games, so I thought it would be unfair to pro-rate Graham's seasons to 16 games. On the other hand, it's not his fault that the league only played 12 games back then, and it's certainly easier to accumulate a ton of yards above league average when you play four more games. So I split the baby, and pro-rated all non-16 game seasons to the difference between 16 games and the number of games in the season. Graham's numbers are pro-rated to a 14 game season, Dan Fouts' great 1982 season is pro-rated as if it was a 12.5 game season, and so on. This just felt right to me, even if it's sort of a cop out.
I then took a page out of Doug's Approximate Value book and weighted each season in a QB's career. So when calculating a quarterback's career value, I gave him 100% of his rating in his best year, plus 95% of his rating in his second best season, plus 90% of his rating in his third best year, and so on. This strikes a nice balance between rewarding the guys who played really well for a long time, but without killing guys with really bad rookie years or seasons late in their career.
With all that out of the way, there are three variables I chose to focus on when ranking the QBs. I think you could make a decent argument for including or not including any of these stats. Let me go into a bit of detail here.
First, we could choose to include rushing yards or not. I've never really seen a study that properly incorporated rushing data into a QB's overall statistical bio. What I did in my History of the Black QB: Part II post was use "Rushing Yards Over 4.0." So a QB that runs 100 times for 600 yards will get a bonus of 200 yards added to his score. I like that system, and I'm going to also add 10 yards for every rushing TD, too. After all, there's no reason why a QB shouldn't be rewarded when he runs for a score. Now, a hypothetical QB with 500 attempts and 7.14 AY/A (and say, 50 rushes for 75 yards) would be equivalent to a QB with 450 attempts, 100 rushes, 500 rushing yards, 5 rushing TDs, and 6.95 adjusted yards per pass. Both would come in at 645 yards above average. Once again, the 4.0 number is pretty arbitrary, but I feel okay about it. I want to reward good runners, but I also don't want to penalize QBs that have a lot of kneel downs, either.
Second, we could choose to lower the baseline from "league average" to three-quarters of league average. The main reason to do this is you can get some pretty quirky results the other way. For example, is Drew Bledsoe not one of the top 100 QBs of all time? Is Trent Dilfer one of the fifteen worst QBs ever? (Well, maybe, maybe not. We'll table that until next week). The bottom line is, QBs that are exactly average have some value. They just do. If the Minnesota Vikings had an average QB last year, they would have made the playoffs. On the other hand, there are good reasons for preferring to keep the league average baseline -- more on this later.
Third, we could choose to include sacks allowed and sack yards lost data into our system. I didn't have this data last time around, and while it's controversial, I think it should be used. After all, my current system would penalize a QB for an incompletion, but not for a 10-yard sack. QBs that throw the ball away and avoid sacks are actually pretty valuable. While sacks certainly aren't solely the fault of the QB, I think including the data do more good than bad. So we can use adjusted net yards per attempt, instead of regular old AY/A (net means we subtract sack yards lost from our numerator, and add sacks to our denominator).
So far, I've been talking mostly in the abstract. Let me give you some examples.
Including Rushing Stats: Not surprisingly, including rushing stats helps Randall Cunningham, Kordell Stewart and Mike Vick the most. It also gives a big boost to Bobby Douglass, Vince Evans, Archie Manning, Steve Young, Steve McNair, Fran Tarkenton, Donovan McNabb, and Jake Plummer.
No QB really drops once you include rushing stats. That's because I don't assign any negative points for being a bad rusher. I'm pretty happy with how this turned out. Including rushing stats hurts Marino and Montana, as they may get jumped by some running QBs. But mostly, excluding running stats doesn't help anyone, and really undervalues some good runners.
1.00 or 0.75: Lowering the baseline from league average to three-fourths of league average is obviously going to help compilers, but maybe not as much as you'd expect. Using the 100%/95%/90%/85%/etc. formula, the value of compilers is automatically lowered. Vinny Testaverde is helped by lowering the baseline, but it only bumps him from the 50s or 60s to the 30s. Jon Kitna is the QB that's helped the most by using the league average baseline. Trent Dilfer, Jake Plummer, and Drew Bledsoe are, too. It doesn't necessarily helps compilers, but it helps guys who had seasons with a lot of attempts where they were just under the league average.
Brett Favre only ranks incredibly high on these lists if you use 75% of league average, and not league average. Warren Moon and John Elway also get big bumps by dropping the baseline. Do we think of them as compilers? I'm not so sure.
On the other hand, some old timers are helped a bunch by using the league average baseline. Otto Graham shoots up the list. Sid Luckman, Sammy Baugh and Norm Van Brocklin do, too. Guys that were great for short periods -- Steve Young, Bart Starr, Roger Staubach, Kurt Warner, Bert Jones and Len Dawson -- all receive nice boosts by using the league average baseline. On the whole, I like rewarding the Otto Grahams of the world and not the Jon Kitnas of the world. I'm sure Favre fans would prefer we use three-fourths of league average, but I slightly prefer using league average. Additionally, Favre gets only a small penalty for his awful 2005 season because of the weighted average scale, so his fans can't complain too much.
Including Sacks -- net data: Who moves up if we switch from adjusted yards per attempt to adjusted net yards per attempt? Well, Dan Marino gets a huge boost. So do Dan Fouts and Jim Hart. Jim Everett and Favre didn't get sacked often, so they look better, too. Deberg, Aikman, and Montana get sizable bumps, as well.
Conversely, some guys got sacked a lot. Steve Young and Randall Cunningham fall a bit once you penalize for sacks. So do Jeff George, Ken O'Brien, Phil Simms, Mike Vick and Fran Tarkenton. It's clear that a sack isn't entirely the fault of the QB, but guys like Marino should get credit for avoiding the sack. Young, especially since he played in the WCO, really should be penalized for having quite a few more sacks than I remember. And I think we're all happy (Jason Whitlock doesn't read this blog) that Jeff George ranks as the 93rd best QB of all time, and not the 60th best, as he would if sack data were excluded.
Final thoughts
There is some overlap here, of course. Using the weighted scale tends to actually help some compilers, because a lot of them had some really bad years. Therefore, I feel better about using league average as the baseline, as compilers would probably get too much credit any other way. And because of the way using 1.00 of league average helps the old time QBs, and because accounting for era is really, really important, I think using league average is slightly preferable.
On the other side, the decision to use rushing yards is connected with the decision to use net yards. Cunningham and Tarkenton had a ton of rushing yards...but also, because they were scramblers, they were sacked a lot. If you're going to give them credit for making something out of nothing and gaining 16 yards on a scramble, you should penalize them when they try to make something out of nothing and lose 8 yards on a sack (instead of throwing the ball away). Because rushing flat out matters -- it's silly to pretend that Dan Marino and Steve Young are equal -- I'm going to use rushing and also use net adjusted yards. I think that's a fair compromise.
With these three binary variables -- Rushing Included (or not), Sack Data Included (or not), and League Average Baseline (or 75% of league average baseline) -- we can get 8 different lists. The one I'm going to use is rushing included, sack data included and the league average baseline. From time to time next week, I'll mention where some QBs rank on other lists. And while I'll focus mostly on just one of those lists, I'm not sure there's such a thing as a "best" list. They're merely different tools, just like a hammer, a screwdriver and a saw.
Using no rushing bonus and 100% of league average is probably best for 'Best Passer Ever' lists. Using 0.75 of league average, and a rushing bonus, is probably best if you're judging career value -- like whether or not a player was a good pick. A guy who was exactly average for 15 years was a pretty good draft pick, so you need to lower the baseline there. Dan Pastorini, Trent Dilfer and Kerry Collins deserve a bit of credit when deciding if they were good draft picks, but they don't need to get much credit when we're ranking the greatest QBs ever. Using 100% of league average, no rushing bonus, and including sacks data is probably best for a unique look at the worst passers of all time. So really, you have to decide what you want to look at, and all of the eight lists serve some value in some circumstances. You just have to decide what you want to study.
I'd love to hear your thoughts. And be sure to check next week for lists of the best and worst QBs of all time.
This entry was posted on Friday, June 20th, 2008 at 8:05 AM and filed under History, Statgeekery. Follow comments here with the RSS 2.0 feed. Skip to the end and leave a response. Trackbacks are closed.
Did I miss something, or is are fumbles not included? It seems you've gone to great pains to include every stat a QB can be remotely accountable for, except for this very important case.
Posted on 20-Jun-08 at 10:04 am | PermalinkRe: including/not including fumbles: 1. Prob. difficult to get fumble data from older QB's. 2. There is a big difference obviously, between a fumble and a fumble LOST. 3. There is a big difference WHERE the fumble takes place. If the QB is "strip-sacked", but falls on his own fumble, the net result is nil. (He was being sacked--this is already accounted for in the formula.) If he fumbles the snap (either regular or shotgun), even if he recovers, that is a negative play because of the loss of down (plus some lost yardage). If he LOSES a fumble after a 12 yd. run because he didn't slide after picking up the first down, then he has hurt his team even more. I can see why Chase would not include this as teasing out this data would be EXTREMELY monotonous, time consuming, etc.--not to mention that it prob. wouldn't do more than move a couple of guys up/down 1-2 spots. (unlike the rushing/sack totals that he is adding in)
Posted on 20-Jun-08 at 11:32 am | PermalinkI'm not a statistician (nor a grammar expert), but i have one suggestion.
Posted on 20-Jun-08 at 12:13 pm | PermalinkHow big of an operation would i be to include the career values of the QB's teammates? I just read your 2006 version of QB-rankings and wondered what Elway was doing at #10. But he played on some bad teams... BAD! So I'm urging, in order for this to be perfect, to try to find some nifty way to weigh in the importance of teammate production. It might be an uncompleteable task, but if it's possible, I think it would be more "right".
Also, Dan Reeves held Elway back - but thats impossible to account for.
Hey Jason W,
No, I didn't include fumbles. Not for any amazing reason, except I didn't want to go crazy with the formula just yet. Things get pretty complicated, but the simpler the better here, I think. I'm also not really sure exactly how to weigh fumbles, and I'm not convinced that fumbles are as attributable to a QB as yards, TDs and INTs are. In short, I'm just not extremely confident and comfortable with the fumble data yet. At some point, though, for sure.
Posted on 20-Jun-08 at 1:05 pm | PermalinkHey Danish,
The value of a QB's teammates is a tricky one. I'm sure it will be discussed quite a bit in the comments next week
Posted on 20-Jun-08 at 1:09 pm | PermalinkIMHO, I think fumbles are at least as attributable to the QB as sacks -- yeah, sometimes they're caused by a blindside hit and the QB never had a chance, but sometimes they come when he's running or gets hit clean from the front, etc. There are definitely QBs who fumble more than others (see Warren Moon, Dave Krieg, Daunte Culpepper), so I don't know how it couldn't be considered attributable to a QB.
As for trying to break them down by type, where they occur, whether the QB falls on it right away, etc., seems to me you could use situational mechanics like that to account for just about anything. An interception 40 yards downfield on 4th down or on a hail mary? Meaningless. A TD on a tipped ball? Not much to do with a QB's skill. If you try to take context into everything, it'll drive you crazy. Just assign a value to it and put it in.
Of course, if you don't have that data for all the older QBs, that's another issue entirely (though it seems to be there for pretty much every QB dating back to the merger).
Posted on 20-Jun-08 at 2:39 pm | PermalinkSolid points, Jason. At some point down the road, we'll get those incorporated.
Posted on 20-Jun-08 at 2:42 pm | PermalinkIf you don't factor in Playoff games,then you will likely end up with results like Doug did-Mark Brunell and Troy Aikman ending up with a similar VALUE, and yet as the games got tougher and the Championships closer, one guy was great a MAJORITY of the time while the other guy was mediocre to bad in his BIGGEST GAMES. Playing GREAT a MAJORITY of the time in the Playoffs (especially in the BIGGEST GAMES-Conf. Champ. and S.B. games) are what separate the GREAT QB's from the rest of the pack.
Posted on 20-Jun-08 at 3:47 pm | PermalinkI'll shatter the suspense, Denny. Aikman and Brunell are indeed ranked close to one another, although Aikman does come out ahead.
Posted on 20-Jun-08 at 4:55 pm | PermalinkChase, would you mind reviewing the rationale for +10 yards per TD and -45 yards per interception? I know the Hidden Game of Football makes a case that an interception is equivalent to a loss of 45 yards... and I remember discussing the +10 yards with you at FBG but can't find the thread. If I recall correctly, unlike the -45 yards for interceptions, the +10 yards for TDs was not based on analysis but rather on curve fitting and/or "feels right" kind of thinking. If that is still the case, then I'm not sold on that part... but perhaps I'm just not remembering.
Related point. Whatever the values, isn't it possible that the "right" values for TDs and interceptions differ across the years? Especially given the radical differences in the passing game across eras...
Aside from that, let me say I really enjoy this work. That said, I think you should find a way to work in postseason value in your next iteration. There is simply no way to accurately rank a QB like Montana without considering postseason. And he obviously isn't the only one.
Posted on 20-Jun-08 at 11:34 pm | PermalinkTake it from someone who knows (click my name), Tom Brady had better be ranked higher than #54 all-time or you'll have scores of very angry Patriots fans threatening you bodily harm. And the worst part is that I'm a huge Pats fan as well -- they just don't care! Seriously, though, I think his monster season last year probably did you a huge favor and bumped him up quite a few spots. Lucky SOB.
Posted on 20-Jun-08 at 11:40 pm | PermalinkMy head nearly exploded. Since I thoroughly enjoyed the original Best/Worst QB posts, I very much look forward to all this...lord knows I'll probably understand better what the hell you're talking about!
Posted on 21-Jun-08 at 12:01 am | PermalinkHmm... I'm sure the numbers will come out good, but something about the system bugs me. It seems like those who compile a bunch of decent seasons are being penalized for the wrong reason. They're good BECAUSE they had a bunch of seasons, so giving them less than full value for season #17 seems wrong. If one wants to penalize them, it seems to me that it should be for the lack of spectacular seasons, not the fact that they had a long career. I'll get back to this momentarily.
And regarding the average vs magic number * average... I like the idea of average simply because it doesn't have a magic number. But on the other hand, it seems an awfully high bar to set for an entire career. One would expect a career to be kind of parabolic, rising to a peak and then tailing off. And using the average kind of penalizes players who were thrown to the wolves in their rookie year (Young, Manning) and helps players who had a couple years of seasoning first (Rivers, Romo).
So here's a rough idea. It feels more "right" to me, but what the hell do I know? The numbers are just WAGs
best - avg * 1.10
2ndbest - avg * 1.05
3rdbest - avg * 1.01
4thbest - avg * 0.98
5thbest - avg * 0.96
6thbest - avg * 0.95
...
I don't see why one can't expect that a "best" season is above average. That'd penalize the compilers for the lack of spectacular seasons, and it'd probably significantly increase the reward of a Marino 85 or Manning 04 season because they're so absurdly good compared to most other players "best" seasons. It'd reward the compilers for having a long productive career which is their real strength. A compiler's best year could in theory have a negative value while their other, lesser years are positive. But this makes more sense logically to me. I'm not sure WHAT it'd do to the Dilfers Leafs and Harringtons but it'd be fun.
Or if one wanted to keep an average average (ha!), one could designate the middle season as average (1.0), then increase/decrease the magic number as one moves away from the middle.
Other random thoughts:
Your system probably overrates dome quarterbacks too simply because they have to deal with wind and rain less often. But short of normalizing data stadium by stadium, game by game, I don't know what else one could do.
If one were going to do that, one could normalize by the quality of the pass defense too, but I imagine that'd be insignificant over the course of a career.
One tricksy thing about sacks and fumbles is it probably depends on a lot of factors other than the quality of the quarterback. A lousy line will get a QB killed (Carr?). A WCO would probably decrease sacks and fumbles because there's less time for them to happen. The Run and shoot would probably increase sacks and fumbles because the QB has less protection.
Oh, another possibility is increasing the reward/penalty for extreme values. If the AYA is 5, a value of 7 might be more than twice as valuable as 6, and a value of 3 might be more than twice as bad as 4. That'd also help value those with 5 spectacular seasons over somebody with 20 "decent" ones.
Posted on 21-Jun-08 at 12:15 am | PermalinkHey JWB,
Yes, the -45 yards is from the Hidden Game of Football. You can read about the 10 yard rationale here: http://www.pro-football-reference.com/blog/?p=404
And yes, post-season data at some point should be incorporated. Baby steps, though. Surprisingly, Montana ranks pretty high without a post-season adjustment. That's because he wasn't just awesome because of his playoff accomplishments.
Hey Neil,
Yes, Brady's big season definitely did me a favor. He ranks a lot higher than #54.
Rock,
It should be fun this week. I'm looking forward to it. Some interesting results.
Posted on 21-Jun-08 at 8:52 am | PermalinkInteresting thoughts, Mattie Shoes.
I'm generally with you about thinking it's unfair not to reward a player's 17th best season if it was awesome. The thing is, that just generally doesn't happen too often. And even if it does, that player is going to rank really, really high anyway, unless his best years weren't incredible. And in that case, people might not think he deserves to rank that high since he was never truly elite. It's a tricky question.
Doug uses 100/95/90 with AV, and I think it works. I think it matches perceived perception, which may or may not be a good thing. I like some of your suggestions though, although they may not be easy to implement. Interesting, nonetheless.
Posted on 21-Jun-08 at 9:09 am | PermalinkChase,
Thanks for the link to Doug's post. Unfortunately, he made the same argument in our FBG discussion, and I don't really buy it. The -45 yards per interception is derived from statistical analysis. The +10 yards per TD isn't... in fact Doug's post ends with this: "I don’t know if ten is exactly the right answer, but it feels about right." Given that the rest of your ranking is based on sound analysis, this element just doesn't seem solid to me.
Also, you didn't address my second point, though I might not have been clear. Let's focus on the -45 yards per interception. The Hidden Game of Football did a nice job of proving why that number made sense, based on their analysis of the 1997 season (IIRC). But what if the analysis of the other seasons would yield a different number? Perhaps most relevant due to rule changes, isn't it reasonable that the number could be different for seasons before 1978?
If it is reasonable that the number could vary, then it is reasonable for a perfect formula to take that into account, both for interceptions and TDs.
Though I know that would be a heck of a lot more work...
Posted on 21-Jun-08 at 11:33 am | PermalinkHey JWB,
I understand your point. It's definitely difficult putting a value of interceptions and touchdowns. You almost never have to, except when you're trying to use just one stat. Nearly all the time, we have a range of stats at our disposal. But when trying to compare every QB in NFL history across era, we have to boil their performance in each year down to one stat, calibrate that to his era, and go from there. Not easy.
I think you're probably right in that an interception hasn't always been worth -45 yards. I'd suspect interceptions are more costly now, for two reasons. One, INTs happen much less frequently, which signals a market answer that interceptions now must be avoided at all costs. And two, the reason INTs must be avoided at all costs now, is because offenses are more effective. The opportunity cost of an interception on first and 10 is much more detrimental now than before, since your offense's chances of scoring now are higher than before. Further, it puts your defense in a worse position because they have to go up against tougher offenses. But this is just a guess -- I don't really know the answer. As such, I'm hesitant to make any widespread changes.
10 feels right to me, as it does to Doug, and as I'm sure it does to some people. It feels way off to others. All I can say is, 'I dunno.' We have to use some number, and we don't want to penalize guys like Aikman who didn't throw a ton of TDs. The important part isn't the actual TD, but getting your team down to the one yard line. For what it's worth, IIRC, the Hidden Game of Football also used 10 yards for a TD.
I'll stop now and just say my formula won't be perfect. It's gotten much more complicated (and I think, precise) than it was two years ago, when many people said it was way too complicated. Making any more changes that aren't mainstream will be tough to implement, I think. Maurile has made a good argument for completion percentage being relevant, but I'm not sure how I want to include that yet, either. These lists are always a work in progress.
Posted on 21-Jun-08 at 11:45 am | PermalinkJWB, the 10 yard TD bonus comes from exactly the same place, and is based on the same kinds of computations as, the INT penalty. Namely...
1. estimate the expected point value of possession of the ball at any point on the field
2. look at the difference between a TD and a same-yardage play that isn't a TD
3. convert the units from points to yards.
Possession at the inch-line has an expected net point value of around 6. A TD has an expected net point value of around 6-and-a-half (7 minus giving the other team the ball at around the 25). So the marginal value of the TD is around half a point which roughly translates to 10 yards. That's the main idea.
You may or may not buy that methodology, but I don't see why you'd buy the 45 and not the 10.
The post Chase linked to wasn't the theoretical justification for using 10. It was intuitive explanation for why the theoretically-computed value makes sense.
On another note, your point about differing values for different seasons is definitely a good one.
Posted on 21-Jun-08 at 12:55 pm | PermalinkDoug,
Actually, I'm pretty sure I remember that the Hidden Game of Football shows that 1 point equates to roughly 12 yards. That would imply that half a point equates to 6 yards, obviously... not 10 yards.
Posted on 21-Jun-08 at 2:15 pm | PermalinkI pulled out the Hidden Game of Football to find the relevant info.
First, on page 54, they state that "point potential increases by about a point every 12.5 yards." And the next sentence says "on average it takes 15 or 16 yards to produce a point." So half a point should be worth about 6 to 8 yards, based on their analysis. At least if this were the method, I could understand the analytical basis that brings us to that value.
Second, on page 111, they advocate a 10 yard bonus for passing TDs, with no analytical basis provided. They state "We advocate a 10 yard bonus for each TD pass. It's still higher than the yardage on a lot of TD tosses, but it allows for the fact that yardage is a lot harder to get once a team gets inside the opponent's 25."
So, unlike the -45 yards for an interception, the HGF authors don't really justify the 10 point bonus with underlying analysis. Unless they do so elsewhere in the book and I didn't find it in my quick look.
Posted on 21-Jun-08 at 2:36 pm | PermalinkYou're right, JWB. One (hypothetical average) point is indeed worth about the same as 12 (hypothetical average) yards. [From here, your own one-inch line is worth -2, the opposing team's one-inch line is worth about +6, so 100/(8) is around 12.]
I don't have the book in front of me, but my guess is that they're justifying the increase from 6 to 10 by claiming that red zone yards are more valuable than typical yards.
I guess I didn't realize you were suggesting 6 instead of 10. I figured you were thinking 30 or 50 or something. Most people think 10 is too low.
Posted on 21-Jun-08 at 3:30 pm | PermalinkI just quickly thumbed through THGOF, too, and didn't find anything more specific than what JWB posted.
However, the fact remains that the analysis done in THGOF (and then replicated later by Romer with more sophisticated methods) immediately leads to a value of somewhere around 10. (NOTE: I consider 6 or 8 to be somewhere around 10)
Posted on 21-Jun-08 at 3:41 pm | PermalinkI'm with Doug; I'm hesitant to give fewer than 10 points per passing TD. Why? Because as it is, a 1-yard TD only negligibly helps a QB's AY/A. If he throws incomplete on first down, and a TD on second down, in most cases, that will lower a good QB's AY/A. It's a delicate balance.
Posted on 21-Jun-08 at 4:15 pm | PermalinkDoug/Chase,
I really didn't set out to propose a figure that is higher or lower. I just wanted it to be a figure that is justifiable based on some analytical basis. I see a basis for 6-8, but I don't see any basis for 10 other than "feels right" or some kind of fitting (e.g., Chase's last post).
As for why they propose 10 instead of 6, like I said, they don't really explain. It is stated in the context of discussing how to improve upon the NFL QB rating system, and follows their statement that giving a bonus of 80 yards per TD, the NFL's method, is far too much. They never try to reconcile it with the other analysis performed elsewhere in the book.
Posted on 21-Jun-08 at 4:25 pm | PermalinkAnd on this: "I consider 6 or 8 to be somewhere around 10."
If that is the case, implying that using 6 or 8 wouldn't change much, then why not use the more accurate number? If it would change much, then we're back to what justifies using 10 instead.
Posted on 21-Jun-08 at 4:27 pm | Permalink"what justifies using 10 instead"
The last yard is more valuable than an average yard.
Posted on 21-Jun-08 at 4:38 pm | PermalinkLOL. Two points on that.
1. Of course the last yard is more valuable. The whole subject we've been discussing is giving bonus yards for TDs on top of the passing yards accumulated by the QB. Giving 6 or 8 yards per TD is already giving a bonus for that last, more valuable yard.
2. Using this logic, should we give less value than an average yard for yards gained by the QB deep in his own end?
Posted on 21-Jun-08 at 7:54 pm | PermalinkThe yards at both ends of the field are more valuable. It's the yards in the middle of the field that are less so.
But I'm guessing there's no way to determine the "right" number short of an exhaustive play-by-play analysis of a LOT of games, and even then, it'd only be an average -- there is no exact, consistent value. Ten seemed really low when I first read the idea (probably in the original "Best QB of all time" post or perhaps an AY/A post). But now I'm a believer. 6 seems too low, 14 seems too high, but ten feels just about right.
The one that gets me is interceptions. It's just a gut feeling, but they seem over-penalized. I understand that turnovers are bad, but... I guess they're very sitaution-specific too. An intercepted screen pass is worse than a downfield interception, and interceptions near either goal line are very bad. But offenses seem most likely to pass in the middle areas of the field, and I'm guessing a lot of interceptions are downfield passes on third and long, and they may not be significantly worse than an incompletion and a punt.
I just keep thinking "Would I rather have 50 touchdowns and 20 interceptions (-400) or 7 touchdowns and 10 interceptions (-380)?" Marino loses 285 yards for his 48 TDs and 17 INTs. That seems a bit harsh for one of the best passing seasons in history, you know? Terry Bradshaw loses over 7,000 yards in his career (212 vs 210). George Blanda made the Pro Bowl in 1962 but loses over 65% of his total yards that year due to interceptions (42 interceptions is a bit extreme, I'll grant you)
Oh, another idea regarding average vs magic number * average... Rather than using the league's average on passing, one could use the league's average per play (running and passing combined). It may help those who played in a running dominated era, and it doesn't complicate the formula (at least on the surface -- average yards per play would be more difficult to compute).
Other ides that would complicate the formula for no real gain but I seem completely unable to stop making suggestions:
grey ink/black ink type stuff. But it kind of screws up the statistics bent. But leading the league is still leading the league, you know?
Some sort of adjustment based on age and experience. Part of the reason Marino's 1984 performance was so impressive was because it was his first complete season as a starter. Part of the reason Favre's 66+% completions, 4000+ yards, and 28 touchdowns last year was so impressive is because he was 38 years old. Look at Marino or Montana's age 38 years. I don't hold Peyton Manning's rookie year against him because he was a rookie -- You expect shaky performances from a rookie. Etc. etc. etc. The problem would be that you're slicing already sparse data even thinner.
Posted on 23-Jun-08 at 1:38 am | PermalinkMS, you are not accounting for the fact that QBs also get positive credit for the yards gained on their TD passes.
Also, there is an analytical basis for why interceptions cost a team the equivalent of 45 yards, at least based on the 1997 season. It's detailed in the aforementioned Hidden Game of Football.
Posted on 23-Jun-08 at 6:24 am | PermalinkI guess this discussion is over. Recommend you consider adjusting the TD bonus next year, Chase.
Posted on 24-Jun-08 at 6:54 am | PermalinkMy Buddy Derek M. has a man crush on Elway and is pissed that he is not ranked higher. He claims that if you incorporated "wins" into the formula that Elway would be #1.
Please stick a pin in his head and explain why "wins" are not directly part of your calculation.
Thanks
Posted on 24-Jun-08 at 8:13 am | PermalinkI've always wondered if prorating to game lengths, either halfway or all the way, properly adjusts for era. What if an adjustment was made to a specified number of attempts rather than a total number of games?
ie if the adjustment was to 600 attempts/year and the average team in a given year attempted 500, then the adjustment factor would be 1.1 as that would adjust the average team up to 550 attempts (halfway between 500 and 600).
Quarterbacks in the 70's just didn't have the same attempts/game and so they didn't have as much of a chance to showcase their talents despite playing almost as many games. I think that it'd be interesting to see how the lists turned out. (likely fairly similar, with some juggling)
Posted on 24-Jun-08 at 10:43 am | PermalinkChase,
By the way, I forgot to respond about fumbles previously. TGHOF uses the same analysis used for -45 yards per interception to show that fumbles cost on average -50 yards. This is intuitive, since interceptions typically happen further downfield. Of course, that is an average for all players, not just QBs, so in reality an average fumble lost by a QB probably costs more than 50 yards... but at least it is an easy starting point that can be refined later if desired.
You mentioned that sometimes they aren't the QB's fault, but that can also be said about interceptions, and you're counting all of those, so I'd assume you should just go ahead and count all fumbles lost.
Posted on 24-Jun-08 at 1:30 pm | PermalinkOK JWB, you've shamed me into it.
I'm going to use the Romer chart to try to figure the TD bonus.
By my eye, it looks like the 99 yard line is worth 5.6 and the 1 yard line is worth -1.6. So that's 7.2 points, spread over 98 yards, or 13.6 generic yards per point.
Also by my eye, the 27 yard line is worth about 0.6, so a TD is worth about 6.4.
OK....
Let's look at a 50-yard TD pass. The change in point values is +4.4. Converting to yardage, that would be equivalent to 4.4*13.6 =~ 59.8 yards. Since the QB has already been credited with 50 passing yards, he should get a bonus of about 9.8 yards for the TD.
Posted on 24-Jun-08 at 7:13 pm | Permalink....
In general, the bonus should be (6.4-X)*13.6 - Y, where X is the point value of the place from which the TD pass was thrown, and Y is the distance of the TD pass.
....
What about an 86-yard pass (that one is easy to read off the chart)? The appropriate bonus should be (6.4-0)*13.6 - 86 =~ 1. Only a 1-yard bonus.
.....
How about a 14-yard TD pass? The appropriate bonus there should be (6.4-4)*13.6 - 14 =~ 18.6.
.....
So different lengths of TD passes deserve different bonuses. Now all we have to do is take the average of all TD passes.
.....
Databases are a wonderful thing. In a few microseconds, I was able to average the appropriate bonus for every TD pass thrown between 1940 and 2007.
......
ANSWER: 14.5 yards.
......
Of course I was just eyeballing. If a TD is worth 6.2 instead of 6.4, then that drops to 11.6. If I put TDs back at 6.4, but change the function so that the 99-yard-line has a value of 5.7 instead of 5.6, then it's 13.3.
.......
So maybe 12 or 15 would be a better number.
Doug,
Great stuff. I didn't know what the right number was, but now at least there is a basis for the number. I wonder how use of 14.5 (or 12 or 15) would change the rankings?
Thumbs up. I'll look for this in next year's iteration of the formula, along with fumbles lost and postseason stats. Right, Chase?
Posted on 24-Jun-08 at 8:43 pm | Permalink