SITE NEWS: We are moving all of our site and company news into a single blog for Sports-Reference.com. We'll tag all PFR content, so you can quickly and easily find the content you want.
Also, our existing PFR blog rss feed will be redirected to the new site's feed.
Pro-Football-Reference.com ยป Sports Reference
For more from Chase and Jason, check out their work at Football Perspective and The Big Lead.
ShareThisWhat’s a starting QB worth? Part III
In Parts 1 and 2, I attempted to figure out what a starting QB --- an actual regular starting QB, not a placeholder like say Chris Redman --- is worth to a team.
I came up with an empirical answer of 2.3 points per game, and I used a sort of thought experiment to convince myself that that is about right.
Ultimately, though, we don't really care about the points. We care about the wins. So I translated the points into wins (just about exactly one win per season, it turned out). But another way to go would be to skip the points step and go straight to the wins. That is, instead of looking at how many points per game a team lost when forced to play its backup, look directly at how many "wins per game" they lost.
When I do that with the same data set as the previous study, I get an average drop of .038 in winning percentage, which comes out to 0.6 wins per season. That's lower than the estimate based on points, which might indicate that, as some commenters speculated, the points go down due to a change to a more conservative game plan and/or possibly extra effort by the defense (though I checked, and defenses did not allow fewer points in games started by the backup.)
I then decided to make absolutely sure I wasn't biasing the results toward the backups by intentionally trying to bias them toward the starters. In particular, I threw out all the teams whose starting QBs had a below-.500 record. So I was looking at all teams since 1990 whose game one starting QB started at least eight games and whose passing stats were at least league average and whose record was at least .500. That sample includes 31 teams. Using those teams, the average difference in winning percentage between starter and backups was .108, which would imply 1.7 wins per year.
Make of that what you will.
Another interesting comment from the previous threads is that perhaps a backup quarterback will do better in his first game or two than he will in later games when opposing defenses presumably have a better idea what to expect from him.
To test this, I ran a regression. I took each game involving the teams in the original sample and recorded the following bits of data about it.
1. The overall season-long offensive quality of the team, as measured by offensive SRS.
2. The quality of the defense they faced, as measured by defensive SRS.
3. Whether the starting quarterback was the starter or a backup.
4. If a backup, whether it was his first week starting or not.
[I should have included home field here, but just forgot.]
For the output variable, I used points scored (for that game) above the league average for the year.
Results:
1. The coefficient on the starting QB variable was 2.4 (points per game), which matches up very well with our estimate from yesterday. That's reassuring.
2. The coefficient on the "first week starting" variable was -0.1 and was nowhere near being significant. So no evidence for a surprise effect.
Finally, I used a regression to get another estimate on the winning question.
Inputs:
1. offensive quality - defensive quality
2. starting QB or not
Output:
Win or loss
The best-fit formula is this:
Est. probability of winning =~ 1 / (1 + e^(.3083 - (pointdiff)*(.1649) - (starting QB?)*(.2842)))
So for example, if your offense was average and your opponent's defense was 5 points better than average (pointdiff = -5), and you had your starting QB (starting QB = 1), your probability of winning would be about 30%. Against the same defense without your starting QB, your chance would be about 24%.
If your offense was average and your defense was average, then your probability of winning drops by about seven percentage points depending on whether your starter is playing or not.
Six or seven percentage points times 16 games yields, again, almost exactly one win per season. More verification of the prior estimates.
This entry was posted on Wednesday, August 20th, 2008 at 3:45 am and is filed under General, Statgeekery. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.
Excellent work, Doug.
Doug, since a starter can be counted on for one extra win a year than the backup, can it be deduced that a "better starting QB" could be expected to produce one additional win per season?
-
Put more personally, I wanted Favre to play for the Bills. I believe Edwards/Losman should produce 7 wins. From your figuring, would 8 wins be the expected value if Favre was here? If so, no big deal that he's not here.
Very cool. But I think there might be one or two (big) wrinkles in the logistic regression. To get unbiased results, you'd need to exclude the current case game result from the SRS calculations. Otherwise you are partially 'assuming the conclusion.'
Plus, there might be a far larger problem. The season-long SRS of the team is affected by the change of QB. To get unbiased results, you'd need to use SRS up to the point of QB change as the estimator, instead of season-long SRS.
Both effects would serve to overestimate the effect of SRS, and therefore rob the QB variable of a lot of variance. My guess is that an unbiased estimate would yield a much larger coefficient for the QB variable.
Another way to put this is that SRS is an intervening variable between QB skill and winning. It would work like this: QB skill ----> team performance (SRS) ----> win/loss outcomes.
Your model is like this:
QB skill ----> win/loss outcomes, and
SRS ----> win/loss outcomes.
Part of the true effect of QB skill will end up being robbed by the SRS coefficient, and part of it will end up where it belongs in the QB variable.
Brian, you're right. If the backups and the starters played similar number of games, then the problem wouldn't be a problem. But they didn't so it (potentially) is.
.
I never manage to get regression models right on the first try (and the model in that post wasn't even the first try).
.
Do you agree that the right model would be:
.
INPUTS:
avg points per game of this team WITH STARTER
defense quality
home/road
dummy for backup QB
.
OUTPUT:
points
.
I'm not going to start wading into partial-season SRS, so I guess I'll have to switch from SRS to regular old PPG, but I can live with that.
.
I'll see if I can do that for tomorrow or Friday. Thanks for the comment.
I think that would be a valid model. I don't think switching from SRS to PPG costs you much in this case. One of the great things about SRS is that it accounts for opponent strength across the league. Opponent strength will obviously vary for the starter and his backup QB within the season.
But, if your data set is large enough (and it wouldn't have to be that big), then opponent strength would average out. Only if you were looking for the delta in expected wins in one particular instance would it cost you, such as 'how much did going from Bulger to Frerotte cost the Rams in '07?' If the question is, 'how much does going from the starter to a backup cost in general?' then you've lost very little.
One other suggestion would be to make OUTPUT point differential, instead of offensive points scored. Plus, I would use a non-point method of accounting for defense quality, such as yds per play. That way, you can see how the difference in QB affects total point differential. A bad QB can really hurt his team's points allowed--for example, interceptions cost about 4 points a pop on average.
Out of curiosity, I tried looking at this from the other end -- I took a glance at teams whose offenses fell apart from one season to the next -- their PPG dropped by 10 or more points (totally arbitrary). I found 9.
Three of those are pretty obvious -- Steve Young's retirement (and Hearst leaving, though Charlie Garner did well), John Elway's retirement (which coincided with Davis getting injured), and Kurt Warner's implosion in 2002.
The Raiders coming off their superbowl loss also shows up. Gannon was injured but he wasn't doing well even before that. I honestly don't know why they fell apart, but it may have just been age catching up with them. Gannon(38), Rice(41), Brown(37), Garner(31)...
Dallas from 1983 to 1984 also shows up, and they did change starting QBs that year, though they changed back partway through the season. I don't know anything about Danny White and Gary Hodgeboom, before my time.
As for the rest:
Falcons 1973 -> 1974 - nope
Bills 1975 -> 1976 - nope
Redskins 1991 -> 1992 - nope
Redskins 1999 -> 2000 - nope
Given all the QB changes in that time, it seems to support the idea that losing your average starter really isn't a crushing blow, but losing a phenom is. Two hall of famers, Warner coming off a 4800 yard season, and Gannon coming off a 4700 yard season are definitely not average. And the list of teams who fell apart with the same starting QB and RB seems to agree with the notion that the QB and RB are only small pieces of a larger puzzle.
Just thought I'd share.
Mattie-----As for Elway, the Broncos, and Brian Griese----IMO, the Broncos downfall in '99 was caused by Griese's year-long performance (or lack-of) much more than a drop-off by the running backs. Yes, Gary had 1000 less yards for the year, but break that down, and it comes to 30 y.p.g. or 1 more yd. per carry less than Davis was giving them. On the other hand, Griese was giving them much less from a QB: his turnovers were equal to his TD Passes, and his Rating was 20 points less than Elway's. By far, the bigger difference in performance was at the QB position. I strongly DISAGREE with your statement that the QB is only a small piece of a larger puzzle. The QB is a VERY LARGE piece of that puzzle, IMO.
Sandy----What makes it a tough argument is the fact that Davis scored so many TD's (21) in '98. Elway had 22 TD passes that year. But--that is as unusual as what happened in '07 with the Pats. Brady had 50 TD passes compared to his leading back (Maroney) having only 6 rushing TD's. Both cases are far from the norm. But I do agree with you about the QB being a very large piece. Just like Kurt Warner proved in '99---a good QB can overcome lapses by the Defense (the Rams won their 1st PO game in spite of giving up 37 points to the Vikes), and the lack of a running game ( Marshall Faulk had an avg. of only 27 yds. per game rushing in the '99 PO's.
We know how many points the Pats scored last year, and they have the same offense this year minus Brady, so we will see how many points, and wins, Brady was worth.
Andrew, thought i would also check in and comment about this, however last year pats points and record arent a very good indicator as you have to already assume a regression to the mean since they were so far out there last year and on top of that their running up the score stats. Assuming regression to the mean and using last years stats in a regression model they would have been expected to score about 53 total tds this years and win about 11-12 wins (12 by football outsiders). That is a better baseline comparison to really see how much value he might have
Peyton Manning had 49 TDs and a 121 QB rating in 2004, along with 4,500 yards passing. In 2005, he threw for 21 less TDs, 800 less yards, and had a QB rating 17 points lower. This is with the same wide receivers and same RB.
Dan Marino threw for over 5000 yards and 48 TDs in 1984, with a 109 QB rating. In 1985, he threw nearly 1000 less yards, had a QB rating in the 80s, threw 18 less TDs and 4 more INTs.
So for the sake of argument, lets say all these players are about the same, talent-wise, and they all had career-type years in 1984/2004/2007. That would suggest to me that one would expect Brady to have thrown for, say, 3900 yards, a QB rating in the high 90s, and say 30 TDs this year.
Of course, he might have done better than that, or he might have done worse. We'll never know. But if their backup throws for 25 TDs, you can't just assume Brady would have doubled that. MAYBE he would have, or maybe he'd have thrown even more than 50, but more likely he'd have thrown for 30.