I came up with an empirical answer of 2.3 points per game, and I used a sort of thought experiment to convince myself that that is about right.
Ultimately, though, we don't really care about the points. We care about the wins. So I translated the points into wins (just about exactly one win per season, it turned out). But another way to go would be to skip the points step and go straight to the wins. That is, instead of looking at how many points per game a team lost when forced to play its backup, look directly at how many "wins per game" they lost.
When I do that with the same data set as the previous study, I get an average drop of .038 in winning percentage, which comes out to 0.6 wins per season. That's lower than the estimate based on points, which might indicate that, as some commenters speculated, the points go down due to a change to a more conservative game plan and/or possibly extra effort by the defense (though I checked, and defenses did not allow fewer points in games started by the backup.)
I then decided to make absolutely sure I wasn't biasing the results toward the backups by intentionally trying to bias them toward the starters. In particular, I threw out all the teams whose starting QBs had a below-.500 record. So I was looking at all teams since 1990 whose game one starting QB started at least eight games and whose passing stats were at least league average and whose record was at least .500. That sample includes 31 teams. Using those teams, the average difference in winning percentage between starter and backups was .108, which would imply 1.7 wins per year.
Make of that what you will.
Another interesting comment from the previous threads is that perhaps a backup quarterback will do better in his first game or two than he will in later games when opposing defenses presumably have a better idea what to expect from him.
To test this, I ran a regression. I took each game involving the teams in the original sample and recorded the following bits of data about it.
1. The overall season-long offensive quality of the team, as measured by offensive SRS.
2. The quality of the defense they faced, as measured by defensive SRS.
3. Whether the starting quarterback was the starter or a backup.
4. If a backup, whether it was his first week starting or not.
[I should have included home field here, but just forgot.]
For the output variable, I used points scored (for that game) above the league average for the year.
1. The coefficient on the starting QB variable was 2.4 (points per game), which matches up very well with our estimate from yesterday. That's reassuring.
2. The coefficient on the "first week starting" variable was -0.1 and was nowhere near being significant. So no evidence for a surprise effect.
Finally, I used a regression to get another estimate on the winning question.
1. offensive quality - defensive quality
2. starting QB or not
Win or loss
The best-fit formula is this:
Est. probability of winning =~ 1 / (1 + e^(.3083 - (pointdiff)*(.1649) - (starting QB?)*(.2842)))
So for example, if your offense was average and your opponent's defense was 5 points better than average (pointdiff = -5), and you had your starting QB (starting QB = 1), your probability of winning would be about 30%. Against the same defense without your starting QB, your chance would be about 24%.
If your offense was average and your defense was average, then your probability of winning drops by about seven percentage points depending on whether your starter is playing or not.
Six or seven percentage points times 16 games yields, again, almost exactly one win per season. More verification of the prior estimates.
This entry was posted on Wednesday, August 20th, 2008 at 3:45 am and is filed under General, Statgeekery. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.