Posted by Chase Stuart on February 22, 2007
Yesterday, we looked at lots of combinations of playoff games featuring a mix of regular season records and prior post-season coaching records. Today we're going to get a bit more precise as we conclude the study, and take a quick look at what happened in 2006.
I think its important to be especially clear on what our goal is. This blog has noted the distinction between retrodictive and predictive systems a few times, and in this comment, PFR reader Jim A provided a very useful link. The basic difference is that retrodictive systems answer the question "which team or coach has accomplished the most in the past" while predictive systems answer the question "which team or coach is most likely to win in the future?" What we're trying to create is a predictive system. There's no denying that Bill Belichick (13-3 playoff record) and Joe Gibbs (17-6 career playoff record) have been much more successful than Marty Schottenheimer (5-13) or Jim Mora Sr. (0-6). But that's as obvious as it is uninteresting. Any retrodictive system would have to place Gibbs and Belichick at the top, and Marty and Mora at the bottom.
But when we're talking about whether Schottenheimer should have been fired, we want to know whether he'll win in the future. We want to know the predictive ability of our system. If we find out that a coach's past post-season record is a useless indicator of his future post-season success, it doesn't mean that Schottenheimer is as accomplished as Belichick; it just means that going forward, we have no reason to expect Belichick to be better than Schottenheimer. Those two statements are very different, and that difference is essential to understanding where we're going with this.
Yesterday, I gave a preview of what we're going to look at today -- multiple regression analysis. For each of the 346 playoff games from 1970-2005, I recorded three input variables and one output variables. The output variable is win/loss; the input variables are: 1) each team's head coach's prior playoff record, 2) the difference in winning percentage of the two teams in the regular season, and 3) where the game was played (home, away or neutral (the Super Bowl)).
Before looking at the variables together, let's look at them individually. Home field advantage is strongly correlated with winning -- the Pearson correlation was 0.362 and the correlation was significant. The difference in regular season winning percentage was even more correlated, 0.442, and significant. As for our third variable, past playoff record? The correlation was just 0.03, and was not significant (0.386 on a 2-tailed test). In other words, there is no historical relationship between a coach's prior post-season record and his future post-season performance in a playoff game.
When you run a least squares multiple regression analysis, the following formula is created:
0.436 + 0.13*HFA + 1.32*RegSeaWin%Diff + 0.01 * PastPlayoffWinDiff
So we might say that a team at home (HFA = 1) that won 2 more games than its opponent (RegSeaWin%Diff = 0.125) and with even head coaches, should be expected to get 0.73 wins (or if the game is played 100 times, should win it 73 times). Notice how small the coefficient for past playoff record is -- the differential among the coaches is going to have minimal predictive power. Further, the P-value for past playoff win differential was 0.15, making it not statistically significant.
So what do you say to your friends who won't believe you when you say a coach's past post-season record is irrelevant to predicting his future post-season success? For starters, they'll probably cite some examples. Maybe the Patriots over the Chargers (2006), the Patriots over the Colts (2003, 2004) or Joe Gibbs' Redskins over lots of teams. But if they try and name several examples, remind them that over 350 playoff games have been played since 1970, so individual examples aren't going to prove much. Then throw out these five examples going the other way:
1) In 1982, Chuck Noll had a 14-4 career post-season record and 4 Super Bowl titles to his resume, while Don Coryell was a choke artist that had gone 2-5 in the playoffs. Coryell's team won in Pittsburgh, 31-28.
2) Tom Flores was coaching the defending SB Champions, had won 2 Super Bowls, and owned a sparkling 8-1 career playoff record. His 11-5 team lost in Seattle (12-4) to Chuck Knox, who had been 6-8 in the playoffs prior to that game.
3) The Great Tom Landry, owner of two SB rings and a 20-14 career post-season record, was coaching another great Cowboys team that went 12-4 in 1983. Hosting the 9-7 Rams, John Robinson in his playoff debut went into Dallas and won, 24-17.
4) Bill Walsh was 7-1 in the playoffs and had won two Super Bowls. His defending champion 49ers team played a Giants team with the same 10-6 record, and a coach in Bill Parcells that had a 1-1 career playoff record. But Parcells' Giants won in 1985, 17-3. (And before you start thinking Parcells shouldn't count as a choke coach because "he's Bill Parcells", note that Parcells lost all three times he had a five game advantage over his opponent. In 1994 (8-3 career playoff record at the time) he lost to Bill Belichick in his first post-season game, in 2003, Parcells (11-6) lost to John Fox in his first playoff game, and in 1989 Parcells (5-2) lost at home to John Robinson, who had an ugly 3-5 playoff record before that game.
5) Don Shula, who had coached in four Super Bowls and won two of them at the time, hosted a New England team in 1985 that was coached by Raymond Berry. Raymond Berry's first full season as a head coach was that year. But Berry's team went into Miami and won, 31-14.
The results are clear: the correlation between past playoff success and future playoff success is extremely small and not statistically significant. But let's take it one step further, as I think you should with almost any study that looks at the post-merger NFL: what's going on lately?
I eliminated all playoff games from before 1993, and ran the same numbers. Now we have a look at the modern, post-free agency era. The Pearson Correlation of past playoff records and winning the next game was 0.000, and of course, not significant. Home field was slightly more correlated than before (0.381) and significant, and regular season record was slightly less correlated (0.417) and significant. Running the least squares multiple regression, we get:
0.500 + 0.00*HFA + 1.32*RegSeaWin%Diff + 0.01 * PastPlayoffWinDiff
Once again, past playoff performance is practically irrelevant, and any effect is not significant statistically (0.21 p-value). What's most curious is how home field advantage has been zeroed out. Perhaps one of our readers can help me out, but the big problem I see is that home field advantage is very closely tied to regular season records: there have been only five games out of 130 where the home team had a worse record than the road team. So I believe what the regression is telling us is that once we know the regular season win differential between the two teams, knowing which team is home isn't very useful. Running the regression with only two variables (removing the HFA variable) does not make past playoff record any more useful.
All the statistical tests I've performed make it clear that in terms of a predictive system, knowing a coach's past post-season record is useless to predicting how he will do in a future playoff game. But for fun I thought I'd look at the 2006 playoff results now.
Here's how the first row in the table can be read. When Bill Belichick played Marty Schottenheimer, Belichick (coach 1) had a +17 playoff win differential (Belichick was 10 games over .500 at 12-2, while Schottenheimer was 7 games under .500 at 5-12), a -2 regular season win differential (New England went 12-4 this year, San Diego went 14-2), was on the road (0 = road, 1 = home) and won (0 = loss, 1 = win).
Coach1 Coach2 PWD RWD HFA W/L
BeliBi0 SchoMa0 17 -2 0 1
BeliBi0 DungTo0 12 0 0 0
BeliBi0 MangEr0 9 2 1 1
BillBr0 DungTo0 5 1 1 0
HolmMi0 SmitLo0 4 -4 0 0
ReidAn0 CougTo0 3 2 1 1
ReidAn0 PaytSe0 3 0 0 0
ParcBi0 HolmMi0 2 0 0 0
EdwaHe0 DungTo0 2 -3 0 0
SmitLo0 DungTo0 1 1 0.5 0
PaytSe0 SmitLo0 1 -3 0 0
Of the 11 games this year, only three times did the coach with the better playoff record win the game: Belichick over Schottenheimer and Mangini, and Reid over Coughlin. It's not much of a stretch to say those latter two games weren't surprising; the Eagles and Patriots were more than a notch above the Giants and Jets this year. The Schottenheimer/Belichick game will forever give ammunition to those who believe that past playoff performance is a strong predictor of future playoff performance -- after all, the most clutch coach ever took a worse team on the road and beat the least clutch coach ever. But let's remember that it was still just one game, and one game that could have very easily gone the other way. Belichick lost to Dungy, Billick lost to Dungy at home, and youngsters Sean Payton and Lovie Smith beat successful playoff coaches Andy Reid and Mike Holmgren.
I thought I'd close things today with just a little bit of anecdotal evidence. For all the Marty-bashing that goes on, his 5-13 record could easily be a lot better. The first five games I think of that he's lost in the playoffs all turned on a single play. If John Elway doesn't have The Drive (thanks to a 3rd and 18 completion), if Byner doesn't commit The Fumble, if Lin Elliot doesn't miss 3 field goals (KC loses 10-7), if Nate Kaeding hits a 40-yard FG in overtime, or if Marlon McCree falls down, Schottenheimer would have been 10-8 instead of 5-13. In terms of retrodictive analysis, that stuff's pretty irrelevant: it happened, and Schottenheimer lost. In terms of predictive analysis, I don't know if Marty would have had to have been any better a coach to have a career winning record in the playoffs.
Schottenheimer also lost a 14-10 game, a 17-16 game, a 24-23 game, and a 24-21 game. He's been in lots of close playoff games, but hasn't come out victorious in many. But considering he's got 200 career wins, and an extensive empirical study shows no correlation between past playoff success and the predictability of future playoff success, I have no doubt that Schottenheimer would have had an excellent chance to win a Super Bowl with the Chargers this year.