Posted by Doug on April 4, 2007
[NOTE: this post was edited to correct an error a couple of days after its original posting. The error is explained here.]
They claim to have shown that having the first draft slot is a disadvantage. I claim they've shown that all draft slots in the first three rounds are essentially of equal value. That's not because first round players aren't better than third round players --- they are, at least on average --- it's because first round, second round, and third round players will, on average, outperform their contract by roughly equal amounts. In a salary capped world, that makes those picks equally valuable.
But that's all theoretical. What I want to know is, does it translate to wins and losses? If, as Massey and Thaler claim, high first round picks are a liability compared to later first round picks and second round picks, then teams with lots of valuable (according to their study) picks should improve more quickly than similar teams without such picks.
This is a sticky question because so many factors are involved. For example, consider the following table, which consists of actual data:
All teams with 3 or fewer wins in Year N-1
Avg wins in Year: N N+1 N+2 TOT ============================================================= Low M-T draft value in Year N 6.0 8.0 7.0 21.0 High M-T draft value in Year N 6.0 8.5 7.6 22.1
"High" and "low" are defined by "above the median" and below the median." In other words, I looked at all teams that won 3 or fewer games in Year N-1 and sorted them by M-T value in the Year N draft. I cut the list in half and called the top half the High M-T Value group and the bottom half the Low Value group.
This would seem to indicate that M-T value does play some small role in team improvements during the three year period. But the problem is if you do the analogous breakdown by NFL pick value chart value, you'll get similar results. That's to be expected, because chart value and M-T value are positively associated. Teams that have a lot of picks generally have a lot of both kinds of value. In order to put the Massey-Thaler theory to the test, we need to separate the two.
The only way I know to do that is with regression. As I've mentioned before, regression is an unbelievably complex subject. Like most people, I don't fully understand all its intricacies. Also like most people, I'm going to go ahead and use it anyway. The only difference is I'm going to warn you that I'm not completely sure if what I've done is OK. So take it for what it's worth.
I ran a regression with the following input variables:
- Team record in year N-1
- Total Massey-Thaler draft value in the Year N draft (described in Friday's post), in millions of dollars of surplus value
- Total NFL draft pick value chart value in the Year N draft (also described in the above-linked post) in their usual units. The first pick is 3000, the 2nd pick is 2600, and so on down to the last pick, which is worth essentially zero.
The output is team record in Year N. Here are the results:
Wins in Yr N =~ 5.2 + .32*(Wins in Yr N-1) + .000040*(NFL pick value) + .056*(M-T value)
The coefficients on the last two inputs were not significant and were not anywhere close to being significant.
Alright, so maybe it takes more than a year for the value of these draft picks to materialize. Here is another regression:
Wins in Yr N PLUS Yr N+1 =~ 11.4 + .53*(Wins in Yr N-1) + .00029*(NFL pick value) - .017*(M-T value)
Again neither of the draft-related coefficients is significant.
When you look at teams' records over the next three years, you get similar results:
Wins in Yr N PLUS Yr N+1 PLUS Yr N+2 =~ 17.8 + .65*(Wins in Yr N-1) + .000025*(NFL pick value) + .32*(M-T value)
Again, neither coefficient is significant. Even if they were statistically significant, they're small enough that it's clear that they have no practical significance, as the following thought experiment shows.
Suppose the Raiders swapped their #1 overall pick straight up for the Colts' #32 overall pick. In giving up #1 and getting back #32, the Raiders gain a net of about .13 million in Massey-Thaler value. Multiplying .13 by the appropriate coefficient (.32) yields about .01 wins. In a three year period.
The R^2 of this regression was about .07 which means that if you want to predict an NFL team's wins during the period 2007--2009 using its 2006 record and the M-T and NFL values of its 2007 draft picks, you're not going to be very successful. But that's pretty obvious.
Assuming the regression is technically OK, these results validate my criticism of the Massey-Thaler paper. Namely, teams do not appear to be able to the translate the theoretical surplus value they get from their draft picks into surplus production on the field. That's probably because the difference in theoretical average value between draft picks is so small that it's swamped by other factors. One of those factors, of course, is what the teams actually do with those picks. In other words, it's much, much less important for a team to know that pick #1 is on average less valuable than pick #30 than it is for them to know that Peyton Manning is better than Ryan Leaf (if they're picking #1) or that Chad Johnson is better than Quincy Morgan (if they're picking #30). What Massey and Thaler's paper shows is that the NFL draft is a meritocracy, or maybe a luckocracy, but it's not in its present form a mechanism for promoting parity.