SITE NEWS: We are moving all of our site and company news into a single blog for Sports-Reference.com. We'll tag all PFR content, so you can quickly and easily find the content you want.
Also, our existing PFR blog rss feed will be redirected to the new site's feed.
Pro-Football-Reference.com » Sports Reference
For more from Chase and Jason, check out their work at Football Perspective and The Big Lead.
ShareThisThe Draft Value Chart: Right or Wrong?
I'm sure most of the PFR readers have seen the NFL draft value chart, sometimes referred to as the Jimmy Johnson draft chart. Lots of people have discussed whether it's accurate, and whether it's still valuable in an era of escalating salaries. I'll sidestep the salaries issue today, and just focus on the actual draft value chart.
For one, how would we know whether or not it's accurate? I suppose there are a few ways of analyzing that, but you need to assign some basic value to each draft pick. We know that Pick N is always better than Pick N+5, but how big is that difference if N = 5, or N = 25, or N = 100?
I looked at every draft from 1970 to 1999, giving me thirty years of drafts. I then assigned the approximate career value of each player to his rookie draft slot. So for the number one pick, we've got 133 points of value from Peyton Manning, 77 points of value from Keyshawn Johnson, 32 points of value from Kenneth Sims, and the value from all the other number one picks from 1970 to 1999. If you do this for the first 224 picks in every draft, and you can then get an average value for each draft pick.
There are some bumps in the data, of course. The seventh pick in the draft has an average value of 39, and the eighth pick an average value of 51, over the thirty years. The 7th pick has a lot of busts (Reggie Rogers, Brian Jozwiak, Joe Profit and Andre Ware) and not that many stars (Phil Simms, Champ Bailey and Bryant Young are the best players). The 8th pick has Ronnie Lott, Willie Roaf, Leslie O'Neal, Otis Anderson and Mike Munchak, and fewer busts.
Since the approximate values of the players that correspond to the draft picks fall off exponentially, I used a logarithmic formula to best fit the data. The formula to predict any NFL draft pick's approximate value in the NFL is:
Approximate Value = -12.583 * Ln(draft pick) + 73.195
You end up with a list that looks like this:
1 73 2 64 3 59 4 56 5 53 6 51 7 49 8 47 9 46 10 44 11 43 12 42 13 41 14 40 15 39 16 38 17 38 18 37 19 36 20 35 21 35 22 34 23 34 24 33 25 33 26 32 27 32 28 31 29 31 30 30 31 30 32 30 33 29 34 29 35 28 36 28 37 28 38 27 39 27 40 27 41 26 42 26 43 26 44 26 45 25 46 25 47 25 48 24 49 24 50 24 51 24 52 23 53 23 54 23 55 23 56 23 57 22 58 22 59 22 60 22 61 21 62 21 63 21 64 21 65 21 66 20 67 20 68 20 69 20 70 20 71 20 72 19 73 19 74 19 75 19 76 19 77 19 78 18 79 18 80 18 81 18 82 18 83 18 84 17 85 17 86 17 87 17 88 17 89 17 90 17 91 16 92 16 93 16 94 16 95 16 96 16 97 16 98 16 99 15 100 15 101 15 102 15 103 15 104 15 105 15 106 15 107 14 108 14 109 14 110 14 111 14 112 14 113 14 114 14 115 13 116 13 117 13 118 13 119 13 120 13 121 13 122 13 123 13 124 13 125 12 126 12 127 12 128 12 129 12 130 12 131 12 132 12 133 12 134 12 135 11 136 11 137 11 138 11 139 11 140 11 141 11 142 11 143 11 144 11 145 11 146 10 147 10 148 10 149 10 150 10 151 10 152 10 153 10 154 10 155 10 156 10 157 10 158 9 159 9 160 9 161 9 162 9 163 9 164 9 165 9 166 9 167 9 168 9 169 9 170 9 171 8 172 8 173 8 174 8 175 8 176 8 177 8 178 8 179 8 180 8 181 8 182 8 183 8 184 8 185 8 186 7 187 7 188 7 189 7 190 7 191 7 192 7 193 7 194 7 195 7 196 7 197 7 198 7 199 7 200 7 201 6 202 6 203 6 204 6 205 6 206 6 207 6 208 6 209 6 210 6 211 6 212 6 213 6 214 6 215 6 216 6 217 5 218 5 219 5 220 5 221 5 222 5 223 5 224 5
How does this compare to the NFL draft value chart? I plotted them both on the same graph, with the NFL chart in blue and my chart in red. Take a look.
Both start out very high, and drop off pretty fast. However, from picks 4 to 100, the NFL draft chart is pretty perfect. Picks 1, 2 and 3 are very overvalued, in increasing order. After about pick 100, every pick is overvalued again, in increasing order. So pick 180 is pretty undervalued by NFL GMs, but pick 120 is just mildly undervalued.
Now if the really, really early picks are overvalued, and the last four rounds are overvalued, why did I call the picks in the middle set of picks perfectly valued? While they're obviously undervalued with respect to picks not in the four to ninety-five range, the drop-off rate is captured almost perfectly by NFL draft charts. So the Jimmy Johnson chart does an excellent job of capturing the marginal value of the difference between pick 8 and pick 37, or between pick 46 to pick 85. While the whole group of picks is undervalued, they're pretty much all undervalued by a similar amount.
Here's an example. According to the NFL draft value chart, picks 2, 161, 162 and 163 are worth 2,680 draft points. Similarly, picks 26, 27, 28 and 29 are worth 2,680 points. But according to my chart, the first set of selections is worth 91 points in approximate value, and the second set of picks is worth 126 points in value. That's pretty significant -- it's the difference between Emmitt Smith or Junior Seau, and Roger Craig or Greg Lloyd. The middle of the road #2 picks have been Kevin Hardy and Tony Casillas. At 161-163, you're looking at someone who's not likely to ever be a starter. Forty-two of the 90 players drafted at 161, 162 or 163 had zero approximate value points for their career; thirty-eight of them never played a down in the NFL. Picks 26 through 29 are a bit more valuable. Mark Ingram, R.W. McQuarters, William Floyd and Darion Conner might be your four picks at those spots.
I've been thinking about this for a bit, and I've come to the conclusion that my draft chart is correct... and so is the NFL one. How can that be? In short, they measure different things. My draft value chart is excellent at telling you the average production you can get from any draft spot. That's important to use when running a study that deals with the expectations of rookies.
But NFL teams aren't always concerned with what the average pick will yield. Why are the bottom draft picks overvalued? If a team's sixth round pick is terrible, as opposed to just pretty bad, that's not going to matter much to the team. He won't play very either way. There's some minimum baseline a player needs to reach in order to be able to play, and some late round picks will fall below that spot. But since NFL teams aren't really hurt by late round picks that are say, really awful, they can be a bit risky. But in my system, the 200th pick will get knocked down quite a bit if a guy stinks.
Suppose that all seventh round picks have a distribution that looks something like this: 30% of the time they're a 1 out of 10, 30% of the time they're a 2 out of 10, 30% of the time they're a 3 out of 10, and 10% of the time they become something like a 5 out of ten. On average, that means 7th round picks are a 2.3, and that's what the Chase draft value chart would say. But assume that NFL teams won't give playing time to a guy that's not a three out of ten. So for NFL team's purposes, 40% of the time they'll have this 7th round pick be a 3 or a 5, and the other 60% of the time they'll play some other guy at that position. The NFL team doesn't have to eat the costs of the ones or twos out of ten. When it comes to late round picks, high upside is a lot more important than average position. This is why the NFL pick value chart "overvalues" later round picks compared to my draft chart. I suspect every NFL GM would take a "6" and a "1" from two 7th round picks than two "four". But my system would tell you the two "fours" are more valuable. We need to keep in mind that the two charts simply measure things, and having different values assigned to different ratings is perfectly fine.
What about those first three picks? Well, sort of the same thing is going on there. As a general matters, I think teams shoot for upside. Most fans and GMs would prefer a chance to win the SB and a chance to not make the playoffs, over a guaranteed playoff team with little chance of winning it all. Upside risk is especially important when you're talking about the top picks, because those can be the guys that take you to the Super Bowl (Manning, Elway, Bradshaw, Pace, and Aikman, e.g.). Further, I think GMs probably have a bit of overconfidence when they're trading up for the top pick. If a team trades up to get the first or second pick in the draft, they probably don't think they'll end up with Russell Maryland or Tony Boselli; they think it's going to be O.J. Simpson or Peyton Manning. So even if they know the average #1 pick is worth X, they probably think they're going to get the next Hall of Famer. And they're willing to take some risk on the top picks, and "overpay" compared to how they'll perform on average, because to NFL GMs, variance matters. If you asked a GM would they rather get an "8" from the second pick in the draft, or a 50/50 shot at either a "10" or a "5"...I think most GMs would flip the coin. The 10 can be a franchise changing pick, and they'll take on added risk even if it's not a positive expected value play. Once again, this is why it's perfectly fine that the NFL draft chart "overvalues" the top picks relative to my chart.
This entry was posted on Wednesday, May 21st, 2008 at 5:37 am and is filed under Approximate Value, NFL Draft, Statgeekery. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.
Awesome post. I think you're hitting the nail on the head with the "upside risk" analysis. I can't tell if you're aware of this, but you are describing utility theory almost perfectly when you discuss the guaranteed 8 vs a 50/50 shot at a 10 or 5.
There is a point of "indifference" in the gamble you describe at which the typical GM would be just as happy with with the guaranteed 8 as with the chance for the 10 vs the 5. The point of indifference might be 60/40, or 30/70, or whatever, but it's something and it could be measured to build a true utility curve for the draft.
Another possible explanation for the apparent overvaluing of top picks may lie in decision theory. GMs are not only making decisions under risk (as you describe above), but also under uncertainty. In other words, they're not really sure what the risk level is, so they end up favoring alternatives with the highest minimum probability of payoff (the risk floor), which would be the top picks. A #1 pick may not pan out the way a #1 pick would be expected to, but there is a very low chance he would not be a contributing player at all, at least compared to picks further down the draft.
The cool thing is, guys like you can help quantify what the risks actually are, minimizing uncertainty. If GMs and coaches paid attention the overvaluing of top picks would be reduced.
I've always been somewhat skeptical that GMs (or draftniks) can indeed distinguish between players with "upside" (i.e. either a 5 or a 10) and players without upside (an 8) but the same mean projection, especially in the later rounds. To me, citing upside is often merely a weak justification for a reach, drafting a guy earlier than his ability/performance warrants due to intangibles. Sure, some high-profile players may truly be higher risk due to a career-threatening injury or a criminal history, but I wouldn't think there are enough of these players to warrant a systematic shift in the value chart. And it isn't even clear that such players are overvalued rather than undervalued.
Somebody ought to do a study of players noted with the upside label to see if their career value is really more varied than those without upside (that is, if we can even agree on which players actually have upside). Then maybe we could put some real numbers into quantifying the risks.
We know that Pick N is always better than Pick N+5
I know we're sidestepping salaries for the moment, but to make it clear I'd say that that the player selected at Pick N is always (expected to be) better than the player selected at Pick N+5. (And even this is an oversimplification since some teams draft for need instead of taking the best player available.)
If we're talking about the picks themselves, rather than the players, the Massey & Thaler paper argues that Pick N+5 is often indeed more valuable than Pick N.
If a team trades up to get the first or second pick in the draft, they probably don’t think they’ll end up with Russell Maryland
This one is true quite literally. When the Cowboys traded up two days before the 1991 draft, it was to draft Raghib Ismail with the first pick. But on the morning of the draft, Ismail announced that he would sign with the CFL's Toronto Argonauts.
Nice analysis.
One observation - in the free agent era, the draft only guarantees a team the use of a player for 4-6 years. After that, he is at least theoretically available to every team, so AV accumulated after that is not acquired through the draft (although compensatory picks for losing a FA should count as value gained thru the draft, I guess). What would the pick values/curve look like if they only assigned the first 6 years (or pick a better number - maybe max contract length for each pick under the current CBA) of AV to the draft slot?
Good point, mrh. I'll be posting a chart eventually with the other two types of AV used, the full career chart and the draft value chart. The full career chart weighs each season the same, and the draft value chart weighs the first four seasons at 100%, the fifth at 90%, the sixth at 80%, etc.. That probably fits what you want a bit better.
(The one I used here weighs the first season at 100%, the second at 95%, the third at 90%, etc..)
I did some similar research, although looking solely at the performance of 1st round picks against the draft value chart.
Taking the period 1967, the start of the AFL/NFL common draft, through to 1996, a period of thirty years. Looking at that period I charted the number of HOFers, number of probowlers, average length of career as a starter and busts and the average number of probowls each probowler was selected to.
For HOFers I have deemed players with 5 or more Probowls as HOFers in order to allow for players who's careers have finished recently but haven't been inducted yet. 5 PBs seems to be the starting point for inclusion in the hall with a few exceptions so I think I'm on reasonably safe ground to say that most players with 5 or more PBs will eventually make their way into the HOF.
I've labelled anyone who had less than three years as a starter a bust, which I think is a reasonable expectation for a 1st round pick.
I've broken the picks into groups of 4 as when I did my data crunching that seemed to work out as the best way to group the picks, but have posted the data for the 1st four overall.
1967 to 1996
#1 Overall
HOFers: 9
PBers: 19
Busts: 3
Avge seasons as starte: 8.7
Avge No of PBs selected to: 4.3
#2 Overall
HOFers: 9
PBers: 16
Busts: 5
Avge seasons as starter: 7.8
Avge No of PBs selected to: 4.7
#3 Overall
HOFers: 4
PBers: 19
Busts: 6
Avge seasons as starter: 7.6
Avge No of PBs selected to: 3.7
#4 Overall
HOFers: 12
PBers: 19
Busts: 6
Avge Seasons as starter: 7.8
Avge no of PBs selected to: 5.3
# 1-4 Overall as a group:
HOFers: 34
Pbers: 73
Busts: 20
Avge seasons as starter: 7.98
Avge No of PBs selected to: 4.50
# 5-8
HOFers: 13
Pbers: 54
Busts: 37
Avge seasons as starter: 6.37
Avge No of PBs selected to: 3.62
# 9-12
HOFers: 11
Pbers: 43
Busts: 38
Avge seasons as starter: 5.94
Avge No of PBs selected to: 3.45
# 13-16
HOFers: 10
Pbers: 41
Busts: 44
Avge seasons as starter: 5.71
Avge No of PBs selected to: 3.62
# 17 to 20
HOFers: 10
Pbers: 35
Busts: 43
Avge seasons as starter: 5.15
Avge No of PBs selected to: 3.31
# 21 to 24
HOFers: 6
Pbers: 33
Busts: 50
Avge seasons as starter: 4.96
Avge No of PBs selected to: 2.44
# 25 to 28
HOFers: 6
PBers: 22
Busts: 63
Avge seasons as starter: 4.44
Avge no of PBs selected to: 2.86
#29 to 32
HOFers: 6
PBers: 16
Busts: 65
Avge seasons as starter: 3.95
Avge no of PBs selected to: 3.04
Without assigning a value to the pick the first round shakes out very much like the overall draft in that it is logarithmic in that there is a rapid drop off then a levelling out.
For me the most interesting thing was that the further back you go in the draft you are significantly increasing your chances of striking out completely.
The very surprising thing for me was that the #4 overall was hugely more productive than the #3 overall, and is probably the best performing slot to pick from. I have no idea if there is a reason for this or if it's merely a statistical anomaly.
Could you prorate this by year?
That is, after one year in the league, what is the AV by pick, then after two years in the league, then after three, and so on up to "career in league" shown above.
This would provide a useful benchmark of not just the final average AV by spot for a career, but also a sense of the "average contribution pace" by year-in-league by spot.
With that, you could very easily start to measure how far ahead or behind "pace" any given draftee is relative to the average at his draft spot, after even just a couple years.
You could then say things like: "It's still early, but by year three, the AV for a guy taken 18th is on average higher then so-and-so." Also "It's only year two but with his AV, so-and-so is already way above the AV that is typical for players after two years who were taken 36th overall."
Would you share the statistical properties of your formula?