Pro Football Reference Blog

Baseball analysis pioneer Bill James had a tool called the Value Approximation Method. In the 1982 Abstract, he introduced it thusly:

The value approximation method is a tool that is used to make judgements not about individual seasons, but about groups of seasons. The key word is approximation, as this is the one tool in our assortment which makes no attempt to measure anything precisely. The purpose of the value approximation method is to render things large and obvious in a mathemtatical statement, and thus capapble of being put to use so as to reach other conclusions.

[The emphasis was in the original.]

James then goes on to describe the method a bit. He used basic stats like batting average, RBI, stolen bases, pitching wins and losses, strikeouts, and so forth, to assign an integer to each player season. A typical MVP would be around 16 or 17, and all-star around 13, an average starter about 10, and so on. He continues:

These approximations are not intended to tell you anything at all about the player that you do not already know. It is not essential that you accept the individual evaluations; there are cases where 10 or 11 points seasons will turn out, under careful scrutiny, to be better than 12 or 13 point seasons. The approximations are intended only to distinguish as quickly and reliably as possible between large contributions, very large contributions, gigantic contributions, medium-sized contributions, small, smaller, and negligible contributions.

...

The value approximation method enables one to set down on paper a simple representation of all of the things that one would otherwise have to hold in mind.

Basketball analysts will occasionally use a simple points-plus-rebounds-plus-assists-plus-some-other-stuff metric to accomplish the same goal. A few circumstances in which such a method might be useful here at this blog:

• Which teams have done the best jobs of drafting? To answer this, we'd need a tool that measures value across positions.
• Likewise, this post about how teams are built could be made more accurate. Instead of simply counting a starter as a starter, we could weight the more important starters more heavily, and we could include the non-starters as well. In other words, instead of saying things like "Team X got 4 of its 22 starters in the first round", we could say more meaningful things like, "Team X got 31% of its contributions from first round picks."
• What is the "real" age of a team? If you simply average the ages of the players, you might have a backup QB or a kicker or a couple of veteran linebackers who only play a few plays a game skewing the average. If you only use the starters, or something like that, you lose information about whether the depth is old or young. It would be nice to weight the average according to how much each player contributed.
• Do players from big name colleges tend to be overdrafted compared to players from smaller schools?
• If, for some reason, you want to know which college award has produced the best pros, you need a tool like this that cuts across eras and across positions.

In the above-linked post, I did whip up an ad hoc value approximation method, and I set it up using language similar to James' language above:

So, with all the players in place, I need a way to measure their NFL success. As we go through it, keep in mind that it is not meant to be a precise metric, but rather an approximate measure of value. Comparing a linebacker who has been in the league for nine years to a running back who is in his second year is very tough to do, so all I’m hoping to do is group guys into broad categories that seem reasonable. I’m going to put a number between 0 and 18 on each player. Peyton Manning is an 18. So are Warren Sapp and Randy Moss. Julius Peppers is a 14, Garrison Hearst a 12, Terrell Suggs a 10, Dominic Raiola an 8, E.J. Henderson a 6, Rashaan Salaam was a 4, Byron Hanspard a 2, and Eric Crouch a zero.

Again remember that the goal here is not to forever put an end to the debate about whether Daniel Graham or Antonio Bryant has had the better career. That’s too ambitious a goal. We simply want to classify them both as being a bit better than Michael Bishop or Travis Dorsch, but not as good as Terry Glenn or Carson Palmer.

Despite the disclaimer, many people took exception to my method, and not unjustifiably. If we want to do this for football, we've got a big problem that James didn't have. Namely, there are only a precious few objective pieces of data that are recorded for all players. James could use homers, RBIs, stolen bases, etc. for position players, give a few bonus points for playing the more important defensive positions, and now he's got all his position players rated. He can use wins, strikeouts, saves, etc. to rate his pitchers. Then all he has to do is find a way to interlace the two lists, and it's not too hard to come up with some intuitive justification for how to do it.

The only objective stats by which we can compare Randy Moss and Mike Singletary are games and games started. Pro Bowls are, I suppose, an objective piece of data in hindsight, but there was obviously a great deal of subjectivity involved in determining that number. Remember, we're trying to very approximately rate all players from all positions in one big list here.

Are these, even approximately, the best 10 players in NFL history?

+-----------------+-------+
| player          | games |
+-----------------+-------+
| Morten Andersen | 382   |
| Gary Anderson   | 353   |
| George Blanda   | 340   |
| Jeff Feagles    | 320   |
| Jerry Rice      | 303   |
| Bruce Matthews  | 296   |
| Darrell Green   | 295   |
| Sean Landeta    | 284   |
| Jim Marshall    | 282   |
| Trey Junkin     | 281   |
+-----------------+-------+

How about these? (Note: games started aren't quite complete in my database, but please play along.)

+-----------------+---------------+
| player          | games_started |
+-----------------+---------------+
| Bruce Matthews  | 292           |
| Jerry Rice      | 284           |
| Jim Marshall    | 282           |
| Bruce Smith     | 267           |
| Darrell Green   | 258           |
| Mike Kenn       | 251           |
| Lomas Brown     | 251           |
| Clay Matthews   | 248           |
| Dan Marino      | 240           |
| Mick Tingelhoff | 240           |
+-----------------+---------------+

This is probably better:

+------------------+---------------+
| player           |    pro_bowls  |
+------------------+---------------+
| Merlin Olsen     |            14 |
| Bruce Matthews   |            14 |
| Jerry Rice       |            13 |
| Reggie White     |            13 |
| Jim Otto         |            12 |
| Junior Seau      |            12 |
| Randall McDaniel |            12 |
| Will Shields     |            12 |
| Ken Houston      |            12 |
| Bruce Smith      |            11 |
+------------------+---------------+

A combination of those three stats is what I used in the college award winners post, because that's all we have. I'd like to try to do better. But it's going to get complicated.

The main idea is very similar to another of Bill James' concoctions: Win Shares. The output and the goal of the Win Shares method are in some sense similar to those of the value approximation method: put a number on every player-season so that we can compare across years and across positions. But the method itself is completely different. Approximate values are simple, intuitive, and approximate, Win Shares are complicated and precise (or at least as precise as the available data allows). Here's the main idea, from the Win Shares entry at wikipedia:

Win shares is a top-down approach which starts with the number of games a team won, and then attempts to assign credit to players, proportionally based on their statistics.

I'm not going to do exactly that, but I'm going to use the same idea. I'm not going to get into too many specifics here because I want to introduce the main ideas without getting too bogged down, but here are the main steps.

The first thing we do is measure each teams's offense. Based on how good the team's offense is, we determine how many "points" are to be split among the players on that team's offense. Let's declare that an average team should have 100 points to split. If the 2007 Patriots' offense was, say, 60% better than average according to whatever metric we decide to use, then Brady, Welker, Light, Maroney, etc. would have 160 points to split. Meanwhile, a terrible offensive unit like the 2006 Raiders might only have 45 or 50 points to split.

Now do the same with defenses. The 2002 Bucs defenders will get a lot of points, somewhere around 150, to divvy up. The 2007 Lion defenders will have only 60 or so.

From here on out, I'm going to focus on the offensive side. I'll return to the defenders in a future post.

Now we have to divvy up the points, which is where it gets real dicey. I'm going to lay out a few assumptions that are almost certainly not correct, but whose use I'll try to justify anyway. Here is the first:

Assumption #1: the offensive line is exactly as good as the offense.

I will make no effort to try to determine whether Emmitt, Michael, and Troy made pro bowlers out of Erik, Nate, and Mark or vice versa. I do realize that this will overcredit lines that were fortunate enough to have superstars behind them, and it will overcredit runners and passers who really did have great lines in front of them, but I don't see what choice we have, because we don't even know who the really great lines were. I'm trying to build an objective method here. I can't be adding extra credit to the 90s Cowboys offensive linemen simply because everybody knows they were opening huge holes for Emmitt. Good offenses have good offensive lines. Bad offenses have bad offensive lines. Approximate Value.

Assumption #2: the offensive line is equally important in the running game as it is in the passing game.

I will let you to try to convince me otherwise, but I think Assumption #2 is a good null hypothesis unless I see some evidence to the contrary.

Putting these two assumptions together, we declare that every team's offensive line will get the same proportion of its team's offensive points. What proportion is that? We'll talk about that later.

Now how do we award points to the individual offensive linemen? First we award "pre-points" to each linemen based on (1) how many games he played, (2) how many games he started, (3) whether he was a tackle (as opposed to a guard or center), and (4) whether he made the pro bowl. Remember, I'm avoiding details in this post, so I'm not going to tell you exactly how I do that. Add up all the pre-points for each team's line and then divvy up the actual points proportionally. If a team's line is given, say, 40 points to distribute among them, then a player who had 25% of the team's pre-points would get 10 points.

OK, now the offensive linemen are done. Just 17 more positions to do.

The next step is to determine how good each team's offense was in the running game compared to the passing game. Once we've decided that, we take the team's remaining offensive points (the ones that haven't already been given to linemen), and we divide them into two categories: (1) running game points, and (2) passing game points. Exactly how we make that split is subject to some debate, and we will have that debate, but not now. For now, just note that in most cases we'll have to give a lot more passing game points than running game points because there are a lot more people that share in the passing game. In a lot of cases, virtually all of the running game points will go to a single individual (remember, the offensive line has already been credited), whereas the passing points will be generally be split among four or five key guys, as well as another half-dozen or so bit players.

On the rushing side, we award the running game points proportionally to the players according to how many rushing yards they had, or some similarly basic metric.

On the passing side, we have to make another split, and so here comes another assumption:

Assumption #3: the ratio of pass-thrower importance to pass-catcher importance is constant from team to team.

Rather than being merely shaky, this assumption is obviously wrong. As Chase pointed out to me, Ryan Fitzpatrick throwing to Bruce and Holt might produce numbers similar to Tom Brady throwing to Jabar Gaffney and Reche Caldwell, but in the former case everybody knows that Bruce and Holt are responsible while in the latter case it's obviously Brady. Believe me, I'd like nothing more than to be able to separate the contributions of the passers from the catchers on a given team. But that's basically the Holy Grail of football analysis and as far as I know no one is remotely close to knowing how to do it. So rather than go through every team in NFL history and put a flag by the ones where we know that either the QB or the receiver group is better than the other, I'm just going to point to the word "approximate" in the title of the post, hope things will even out in most cases, and use Assumption #3.

In keeping with the spirit of this post, I'll decline to talk about what the constant pass/catch ratio is. But assuming we've got that figured out, we now award the passing points proportionally according to passing yards (or maybe some slightly more interesting metric) and the catching points proportionally according to receiving yards. Note that many running backs will pick up a few points here to add to their rushing points.

That's it. That's all there is to it.

What's left to talk about? Oh yeah, lots of stuff:

• What metric do I use to determine offensive points at the team level?
• What fraction of points should go to the line?
• What is the pass/run split?
• On the passing side, what is the throw/catch split?
• We need to figure a way to give some of those offensive line points to fullbacks and tight ends, many of whose jobs include a lot of blocking.

Aside from that, we're finished. Oh wait.

• We have to go through all this stuff with defense too.
• Kickers, punters, returners?

As you can tell, it's going to take a few posts to get through this, so I'll stop this one here. But be aware that this isn't one of those posts where I conjure up some grand idea and then don't follow through on it. I actually have done most of the programming and even have lists. But I've been doing this long enough to know that, once I post the lists, no one will look at anything but that. I'd like to iron out some of the methodology without that as a distraction. If you have any comments so far, I'd love to hear them.