Posted by Chase Stuart on July 7, 2008
Maybe it's the recent inundation of data that PFR now posseses, but I'm extremely skeptical whenever I see any study at all anymore. Many moons ago, Doug wrote an article on misusing statistics and the number of people distorting data has only increased since then. There are tons of sources on the internet that use a little bit of statistics and a lot of persuasive writing to convince you of things that aren't always true.
There's nothing wrong with statistical analysis, even when it's just a snap-shop of the pie. The problem is that people don't correctly view the analysis as meaningless trivia, but rather as really useful information. The lawyers in the crowd might draw an analog to Federal Rule of Evidence 403, which keeps out certain relevant evidence "if its probative value is substantially outweighed by the danger of unfair prejudice, confusion of the issues, or misleading the" reader.
The problem isn't that there is something wrong with random evidence that's offered, but that data is highly prejudicial. People infer far more from it than they should. And that's why it should be left out. Understanding what's neat trivia and what's causal, valuable information is often tricky.
Why am I on this tangent today? As a Jets fan, you might think I was happy to read Pat Kirwan's recent article on NFL.com. As any football fan knows, Pat Kirwan is an incredible writer who has probably forgotten a bunch more about the NFL than I'll ever know. That being said, here's what he wrote:
How much will the Steelers miss Alan Faneca and how much improvement can the Jets expect to have with Faneca in their lineup?
To project the answer to these questions, I went back and looked at three elite guards who recently left their original teams and moved on for financial reasons. Larry Allen left the Cowboys after the 2005 season for the 49ers; Steve Hutchinson left the Seahawks for the Vikings after the 2005 season; and Eric Steinbach bolted from the Bengals after 2006 for division-rival Cleveland. The outcome of these three departures should be a good indicator of the impact Faneca will have in 2008.
Right there, bells go off in my head. That's why I titled this post, "I've got a habit." I can't even read the rest of the post without wondering why did he choose to look at only the last three years, and why did he choose those three guys instead of some other guards? Let's move on to the rest of the article.
In 2005, the last season Allen played for the Cowboys, Dallas ranked 13th as a rushing club in the NFL. In 2006, they remained the 13th-ranked rushing team and continued to rush the ball at 3.7 yards per carry. Not much of an impact on the Cowboys when they let Allen go -- but his impact on the 49ers offense was significant. The year before Allen arrived in the Bay Area, the 49ers were the 17th-ranked rushing team. Allen's first season saw the team jump all the way up to no. 6 rushing the football.
In 2005, the Seahawks were the no. 3 rushing team and most of the running plays were designed to follow Hutchinson. The year after he left for Minnesota, the Seahawks dropped to 14th and the rushing average dropped from 4.7 yards per carry to 4.0. Hutchinson's arrival in Minnesota in 2006 spiked the Vikings' running game, which was no. 27 before he got to there, to 16th in his first season at guard.
When Steinbach left the Bengals, they continued to be a below-average running team, but the Browns went from no. 31 in the league to 10th -- and Steinbach led the way on many of those plays.
It appears the Steelers will probably drop off. As for the Jets, they should expect their running game to jump from 19th to somewhere closer to 12th. Thomas Jones should have a lot more good-sized holes to run through with Faneca blocking the point of attack. The average jump in league ranking when the elite guards switched teams was 14 spots in the team ranking in the first year , so a Jets move of seven spots could be considered conservative.
Once again, I don't want to make this into an anti-Kirwan rant, because I like the guy. But we've got six teams to look at, and only four of the six changed in the direction the writer wants us to believe. That's already a pretty bad start. Then look at some of these examples -- The 2007 Browns added Eric Steinbach...but they also added Joe Thomas and Jamal Lewis. I don't think it's even reasonable to conclude that Steinbach was the main reason for the improvement, and that doesn't even include the addition of Derek Anderson. Surely Joe Thomas and Jamal Lewis played a huge role in why the '07 Browns improved at running the football.
What about the '05-'06 Vikings? Well yes, Minnesota added Steve Hutchinson. But they also added Matt Birk, who has made 6 Pro Bowls in career, and missed all of 2005 with injury. And they added Chestor Talyor, after having Mewelde Moore and Michael Bennett split carries in 2005. And All Pro FB Tony Richardson was thrown into the mix, too. Certainly we're not comparing apples to apples when you have all those significant changes.
What about Larry Allen? Well we can't look at the '05-'06 Cowboys, since Dallas was a better rushing team after Allen left. What about the 49ers? Well yes, San Francisco moved up a lot in the rankings...but do you think that might have had something to do with Frank Gore starting one game in 2005, and 16 games in 2006? And while the 49ers rushing attack was great with Allen in '06, when he started 11 games, in '07 -- when Allen started 16 games -- the running game was worse than without him in '05. The 49ers running game probably overachieved significantly in 2006, but I have a hard time giving Allen much of that credit when they were worse the next year with "more Allen" on the team, and Gore barely playing the year before Allen arrived.
That leaves just one example, the Seahawks falling after Hutchinson left. I don't know if you noticed, but after Shaun Alexander missed six starts in 2006, and Hasselbeck missed four that year, as well. Alexander was incredible in 2005, but hit the wall (and injury) in 2006. I'm sure Hutchinson hurt him somewhat, but this doesn't come close to representing a lab experiment.
Once again, let me say that I like Kirwan a lot, and think he's a very good writer. But I'm using this as a example to keep your eyes open whenever you see a writing using some arbitrary limits and drawing some strong conclusions. The theory is when good guard leaves, old team suffers, and new team improved. 33% of the time, that theory didn't even hold water. The rest of the time, the team had a different RB in all four cases (Gore, Alexander with an injury, Taylor, Lewis), and a different stud lineman added in two others (Thomas, Birk). It's a lot less convincing, but no less true, to say that "When a great guard changes team, one-third of the time the counterintuitive result happens, one-third of the time the other team adds another star guard, and one-third of the time the team changes the RB and the stats change."
I wondered why Kirwan stopped after going back just three years. I thought maybe there were some ugly examples from prior years, but maybe he just didn't spend that much time researching this. You should always be skeptical when writers choose arbitrary limits on the length of the study. So I went back, and looked at some other pro bowl guards that switched teams.
Marco Rivera, GB to DAL, 2005: Dallas replaced its top RB (Eddie George) with Julius Jones, and its backup RB (Jones) with Marion Barber. Not really a fair comparison, and Dallas' rushing game was pretty similar both years. GB's running game dropped off significantly in 2005, but Ahman Green missed 11 games due to injury that year.
Ruben Brown, Buffalo to Chicago, 2003: The Bills running game slightly improved in 2004 once Brown left, but they also switched from Travis Henry to Willis McGahee. The Bears running game slightly took a step back, and they also went from Anthony Thomas to Thomas Jones.
Ray Brown, San Francisco to Detroit, 2001; Ron Stone, New York to San Francisco, 2001: Garrison Hearst and Kevan Barlow stayed as the main guys, and produced very similar results in 2002. But they replaced one Pro Bowl Guard with another. The Lions improved slightly, but also had James Stewart for a few more games. The Giants (who lost Stone) improved a bit, but they also gave a lot more carries to Tiki Barber than Ron Dayne.
Randall McDaniel, Minnesota to Tampa Bay, 1999: The Bucs running game improved quite a bit, and arguably due to McDaniel's arrival. But the Vikings running game improved even more, although that's at least partially, I'd suspect, due to Daunte Culpepper's insertion into the lineup.
I'm sure that the Jets will be better at running this year, thanks to Faneca. But the evidence doesn't really give us any indication of how much better New York will be, or if it's really going to be because of Faneca. Maybe Damien Woody and Tony Richardson will help just as much. Maybe Thomas Jones will get hurt, and Leon Washington will have a huge year. Intuitively, it makes a ton of sense that a star left guard will make a team better. But be careful when trying to figure out exactly how much better they'll be.
More generally, keep your eyes open when reading any studies that involve statistical analysis, this blog included. Thinking critically when you see a writer try to prove some relationship will almost always lead you to discover some flaws in the study.