Craig Button isn’t buying it. Anecdotally, the theory makes sense, but he needs more. The idea posed to him: that NHL draft scouting has become more accurate in the past few decades and that we may never see another Alexandre Daigle, Patrik Stefan or Nail Yakupov again, because we simply have more information than ever at our disposal.
“How is it more accurate?” Button challenges. “What’s your benchmark for defining a draft bust? Is it 400 games? Because if that’s your benchmark, it’s not true. The success rate hasn’t changed.”
You know that lovely feeling when you have a theory, you crunch the numbers, and they reveal exactly what you hypothesized? This is not one of those times. The assumption was that scouting has become more accurate, but the numbers don’t support it definitively – nor do they completely disprove it. Scouting isn’t necessarily better or worse, but it’s certainly more complicated.
For the sake of this study, we couldn’t simply look at which players from a given draft class reached the NHL. About 10 percent more players make it now than in 1990, but that’s an artificial increase, as the draft has shrunk from 12 to seven rounds, and expansion has spawned 10 extra teams since 1990, meaning we’ve gone from 252 players competing for 21 rosters to 217 competing for 31 rosters. Instead, we studied the first round of the 20 NHL drafts between 1990 and 2009. We capped the sample there because players drafted in 2010 or later haven’t had enough time yet to develop their careers.
We defined “busts” as: top-three picks who didn’t reach 700 games; top-five picks who didn’t reach 500; and, first-rounders who didn’t reach 200. A “super bust” failed to reach 100 games, and a “washout” failed to reach 10.
Contrasting the 1990s with the 2000s does reveal subtle improvements in scouting accuracy when it comes to the epic draft-day disasters. The percentage of super busts declined from 28.2 to 25.6, and the washout rate shrunk from 14.3 to 13.7. Why?
Lou Lamoriello has been an NHL GM every season since 1987-88 between the New Jersey Devils, Toronto Maple Leafs and New York Islanders, and the biggest change he notices in scouting is the amount of opportunities teams get to see players. “It’s a 12-month process right now,” he said. “There was time off. Now, the young players, whether they’re under 17, whether they’re 18, whether they’re 19, whether they were drafted or weren’t drafted, at a very, very young age, they’re identified. They’re followed because of the number of tournaments that are played throughout the country and in Europe.”
Lamoriello adds that technology allows scouts to observe footage of players at much lower age groups now – and that the rise of non-official “experts” in the online community might offer nuggets that influence how a scouting staff views a given prospect.
The other most prominent NHL GM of the past three decades is Ken Holland, who’s in his first year with the Edmonton Oilers after leaving the Detroit Red Wings. For Holland, the reason fewer players wash out is because of the post-draft resources invested in them. “Pretty much every NHL team has two, three, four people in player development,” he said. “We’re drafting these players, and every NHL organization has a group of people to educate them about what they need to do to become NHL players.”
Holland also points out that staffs have grown on the scouting side, not just the player development side, and that players have more people working with them at every step on their path to the pros, from shooting coaches to agents to nutrition consultants.
So do more information and resources mean more accurate scouting? Not always. Under the softer parameters grading first-rounders by the 700/500/200-game model, busts actually increased from 38.5 percent in the 1990s to 39.3 percent in the 2000s. Under the benchmark set by Button, an NHL scout and GM turned TSN director of scouting, 12.0 percent of first-rounders in the 1990s played fewer than 400 games versus 13.5 percent in the 2000s.
So modern scouting does a better job weeding out players who have no business sniffing the NHL – but is actually worse when it comes to the smaller misses, the gaps between picking a plugger and a star, the disappointments in the vein of, say, drafting Gilbert Brule when Anze Kopitar was still on the board.
Scouting, then, is still pretty darned hard. One reason why: more information also means more clutter, more red herrings, and more conflicting reports on young players, forcing scouts to differentiate between “signal and noise,” as Button puts it, paying homage The Signal and the Noise, a book by statistician Nate Silver on why predictions fail.
Analytics remain in their relative infancy, especially when it comes to the information available on pre-NHL players. Another scouting challenge is one unique to sports: a player drafted into the NHL can’t exactly “intern” there first as part of a high-school co-op program to find out if he’s a fit. “In the NHL and Major League Baseball and the NBA and football, they’re looking at you and projecting you to the next level without ever being able, in 99 percent of the cases, to evaluate you against the players you’re actually going to be playing against,” Button said. “You never get that opportunity. Ever. Ever!”
We thus have to keep expectations realistic when it comes to how much scouting can improve in the foreseeable future, at least until advanced stats reach a new threshold of accuracy. As Holland puts it, the salary-cap era has heaped newfound pressure on scouts because teams depend on players signed to entry-level deals so much for their success, yet scouts aren’t yet equipped with the tools to deliver immense breakthroughs in their results. They can only take baby steps for now when it comes to evaluating the babies.