since I can not copy and paste the URL due to the swear in the URL I have copy and pasted the entire article and bolded a few areas. Ive been saying for close to 10yrs now what a crap stat the +/- system is and how ts more of a team stat then an individual stat, and this article further more proves my point in how flawed the system is.
A five-year study on team-based plus-minus numbers indicates it gives plus and minus marks to undeserving players at least one third of the time…Long-time NHL executive Brian Burke has called the NHL’s official plus-minus number a “horse ****” stat.
The now venerable stat has long been the object of criticism from NHL players, coaches, general managers, reporters and, more recently, members of the advanced stats community.
One major and recurring criticism has been voiced since the stat became an official NHL number in 1967, namely that a good player who is surrounded by weak players will likely have a poor plus-minus number through little fault of his own, while a weaker player who is surrounded by strong players will have a good plus-minus number through little credit to himself.
Just how commonly does the official NHL plus -minus system award get it wrong, assigning false positives and false negatives?
In a five year study — compiled through game in, game out video analysis of every goal scored for and against the Edmonton Oilers from 2008 to 2013 — it’s apparent that on goals for, about 70 per cent of the plus marks are correctly assigned to players who make some contribution, major or minor, to the goal. But 30 per cent of the plus marks are awarded to players who make little or no contribution at all to the goal being scored.
On goals against, team derived plus-minus systems are even less fair and accurate. Half of the minus marks on goals against are handed out to players who make a major or minor mistake on the goal against, but half of the minus marks go to players who either had no impact on the play or were doing their job defensively but nonetheless are assigned a minus mark due to an erring teammate.
In the end, it’s safe to say that one third of the plus and minus marks handed out under the official plus-minus system are assigned to players who don’t deserve them, and the problem is greatest at the defensive end of the ice. This number of incorrect assignments is likely as high as 40 per cent of all plus-minus marks handed out.
The road to a useful stat on two-way play
The NHL’s road to a useful stat to measure two-way play started with a focus on how to best measure all of those players who make a major contribution to a goal scored.
Newspaper writers listed most goal scorers in major hockey games through the 1880s and 1890s. Hockey became more organized and professionalized in the first decade of the 1900s. More official goal-scoring records were kept, which led to some angst from league officials. In 1913, hockey historian Bill Fitsell reports, National Hockey Association president Emmett Quinn suggested official scorers be abolished as scoring rivalry led to hogging the puck.
The focus of stats is to fairly record what is most significant in a game, so to recognize the contribution of puck passing specialists, the International Hockey League, a top American pro league, started to count up “helpers,” or assists on goals, in 1906-07, says hockey historian Ernie Fitzsimmons. A limit of one assist per goal was counted. That long ago season, Lorne Campbell of the Pittsburgh Pros led the IHL in scoring, with 35 goals and 25 helpers in 24 games.
Through the 1910s, hockey’s various leagues started to allow forward passing in all but the offensive zone. More leagues adopted the assist stat. In 1916, Fitsell says, National Hockey Association president Frank Robinson stated: “The man who assists in scoring a goal is entitled to quite as much credit as the man actually scoring it and that should be impressed upon players with a view to encouraging team play.”
Through the 1920s, it was much more common for players to be given credit for a goal than an assist, with about 0.6 assists assigned per goal.
In 1929, in an effort to rid the NHL of the smothering defensive play that had stifled goal scoring, forward passing in the offensive zone was also permitted. At once, assist totals rocketed up.
In 1926-27, Bill Cook of the New York Rangers won the scoring title with 33 goals and 4 assists in 44 games. By 1935-36, however, Sweeney Schriner of the New York Americans won it with 19 goals and 26 assists in 48 games.
As many as four assists were allowed on each goal. In 1936, the modern rule of awarding an upper limit of two assists per goal was put in place. In recent decades, there’s been about 1.7 assists for every NHL goal scored, 2.7 points on every goal.
The method of handing out goals and assists has always been mechanical. The last player who touches the puck before it enters the net is awarded the goal, even if he’s not primarily responsible for the goal going in. Assists are also awarded in mechanical fashion. So some player might set a massive and effective screen on the goalie, or he might make a monster check behind the net to win the puck in a dangerous spot, but unless he also touches the puck during the sequence, he is ineligible for an assist, even with his critical contribution to the resulting goal.
Of course, these are obvious flaws, but for all its flaws, this mechanical system of awarding points has generally worked pretty well to help fans identify those players most responsible for goals being scored.
The Roth- Irvin System
The search for the Holy Grail of hockey stats — a useful metric to measure two-way play, not just offence but also defence — heated up in the 1940s, when Montreal tie salesman Allan Roth approached Dick Irvin, the Hall of Fame coach of the heralded Montreal Canadiens, about compiling a few stats during games.
It’s not known exactly who thought up what, but Roth started to provide Irvin with reports where he had counted up such things as faceoff wins and losses, hits and whether or not Montreal stars like Rocket Richard, Toe Blake and Butch Bouchard were on the ice for goals for and against Les Habitants.
The Roth-Irvin plus/minus system was born, with a plus mark given to every player present on the ice for an even strength goal for and a minus mark going to each player out for a goal against. It was this plus-minus metric that represents the first major stab at coming up with that most precious and elusive of all hockey stats, a metric that fairly and accurately measures defensive play as well as goals and assists measure offensive play.
Even in a sport as static and orderly as baseball, it’s hard to measure a player’s defensive play, but this difficulty is greatly multiplied in a more chaotic, flowing sport based on intricate teamwork such as hockey.
“Defense is inherently harder to measure,” said stats expert Bill James, the prime mover of baseball’s Moneyball movement and a speaker at the MIT Sports Sports Analytics conference last year. “And this is true in any sport. In any sport the defensive statistics are more primitive than the offensive statistics. It’s not just sports. It’s true in life. It would be true in warfare and true in love.”
This does not mean it’s impossible to get useful defensive stats in hockey, James said, but analysts in search of them have to be more intelligent and disciplined in their analysis.
Hockey has had a few such sharp thinkers, starting with Irvin and Roth, who would go to become the official scorer for the Montreal Royals Triple A baseball team, then moved to Brooklyn in 1947 to become one of baseball’s stats gurus. When Roth was with Dodgers, the team developed a taste for keeping track of pitch counts and focused on a batter’s power and on-base percentage, not on his batting average.
Meanwhile, his work continued in Montreal, with Irvin hiring his son Dick Jr, who went on to be a well-known Hockey Night in Canada host, to do stats work for the 1951 season.
“They weren’t official,” Irvin Jr. has said of the stats he counted. “They never went to the league. It was for his own benefit.
“He just thought there were things he should know about the game, so he could sit down after the game and look at who won and lost the face-offs, who was doing the bodychecks and who was on the ice when the goals were scored. When you’re a coach behind the bench you can’t remember sometimes.”
With the development of the Roth-Irvin plus-minus, for the first time if a player was out on the ice for numerous goals against, a number reflected that and might indicate the player was a defensive liability.
In 1967-68, the NHL made the Roth/Irvin plus-minus an official stat. Over a player’s career, this plus-minus number is generally accepted to be a useful indicator of his two-way prowess. The top five all-time are Hall-of-Famers Larry Robinson, +730, Bobby Orr, +597, Raymond Bourque, +528, Wayne Gretzky, +518 and Bobby Clarke, +506.
But when you look at the stat in single season chunks, the stat presents major problems, most crucially that of quality of teammate, the impact consistently good or bad teammates will invariably have on a Roth-Irvin plus-minus number.
For example, here are two lists of 1970s and 1980s-era National Hockey League players, the players on both lists having something in common.
1. Andre Dupont. Mike Krushelnyski. Dallas Smith. Brian Engblom.
2. Bobby Clarke. Wayne Gretzky. Bobby Orr. Larry Robinson.
The first group of players were just average players. The second group has some of the greatest NHL stars from the past 40 years. But all of these players, from ‘Moose’ Dupont to ‘Big Bird’ Robinson, are on the list of top 100 single-season plus/minus players in NHL history.
Orr had the single best year when it comes to plus-minus, with a stupendous plus-124 rating in 1970-71. Not far behind him, though, was his defence partner from that season, Dallas Smith, who scored the fourth highest plus-minus ever at plus-94.
In other words, Smith, a decent player, got lucky enough to put up as superstar plus-minus number, just as Dupont did in his successful plus/minus year playing on the Flyers with Bobby Clarke, Bill Barber and others, as Oilers stalwart Mike Krushelnyski did playing with Wayne Gretzky and Jari Kurri, and as Montreal Canadien Engblom did playing with the likes of Robinson.
In the 1970s, it quickly became apparent to close observers of the game that if a player is so lucky as to be teamed up with good players, his plus/minus goes up. If he’s with bad players, it goes down. Engblom admitted as much in an article he wrote for ESPN.com. “We had a strong team (in Montreal), and I led the league in 1980-81 with a plus-63, followed by a plus-78 the following year. I was always the beneficiary of a team that possessed the puck and scored goals.
“Playing on a good team or a bad team impacts a player’s plus/minus ratings in a big way. I don’t care if you are a Hall of Fame player.
“The best players on a bad team will really get hurt.”
The criticisms of the Roth-Irvin plus-minus continue to this day.
“What can I say about (plus/minus)?” Montreal forward Alexei Kovalev once said to ESPN.com. “If I make a good play and we don’t score, then one of our other players makes a mistake and the other team scores, I get a minus.”
In his article, Engblom noted how a player, Jon Klemm, had been minus-3 one game, but it might not mean he had a bad game. “Perhaps none of the goals scored during Klemm’s shift came on his side, or perhaps all of them did.”
The final word here goes to former Bruins defender Don Sweeney, who finished his career at plus-112. In Stan Fischler’s book, Boston Bruins: Greatest Moments and Players, Sweeney said, “The plus/minus can be a very misleading statistic. You can be totally uninvolved in a play, but there’s a breakdown some place else and the puck is in your net. Then again, you could be the sole cause of a goal against your team, but everyone else on the ice with you has to share it.”
Neilson’s technological and conceptual breakthrough
The first to come up with a solution to the quality-of-teammate quandary was Hall of Fame coach Roger Neilson in the 1970s. Through the use of a new technology — video analysis, one not available to either Roth or Irvin in the 1940s — Neilson started to closely study individual contributions to goals and scoring chances, dissecting plays to figure out which players were at the heart of hockey’s most important on-ice sequences.
Neilson, a former high school math teacher, wanted to develop a superior plus/minus system to rate players. After every game, he or one of his assistant coaches would go over videotape of the game, identifying all the scoring chances (hard shots from the slot that hit the net), then giving out plus marks to those players — and only those players — who contributed to scoring chances, as well as minus marks to only those players who made mistakes on goals against.
Neilson’s system of rating goals and scoring chances has since been used by numerous NHL teams, and I’ve employed it in my work at The Cult of Hockey.
In the past five years, I’ve studied 1,556 goals for and against the Edmonton Oilers when the team has been at even strength, as defined by the NHL’s official plus-minus system. It uses the methodology designed by Roth and Irvin in the 1940s, and hands out marks on all even strength situations, as well as plus marks to players who score short-handed against five opposing players, and minus marks to the five power play culprits.
Five Year Findings
Edmonton Oilers 2008-13
Year Official +/- Neilson +/- Percentage
2008-09 1673 974 58%
2009-10 1821 1064 59%
2010-11 1659 999 60%
2011-12 1644 1006 61%
2013 956 544 57%
Part 1: Goals for
Number of goals for the Edmonton Oilers 2008-2013: 713
Plus marks awarded to position players under official NHL system on 713 goals for: 3,525, 100 per cent of players on the ice.
Plus marks awarded to position players under Neilson system on 713 goals for: 2,504, or 71 per cent of players.
Finding #1
On goals for, 71 per cent of plus marks are handed out fairly and accurately under the official NHL system, with 29 per cent false positives.
Part 2: Goals Against
Number of goals against the Edmonton Oilers 2008-2013: 843
Minus marks awarded to position players under NHL’s official system on 843 goals against: 4,228, 100 per cent of players on the ice.
Minus marks awarded to position players (non-goalies) under Neilson system on 843 goals against: 2,083, or 49 per cent of the players on the ice.
Finding #2
On goals against, 49 per cent of minus marks are handed out fairly and accurately under the official NHL system, but 51 per cent are false negatives.
Part 3: Total Goals
Total number of goals scored for and against the Edmonton Oilers 2008-13: 1,556
Total number of plus and minus marks handed out under NHL’s official plus-minus system on 1,556 goals: 7753
Total number of plus and minus marks handed out under Neilson system on 1,556 goals: 4,587
Finding #3
On goals for and against, 59 per cent of plus and minus marks were handed out fairly and accurately under the official NHL system, 41 per cent were not.
Discussion
The Neilson system seeks to correct the flaw fatal flaw inherent in the mechanical official system. It doesn’t assume collective responsibility for goals or scoring chances against, but endeavours to identify individual responsibility.
It does so by introducing human judgement into the process. The evaluator uses his hockey knowledge to determine who deserves a plus and who deserves a minus. In my case, I’m a passionate amateur and I use a set criteria to make my decisions. At the same time, I am not an NHL coach. I have no doubt that the assignments given by an NHL coach on these goals for and against would be superior to my own assignments.
My own understanding of the game and the Neilson system has been assisted by discussions with such hockey experts as Clare Drake, formerly of the University of Alberta, Billy Moores of the Oilers, Dave King of the Phoenix Coyotes, Craig Ramsay of the Florida Panthers, and Ron Smith, who was Neilson’s right-handed man for many years.
It’s worth noting that on most goals, one or two players will deserve full marks, while one or two others deserve only partial marks. I error on handing out a plus or a minus mark on a goal even if a player’s contribution to the goal is relatively small. This may inflate plus and minus totals of player somewhat, but it strongly suggests that if you don’t get a mark under the Neilson system, you have had no involvement in the scoring sequence at all. You have really earned that zero. That’s what this particular system is designed to do.
In the five seasons, the Neilson system has found that 58, 59, 60, 61 and 57 per cent of the plus and minus marks assigned in the Roth-Irvin system were warranted.
Why did the percentage go up? In the middle two-seasons, my Cult of Hockey colleague Bruce McCurdy of Oilers Analytics would often, though not always, provide a second opinion on which players deserved plus and minus marks on goals. Most often, I agreed with Bruce’s assessments, and this saw me add a plus or minus marks, which boosted the overall totals in those two years. The thoroughness of this approach suggests we missed little when it came to identifying all possible players involved in those goals.
In the first two years of this study, and during this past year, I did the work on my own. Without this second set of eyes, no doubt I missed a handful of plus or minus marks.
In this regard, the best practice for Neilson’s system would be to have multiple raters using the same criteria to rate each goal.
Three raters will certainly provide more robust data than one rater, with a plus or minus mark only being assigned if two out of the three raters agree it’s deserved. If you had three raters looking at each goal scored — and if you employed a system where you’d need agreement from two raters to hand out a plus or minus mark — it’s my best estimate that you would end up in the range of 58-to-60 per cent confirmation of the official plus-minus assignments.
Of course, this practice goes to what is a best practice for using Neilson’s approach. There’s little doubt that work needs to be done in perfecting the way Neilson assessed hockey. However, just because some honing needs to be done, it doesn’t mean that the Roth-Irvin numbers is any more valid. Whether it assigns false positive or false negative plus and minus marks just 35 per cent of the time or 45 per cent of the time, this is an uncomfortably large number of inaccurate readings for the NHL’s official system.
