When a sports writer asked hockey coach Mike Babcock about a particular hockey analytic, Babcock responded, “We don’t know if any of it’s true.” What did he mean by that?
The analytic in question was the Corsi statistic, and it has to do with the number of shots each team makes. It’s just a matter of counting shots; so it’s in no way mysterious. A higher Corsi number is presumed to be better. The metric is simple and objective; so how could Babcock question whether it was true?
The issue is not the statistic itself; the point of any analytic is that it’s meant to help with decisions. Should the coach make the decision to use players who have high Corsi numbers and not use the players with low ones? Babcock was suggesting that it’s not that simple and we that don’t know how good that statistic is at identifying who the best players are. He was pushing back against the simplistic notion that all we need to do is look at the data because it gives us the answer.
You don’t need to be a hockey fan to imagine there are all kinds of things that affect the number of shots that are not a result of the skill of the players. It depends who they are playing against; it depends on the strategy; it’s depends on who else is on the team. A good coach will look at all those factors; the Corsi score is a convenient piece of data; but doesn’t tell you who the best players are, it just points out one particular thing.
This observation that we can’t over rely on analytics would be so obvious that it’s not worth saying except for two things:
- Some people get so passionate about analytics that they think they capture everything about the world; they think they only need to look at the analytics to decide who to play.
- Some people, aware of the subtleties, get fed up with the numbers junkies and tend to dismiss the value of analytics as a way to inform judgement.
In most business situations, analytics are absolutely useful for informing judgement and absolutely not useful for replacing judgement. Babcock’s right, we don’t know how “true” the implication of a particular metric is and need to use judgement in decision-making.
Looking to the Future
The next interesting question is at what point, despite the subtleties, do analytics on their own lead to better decisions than human judgement informed by analytics? Sticking to hockey, a popular metric to use for goalies is “goals against average,” i.e. an average of how many goals they let in per game. It’s easy to see the limitations of that metric; if the team is poor, then even an excellent goalie will have a poor goals against average. However, sports analytics enthusiasts are taking the analysis much further and looking at the difficulty of each shot the goalie stops or doesn’t stop. Just to keep things simple, you might decide that looking at how many bad plays a goalie makes (letting in easy shots) and how many good plays a goalie makes (stopping a difficult shot) is all you need to know to decide what goalie to play.
This more advanced analysis requires some judgement around how to decide whether a shot is easy or difficult. It also leaves out some relevant factors such as whether the goalie has been playing so many games that they are overly tired. However the fact that the new level of analytics is imperfect is not the question; the question is whether it is good enough to be better than human judgment. In some business situations, like scheduling employees, an algorithm can often be better than a human, and letting the human add their own judgement just makes things worse.
So here are the takeaways for managers:
- Remember that it all comes down to what decision is being made; in this example about hockey stats we are making the decision on which players to use or which goalie to hire. The issue is how do we make that decision?
- The right way usually is to get what metrics you can and the use that to inform judgement. This is better than worshiping the metrics and it’s also better than ignoring them.
- However, there are times when analytics + human judgement will be worse than analytics on its own. We will find more such cases in the future as analytics gets better.
We are entering an age where “analytics savvy” means understanding the intersection of judgement and data. That can be a tough skill to learn.
∼∼∼∼∼
Special thanks to our community of practice for these insights. The community is a group of leading organizations that meets monthly to discuss analytics and evidence-based decision making in the real world. If you’re interested in moving down the path towards a more effective approach to people analytics, then email me at dcreelman@creelmanresearch.com