What Gets Measured Gets Counted
Thursday, 22 April 2010
On Wednesday, March 03, 2010, the ABC station in New York ran a report about a New York City police officer who went public about quotas. Apparently, the police are given specific targets to meet for arrests and summons. The complaint was that the quotas were being enforced blindly…so officers had no choice but to miss their numbers (and be disciplined) or just arrest people indiscriminately to keep their numbers up. (The original story.)
Well, are performance measurements and quotas bad? Large organizations need to manage by the numbers to keep things fair. Don’t they?
From a human performance perspective, there is actually a lot wrong with this approach, whether it is used in law enforcement or other businesses. For one thing, the numbers need to be connected to the desired performance and they need to be under the control of the performer. In this case, the measure doesn’t track the real desired results and it isn’t in the control of the performance because it doesn’t account for situational differences.
First of all, the measure is tracking activity, not results. The number of arrests look like results but it is more like measuring the number of proposals a sales person generates. Yes, there is a relationship between arrests and crime just as there is a relationship between the number of proposals and sales. But, what we are really looking for is a measure that tracks the amount of good arrests or, ultimately, the amount of actual crime.
Counting arrests is a problem because it assumes a constant volume of crime. To establish a required number of arrests for all police officers on all shifts in all areas implies that there is a stable amount of crime and the police can reasonably be expected to solve a certain amount of it. This may be approximately true over time (but probably not) but can’t possibly be true on a daily basis. The performer cannot control their performance on this metric. Unless they cheat.
One part of the bad news is that holding performers accountable for measures that they can’t control breeds cynicism and actually harms performance.
Imaginary dialog. (Italics/parens indicate what the individual is thinking but not saying.)
Sargent: Here is your quota. (I have a pretty good idea about what goes on in his area…I’m glad I don’t have to meet these targets.)
Officer: But we’ve been patrolling heavily and crime is down. I don’t think I can hit those goals, especially during the day. (Surely he knows this isn’t reasonable.)
Sargent: I don’t want to hear about it…just hit the numbers. (I have to maintain a firm hand as a leader. Besides, the people I answer to are so far removed from the daily problems of the beat officer that they won’t hear anything I say about the quotas either.)
Officer: Yes sir. (My only hope is to cheat.)
The result is wasted time, money, effort, and also injury to innocent people. With the side benefit of misleading statistics on record.
This sounds like lots of businesses actually…the farther away from the actual work you are, the less you will be able to understand the issues behind the numbers and the more likely you are to turn it into a clear cut, simplistic question. “Did you make the numbers? No? Then start making the numbers. I insist.” It highlights the importance and the sad lack of knowledgeable management. Maybe even worse is that there is management training out there that will teach you NOT to “take on your employees problems.” Which some people translate into meaning “don’t listen to any explanation or get dragged into troubleshooting.” Which is really not helpful and not managing either.
But back to the measurement question. What should they measure then?
I’m not an expert on law enforcement but if you are trying to measure police performance, how about measures that measure what you really want to improve that are also in the control of the performer? By the way, this isn’t easy but here are a few ideas.
- You really don’t need more arrests. You either want more convictions (as an indicator that the right person was arrested) or, ultimately, reduced crime, maybe based on reports or complaints by citizens. Some kind of ratio would be a good place to start.
- A ratio comparing arrests with convictions (or plea bargains) to indicate the quality of the arrests.
- Ratio of crimes reported to crimes solved.
- An index incorporating crime per capita, arrests (or convictions), complaints, and feedback from the public.
In a business, besides looking at the performance and setting measures based on indicators that the performer can legitimately control, it is also critical to incorporate knowledgeable managers who can understand the context of the performance and make allowances where appropriate. There is a point as you progress up the “food chain” though, where you lose touch with the day-to-day issues…or even to where the management never had the know-how in the first place. (For example, how many mayors or public commissioners are former police officers?) In those cases, the higher-ups need to listen to the people on the ground and develop enough trust to have rational discussions about performance.
If you want the measures to drive performance, it is critical to define them carefully and consider the possible unintended consequences because people will really try to make the numbers…even if it might be better if they didn’t.