Wanna Play?

October 29, 2005

Can you design a management control that cannot “be gamed”?

Rate programmers on the number of bugs, and the number of bugs may fall, with minimal improvement in quality. Risk-taking and productivity will also drop. Also, a new pastime will be born: the post-mortems of bugs: “this one is not really a bug”, “this one existed before I touched the program”, “this one was a bug in the specification I received”.

Measures of productivity are even worse. They work okay when one is analysing the past, but if they’re applied to ongoing tasks, they’re easily gamed. Lines of code … too easily gamed to even talk about it. Even “function points” are gamed by “deflating” the content of a function point.

Wait… what is “gaming the system”? The essentials are:

  • Some measurement or calculation is used as a proxy measure of something else. (E.g. # of bugs == quality; or, function-points == productivity).
  • You “game” the system by doing great on the proxy measurement, but not on the underlying nebulous “thing”.

Simplistic contols don’t work: Is there a solution? Is it just about designing a good system?
Let’s take a non-software example. A factory makes shoes. We want to measure employee productivity. Simple enough — count the number of shoes produced per day.

We find that some are hurrying though shoes, so that they get “good numbers”. So, we only only count shoes that make it past quality control!

Wait… what if producing [50 passed and 1 failed] shoes takes a person more time as producing [51 passed and 2 failed] shoes? An employee with an eye on the numbers would prefer doing the second, because the only number being counted is 51 vs. 50. By speeding up just a little, the employee gets more failed shoes, but also more passed shoes.

Complex Controls are… complex: What if the cost of material for the one additional failed shoe is (say) $10 and the employee’s average wage per shoes is $5? Then, we’d rather an employee made 50 and wasted just one. So, we tweak our measure again. We need to adjust our measure by subtracting some “points” for each failed shoe.

Also, a shoe might pass QC with flying colors or it might just scrape through. If we assume that the firm prefers to produce better quality shoes, rather than just “passable” shoes, how do we factor that into our measure? We might tweak it again, by having different levels of quality.

Now, what if there are some employees who are gentle on the shoe-making equipment while others aren’t? The former will cost us less in the longer run. How do we compensate them for that?

Complex Controls don’t work ALONE: You could go on designing more complexity into the system of rewards, but that doesn’t always work either. Firstly, complexity and bureaucracy of the control system begins to impact the system being measured. Secondly, people continue to resent the unfairness of the (say) 2% error in the system. Also, the “de-skilling” of the control system is taken by some to be an invitation to give up thinking for themselves.

So, can there be a good system of controls? Yes, it is possible when work is objectively measurable. Where it isn’t, a “forced objectivity” is not the answer. There proxy measures are good for self-understanding, and that’s all. They are not overall measures. Much more important is to create a “sense of ownership” and doing what is right. This means laying out values and goals. It means communicating them clearly. Even the best measurement and control system works in a context of shared values with the bulk of employees not trying to game it.

Updates:  Here’s a good article about trying to measure programmer productivity.