Predictor vs. control metrics
For some time I have been looking for a good example to illustrate the difference between predicting some software attribute and controlling that attribute. A recent study comparing two methods of predicting a person’s height looks like it might be what I am after.
The study compared the accuracy of using information derived from a person’s genes to information on their parents’ height.
If a person is not malnourished or seriously ill while growing, then their genes control how tall they will finish up as an adult. Tinker with the appropriate genes while a person is growing and their final height will be different; tinker after they have reached adulthood and height is unlikely to change,
Around 150 years ago Francis Galton developed a statistical technique that enabled a person’s final adult height to be estimated from their parent’s height. Parent height is a predictor, not a controller, of their children’s height.
A few metrics have been found empirically to correlate with faults in code and a larger number, having no empirical evidence of a correlation, have been proposed as fault ‘indicators’. Unfortunately many software developers and most managers are blind to the saying “correlation does not imply causation”.
I’m sure readers would agree that once a baby is born changes to their parents height has no effect on their final adult height. Following this line of argument we would not expect changing source code to reduce the number of faults predicted by some metric to actually reduce the number of faults (ok, fiddling with the source might result in a fault being spotted and fixed).
The interesting part of the study was the finding that the prediction based on existing knowledge about the genetics of height explained 4-6% of subjects’ sex- and age-adjusted height variance, while Galton’s method based on parent height explained 40%. Today, 60 years after the structure of DNA was discovered our knowledge of what actually controls height is nowhere near as encompassing as an attribute that is purely a predictor.
Today, do we have have a good model of what actually happens when code is written? I think not.
How much time and money is being wasted changing software to improve its fit to predictor metrics rather than control metrics (I have little idea what any of these might be)? I dread to think.
Interesting. To go further, there is a still a lot medically that we just do not understand about the human body. Sure, we have been able to produce some medical knowledge based on cause-effect – take this drug, monitor effects, does it resolve the issue; but we have no clue why the drug/etc may work. In other words we’ve learned how to stimulate the response we want based on a known input without understanding why the stimulation work – kind of like knowing that if you hit your knee just right it will cause your lower leg to kick out without understanding how the reflex nerves/etc function to make it happen.
As you point out – we’re not that different in software. We know we can do certain things to resolve certain issues, but all too often we don’t delve into the problem deep enough to find the real underlying cause. So instead we end up with band-aid after band-aid.