The Problem With Experimentation in the Real World

Government policies—from teaching methods in schools to prison sentencing to taxation —would also benefit from more use of controlled experiments. This is where many people start to get squeamish. To become the subject of an experiment in something as critical or controversial as our children’s education or the incarceration of criminals feels like an affront to our sense of fairness and our strongly held belief in the right to be treated exactly the same as everybody else. After all, if there are separate experimental and control groups, then surely one of them must be losing out. Well, no, because we do not know in advance which group will be better off, which is precisely why we are conducting the experiment. Only when a potentially informative experiment is not conducted do true losers arise: all those future generations who stood to benefit from the results. The real reason people are uncomfortable is simply that we’re not used to seeing experiments conducted in these domains. After all, we willingly accept them in the much more serious context of clinical trials, which are literally matters of life and death.

Timo Hannay observes that we cannot experiment with classrooms and prisons because finding one experiment works means another didn't, creating winners and losers and offending our sense of justice.