Designing experiments is deceptively simple. After all, you know what’s going on, right? So you just design an experiment that manipulates or otherwise examines the variable of interest, with an appropriate control, then show the pattern you expected, write it up and publish. Easy.
Or not? What if you assume you are always wrong and just need to design your experiments to reveal the way in which you are wrong. What additional controls might you think of? What confounding variables might hard thought reveal? What if you did think about all those issues that might mess up your work right from the start? Could you ever do the experiment?
There is a hard balance between paralyzing yourself as you struggle to design the perfect experiment and doing the less than perfect experiment only to discover a comparatively easy control might have revealed certain problems a lot earlier.
We struggle with this balance. All too often an additional control becomes obvious only after an experiment is complete. This means we have to do it all over, or redo part of it. This is an issue that comes up a lot in experimental evolution, where density and genetic relatedness interact. It can come up in ecological experiments. It is important even in experiments that use the comparative method without actually manipulating living things.
I do not know the answer to efficiently designing the perfect experiments. I do know that most ideas are eventually proven wrong, incomplete, or irrelevant, so keeping that humbly in mind might help one develop a certain humility and caution in experiments. And never forget that the most difficult to explain results might point to the next big idea.
