Hypothesis generation

If this were a textbook of statistics then this chapter would not exist. Modern statistics is almost entirely concerned with testing hypotheses, not developing them. Such a bureaucratic approach views science as a compartmentalised activity in which hypotheses are constructed by one group, data are collected by another group and then the statistician confronts the hypothesis with the data. Since this book is a user's guide to causal modelling, such a compartmentalised approach will not do. One of the main challenges faced by the practising biologist is not in testing causal hypotheses but in developing causal hypotheses worth testing.

If this were a book about the philosophy of science then this chapter might not exist either. The philosophy of science mostly deals with questions such as: how can we know if a scientific hypothesis is true or not? What demarcates a scientific hypothesis from a non-scientific hypothesis? For most philosophers of science, the question of how one looks for a useful scientific hypothesis in the first place is someone else's problem. For instance, Karl Popper, in his influential Logic of Scientific Discovery (Popper 1980: 32), says that ‘there is no such thing as a logical method of having new ideas, or a logical reconstruction of this process. My view may be expressed by saying that every discovery contains “an irrational element”, or “a creative intuition”…’ Later, he says that ‘[scientific laws] can only be reached by intuition, based on something like an intellectual love of the objects of experience’. Again, one gets the impression that science consists to two hermetically sealed compartments. One compartment, labelled hypothesis generation, consists of an irrational fog of thoughts and ideas, devoid of method, out of which a few gifted people are able to extract brilliant insights. The other compartment, labelled hypothesis testing, is the public face of science. Here, one finds method and logic, in which established rules govern how observations are to be taken, statistically manipulated and interpreted.

At a purely analytic level there is much to be gained by taking this schizophrenic view of the scientific process. After all, how a scientific idea is developed is irrelevant to its truth. For instance, the history of science documents many important ideas whose genesis was bizarre.