How does evolution grow bigger brains? It has been widely assumed that growth of individual structures and functional systems in response to niche-specific cognitive challenges is the most plausible mechanism for brain expansion in mammals. Comparison of multiple regressions on allometric data for 131 mammalian species, however, suggests that for 9 of 11 brain structures taxonomic and body size factors are less important than covariance of these major structures with each other. Which structure grows biggest is largely predicted by a conserved order of neurogenesis that can be derived from the basic axial structure of the developing brain. This conserved order of neurogenesis predicts the relative scaling not only of gross brain regions like the isocortex or mesencephalon, but also the level of detail of individual thalamic nuclei. Special selection of particular areas for specific functions does occur, but it is a minor factor compared to the large-scale covariance of the whole brain. The idea that enlarged isocortex could be a “spandrel,” a by-product of structural constraints later adapted for various behaviors, contrasts with approaches to selection of particular brain regions for cognitively advanced uses, as is commonly assumed in the case of hominid brain evolution.