Hypothesis testing and analogical reasoning in experimental archaeology

The first step in the development of an archaeological experiment entails hypothesis creation. The method by which one generates ideas should be clearly stated, as the interpretation of an experiment's results is directly dependent on the authors’ underlying assumptions (Domínguez-Rodrigo Reference Domínguez-Rodrigo2008). The proposed hypothesis must be formulated in a falsifiable cause-and-effect scheme that consequently implies an opposing null hypothesis.

We follow the current best practice for archaeological experimentation proposed by scholars including Lin et al. (Reference Lin, Rezek and Dibble2017) and Eren et al. (Reference Eren, Lycett, Patten, Buchanan, Pargeter and O'Brien2016). To the extent that Holen et al.’s logic of hypothesis construction is transparent or reproducible, we find it to be problematic. Instead of setting up the experiment with two opposing hypotheses, the authors have conducted their experiments and summarily confirmed that the observed patterns match those from the Cerutti site. While the authors do acknowledge alternatives, they do not evaluate the other viable explanations for the observed patterns (see also Braje et al. Reference Braje, Dillehay, Kelly, Klein, Meltzer and Rick2017, and the rebuttal by Holen et al. Reference Holen, Deméré, Fisher, Fullagar, Paces, Jefferson, Beeton, Rountrey and Holen2018c). In order to suggest that the bones were, in fact, modified by humans, Holen and colleagues must also quantitatively compare lithic and faunal remains—modified via other taphonomic processes—against their experimental dataset.

Beyond hypothesis construction, it is critical to justify the analogues that scaffold the experiment. In their 2017 publication, the authors focus much of their attention on the use of a hafted hammer stone. The origin of this analogy should be explained—beyond a passing reference to late prehistoric Plains peoples (see Holen et al.’s Reference Holen, Deméré, Fisher, Fullagar, Paces, Jefferson, Beeton, Cerutti, Rountrey, Vescera and Holen2017 supplementary information 5, p. 22). The replication of patterns from the La Sena Mammoth site (see also Holen et al.’s Reference Holen, Deméré, Fisher, Fullagar, Paces, Jefferson, Beeton, Cerutti, Rountrey, Vescera and Holen2017 supplementary information 5, p. 22) represents an uncontextualised comparison that appears only tangentially related to the Cerutti Mastodon site—and nearly 100 000 years removed. The experiment should be set up so that the variables are as close as possible to the observation that one is attempting to replicate. In this case, the experiments presented are not only too variable and inconsistent for any meaningful assessment of hypotheses, but are also built on foundations that are insufficiently comparable to the archaeological context in question.

Experimental control

When designing and implementing an experiment, it is important to exert control over variables at a level justified according to the experimental design (for a discussion on experimental validity, see Mesoudi Reference Mesoudi2011, and for an illustration of its spectrum, see Eren et al. Reference Eren, Lycett, Patten, Buchanan, Pargeter and O'Brien2016). This ensures that the observed results reflect manipulation of those variables associated with the hypotheses, rather than the interaction of nuisance variables (sensu Lin et al. Reference Lin, Rezek and Dibble2017, but see also a discussion of the role of randomisation in mitigating nuisance variables on page 14 of the same manuscript). Based on a review of Holen et al.’s experimental design, we conclude that there was substantial variation not only in the raw materials used, but also the ways in which the experiments were executed; these variables detract substantially from the arguments made by the authors.

In reporting their study, Holen et al. (Reference Holen, Deméré, Fisher, Fullagar, Paces, Jefferson, Beeton, Cerutti, Rountrey, Vescera and Holen2017) list the raw materials and mass of individual tools from the Cerutti site, as well as their experimental hammer stones and anvils. While the individual archaeological hammer stones range in mass from 7.6–14.45kg (without refits; up to 18.25kg with refits), and are made of andesite and pegmatite, the experimental hammer stones range from 1.7–14.7kg and are of andesite or granite. Similar differences characterise the anvils. The variation in raw material, size and mass make it difficult to compare the results of the experiments with confidence. In turn, extrapolating to the archaeological record and the Cerutti assemblage becomes more tenuous.

Acknowledging the rarity of elephant remains with which to conduct such experiments, we recommend that scarce elephant bones not be used to confirm the hypothesis as presented by the authors, but rather function as benchmarks against which more refined experiments may be compared. For instance, after establishing whether or not the different sizes of hammer stones can be used effectively in the processing of large proboscidean bones, it would then be productive to establish a more robust experimental procedure using the bones of cattle or other large mammals.

Holen et al.’s four reported experiments were executed inconsistently. Of these experiments, two were conducted on elephant bones, while the other two were undertaken using cattle and/or kangaroo bones. In the first case, the elephant bones were propped up on a wooden block and struck with a hafted stone. In a second experiment conducted on an elephant bone, a larger hammer stone was used after the failure of another hafted implement. The specifics of the subsequent lines of experimentation, carried out with cattle and kangaroo bones, are less clear, and require more detailed reporting to allow meaningful evaluation or repetition. We suggest future iterations of this experimentation begin by employing both hammers and anvils of approximately the same dimensions, composition and size. These should be randomly assigned to treatment groups, in order to create results that may be compared with one another and with outgroups of assemblages created through the testing of alternative hypotheses.

Once the raw materials used in the experimentation are held constant, it would then be beneficial to ensure that the ways in which the raw materials are made to interact (e.g. hammer stone on bone impacts) are also systematised. In reviewing the highly controlled experiments of Dibble and Rezek (Reference Dibble and Rezek2009), and work by Magnani et al. (Reference Magnani, Rezek, Lin, Chan and Dibble2014), we understand that strict laboratory conditions are not always attainable, or desirable, in experimental archaeology. At a minimum, however, we suggest ensuring that the bones be struck and supported as consistently as possible. Ideally, the force would be regular and dealt from similar angles, a set number of times. The reported experiments make no apparent effort to control for any of these variables, casting doubt on the internal validity of the results (sensu Lin et al. Reference Lin, Rezek and Dibble2017).

Quantification of results

The experiments reported by Holen et al. (Reference Holen, Deméré, Fisher, Fullagar, Paces, Jefferson, Beeton, Cerutti, Rountrey, Vescera and Holen2017) are entirely qualitative, producing a good initial pilot study and the basis for designing further archaeological experiments, but limited inferential value (Lin et al. Reference Lin, Rezek and Dibble2017: 680). When designing replication experiments, quantification is a basic requirement to enable the comparison of excavated artefacts with experimental results using standard statistical methods (Eren et al. Reference Eren, Lycett, Patten, Buchanan, Pargeter and O'Brien2016: 108). Quantification allows researchers to assess alternative hypotheses explicitly. Qualitative visual comparisons are insufficient to draw meaningful conclusions regarding differences in the morphological characteristics of bone and stone tool assemblages. Fortunately, there are numerous options for Holen and colleagues to consider. For stone tool morphometrics, for example, Caruana et al. (Reference Caruana, Varvalho, Braun, Presnyakova, Haslam, Archer, Bobe and Harris2014) suggest the use of digital elevation models in ArcGIS to quantify the shape of pitting on stone artefacts ($\displaystyle{{perimeter^2} \over {volume}}$), in order to distinguish anthropogenic traces (whether ancient or modern) from natural characteristics of river cobbles. The authors may further wish to evaluate the morphologies of the bone flakes and impact notches from the Cerutti site, in relation to experimental datasets, employing both dimensional measurements standard in the assessment of anthropogenic bone modifications (Galán et al. Reference Galán, Rodríguez, Juana and Domínguez-Rodrigo2009), as well as more recently developed geometric morphometric techniques (Yravedra et al. Reference Yravedra, Aramendi, Maté-González, Courtenay and González-Aguilera2018). The use of any or all of these quantitative approaches would better enable the archaeological community to evaluate the authors’ findings.

Finally, quantitative measures depend on assumptions of a sufficiently large experimental sample to ensure an acceptable level of statistical confidence (Drennan Reference Drennan2009: 126). In order to assess the proper sample size needed for an experiment, it is standard across experimental fields for researchers to perform a power analysis: a measure of how well a particular statistical test can distinguish a difference between groups, if indeed there is one (Cohen Reference Cohen1992: 100). After establishing appropriate controls on the raw materials and other experimental parameters, we therefore recommend that the authors employ such an analysis, expanding their experimental sample size appropriately to facilitate meaningful evaluation of their data.