4.1. The data generator–user gap

Here is my argument that in some contexts of data modeling, adequacy-for-purpose won’t be a good evaluative tool.

Premise 1 comes from the PR view straightforwardly. I will draw support for premise 2 from the PR view in addition to broader literature on model evaluation and adequacy considerations. The specification of purposes, for instance, is an important task which requires “significant unpacking and reflection”: “to judge a model’s adequacy-for-purpose, the evaluator has to understand what the purpose is and what would count as achieving it” (Parker Reference Parker2020, 461). I focus my critique on the claim that the purposes in data modeling contexts are particular. Bokulich and Parker write that the aims of data users are “typically rather specific and circumscribed” (Reference Bokulich and Parker2021, 10); Parker writes that “the purposes for which models are constructed and used are often rather circumscribed and local” (Reference Parker2020, 461). For theoretical models, this seems reasonable: “[B]y and large, the adequacy hypotheses put forward on behalf of models are highly local and highly specific. If they weren’t so specific,” writes Anna Alexandrova, “they would be either untestable or false” (Reference Alexandrova2010, 300). This feature of model evaluation—that purposes should be circumscribed, local, specific—is supposedly true in data modeling as well as theoretical modeling.

Why the purpose has to be circumscribed, specific, and/or local is evident through Bokulich and Parker’s use of a modeling problem space: a data model must stand “in a suitable relationship” with a target T, user U, method W, background circumstances B, and goal (or purpose) P jointly (Reference Bokulich and Parker2021, 11–12). On my reading, because the problem space is constrained by T, U, W, and B, the specified goal, the purpose P, must be circumscribed. A less circumscribed purpose wouldn’t jointly satisfy all the constraints in the problem space. The data model cannot be a solution unless the purpose P reflects the various constraints of the problem space. Intuitively, to evaluate a data model based on adequacy-for-purpose one must have some purpose in mind. Much more could be said about such purposes, but all I need to establish here (to support premise 2) is that purposes need to be particular, whether that be understood as being local, circumscribed, specific, etc.

Premise 3 is my main argumentative claim. I will defend it by arguing that a lack of specific/local/circumscribed purposes can happen when there is a gap or separation between data generators and data users. Thus, the source of the challenge to specifying particular data model purposes is what I will call the data generator–user gap. There are at least two ways to illustrate this gap, one which emphasizes scope and another which emphasizes iteration. I will use the problem space to demonstrate both senses of the data generator–user gap.

Sometimes the data generator–user gap is a gap in people. They might just be different groups of researchers. Microbiome science, for instance, is often called an observational field because so much of the research is “mere” description about which bugs are where and how many of them there are. One justification for funding such descriptive research is that others might use the data to achieve a downstream goal such as treating a disease. The data modeling problem space, then, might look like figure 1a. When those generating and processing the data are not the only users of that data, there can be a scope gap in their purposes. The purposes of the generator are more likely to be (but not always) global, general, and/or uncircumscribed (broader scope). This might be because they want the data to be usable for unknown others and unknown purposes. Downstream users usually have specific purposes (narrower scope). In figure 1a, the data generator must evaluate the data model. When there is a separation between the upstream data generator and the downstream data user, the generator must make decisions about data modeling without knowing the purpose(s) to which the data might be put. Call this the scope gap.

Figure 1.

a. Scope gap. A data generator may have a broader purpose during data modeling, as compared to the specific, yet unknown, purposes of downstream users. b. Iteration gap. When data modeling occurs in an iterative pipeline, purposes might be of similar scope yet still differ. B = circumstances, T = target, G = data generator, U = user, W = method, P = purpose, (D)M = data model.

Figure 1. Long description

Panel A: A flowchart depicting the scope gap in data modeling. A data generator with a broader purpose influences the data model, which is then used by multiple users with specific, yet unknown purposes. The flowchart includes labels for circumstances, target, data generator, user, method, purpose, and data model. Panel B: A flowchart illustrating the iteration gap in data modeling. Data modeling occurs in an iterative pipeline where purposes might be of similar scope yet still differ. The flowchart includes labels for circumstances, target, data generator, user, method, purpose, and data model, showing multiple iterations of data modeling and user purposes.

A second way to think about the data generator–user gap is in terms of data processing iteration (figure 1b). Bokulich and Parker note that the term ‘raw data’ is often used to contrast with data models or data products, and further that “in practice, such terminology frequently tracks not an absolute or intrinsic difference, but a relative one: datasets that are taken as input to a given study might be considered ‘raw’ data, even if they are the product of substantial prior processing” (Reference Bokulich and Parker2021, 8). As instruments themselves black box more and more computational processing, it makes sense that scientists tend to consider data to be raw whenever it is the input to their own contribution. Data generation pipelines may thus involve multiple iterations of data processing, where different researchers with different sets of expertise are responsible for sequentially modeling the data.

Here, the problem isn’t that data generators’ purposes aren’t specific or particular, but that they can differ across iterations. Earlier data modelers might be concerned with correcting for particular instrument biases, whereas later ones might want to adjust the data based on information gleaned from positive and negative experimental controls. Thus, when data modelers are part of an iterative pipeline, purposes might be of similarly narrow scope but still differ from each other. Modifications to data models or other decisions with regard to data model evaluation can then propagate down a pipeline, potentially becoming entrenched.

According to the PR view, McLaren, Willis, and Callahan should evaluate their data model with respect to a particular purpose. What might the purpose of the MWC model be? Let’s use the problem space. The target of the data model seems easy enough to specify. The target is just whatever microbial community was or will be sampled for a given experiment. As McLaren, Willis, and Callahan deal with a few specific datasets in their manuscript, the targets would be whatever communities those samples came from. More generally, though, McLaren, Willis, and Callahan are not concerned with any particular microbial community. Instead, they attempt to provide a computational, data modeling solution for a problem that impacts use of MGS pipelines in microbiome research more broadly. Considered from this viewpoint, the target could be cashed out more generally as (micro)biological communities! As long as a microbial community is amenable to MGS methods, it could be considered a potential target for the data model that McLaren and colleagues propose.

Now to method and circumstances. These are also relatively straightforward. In fact, McLaren, Willis, and Callahan build their model from mechanistic understanding of the methods used for sequencing pipelines and the circumstances typically encountered in such experiments. I will note here, though, that McLaren and colleagues don’t need to capture too much about how particular labs run their MGS protocols; instead, the level that they address is rather a model of the general measurement process.

Now to the user(s). My argument in this paper is about a data generator–user gap, and now is when that separation is evident. Consider the following question: In the problem space of my bioinformatics example, who are the users? The identity of the user constrains the specification of the purpose or goal. So, who are they? The most obvious answer is the data modelers themselves: McLaren, Willis, and Callahan. They are developing the data model, exploring its features by applying it to real data, and arguing for its merits. They need to evaluate the data model, in order to know whether and how to advocate for its use (and to defend its worthiness to be published).

McLaren, Willis, and Callahan, however, produced a data model that is meant to aid other users in correcting their own data for this commonly ignored bias. So, it seems like the purpose should not be specified narrowly as correction of any particular dataset. The goal should be specified more broadly, otherwise it cannot satisfy the user constraint. But if the users are McLaren, Willis, and Callahan, then the purpose within the problem space, the one which should be considered for evaluating adequacy for purpose, needs to be theirs. Their purpose, however, is not circumscribed to their own uses. Thus, a separation between data generators and data users can prevent data modelers (like McLaren, Willis, and Callahan) from evaluating their data model based on adequacy-for-purpose alone.

In sum, because the PR view advocates evaluating data models in terms of adequacy- or fitness-for-purpose, evaluation requires specifying a particular purpose. I used an example from microbiome sequencing bioinformatics to illustrate the difficulty of such specification. Within the problem space for the MWC data model, an overly narrow purpose fails to capture the stated motivation for the project, and a more broadly specified purpose will make the evaluative task inaccessible to upstream data model generators.

4.2. A tripartite disambiguation of ‘data model’

Questions about specifying the purpose of the MWC data model are sensitive to which entity is meant by the term ‘data model’. I will now use the bioinformatics example to develop some distinctions in the meaning of ‘data model’ which have gone underemphasized in the literature. Table 1 summarizes how the three roles may impact data model evaluation.

In McLaren, Willis, and Callahan’s manuscript, there seems to be three roles for data models. First, there is the data model qua model of the measurement process, illustrated in their manuscript as an idealized depiction of a sequencing experiment (McLaren et al. Reference McLaren, Willis and Callahan2019, 3). The second role is the data model qua representation of a data processing procedure. This is the formal model itself, ${\boldsymbol O}\; {\boldsymbol \sim} \;{\boldsymbol A}{\boldsymbol \cdot}{{\boldsymbol B}^{\left( {\boldsymbol P} \right)}}$ , capturing the way bias may impact measurements across vector abundances. Correcting for bias using this model is a process that is applicable across experiments and datasets (thus there are general and specific instantiations). The third role is the data model qua processed or corrected dataset. Clearly, McLaren, Willis, and Callahan—and others attempting to create and evaluate data models that can play the second role—cannot be expected to evaluate their models with respect to the purposes of any subsequent data models playing the third role.

Table 1.

Roles for data models versus features relevant for data model evaluation

When data models play roles 1 and 2, the generator–user gap is less problematic because these roles impact the design of the data processing procedure. Adequacy-for-purpose isn’t a very helpful guide for data model evaluation, though, because downstream uses of the processed data are unknown (or at least less relevant). If a data model is playing role 3, however, then at least some contexts will feature a gap between the data generator and the data user. This makes the PR view, with its adequacy-for-purpose account of data model evaluation, less helpful to the data generator because it fails to distinguish between data models as datasets and data models as elements of an upstream data modeling context.

If a philosophical view of data models is meant to provide evaluative guidance to researchers, my analysis suggests that the PR view falls a bit short. An account of data model evaluation that can be useful to researchers like McLaren, Willis, and Callahan may need a more nuanced philosophy of data modeling as opposed to simply borrowing from the existing literature on pragmatic evaluation of theoretical models. Such an account could clarify these terminological problems in addition to providing better guidance for data model evaluation.

With respect to better evaluation, here is one lesson from the bioinformatics example. There have been many efforts to assess microbiome methods, including sequencing (e.g. (Golob et al. Reference Golob, Margolis, Hoffman and Fredricks2017; Quince et al. Reference Quince, Walker, Simpson, Loman and Segata2017; Knight et al. Reference Knight, Vrbanac and Taylor2018; Sierra et al. Reference Sierra, Li and Pushalkar2020; Abellan-Schneyder et al. Reference Abellan-Schneyder, Matchado and Reitmeier2021)). The PR view suggests that having a better understanding of the purpose to which a dataset will be put could improve data model design. Here, though, I showed that less specific, less circumscribed purposes might also play a role. With respect to broader purposes, researchers might do well to facilitate more collaborations between those who do the bulk of the data generation (from sample collection through bioinformatic processing) and those who will be users of that data, as discussed by Sabina Leonelli (Reference Leonelli2016), for example. However, there are also benefits to be gained by better integrating teams within the data generation process itself. One of the strengths of the MWC data model is that it reflects, as the authors say, “properties of real experiments”. Instead of imposing a bioinformatic model with dozens of parameters decoupled from actual experimentation, the MWC data model links model parameters to stages in the sequencing measurement process (or, more precisely, to aspects of a model of that process). Without an understanding of the actual upstream measurement process, downstream computational folks cannot prioritize such parameter linking.

More communication between those working at the bench and those working in the cloud may allow each to adjust experimental design to account for measurement biases that accrue to distant stages in the pipeline. The MWC data model illustrates how bioinformatic data processing can correct for biases present in upstream steps of microbiome sequencing protocols, but it’s also possible that adjustments to experimental design in the lab could help support more robust bioinformatic analysis in data processing steps.