Reference Birnbaum and WanBirnbaum and Wan (2020) proposed an extension of basic true and error (TE) theory: MARkov True and ERror (MARTER) theory, in which parameters of a risky decision making model change gradually over time according to a random walk, generating different true preference patterns at different times. They presented a computer program that can be used to simulate data according to this model and showed that the simulated data can be properly analyzed by computer programs that fit the TE models.

They also showed that TE analysis can correctly identify whether the generating model had satisfied transitivity or if it violated transitivity, and that methods proposed by others to test properties of binary response probabilities, such as testing Weak Stochastic Transitivity and the Triangle Inequality, for example, as in Reference Regenwetter, Dana and Davis-StoberRegenwetter, et al. (2011) were not capable of correctly distinguishing whether a transitive or intransitive generating model had been used to simulate the data. When the data were generated from a MARTER model, the TE analysis correctly detected the generating model and gave accurate estimates of the parameters used to generate the data, even in cases where other methods of analysis failed to correctly identify the process.

It is often said that all models are false but some models can still be useful. A magnetic compass does not point exactly True North, and flat maps are only approximations of the curved earth’s surface, and yet these models proved useful in navigation. Signal Detection theory, factor analysis, and analysis of variance are similar quantitative models that are regarded as “robust”. That is, they are regarded as useful for answering substantive questions about psychological theory even when their assumptions are only approximations.

In this follow-up to Reference Birnbaum and WanBirnbaum and Wan (2020), we conducted a series of simulations to investigate what happens if we generate data with violations of the MARTER model and then use a TE model that is only an approximation of the generating process. The computer program of Reference Birnbaum and WanBirnbaum and Wan (2020), MARTER_sim.htm, can be used not only to simulate data from MARTER models, but it is also capable of generating data from stochastic processes that violate certain of the assumptions.

We generated data that either satisfied or violated two key TE assumptions. We then asked three related questions: First, would TE analysis correctly detect the violations, so that an investigator would at least be aware of the lack of fit of the model? Second, would violations of the assumptions lead to wrong conclusions regarding transitivity? Third, would violations lead to biased parameter estimates?

We explored violations of these two key assumptions: First, the TE fitting model assumes that people do not change their true preferences in the short time interval of a session; Second, the TE fitting model we used assumed that error rates are independent of true preferences.

We ran 59 simulated sets of data from a hypothetical study of transitivity of preference, each with 10,000 sessions of data with two replications per session, similar to those described in Reference Birnbaum and WanBirnbaum and Wan (2020). Half of the simulated data sets were generated from a transitive process and half from an intransitive process. Half of the data sets were generated from process that satisfied the TE fitting model assumptions, and half violated those assumptions. Further, different stochastic decision models were used and different combinations of error rates were combined with those.

In all of the cases generated by MARTER models (that satisfied the TE assumptions, i.e., “control” conditions), the TE fitting model did an excellent job of identifying the model and recovering the parameters that had been used to generate the data. The TE fitting model always correctly identified whether the generating model was transitive or intransitive. These “control” conditions thus replicate and extend the findings reported by Reference Birnbaum and WanBirnbaum and Wan (2020) to new examples.

In all of the cases generated with violations of the TE fitting model, the tests of fit were large and significant, showing that an investigator could, in principle, realize that there was a systematic violation of the fitting model. That is, the model analysis does not “sweep under the rug” either of these types of violations.Footnote ¹

Despite systematic violations in these cases, the TE model correctly identified in every case whether the generating model had been transitive or intransitive. Because the main purpose of the hypothetical experiment was to test transitivity, this apparent robustness of the TE analysis is potentially the most important result of the simulations.

When the assumption that true preferences are stable within a session was violated, the estimates of incidence of preference patterns were surprisingly accurate, but the estimated error rates were inflated by the fact that true changes of preference were absorbed into the error component of the model.

In the cases where error rates were systematically different for different true preferences, it was found that the estimated incidences of preference patterns were decreased when the error rates associated with those patterns were greater, and relatively increased when the error rates were smaller for those patterns.

In summary, the TE fitting model appears to be quite robust with respect to these two sources of violations of the model. Although our conclusions must be restricted to the examples we have studied so far, we think that these conclusions are more generally applicable, but we are not yet able to provide analytic proofs, nor have we yet been able to deduce formulas to describe the limits (if any) of the conclusions that the TE fitting model is quite robust with respect to these two potential sources of violation.

Methods based on analysis of binary response proportions were unable to detect which datasets had been generated from transitive or intransitive models. All of the datasets would be declared by those methods to be compatible with transitive models, despite the fact that half of the datasets had been generated from intransitive models.

We have written a complete paper with the details of our simulations and results, which is available at the journal’s Website via the following link: http://journal.sjdm.org/20/200413b/supp.pdf