2.1. Linear regression

Regression analysis uses an intuitive idea to discover relationships in indeterminate data. Simply put, a linear regression takes a scatterplot and finds a line of best fit. In this way, indeterminate relationships are discovered between features represented on the $x$ and $y$ axes. Imagine, for example, that a scatterplot is constructed to plot a man’s height ( $y$ ) against the height of his father ( $x$ ).Footnote ¹ The relationship between $x$ and $y$ is not determinate, meaning that given the height of a man’s father, one cannot determine the height of the man. Knowing the father’s height, though, will allow one to make a better estimate of the son’s.

In this example of a simple linear regression, the father’s height is a predictor variable, while the son’s height is an outcome variable. The mathematical relationship between the predictor variables and the outcome variable (this is typically the variable that most interests the practitioner) is

(1) $${Y_i} = {\beta _0} + {\beta _1}{X_{i1}} + \ldots + {\beta _j}{X_{ij}} + {\epsilon _i},$$

where ${Y_i}$ is the outcome variable, ${X_{ij}}$ is the predictor variable, ${\beta _j}$ is a coefficient, and ${\epsilon _i}$ is the error or difference between the predicted value and the actual value. Given some data, the coefficients are estimated to minimize the difference between the model predictions and the observed values for the outcome variable. The value of the coefficient characterizes the relationship between the outcome variable and the predictor variable. For example, if a regression model predicts a man’s height given the height of his father, the coefficient represents the change in a man’s height per unit increase in the height of his father.

However, linear regression cannot model binary outcome variables like sex or mortality because it is a continuous linear function. Logistic regression was developed to address this limitation.

2.2. Logistic regression

The general strategy of logistic regression is to transform the data such that the outcome variable is the log of the odds of a particular outcome rather than a predicted value. For example, if mortality is the outcome variable of a logistic regression, the model yields the log of “odds of mortality” given the predictor variables rather than mortality directly. This is convenient because odds are continuous and so can be modeled with a continuous linear function.

In a logistic regression, the log of the odds (called a logit) is predicted rather than the odds. Whereas odds can range between 0 and $\infty $ , the log of odds ranges from $ - \infty $ to $\infty $ , so the regression model will always produce an outcome that is within range.

The logistic regression applies the following transformation to the outcome variable:

(2) $${\rm{logit}}\left( x \right) = {\rm{log}}\left( {{x \over {1 - x}}} \right).$$

In a logistic regression, it is stipulated that the regression equation estimates the transformed outcome variable rather than the outcome variable directly. The regression equation is then rewritten as

(3) $${\rm{logit}}\left( {P\left( {{Y_i} = 1} \right)} \right) = {\beta _0} + {\beta _1}{X_{i1}} + \ldots + {\beta _j}{X_{ij}},$$

where ${Y_i}$ is the binary outcome variable, ${X_{ij}}$ is the predictor variable, and ${\beta _j}$ is a coefficient. The coefficients again represent the relationship between the outcome variable and the predictor variable, although in logistic regression, they are interpreted as a change in log-odds of success given a one-unit change in the predictor variable. Unlike the linear regression, there is no error term in the mathematical form of the logit model. The indeterminacy of the model is instead captured in the probabilistic interpretation of the outcome variable.

To summarize, the logit model applies the regression framework to binary outcome variables by estimating the log-odds of an outcome rather than estimating the outcome directly.

3.1. Yule’s measure of association

George Udny Yule, (Reference Yule1900) developed his measure of association for binary variables based on change in odds. This measure is called Yule’s Q. Like correlation, it is a measure between $ - 1$ and 1 of the relationship between two variables. A measure of 0 indicates that the variables are independent, whereas a measure of $ - 1$ or 1 indicates that one can infer the value of one variable from the value of the other variable. The odds ratio (OR) scaled between $ - 1$ and 1 is Yule’s $Q$ :

(4) $$Q = {{{\rm{O}}R - 1} \over {{\rm{O}}R + 1}}.$$

Suppose we have a data set of 1,000 observations with two binary features: whether or not one has a PhD in philosophy (philosophy PhD) and whether or not one believes in the external world (external world belief).Footnote ² The data are organized in a table with four cells, such that each combination of values is accounted for (see table 1).

To calculate the measure of association between philosophy PhD and external world belief using Yule’s $Q$ , one must measure whether one is more or less likely to believe in the external world given that one has a PhD in philosophy compared to not having a PhD in philosophy. This is calculated and standardized as follows:

(5) $$Q = {{\left( {15} \right)\left( {10} \right) - \left( {970} \right)\left( 5 \right)} \over {\left( {15} \right)\left( {10} \right) + \left( {970} \right)\left( 5 \right)}} = - 0.94,$$

where $Q = - 0.94$ indicates that the binary variables are strongly negatively associated, meaning that if one has a philosophy PhD, the odds of not believing in the external world increase relative to the odds if one doesn’t have a philosophy PhD.

Yule’s $Q$ is based on the OR. The OR is a measure of how the odds of one variable change relative to the value of a second variable. For example, the OR for our toy example is

(6) $${\rm{OR}} = {{\left( {15/\left( {15 + 970} \right)} \right)/\left( {970/\left( {15 + 970} \right)} \right)} \over {\left( {5/\left( {5 + 10} \right)} \right)/\left( {10/\left( {5 + 10} \right)} \right)}}.$$

The numerator is the odds that ‘Philosophy PhD’ = 1 given that ‘External world’ = 1. The denominator is the odds that ‘Philosophy PhD’ = 1 given that ‘External world’ = 0. The odds ratio can be simplified to the ratio of the cross diagonals:

(7) $${\rm{OR}} = {{\left( {15} \right)\left( {10} \right)} \over {\left( {970} \right)\left( 5 \right)}}.$$

3.2. Pearson’s measure of association

Pearson’s measure of association between binary variables, on the other hand, is based on the assumption that there are underlying continuous variables that govern the observed binary variables. If, for example, the binary variable observed is mortality, some underlying continuous but unobserved variable (perhaps overall health) is assumed to account for the value of the binary variable mortality. At a certain level of the unobserved variable, the observed variable will change values. On the basis of the observed binary data, an underlying continuous distribution is fit that would give rise to the binary observations. The properties of the underlying distribution of each binary variable are used to construct a measure of association between $ - 1$ and 1 Pearson, (Reference Pearson1904).

This method depends on distributional assumptions, whereas Yule’s $Q$ does not.Footnote ³ Despite the fact that his measure introduces more assumptions, Pearson preferred this because it closely mirrors his measure of correlation between continuous variables. Pearson’s correlation coefficient for continuous variables assumes a bivariate normal distribution among the variables (for the most meaningful result). It measures the strength of a linear relationship among the data-generating distributions, not merely among the observed data.

Pearson has other reasons for preferring his measure over Yule’s. Pearson and Heron (Reference Pearson and Heron1913) argue that classifications that appear to be binary actually track continuous quantities. In the case of mortality, for example, they claim that binary data recording death or recovery “were used to measure a continuous quantity—the severity of the attack” (192).

3.3. Reflecting on the Yule/Pearson dispute

Yule was concerned that the additional distributional assumptions suggested by Pearson licensed inferences that went far beyond the observed data. He criticized Pearson in a Reference Yule1912 paper, stating that “apart altogether from the question of its legitimacy, the assumption [of an underlying continuous distribution] is unnecessary and unverifiable” (612). Given that Pearson’s underlying distributions could not be verified, Yule suggested that his own measure of association, based purely on the observed data, should be preferred.

Pearson replied, “If Mr. Yule’s views are accepted, irreparable damage will be done to the growth of modern statistical theory” (Pearson and Heron Reference Pearson and Heron1913, 158). Continuing his response, he states that “the controversy between us is much more important than an idle reader will at once comprehend. It is the old controversy of nominalism against realism. Mr. Yule is juggling with class-names as if they represented real entities, and his statistics are only a form of symbolic logic. No knowledge of a practical kind ever came out of these logical theories” (302).

While Yule is motivated by epistemic conservatism, Pearson argues that classification that does not rest on some underlying continuous variation is meaningless. Thus he accuses Yule of nominalism, while he sees himself as defending scientific realism. Pearson never directly replies to Yule’s epistemological point, however. Yule’s point is that even if binary outcomes are in fact grounded in underlying continuous attributes, the observed discrete data underdetermine them, and their existence cannot be verified.

The issues on display in this old dispute continue to plague the interpretation of statistical techniques in a new context—psychological measurement. The two competing interpretations of the logit model that I discuss in the next section descend from Yule’s and Pearson’s different measures of association. One interpretation of the logit model is epistemically conservative, whereas the other posits the existence of an underlying latent variable that determines the binary outcome variable. This interpretation underlies the use of the logit model to measure unobserved attributes in psychology and economics.

3.4. Two approaches to the logit model

The outputs of a logistic regression model are most straightforwardly interpreted as the log-odds of a particular outcome: ${\rm{log}}\left\{ {\left[ {P\left( {Y = 1} \right)\left] / \right[1 - P\left( {Y = 1} \right)} \right]} \right\}$ . This outcome can be easily converted to the odds of a particular outcome (by exponentiating) or a probability using the sigmoid function:

(8) $$s\left( x \right) = {1 \over {1 + {\rm{exp}}\left( { - x} \right)}}.$$

Although the logit model estimates, just like Yule’s $Q$ association measure, are computed from observed frequencies, there is a further question of how the output should be interpreted. Literally, the output of a logistic regression is a log-odds. But does the log-odds reflect an underlying continuous variable that decides category membership in the same way that Pearson claims that mortality reflects the severity of a disease? If so, then logistic regression can be seen as a model involving hidden variables—a latent variable model. Alternatively, does the log-odds summarize the expected long-run frequencies? Or does it represent a logical connection between the information obtained from predictor variables and the outcome variable? If so, the model is fundamentally probabilistic. The various approaches to the logit model discussed in this article differ with regard to how they answer the question of how to interpret the output of the logit model.

3.4.2 The latent variable approach

The latent variable approach to the logit model defines the outcome variable as a representation of an unobserved attribute that is not reflected in the data. Consider again the logit model that predicts the odds of a convicted criminal reoffending given the number of past convictions. Instead of interpreting the outcome variable as a function of the predictive probability, the latent variable approach legitimizes the interpretation of the outcome variable of the model as a distinct attribute that explains the observed frequencies, probabilities, and odds. For example, the latent variable could be recidivism, the tendency of a criminal to reoffend.

It is generally accepted that this approach to the logit model is a descendant of Pearson’s measure of association based on an underlying continuous variable (Agresti Reference Agresti2002; Breen, Karlson, and Holm Reference Breen, Karlson and Holm2018; Cox and Snell Reference Cox and Snell1989; Powers and Xie Reference Powers and Xie2008). A continuous underlying variable $Z$ is posited. At a particular threshold of $Z$ , the observed binary outcome changes from 1 to 0. If, for example, the observed variable is state of matter and the underlying variable is temperature, at a threshold ${32^ \circ }$ F, the observed variable changes because of the value of the underlying variable.

Mathematically, the latent variable approach to the logit model is expressed as follows. Suppose that $Z$ is a latent variable,

(9) $${Z_i} = {\beta _0} + {\beta _1}{x_{i1}} + \ldots + {\beta _j}{x_{ij}} + {\epsilon _i},$$

and

(10) $${\rm{i}}f{\rm{\;}}{Z_i} \ge 0,{Y_i} = 1$$

(11) $${\rm{i}}f{\rm{\;}}{Z_i} \lt 0,{Y_i} = 0.$$

The error term ${\epsilon _i}$ is assumed to follow a standard logistic distribution with a fixed variance. Because the model is not considered a probability model in this case, it is necessary to express the indeterminacy of the model in an error term. That the variance of the error is fixed will play an important role in my subsequent argument that the outcome of a logit model cannot be interpreted as a measurement of a latent variable. From this distributional assumption, it follows that

(12) $${\text logit}\left( {P\left( {Y = 1} \right)} \right) = {\beta _0} + {\beta _1}{x_{1i}} + \ldots + {\beta _j}{x_{ji}}.$$

Note that unlike the transformational approach, the latent variable approach includes an error term. The outcome variable in the latent variable approach is not considered to be fundamentally probabilistic; rather, it is taken to represent the value of a random variable that is unobserved. Thus the uncertainty is not captured in the outcome variable of the model. To capture the uncertainty inherent in a regression model, then, an error term is tacked on. The assumption that the error term follows a standard logistic distribution defines the relationship between the logit and the unobserved random variable.

The central question of this article is, do statistical models yield measurements? In the social sciences, statistical models are sometimes treated as measurement instruments, meaning that the output of these models are measurements. The latent variable approach to logistic regression is central to the use of statistical models as measurement instruments in the social sciences. According to the latent variable approach, underlying attributes can be represented by statistical models of observed binary data. In fields that aim to study unobservables like psychological traits or economic value, this interpretation of the logit model suggests a way to access the attributes being studied (Everitt Reference Everitt1984; Reise, Ainseoth, and Haviland Reference Reise, Ainseoth and Haviland2005).

Discrete choice theory in economics, for example, makes use of the latent variable interpretation of the logit model to measure attributes of individuals that compel them to make particular choices. Daniel McFadden (Reference McFadden1986, 275), the founder of discrete choice theory, describes it as a way to “model explicitly the cognitive mechanisms in the black box that govern behavior.”

The latent variable interpretation of the logit model also plays a central role in psychometrics, the study of the measurement of psychological traits (Borsboom, Mellenbergh, and Van Heerden Reference Borsboom, Mellenbergh and Van Heerden2003; Borgstede and Eggert Reference Borgstede and Eggert2023; Hood Reference Hood2013). The Rasch model is a psychometric model that is a special case of the two-parameter logistic model (Anderson Reference Anderson, Millsap and Maydeu-Olivares2009; Rasch Reference Rasch1960). Statistician Carolyn Anderson (Reference Anderson, Millsap and Maydeu-Olivares2009, 325) describes the psychometric framework as assuming “a latent continuum” that “may be utility, preference, skill, achievement, ability, excellence, or some other construct” underlying binary test data.

4.1. The comparability constraint

Some procedures can be ruled out as measurement procedures from the get-go. We can, for example, rule out the following physical procedure for measuring one’s height: Take the nearest physical object (no need to report what it is) and count how many times this object can lie end to end from head to toe. Report the number of times. The following inferential procedure for measurement can be ruled out too: Take an instrument indication and multiply it by a random number between 1 and 10. Report that magnitude as your result.

Why do these procedures fail to yield measurements? In each case, the procedure violates a criterion I call comparability. Consider the following: if you report your height to be ten using the “nearest physical object” method, and I report mine to be nine, I am still in the dark about the empirical relationship between my height and yours. Comparability is the preservation of empirical relations between quantities in the numerical outcome of measurement. In the case of height, the empirical relations “shorter than” and “taller than” should be preserved by the measurement procedure for comparability.

In this section, I motivate the adoption of the comparability criterion for measurement: if a procedure violates comparability, then it is not a measurement. In other words, if no empirical relations between the quantities that are the subject of measurement are preserved in the numerical outcomes given by a procedure, then that procedure is not a measurement. Comparability, then, is a necessary condition of measurement.

The representational theory of measurement (Suppes and Zinnes Reference Suppes, Zinnes, Luce and Bush1963; Krantz, Suppes, and Luce Reference Krantz, Suppes and Luce1989; Luce et al. Reference Luce, Krantz, Suppes and Tversky1990; Luce and Suppes Reference Luce, Suppes, Luce and Bush2002) proves to be a useful formalization for my purposes—comparability can be seen to follow from the use of a representational theorem to map empirical quantities to numerical scales that preserve order.Footnote ⁶ According to the representational theory, there are two central problems for measurement: (1) what is the representation theorem that maps empirical relations to numerical relations (finding a representation theorem)? and (2) given a representational theorem, what transformations preserve the relations between the representations (finding a uniqueness theorem)?

The representation theorem maps empirical relations to numerical values. For example, a representation theorem for assigning numbers to lengths of rods will map each rod to the real numbers, the operation of concatenation (the action of linking things together) to addition, and the comparison of physical rods (shorter than, longer than) to the comparison of real numbers (less than, greater than).

The uniqueness theorem defines the transformations that preserve the relations given by the representation theorem. Some scales permit more meaningful transformations than others. For example, a ratio scale can be multiplied and divided, whereas an interval scale cannot be. A ratio scale is a scale with a true zero, such as length. The zero point on the scale indicates an actual absence of length. I can compare length multiplicatively—it is meaningful to claim that the distance from my house to the university is twice the distance from my house to the grocery store. This holds true regardless of whether I measure the distance in meters or yards. Interval scales have an arbitrary zero, such as temperature. Temperatures cannot be compared multiplicatively—it is not meaningful to claim that it is twice as hot today as yesterday, because this relation is not preserved across scales with different zero points (e.g., Fahrenheit vs. Celsius). Ordinal scales indicate order but violate additivity. Ratio and interval scales distribute numerals evenly, such that there is the same distance between 1 and 2 and 3 and 4. Ordinal scales make no such guarantee as to the distance between each numeral, and so additivity is violated.

The representation theorem preserves empirical relations present in the target system: suppose the representation theorem maps the empirical relations of shorter than ( $ \prec $ ) and longer than ( $ \succ $ ) to the numerical relations $ \lt $ and $ \gt $ ; then, the representation theorem must be such that if $a \prec b$ , then $R$ is a representation theorem only if $R\left( { \prec, a,b} \right) \to A \lt B$ . The empirical relations preserved over and above mere ordering (like additivity) determine to which type of scale the empirical relations are mapped.

Comparability is implied when empirical relations are mapped to any scale that preserves order: ordinal, interval, or ratio.Footnote ⁷ Note that comparability is not a formal property of a scale—instead, it is a property of measurement outcomes that is guaranteed by the successful mapping of empirical relations to an ordinal, interval, or ratio scale.

Recall the nonmeasurement procedures described earlier—the “nearest physical object” method of measuring height, for example. What goes wrong in this case is that the “representation theorem” employed fails to preserve any empirical relations in the mapping to numerical relations. Because two individuals who use this method will have different units for measure (depending on the physical object nearest to them), it is not guaranteed that given $a \prec b$ , $R\left( { \prec, a,b} \right) \to A \lt B$ .

What it is to measure is to map empirical relations onto an ordinal, interval, or ratio scale. A measurement procedure performs that mapping while preserving the empirical relations present in the quantity of interest. If what it is to measure is to map empirical relations onto an ordinal, interval, or ratio scale while preserving the empirical relations that are present in the quantity of interest, then comparability is entailed by measurement.Footnote ⁸ A lack of comparability in measurement outcomes, then, indicates that measurement has not taken place.

4.2. Comparability and the foundationalism debates

An attractive feature of the comparability criterion that I propose is that it is relativizable to a theory and thus can be adopted by both foundationalists and antifoundationalists about measurement without challenge to their respective view.

Foundationalism about measurement is the view that measurements reflect pretheoretical observations that form the foundation for scientific theorizing (Thalos Reference Thalos2023, 1). Antifoundationalism about measurement is the view that measurements are dependent on theoretical assumptions (Chang Reference Chang2004; Tal Reference Tal2012; van Fraassen Reference van Fraassen2012). Contemporary discussion of psychological measurement is often focused on this debate (see Michell Reference Michell1993, Reference Michell1997, Reference Michell1999, Reference Michell2005; Hood Reference Hood2013). In this article, though, I offer a critique of a method employed in psychological measurement that sidesteps this contentious issue.

Foundationalism is consistent with comparability as a criterion for measurement because the foundationalist is committed to the view that the use of ordered scales in measurement reflects real quantitative properties possessed by the measurement subject. Given this, it is necessary for the measurement procedure, according to the foundationalist, to accurately map those empirical relations to numerical relations. I have argued that comparability follows from this view.

Antifoundationalist accounts initially appear to be in greater tension with the adoption of the comparability criterion than foundationalist accounts are. According to antifoundationalist views like the model-based theory of measurement advanced by Eran Tal, measurements are made relative to an idealized model of the measurement process. This model formalizes background theoretical assumptions and statistical assumptions about error. For example, in measuring temperature, the measurement depends upon broad background theoretical assumptions (a physical theory of temperature) and also statistical assumptions about the error expected given slight variations in the conditions under which the measurement takes place (Tal Reference Tal2019, 864). Measurement, then, is theory laden, meaning it cannot be done independently of assumptions about the quantity being measured and the process by which the quantity is measured.

There are constraints on which models yield measurement outcomes: measurements must be based on models that are coherent and that produce outcomes that are “consistent with other measurements of the same or related general quantity types performed under different conditions” (Tal Reference Tal2019, 871). A model is coherent when it coheres with the physical theories that are presupposed prior to measurement. A model is consistent when outcomes of different measurement procedures agree within an acceptable margin of error. Consistency of a model establishes the objectivity of a measurement, whereby a measurement is objective when it is “about measured objects rather than about features of the measurement operation” (Tal Reference Tal2019, 866). Tal goes on to state that “without this minimal sort of objectivity, measurement itself is impossible” (866).

Objectivity is model relative, however, meaning that objective measurements can be made only after some initial background assumptions have been accepted. For example, the kinetic theory of matter serves as a theoretical background that justifies the use of thermometers to measure temperature. Our measurements of temperature tell us about the measured object in the world, but always with respect to our background theories.

Like objectivity, comparability can be model relativized in the following way: model-relative comparability is the preservation of empirical relations between quantities in the numerical outcome of measurement given the relevant background theories. So, even when measurements cannot be taken independently of theoretical assumptions, measurements can be deemed comparable given the theoretical assumptions that ground the possibility of measurement.

4.3. Logit model measurement

In this section, I describe how the logit model is used for measurement in psychometrics. Then, I argue that the logit model is not a genuine measurement instrument of a latent trait because it violates the comparability criterion for measurement argued for earlier.

4.3.1. The Rasch model

The latent variable approach to the logit model is used in psychometrics to measure unobservable psychological traits. The Rasch model (developed by Georg Rasch), for example, is a special case of the logit model that is used to measure ability or some other latent trait given dichotomous test data.

The Rasch model takes the probability of correctly answering a question as a function of difficulty and ability (or some other latent trait); that is,

(13) $$P\left( {\text Correct} \right) = {1 \over {1 + {e^{ - \left( {\text ability\ - \ \text difficulty} \right)}}}}.$$

This expression is equivalent to a logistic regression with two predictor variables, ability and difficulty. In the context of a Rasch model, the ability coefficient is interpreted as a measure of a latent trait.

In what follows, I argue that the logit model lacks comparability and thus cannot be used for measurement. This would challenge the use of logit models like the Rasch model for psychological measurement.