In psychology, the term ‘construct validity’ refers to a particular kind of validation problem that faces designers and users of psychological tests: how to establish that a test purporting to measure something unobservable is in fact measuring that thing? Originally introduced in a now famous 1955 paper by the psychologists Lee Cronbach and Paul Meehl, the meaning of construct validity and the extent of its impact on measurement validation practices are each the subject of extensive bodies of methodological and sociological analysis.Footnote 1 This paper complements these works by situating the development of construct validity in the expansion of US psychology in the first half of the twentieth century, focusing in particular on the concept’s origins in the contemporaneous development of the APA’s first code of ethics.
Conflicts over the measurement of unobservable things are not unique to psychology. Historical and sociological accounts describe the mutual influence between material structures of unobservability and the socio-technical problems that people must organize themselves around to explain or control the phenomenon. Distinctive social tensions arise, for example, if the phenomenon is extremely large or small (e.g. particle physics experiments), impractical to access physically (e.g. spacecraft navigation), extremely costly to reproduce (e.g. ballistic missile control), or unethical to test fully (e.g. dose response estimation).Footnote 2 A common explanatory strategy across these multiple forms of unobservability is to show how people accomplish understanding despite material limitations; researchers emphasize the formation of mutual agreements about what counts as evidence, and consider the various organizational and institutional tools that seem to be used for consensus formation. Yet another version of the problem of unobservables arises when the reality of the underlying phenomenon lies in significant doubt. In this type of case the accomplishment of mutual agreement has a distinct tenor. Unlike phenomena which are reasonably consensual but inaccessible to observation, the act of measuring entities or processes that are themselves of uncertain validity – trying to hit a target that may not be there – raises difficult ethical questions. Cases of outright fraud and fabrication clearly conflict with the moral prerogatives of scientists, but what about someone who is confused, or has been misled, or is simply mistaken? Is it a form of misconduct to be wrong about the kind or range of inferences a scientific instrument can support?Footnote 3 These questions often generate social conflict, particularly when they call into doubt the integrity of individual scientists (implicitly or otherwise).
The social complexity of this boundary between the ‘ethical’ and ‘technical’ dimensions of scientific conduct is usefully illuminated by the history of the expansion of psychological measurement in the United States. In the first half of the twentieth century, owing largely to its role in military personnel management around the First and Second World Wars, US psychology underwent a period of rapid expansion.Footnote 4 Alongside a significant increase in the number of academically trained psychologists, the psychological test questionnaire and its attendant scoring methods diffused rapidly across multiple institutional domains, and Americans were increasingly exposed to a vast new world of tests purporting to measure everything from opinion, to intellectual ability, to personality.Footnote 5 In the years following the Second World War, the still-growing use of tests was an increasingly salient issue for academic psychologists working to govern the scientific reputation and public standing of the psychological profession. This led to the formation of the first APA Committee on Scientific and Professional Ethics in 1947 and, six years later in 1953, the publication of the Ethical Standards of Psychologists, the profession’s first ethical code.Footnote 6
In the process of writing the ethics code, members of the committee observed that the growing use of psychological tests, particularly by relatively untrained practitioners, raised ‘technical’ questions about proper test use that they could not address directly. Although they recognized that the use of tests was not completely outside the domain of ethical questions, the specifically technical quality of tests made validation hard to place in existing normative models of professional conduct. For example, was it an ethical matter to unknowingly use a test incorrectly? Could tests reasonably be used for scientific purposes other than strictly validated measurement, such as generating new theoretical perspectives? Were researchers ethically obligated to follow the latest technical trends in test design and analysis? What examples were fair game for technical critique, and in whose language must such a critique be phrased? It seems that, to the ethics committee, answering these questions would require something like a minimum set of technical standards for test users.
In 1950, the American Psychological Association responded by establishing a Committee on Test Standards (hereafter CTS) ‘to prepare a statement on technical standards for evaluating tests and on the contents of test manuals’.Footnote 7 Cronbach was appointed chair of this committee and, after the first meeting of the CTS, took up the task of writing the first draft of its report, which would eventually become the Technical Recommendations for Psychological Tests and Diagnostic Techniques (hereafter Technical Recommendations). A significant feature of the Technical Recommendations is their lack of any requirement that researchers follow concrete, defined standards for test validation. This choice is made explicit in the introduction, which argues that ‘in a developing field, it is necessary to make sure that standardizing efforts do not stifle growth’.Footnote 8 This feature of the APA’s efforts to regulate psychological measurement was not new. The Technical Recommendations directly quote the report of a previous committee on measurement led by James Rowland Angell in 1906 that had also declined to set standards for the content or form of tests:
The efforts of a standardizing committee are likely to be regarded with disfavor and apprehension in many quarters, on the ground that the time is not yet ripe for stereotyping either the test material or the procedure. It may be felt that what is called for, in the present immature condition of individual psychology, is rather the free invention and appearance of as many variants as possible. Let very many tests be tried, each new investigator introducing his own modification; and then, the worthless will gradually be eliminated and the fittest will survive.Footnote 9
That is, the Technical Recommendations are only part of a much longer series of committees that continuously declined a role in articulating enforceable standards of scientific practice.
To understand this somewhat peculiar feature of the governance of technical conduct in psychology, it is helpful to recognize the incorporation of test validation into the governance of scientific integrity in psychology as part of a longer history of the extension of academic organization into the regulation of professional conduct.Footnote 10 This perspective is a consistent theme in prior work on the professional authority of psychology, which emphasizes the field’s intensive use of a logic of mechanical objectivity and the relationships this logic cultivates with external dependencies. Relationships with clients and suppliers, the demands of key patrons, novel technical innovations, and loosely associated applied enterprises claiming psychological authority each create conditions under which rigid, impersonal adherence to procedure creates a basis for public trust in psychologists’ ideas and judgements.Footnote 11 Much of the work of the CTS is well described by this theoretical approach, particularly (as we will see in more detail below) its interactions with psychologists working in applied institutional settings such as hospitals or prisons.
Reciprocally, a crucial point of difficulty in the work of the CTS concerned the relationship between its recommendations and the prerogatives of an important and peculiar professional constituency: elite academic professionals. Contemporary academic fields maintain a pantheon of established senior scholars whose collective reputations as technical experts scaffold investment in the discipline and who play prominent roles in the public portrayal of the field as a coherent science.Footnote 12 Members of this ‘core group’ are among the primary consumers of literary representations of the field.Footnote 13 Additionally, by participating extensively in the world of formal professional committees, members of this elite gain access to the physical spaces where passing connections, gossip, rumours, arguments, disputes and other important forms of informal academic sociality occur. Since academic professionals also tend to differ sharply from one another in their epistemic preferences and outlooks (even, or perhaps especially, within the same discipline), an important obstacle to the articulation of standards of professional conduct is the difficulty of articulating rules that differentiate legitimate variation in the use of common tools from more basic (and therefore more serious) gaps in technical competency.
This final point is a lesson Cronbach learned while seeking feedback on his first draft of the Technical Recommendations. The draft report, titled ‘Test standards: their nature and purpose’, notably includes several critiques of existing tests, including one authored by Gordon Allport, former APA president and a leading figure in personality psychology.Footnote 14 Cronbach’s critique argued that Allport’s test, the Study of Values, interpreted relatively arbitrary test items as measuring deeply held dispositions (e.g. religiosity) without sufficient evidence that this is what the questions measure. This critique bears a striking resemblance to the notion of construct validity developed years later in more abstract terms by Cronbach and Meehl, after the conclusion of the CTS. Hence, although Cronbach later cited committee discussions on projective testing as the original basis for the problem, personality psychology evidently also played some role in Cronbach’s thinking.
Through a chance encounter in Washington, Allport acquired an early copy of the draft report. He reacted negatively to the critical mention of his test, and reacted by sending Cronbach a letter sharply explaining his disagreements with the report’s evaluation of his work. Cronbach could not have been too surprised by receiving pushback on at least some of the draft report’s critiques: all academic committees are social scenes where disciplinary priorities are both formally and informally adjudicated, and although Cronbach was new to the world of top APA committees he was no novice to academic ritual generally.Footnote 15 But he was not well embedded enough in the field’s bureaucratic elite to foresee the social consequences of a potentially untimely move in the field. After some negotiation and editing, the critique of the Study of Values was removed from the final report.Footnote 16 This sequence of events reflects the importance of cases of strategic action gone wrong: moves where advance coordination is not successfully achieved, and alternative forms of diplomatic action are needed to repair ongoing relations.Footnote 17 At the ethical–technical boundary, misunderstandings of these kinds can lead in surprisingly productive directions.
In the following sections, I reconstruct and contextualize the exchange between Allport and Cronbach using archival materials from the personal archives of both Allport and Cronbach, as well as the APA’s archived administrative records on the CTS. I emphasize two important features of the ethical–technical boundary that are made especially visible by this episode of scientific controversy. First, I show that Cronbach and Allport held partially incompatible views on the use and interpretation of psychological tests. More specifically, although they both talked about Allport’s Study of Values in their correspondence, there were key differences in how they construed the object under dispute. At the time, Allport was on the verge of publishing a new edition of the book reporting on the Study and its uses in research after its first release. As he commented repeatedly in his correspondence with Cronbach, the status of the book as a ‘venerable publication’ (i.e. as a business relation with his publisher and the market for his work) was central to his understanding of the critique. But Cronbach shared very little of this literary context with Allport. He worked with the Study as one of many bundles of testing material: a test instrument, a manual explaining the procedures for administering and scoring the test, and often the data sets generated by applying the test in practice. Hence Allport and Cronbach’s respective stakes in the testing enterprise reflected partially incompatible ideas about what kind of ‘technical’ activity tests could be used to perform.
Second, I highlight the role of academic expansion in creating conditions for Allport and Cronbach’s disagreement. Many of the tensions in the work of the CTS originate in the uncertain and changing status of academic authority in an expanding professional field. By mid-century, US psychologists were developing and using a large variety of tests (e.g. projective, fitness, mental, achievement) as measurement instruments in a diverse range of institutional contexts (e.g. schools, hospitals, prisons). The use of tests to back the objectivity of psychological claims in these settings raised a whole series of practical problems for the professional governance of psychological expertise. On one hand, tests offered ambitious psychologists a new terrain of authorship, scientific credit and professional standing. The growing consumption of tests by schools, public agencies and businesses had by mid-century greatly expanded the public profile of the field and marked what many presented as a new level of epistemic maturity.Footnote 18 But the testing enterprise also created new sources of potential reputational exposure for psychology. In practice, academic psychologists had little direct control over the ‘test’ as a written genre; anyone could develop or use a test. Although some academics were eager to frame this matter primarily as a problem of unskilled use by low-quality practitioners, it also raised more challenging questions regarding the practices of disciplinary elites.Footnote 19
Gordon Allport
To understand how Gordon Allport’s work came to be mentioned unfavourably by Cronbach and the CTS, it is helpful to begin the story from his perspective. While he is now most widely recognized as one of the founders of social and personality psychology, many historians of the social sciences have argued that Allport was exceptionally influential in institutionalizing the model of theory-driven empirical social research that had come to dominate US social science by the 1960s and 1970s.Footnote 20 Except for a brief stint at Dartmouth (perhaps not so far afield?) and numerous visiting positions abroad, Allport was thoroughly embedded in the Harvard interdisciplinary complex for his entire academic career; he led the group of social scientists who would become the Department of Social Relations (DSR) in their ambitious quest to absorb the social sciences into a new interdisciplinary entity.Footnote 21 In US sociology, the DSR is perhaps better remembered as the institutional home of Talcott Parsons’s project to remodel sociology around the development of a ‘grand’ social theory of action, but it was also an important vehicle for Allport’s own theoretical ambitions. His intellectual identity was substantially entangled with the lifespan of the DSR, which would cease to exist not long after his death in 1967.Footnote 22
The CTS worked at a distance from the direct influence of DSR’s status culture, but only so far; Allport’s advocacy for theory-driven psychology must be understood in the context of his efforts to cultivate a particular style of academic work. Indeed, Allport would later write that he felt that the purpose of his life was to ‘search for a theoretical system – for one that will allow for truth wherever found, one that will encompass the totality of human experience and do full justice to the nature of man’.Footnote 23 In attempting to create a space in US academia for this type of grand social theorizing, Allport participated in importing a complex blend of European academic professional customs to elite US higher education.Footnote 24 The DSR’s ambition to co-author a grand Weberian theoretical tract culminated in the publication of Toward a General Theory of Action in 1951, but the practical dimensions of writing the book seem to have had a more lasting impact on the social sciences than did any of the particular ideas in it. In particular, the book was authored in a seminar explicitly restricted to the department’s senior faculty. Input and ideas from junior faculty and students were categorically excluded from the writing process; Parsons set up a separate seminar for junior faculty, in which they were to be informed of the book’s draft contents.Footnote 25
Reflecting on his association with the university in an autobiography written for the APA, Allport wrote, ‘In the course of fifty years’ association with Harvard I have never ceased to admire the unspoken expectation of excellence.’ His identification with this culture of excellence is reflected throughout the remainder of the text. Notably, in a section a few pages down commenting on the negative reception of his work by the experimental psychologist Edward Titchener, Allport further remarked, ‘Never since that time have I been troubled by rebukes or professional slights directed at my maverick interests.’ Closer inspection of the historical record reveals a more complicated biography. Allport’s role in the conceptualization of validity reflects just the opposite stance toward criticism: an enduring concern with his representation in the field’s publications and careful management of his reputation as a methodologically competent researcher.Footnote 26
Allport was a prominent senior member of the US psychological field in the early 1950s. He had previously held the APA presidency in 1939, followed by leadership positions in several other regional and national psychological associations. His students were also moving into similar roles, marking another important channel of influence. Among them was Fillmore Sanford, a DSR graduate who took up a position as executive secretary of the APA in 1950. (Sanford’s role in the story will become apparent shortly.) Around the time the Committee on Test Standards began its work in earnest, Allport was finishing contributions to Toward a General Theory of Action. He had also recently finished two other big projects: a long stint as editor of the Journal of Abnormal and Social Psychology, and a revised edition of his Study of Values, a test of values co-authored with Philip E. Vernon and published in 1931.Footnote 27 The Study of Values was fairly popular among academic research psychologists and applied/clinical psychologists alike, but contemporaneous reviews of the use of psychological tests in the US show that it was not as widely applied as other better-known tests, such as the Stanford–Binet IQ test, the thematic apperception test, or the Rorschach test.Footnote 28 Finally, and most crucially, in 1950 Allport had also recently finished a period of service as chair of the APA Committee on Scientific and Professional Ethics.Footnote 29 We next consider the mode of professionalization that this activity represents.
Ethical Standards of Psychologists
In the period immediately following the Second World War, the APA began to take a more active role in the governance of an expanding psychological field.Footnote 30 Of particular concern was the rapid diffusion of psychological tests into US culture, and the perceived consequences (or potential consequences) of a loss of professional control over what was seen as part of the field’s core technical jurisdiction.Footnote 31 This was an eminently material problem: one of the major obstacles to establishing the testing enterprise as scientifically objective was the general availability of testing materials. This raised several concerns, two of which are worth noting here. First, libraries were increasingly likely to own and lend out copies of tests and their scoring guides. This meant that, for the first time, it was possible for psychologists to encounter test subjects who had already seen their measurement instruments. Second, the widespread availability of tests meant that consumers of many new kinds were coming to see themselves as capable test users, despite a lack of formal psychological training.Footnote 32
It was in this increasingly disorganized epistemic milieu that the APA first set out to define a code of ethics for professional psychologists. Unlike earlier efforts to cultivate test validation practices in psychology, the 1950–4 CTS was more explicitly tied to this ethical codification project.Footnote 33 The idea that such a committee was needed came directly from the Ethics Committee, and one member of the CTS (Donald Super) was appointed specifically to act as a liaison between the two committees; the two committees also involved numerous subcommittees seeking feedback on draft reports concurrently.Footnote 34 Given its staffing and organizational form, the early years of the CTS would have been difficult not to associate with the professional-ethics enterprise in the discipline – particularly for the still relatively small network of academics circulating through administrative positions in the APA in the early 1950s.
This context created an important but unanticipated dilemma for the CTS. There is a key distinction in the 1953 Ethical Standards of Psychologists between professional skill and ethical conduct. The 1953 code of ethics frames the matter of proper research practice as a potential domain of ethical concern, but also carefully identifies it as a domain that has its own non-ethical logic. This logic is broadly educational; it imagines a psychological profession that progressively improves through training and selecting skilled researchers:
The psychologist’s behavior in research as in other situations can be a consequence of a variety of influences. Whether or not the research he conducts is adequate in a technical sense may depend either upon his intentions or upon his research skill, his facilities, and the limitations imposed by his sponsor (e.g., populations available, time limits, funds, etc.). Similarly, his relationships with his subjects will be a function not only of his concern for their welfare but also of his capacity to appreciate their needs. In an ethical code we are dealing with certain of these influences only; where the research behavior falls below standard because of inadequate training or insufficient ability it remains a matter of concern – but from the point of view of selection and training rather than of ethics.
This way of bounding the technical obligations of researchers frames technical mistakes and ethical transgressions as separate categories: ‘misleading research which results from causes such as lack of skill or poor judgment is not our present concern’. On the other hand, the need to say this at all reveals the ambiguous nature of this boundary.Footnote 35 What constituted ‘lack of skill or poor judgment’ and whether such matters were ethically relevant were always present concerns.
The problem of articulating ethical standards was the key organizational context for the work of the CTS and the specification of validity as a theory of professional conduct around tests. At first, Cronbach seems to have not really understood this. His assignment to chair the CTS came at the start of his career, as he was just beginning to make his way into the inner networks of the postwar psychological elite. His position at the University of Illinois (and in California and Oregon before that) left him at a distance from the urban centres of gravity in postwar social science. In fact, the committee assignment only came his way through the recommendation of Dorothy Adkins, a psychometrician and student of Thurstone three years Cronbach’s senior at the University of Chicago, who declined participation in the committee due to being ‘currently so swamped that I am following a strict rule of saying no to any additional commitments for the next year’.Footnote 36 Cronbach was friendly with Sanford, his main contact at the APA, but they did not know each other initially, and Cronbach’s early correspondence with Sanford strongly suggests that Cronbach had not previously been similarly engaged.Footnote 37 For example, Cronbach was evidently new to the APA’s business routines; an early letter to Sanford asks ‘the precise routine for getting expense accounts paid’.Footnote 38
On top of Cronbach’s unfamiliarity with the social routines of the APA, the relationship between CTS and the ethics committee was itself left fundamentally ambiguous. Cronbach commented in an early letter to Sanford that he had accepted his appointment to chair the committee without knowing much about the charge of the committee or the context for its work.Footnote 39 Sanford pointed Cronbach to this document (by sending him a copy) and told him to speak with Donald Super, a member of the ethics committee who had led the effort to write this earlier set of test-relevant standards. Cronbach understood this to mean that his committee was ‘expected to cooperate with the Committee on Ethical Standards, but is not regarded as a subcommittee of that group’.Footnote 40 Sanford’s response was noncommittal:
Your committee is a special committee appointed to stand and walk on its own feet. I do not think that anybody has instructed you to cooperate with the Committee on Ethical Standards, but it seems to me that perhaps you might want to as long as such cooperation seems mutually beneficial to the two committees.Footnote 41
This exchange reveals a moment of ambiguity in the ethical–technical boundary: Sanford first says no, and then says well maybe, but only if you both think it’s beneficial. While in the end the conclusions of the CTS closely mirror previous testing committees, the initial framing of the committee’s activities certainly did not guarantee this outcome.
Incident #4
In the first year of the committee’s work, Cronbach met with a small number of his fellow committee members (half were unable to attend) at a conference or retreat centre a short distance from Urbana, Illinois to begin drafting what would become the Technical Recommendations.Footnote 42 The product of this meeting was an early draft of two parts of the report: an introduction, titled ‘Test standards: their nature and purpose’, and a section on interest tests. Notably, this early draft is formatted in a way that closely mirrors the report of the ethics committee on test publication practices.Footnote 43 In particular, like the earlier report, it included a list of ‘incidents’ meant to illustrate the core issues at hand to be addressed by the report’s recommendation.Footnote 44 Portions of this early draft persevered into the final Technical Recommendations, but some of the content is markedly absent in future drafts. In particular, the fourth incident cited in this early draft pertained directly to Allport’s Study of Values. The incident read,
Allport Vernon Study of Values purports to measure six attitudes. These are defined at length, but no evidence is provided that (for example) theoretical-valued persons are indeed ‘pursuing truth, rational’. Hence the test rests entirely on the adequacy of the items. Items stress verbal endorsements of the value of clear thinking plus numerous items on concern for scientific subjects. Items would not allow a person with a ‘theoretical value’ directed toward anthropology, history or law to reveal this. Items do not ask whether the person is a rational thinker, only that he endorses reason in principle. Nor, for those who score high because of scientific interest, do items appear to discriminate between those who enjoy reasoning and theory-building and those who enjoy research for its mechanical side.
The draft further associates this incident with the following recommendation regarding test validity: ‘The field must be defined in a way sufficiently explicit to indicate what activities it includes and what it does not include. The basis for sampling items within the field should be reported so that users can judge whether the score does fairly cover the implied area.’
This unfavorable mention of the Study of Values came just months after the publication of a new revised edition of the study, originally published in 1931. It is unclear whether Cronbach and colleagues would have known this, but the early draft of the report suggests they may not have. Allport published his 1951 revision to the study in collaboration with his student Gardner Lindzey, but the draft report refers to the ‘Allport–Vernon’ test.Footnote 45 Allport made sure to correct Cronbach on this mistake.Footnote 46
Allport encountered the preliminary draft of the report as a matter of chance. It is easy to assume that he would have been privy to the report due to his status as a former APA president. But at the time he was not on the APA Council of Representatives or the APA Committee on Scientific and Professional Ethics, nor was he a representative of a publisher, so the limited number of copies in circulation would have been outside his immediate grasp. Allport’s connection to Sanford was key. They encountered each other in the hallways of the US Department of State in September 1951, and after the report came up in conversation Sanford offered to send him what ended up being the APA office’s last copy of the report. Sanford further sympathetically lamented ‘the statisticians’ need for neatness’ and asked Allport either to send comments to Cronbach directly, or to send them to Sanford so that he could intervene on Allport’s behalf.Footnote 47 This was likely meant to please Allport; Sanford later wrote to Allport that ‘it will probably happen that I will meet with the Committee [on Test Standards] sometime in the future and I think it can be counted on that then, as so often seems to happen, I will be under the influence of some sort of identification with you.’Footnote 48 But it also obligated Allport to say something to Cronbach.
Allport interpreted this technical critique of the validity of his test as an ethical criticism of his conduct as a researcher. His judgement of the situation is apparent in a brief memo from Allport to Sanford reminding him to send a copy of the report to Harvard. In the memo, Allport misstated the name of the Committee on Test Standards, instead referring to the ‘Cronbach Committee on Ethics and Tests’. This lexical slip is doubly revealing. First, Allport situates the CTS in the APA’s efforts to set an ethical code of conduct, despite the committee’s separation from the ethics committee. Second, Allport marked Cronbach as responsible for the effort. Although many signs seem to point to Cronbach himself choosing this example, the distributed nature of authorship on committees like the CTS makes it difficult to know who brought up the Study of Values as a negative exemplar of this kind of validity problem. The first meeting of the CTS at the University of Illinois involved Cronbach, Super and Humphreys; Meehl and Conrad were not in attendance.Footnote 49 This narrows the candidates down somewhat. Officially, Bordin was the appointee to the committee who was supposed to represent expertise in interest testing.Footnote 50 But the institutionally official order of authorship was what mattered. The early draft of the report is attributed to the committee, and Cronbach was the chair of the committee. So it was Cronbach alone who would receive Allport’s ire.
Cronbach was initially unaware of Allport’s views on the report, but he did not remain ignorant for long: in late October, Allport sent Cronbach a letter rebuking the first draft of the report.Footnote 51 In the letter, Allport writes, ‘The publisher of a scale is naturally perturbed to find one of his venerable publications cited as a horrible example of some technical malfeasance on the part of the authors.’ He then proceeds to lay out a defence of the Study of Values that is something of a non sequitur relative to the specific criticism in the report. The critique in the report is that the study assigned general labels to specific survey items in a way that had not been given proper theoretical justification (i.e. explained in the context of a theory of values). But Allport portrays the critique as if it were subjecting the study to an unfairly high standard of statistical performance. His response contends that (1) the study is not intended to predict occupational interests, but also that (2) the study does in fact predict occupational interests.Footnote 52 Allport’s comments also ground his complaints in a logic of field expansion: he remarks further that his ‘broader’ ‘point’ or ‘purpose’ is to enjoin Cronbach from producing a set of standards that ‘silences those who may seek to use alternative hypotheses regarding the nature of personality organization’. He suggests that his test was meant to explore an alternative and ‘unconventional’ set of hypotheses in psychology, and that condemning the test in the report meant ‘running the danger of seeming to impose a type of thought-control that would be misunderstood and resented’.Footnote 53
It may have come as a surprise to Cronbach that Allport even had access to a copy of the first draft. In fact, Allport seems to have been less than forthright with Cronbach regarding where he got it: in the first line of the letter, he names Houghton Mifflin Co. (the publisher of the Study of Values) as his source rather than Sanford. This detail was plausibly true: Cronbach had anticipated that one of the more challenging audiences for the report would be the test-publishing community, and set up a meeting at the annual APA conference in 1951 to circulate the preliminary draft of the report among them for feedback.Footnote 54 But it is much more likely that Allport was looking to spare Sanford some trouble, since Sanford had ‘officially’ asked Allport to write a response to the draft report and offered to bring it to Cronbach if Allport did not want to write him directly.
There is not much direct evidence regarding Cronbach’s reaction to Allport’s letter, but his next couple of steps as chair of the committee are indicative. In his next letter to Sanford, Cronbach asked for ‘a guide from APA legal counsel as to the boundaries of what our standards can do’, and in particular whether the CTS could call out ‘bad examples of publishing practice by name’.Footnote 55 A subsequent message then asked Sanford’s aid in keeping the CTS distinct from the ethical standards committee in conversations with other APA constituents. Writing in reference to Sanford’s annual secretarial report on the APA’s national meeting, Cronbach noted that test publishers
are extremely, nay excessively, touchy, about the fact that the test standards grew out of the ethics committee. They make a vast point of being insulted that people question their ethics; and, indeed, much of the recent ethical standards are proposed guides to professional judgment rather than ethical matters. Anyhow, we found that by discussing our standards as a consensus of technical judgment and dropping the word ethics completely, we got much better reception for our material without weakening the material. I think you can help us in this bit of public relations by keeping us in our separate domicile … Super agreed with this suppression of the ethics notion. I feel we could properly give acknowledgment to the predecessor committees if the matter weren’t so touchy, and that it really is an ethical matter when a professional worker does not employ good judgment.Footnote 56
Notably, Cronbach expresses some resistance to the idea of separating ethics and technical practice, but prioritizes managing the ‘touchy’ nature of the connection over defining what ‘good judgment’ might mean in practice.
Allport was just one of a broader set of outside parties weighing in on the committee’s work; he is listed among sixty-six ‘consultants’ in the 1953 annual report of the committee to the APA board.Footnote 57 But the impact of Allport’s intervention on subsequent iterations of the report is unambiguous: they omit any critical mention of the Study of Values. In a subsequent letter to Cronbach praising a new draft of the report, the study does not come up, but Allport does make reference to the same ‘plea for a bit wider hospitality to exploration and experiment, and a little less threat and policing’ that he expressed in his first message.Footnote 58 The thrust of letter #2 is largely along the same lines, critiquing Cronbach for conflating technical preferences with moral obligations:
I am not clear about the intent of the word ‘Essential’. Does it mean APA censure for a person who slips up on one of the standards? … Does it mean that publishers will be intimidated (unintentionally)? Or does it mean that in the eyes of just your Committee these standards are regarded as really essential for ‘scientific decency’? What I have in mind is the possibility that it might be well to avoid the implication of sanctions that the term ‘essential’ conveys.Footnote 59
The committee seemingly declined to make this change. ‘Essential’ categories of validation practice appear in the final Technical Recommendations, although the full set of essentials has scarcely been followed exhaustively since. But Allport’s message also foreshadows a more important move: the rejection of a hypothetical enforcement paradigm in social-science methodology. The Technical Recommendations were approved at the August 1953 meeting of the APA, and a copy of the report was issued to all APA members as part of their subscription to the Psychological Bulletin; little of the ‘policing and threat’ that Allport feared on the first draft of the report survived into the final report and its successors. In the present, even the forensic style employed by replication-oriented methodological researchers in psychology does not make much use of validity theory as a technical or rhetorical means of critique.Footnote 60
On the other hand, the conflict between Cronbach and Allport over how a test could be critiqued was among the most eventful episodes in the arc of the committee’s work, which otherwise involved largely routine problems of coordination, reimbursement and circulation (i.e. printing copies of the report). Beyond Allport’s intervention, the committee was not especially responsive to complaints about the contents of the report, particularly those received after its publication. After the publication of the Technical Recommendations, Cronbach received a complaint about the report from the author of a test criticized in the report. The author, DeWitt Sell, was a psychologist in the Department of Public Welfare in the Ohio State Division of Corrections; his test, the Ohio Penal Classification Test (OPCT), is the target of detailed technical criticism in the Technical Recommendations. Sell’s letter is similar to Allport’s; he accuses the committee of ‘misquot[ing] procedures out of context for the purpose of having a scapegoat to buttress our paternalistic wisdom!’ and writes that ‘the OPCT has been unfairly maligned before the profession’. In substance, Sell was only somewhat vindicated; Cronbach admitted the report contained a ‘minor’ error, but did not agree that this affected the meaning enough to revise the report. Besides, by the time Sell wrote to the CTS it was a moot exercise; unlike Allport, his position outside the APA’s elite circles had already made it impossible for him to intervene on the report’s contents until after it was published.Footnote 61
The committee also evinced little concern with the practical questions of validation facing test users, even when these practical questions touched directly on construct validation problems. In December 1953, George Middleton Jr, then chief psychologist of the Alabama State Hospitals, sent the committee an inquiry regarding a new set of Rorschach ink-blot prints that the hospital system had purchased from the Psychological Corporation. The new prints were ‘a superior reproduction of the original blots’, but this improvement was problematic for the psychologists of Middleton’s department, who felt that visible differences in ‘texture’, ‘hue, intensity, and brightness of the colors’, and ‘definition of detail’ could significantly impact the pattern of responses elicited by using the test in clinical settings.Footnote 62 In response to Middleton’s inquiry, Bordin declined to take any action ‘other than what you have already done, namely to call the publisher’s attention to the problem’.Footnote 63
Conclusion
The subsequent revision of construct validity into a stand-alone work of methodology arguably led to much more widespread use than it would have seen as part of a technical report, which are not widely read even among scientists. Indeed, the case of construct validity is historically notable in part because of the amount of attention the concept has received both within and beyond psychology. Yet as prior histories of this transformation have emphasized, the scientometric popularity of construct validity is by no means a sign that measurement is being practiced in the manner that the 1955 paper prescribes. So why is construct validity talked about so much? What explains its apparently universal relevance if not measurement practice?
This paper has suggested that construct validity originated in a set of compromises intended to preserve the centrality of the academic profession during a period of rapid jurisdictional expansion. Construct validation specifically accomplishes this by producing conditions under which ‘theory’ and ‘methodology’ can talk about the same measurement objects in mutually inconsistent ways without calling into question the ethical status of the other perspective. ‘Validation of psychological tests [had] not yet been adequately conceptualized, as the APA Committee on Test Standards learned when it undertook (1950–54) to specify what qualities should be investigated before a test is published.’ At first glance, this introductory sentence to Cronbach and Meehl’s paper seems to mean that, prior to 1950, psychologists had not figured out what they meant by test validation. But read against Allport’s intervention and his authoritative position relative to the committee’s work, it could also be read as saying that the practice of test validation had not yet been integrated into the project of developing psychological theories. From this perspective the primary contribution of validity theory is to enable psychological experts to recognize the validation issues associated with interpreting tests while continuing to use them for a wide variety of purposes in practice, notably including theory development.
Reading the development of construct validity this way helps us to understand better some of the durable tensions that characterize psychological expertise up to the present. By the 1950s, psychological tests were technical devices facilitating mass data collection as well as an academic literary genre forming a basis for building scholarly reputations and careers. Multiple uses for tests created new trouble for the logic of academic and professional reputation and the social order of scholarly criticism it presumes, because it required negotiating a challenging boundary concerning the ethical status of technical practices that were previously irrelevant to the project of theory. By making test validation legible to theoretical social science, the compromise between Cronbach and Allport and its aftermath ensured greater legitimacy for methodological expertise within the US psychological profession, but primarily in service of articulating theory rather than directly evaluating its validity. The ideas about validity articulated in the Technical Recommendations reflect a lasting agreement to organize the regulation of method in psychology as a matter of academic inquiry rather than professional ethics. Few elite psychologists at the time would have thought that testing procedures were an especially rich topic for theorizing. But by emulating the ethics committee’s report in genre, Cronbach inadvertently demonstrated that technical questions could easily break the traditional rules of academic reputation.Footnote 64
A residual puzzle in the story of construct validity relates to a distinctive institutional feature of the social sciences relative to the natural sciences: the taken-for-granted division of social-scientific labor into ‘theory’ and ‘methodology’. Many fields throughout science have theorists as well as technicians who operate measurement equipment, but the notion of a ‘methodology expert’ remains largely specific to the social sciences. Although the style of methodological discourse that validity exemplifies is typically associated with the rise of statistical approaches to social knowledge-making, the present case suggests that construct validity might be better understood as a kind of ‘theory by other means’ – as a kind of theoretical argument made in methodological language.Footnote 65 That is, if we view test validation as a means by which psychological measurement became legitimate and routine across many distinct institutional settings, we might then see construct validity reciprocally as the incorporation of theory as a legitimate part of the logic of psychological measurement – that is, as a way of legitimating ‘theorizing’ as part of mechanically objective procedure.Footnote 66 Future work on the history of methodology as a distinctive form of scientific expertise in the social sciences could further consider how this theoretical style of reasoning has shaped how social scientists make knowledge about their methods.
Acknowledgements
I would like to thank Angela Creager, Lara Keuck, Anna Berg, Nicolas Torres-Echevarry, Mikey McGovern, Christopher Phillips, Mitchell Stevens, Brandon Stewart, Janet Vertesi, Fred Wherry, Philipp Brandt, Emily Merchant, and Zach Griffen for sharing invaluable insights throughout the lifespan of this project. I am also grateful to archivists at Stanford University, Harvard University and the Library of Congress, who provided access to crucial materials on the CTS. Research for this project was supported by PhD fellowship support from Princeton University, a graduate research fellowship from the National Science Foundation, and a French government grant managed by the Agence nationale de la recherche under the France 2030 programme (ANR-22-EXES-0014).
Competing interests
The author declares none.
Open access
