Title: Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences

Authors: Abdullah Almaatouq, Thomas L. Griffiths, Jordan W. Suchow, Mark E. Whiting, James Evans, and Duncan J. Watts

Deadline for Commentary Proposals: Monday, January 30, 2023

About Commentary Proposals: When a target article or recent book has been accepted for BBS commentary, the editorial office sends out the call for commentary proposals to thousands of people. Commentary proposals help the BBS editors craft a well-balanced commentary invitation list.

If this target article or book précis interests you as a possible subject for commentary, please download the full pre-print to see if you would like to propose a commentary.

Download Target Article Accepted Manuscript

If you are interested please follow the instructions below. Please keep in mind that we are not asking you to submit a commentary -- but rather, a short proposal in order to be considered as an invited author after the proposal deadline. If you are not interested, no action is required.

Abstract: The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment’s specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. The researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm—and with far greater efficiency.

Keywords: (in)commensurability, cumulative knowledge, generalizability, experiments

Commentary Proposal Submission Instructions

In order to nominate yourself for a commentary invitation, follow the instructions below and submit a commentary proposal via the BBS Editorial Manager Site: https://www.editorialmanager.c...

You may find these instructions available at https://www.cambridge.org/core/journals/behavioral...

If you received the call for commentary proposals, your username and password should have been included inside the email. At the Editorial Manager (EM) site you can register a new user account, update your existing information, or retrieve your username and password.

COMMENTARY PROPOSALS MUST INCLUDE THE FOLLOWING:

1. Name of target article/book précis on which you are submitting a commentary proposal.

2. All proposal authors, including any possible co-authors, listed at the top of your submission document.

3. What aspect of the target article or book you would anticipate commenting on.

4. The relevant expertise you would bring to bear on the target article or book.⁺

>> Please number these sections in your proposal: 1., 2., 3., 4. <<

⁺Including your relevant expertise saves the editors valuable time when evaluating proposals. If one of these requirements is missing, your proposal will be returned for resubmission.

EDITORS' NOTES ON WRITING YOUR PROPOSAL:

In addition to the open “Call for Commentary Proposals,” we invite commentators who do not submit proposals—these include reviewers of the paper, scholars whose work is discussed in the paper, and commentators suggested by the authors. (Obviously, these can be overlapping sets.) Once we subtract this set, only about 20 submitted proposals from the Call for Commentary Proposals can be invited to write a commentary.

Commentary selection is necessarily multifactorial. It must be balanced to a degree across the various fields of cognitive science, point of view of the book, and several other aspects of academic diversity. The number of proposals can vary widely, however, depending on the topic, the range is from 15 to 150! In the latter case, when we can accept only a little over 1 in 10 of the proposals, a few things will facilitate a positive reading of a proposal, and hopefully acceptance, given the constraints:

1. The proposal for the commentary should not be longer than the commentary, 1,000 words. 100-500 is optimal, and we value succinctness. On the other hand, “I intend to comment on X aspect of the target article/book précis” is not enough. Are you for it, against it, or extending it?

2. Under no circumstances should proposers simply write a commentary and submit it to us.

3. Proposers should clearly state what aspect of the target article/book they intend to comment on. It’s quite obvious when proposers are using the commentary forum only to promote their own research and not engage with the target article/book. Such proposals are routinely declined.

4. Concerning “the relevant expertise you would bring to bear”: While the editors have a generally good idea of who is active in the fields of the target article/book, we must cover a wide range and may be unaware of the people who have been most productive and influential in a given area, or the scholars who have engaged in heated debate with the authors in the past. So, the editors will be greatly helped if every proposer states their position in the field and lists between 2-10 relevant publications, again succinctly. On the other side of the spectrum, under no circumstances should an entire CV be included.

5. BUT … it’s not all about articles previously published, or position in the field. It’s not necessary to have published in the area, and it’s not necessary to have a current academic appointment. We make efforts to include proposals coming both from established figures and total newcomers. An engaging idea elicited by the book, an illuminating application of the target article/book concept to an allied field, or a truly clever riposte is often all that’s needed.

6. Being a co-author on multiple proposals directed to one target article/book précis will almost certainly remove one set of your co-authors or the other from contention altogether, which will put you in an unpleasant game theoretic situation with your colleagues. Do this carefully, if at all.

7. We make our choices mostly on quality and fit, but we do want to open up BBS to as many individuals as possible. If you’ve written one or more other commentaries recently, your odds of having another one accepted will correspondingly go down, though not to zero.

HOW TO SUBMIT A COMMENTARY PROPOSAL VIA THE ONLINE SUBMISSION SYSTEM:

1. Log-in as Author

Log-in to your BBS Editorial Manager account as an author: http://www.editorialmanager.com/bbs

If you do not have an account, please visit the site and register. You can also submit a request for missing username and password information if you have an existing account.

2. Submit New Manuscript

Within your author main menu please select Submit New Manuscript.

3. Select Article Type

Choose the article type of your manuscript from the pull-down menu. Commentary Proposal article types are temporarily created for each accepted target article or book. Only select the Commentary Proposal article type that you wish to submit a proposal on. For example: "Commentary Proposal (Author name)"

4. Enter Title

Please title your proposal submission by indicating the relevant first author name of the target article or book. For example: "Commentary Proposal on [Author name]"

5. Co-Authors

Commentary Proposal submissions are limited to a single author. If you are proposing to write a commentary with co-authors, the system will not allow you to enter their information here. Instead, include their names at the top of the Commentary Proposal document you upload. These potential co-authors need not contribute to the Commentary Proposal itself.

6. Attach Files

The only required submission Item is your Commentary Proposal in MSWord or RTF format. In the Description field please add the first author name of the target article or book. For example; "Commentary Proposal on [Author name]"

7. Approve Your Submission

Editorial Manager will process your Commentary Proposal submission and will create a PDF for your approval. On the Submissions Waiting for Author's Approval page, you can view your PDF, edit, approve, or remove the submission. Once you have Approved the Submission, the PDF will be sent to the editorial office.

8. Editorial Office Decision

Note: Before the commentary invitations are sent, the copy-edited and revised target article/book précis will be posted for invitees. In the case of Multiple Book Review, invitees will be sent a copy of the book or ebook to be commented upon if requested. With Multiple Book Reviews, it is the book, not the Précis article that is the target of commentary.

Please do not write a commentary unless you have received an official invitation! If you have any questions or problems please email bbsjournal@cambridge.org.