Hostname: page-component-77c89778f8-fv566 Total loading time: 0 Render date: 2024-07-24T10:00:45.375Z Has data issue: false hasContentIssue false

Open Minds, Open Methods: Transparency and Inclusion in Pursuit of Better Scholarship

Published online by Cambridge University Press:  07 January 2021

Eike Mark Rinke
Affiliation:
University of Leeds
Alexander Wuttke
Affiliation:
University of Mannheim
Rights & Permissions [Opens in a new window]

Abstract

Type
Opening Political Science
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
© The Author(s), 2020. Published by Cambridge University Press on behalf of the American Political Science Association

The idea of “open scholarship” has enjoyed a remarkable career. From discussions around the production, availability, and analysis of data (Moravcsik Reference Moravcsik2014) to persistent publication biases (Franco, Malhotra, and Simonovits Reference Franco, Malhotra and Simonovits2014) and the accessibility of research publications (Bull Reference Bull2016), “openness” has emerged as a unifying concept in discourses about the discipline (Elman, Kapiszewski, and Lupia Reference Elman, Kapiszewski and Lupia2018; Lupia and Elman Reference Lupia and Elman2014; Wuttke Reference Wuttke2019).

Despite its ubiquity, the meaning of “open science” or “open scholarship” (OS)Footnote 1 has been ambiguous and contested to the point where scholars have identified different “schools of thought” (Fecher and Friesike Reference Fecher, Friesike, Bartling and Friesike2014). Our goal in this symposium is to advance an integrated concept of OS. We argue that as an academic reform movement aimed at raising the credibility of political research, the idea of openness is best conceived of as covering two orthogonal dimensions: openness as transparency and openness as inclusion. This conception of OS dovetails with philosophical accounts that locate science’s unique epistemic quality in its social character (e.g., Longino Reference Longino1990; Oreskes Reference Oreskes2019; Solomon Reference Solomon2001). The ineluctably social processes of mutual vetting, cumulative learning, and free exchange of views can only enhance the credibility of our work if we open up the research process and open scholarly discussions to diverse participants. These two components of OS reinforce one another in the credible accumulation of knowledge about particular questions. Thus, a key benefit of OS is that it encourages us, the scientific community, to assess whether we are realizing our epistemic potential by posing a simple question: How far open are the doors to our research and did we invite others in?

Thus, a key benefit of OS is that it encourages us, the scientific community, to assess whether we are realizing our epistemic potential by posing a simple question: How far open are the doors to our research and did we invite others in?

So, what exactly does openness imply for scholarship and what does the OS movement aim to achieve? To understand its purposes, it helps to revisit a foundational question about the nature of research: Why do we consider scholarly knowledge claims to be credible or at least more credible than knowledge claims from other sources?

An important tradition in the philosophy of social sciences, going back to at least Merton (Reference Merton and Storer1973) and taken up recently by Oreskes (Reference Oreskes2019), argues that science’s epistemic value lies in its social character. In this view, academic knowledge production is distinctive in that scholars constantly review, scrutinize, and evaluate one another’s work. As scholars, we expose our ideas to critique and engage in a process of cumulative—usually incremental—knowledge creation. In doing so, we should not only “stand on the shoulders of giants” but also examine them like a scrupulous orthopedist. The point of the social process of science is that it uncovers errors and opportunities for improvement. It allows us to achieve more robust insights by making us aware of the triumphs and missteps of others who came before us. Moreover, the social process of scholarly debate, at least in principle, may approximate a consensus that provides credible certainty to interested observers. When, after thorough vetting, a community of scholars agree on a particular scientific knowledge claim, then it no longer represents a private opinion but rather the best available collective expert knowledge on a specific topic—at least for the time being and within that community.

A revelation of the OS movement is that this social process depends on transparency or, more fundamentally, on the observability of our work. Simply stated, the academic research process can be subject to self-corrective action only to the extent that others can see it. Openness as transparency—the first dimension of OS—enables social vetting and mutual learning. The less information we make accessible and the more reclusive we are toward our peers, the lower the chances that our work will be vetted by them in meaningful ways and the dimmer the prospects for useful knowledge production. Hence, the more open we are about our research, the more effectively science’s social mechanisms of error control and cumulative learning can make our knowledge claims more robust.

Expert consensus—conceived broadly as a multiplicity of valid studies coming to similar conclusions—is another social mechanism that can foster the credibility of research, but it requires openness. Under ideal conditions, scientific consensus is a helpful signal of expert agreement. However, when studies with unwelcome findings remain hidden in file drawers, scholarly debate will reflect a distorted subset of the available information. As a result, the epistemic value of scholarly consensus will diminish because it will no longer represent the best available collective knowledge. Therefore, the more we open up access to all meaningful research outcomes, the more reliable and valid the signal emitted by the scientific consensus on a particular question will be.

Another key recognition of the OS movement is the role of openness in terms of access to scholarly debates. Openness as inclusion—the second dimension of OS—is key to realizing the epistemic potential of science because it enriches social vetting and makes scholarly agreement and disagreement more meaningful. Opening up scholarly debates to the greatest reasonably possible diversity of participants minimizes the risk that errors that may have been obvious—depending on one’s lived experiences—are overlooked. Hence, a complete understanding of OS encompasses a commitment to reflecting on the role of race, class, ethnicity, professional status, religion, and other meaningful group boundaries in defining the standards of recognizing academic contributions. In this way, the outlook of OS is similar to that of other academic reform movements working to make science a more egalitarian process in which a wide range of voices is heard, such as Women Also Know Stuff and People of Color Also Know Stuff. Viewing research as an enterprise the epistemic value of which depends on its social character pushes us toward an understanding of OS that implies a principled commitment to making the academic process more socially inclusive.

Openness as inclusion—the second dimension of OS—is key to realizing the epistemic potential of science because it enriches social vetting and makes scholarly agreement and disagreement more meaningful.

An integrated conception of OS that centers science’s social character therefore views both propositions—opening up the research process and opening up scholarly debates—as reflecting the same set of principles. In our view, OS consists of both opening the door to one’s research and inviting others in. Only when we satisfy both conditions can we hope to achieve a process of proper collective scrutiny, understanding, and assessment of our work as well as cumulative insight building from it.

In this perspective, the OS movement is all about interrupting the process of “normal science” and critically interrogating it together: Do our established norms, values, incentives, and rules encourage openness and observability of our work? OS is asking this question in an inclusive exercise of promoting self-critique through openness.

Critics argue that the OS movement falls woefully short of its own objectives. Specifically, they point out that OS itself is lacking in diversity and that it perpetuates steep hierarchical power structures within the discipline, favoring elite institutions in the northern hemisphere that often focus on quantitative methods (Women’s Caucus for Political Science 2017). We think there is an important point to this critique: it is true that many advances associated with the OS movement focused on making number-crunching more transparent. Moreover, it is clear that “doing OS” requires resources, which simply may not be available to some researchers—especially when it comes to acquiring the new skills it requires, from learning how to properly publish reproduction materials to documenting the research process more thoroughly than was common in the past. OS raises the bar of what is expected of us as researchers—but it also works to provide the tools needed to clear it.

Strengthening the academic commons is a key point of OS, focused as it is on providing everyone with the tools and infrastructures needed to conduct transparent and credible research, free of charge and regardless of hierarchical status. If this spirit of open innovation is coupled with a social-inclusiveness perspective, we expect that OS—in the long run—will flatten, not steepen, hierarchies that permeate our scholarly community.

If this spirit of open innovation is coupled with a social-inclusiveness perspective, we expect that OS—in the long run—will flatten, not steepen, hierarchies that permeate our scholarly community.

Likewise, the dual conception of OS as transparent and inclusive casts into doubt the entrenched notions of essentialist barriers between OS and qualitative research traditions. There is no intrinsic contradiction between OS and qualitative thinking. Indeed, interpretivist epistemologies have long been concerned with issues of transparency. Questions of researcher subjectivities in discussions around research reflexivity and positionality (Bourke Reference Bourke2014; Rose Reference Rose1997) connect seamlessly with the transparency dimension of OS. Additionally, as a continuous exercise of scholarly self-critique, OS necessarily questions academic power relations and, in this sense, aligns well with the analysis of power embedded in critical strains of political research. From this perspective, the principles of OS render it an undertaking that is more diverse in nature than some observers have characterized it.

Based on this broad conception of OS, the objective of this symposium is to extend the discussion of OS to political researchers who have not considered themselves to be active proponents of the movement toward OS. It assembles contributions that, as an ensemble, showcase how we can and should address the two dimensions of OS discussed in this introduction: opening up the research process and our discussions about it. We believe that the measures and advice presented in these articles will increase the epistemic value of political research as an inherently social process. The purpose of this symposium is to challenge the discipline to be transparent and inclusive as we strive to understand the political world around us.

Three articles in the symposium discuss openness as transparency. Kapiszewski and Karcher (Reference Kapiszewski and Karcher2020) present various entry points to transparent research for researchers working in qualitative traditions. The article by Rohlfing et al. (Reference Rohlfing, Königshofen, Krenzer, Schwalbach and Bekmuratovna R2020) demonstrates how transparency can be evaluated and improved in a context-sensitive manner in specific methodological areas. They review the political research literature using Qualitative Comparative Analysis (QCA), not only documenting a problematic lack of reproducibility but also providing a checklist that QCA researchers can use to improve the transparency and reproducibility of their work. Engzell and Rohrer (Reference Engzell and Rohrer2020) offer an interdisciplinary perspective on transparency. Their article discusses what political research can learn from psychology’s “replication crisis” and ensuing “credibility revolution.”

Three more articles address openness as inclusion. Lupia (Reference Lupia2020) explains why opening up our work to the scholarly and general public is of paramount importance for trust in social science. Janz and Freese (Reference Janz and Freese2020) focus on how the core scientific practice of replication can and should be made more inclusive in our discipline. Arguing that the epistemic ends of OS require us to build a cooperative-including rather than a combative-excluding culture, they illustrate how a culture of “fear of replication” has sidelined replication as a key academic practice. In response, Janz and Freese provide actionable recommendations for how we can recenter replication as an inclusive practice. Finally, Breznau (Reference Breznau2020) shows how we can advance political research by opening up to a massively inclusive approach to collaboration, even within the confines of a single project. He shows how a wealth of crowdsourced expertise and perspectives can be brought to bear on research of unique utility that would not have been possible in the traditional “one-to-10 authors” paradigm.

Footnotes

1. Compared to the term “open science,” “open scholarship” encompasses a wider group of referents. We deliberately use the acronym OS throughout this article without specifying it to refer to either term because the presented arguments apply to both open science and open scholarship.

References

REFERENCES

Bourke, Brian. 2014. “Positionality: Reflecting on the Research Process.” The Qualitative Report 19 (33): 19.Google Scholar
Breznau, Nate. 2020. “I Saw You in the Crowd: Credibility, Reproducibility, and Meta-Utility.” PS: Political Science & Politics. doi: 10.1017/S1049096520000980.CrossRefGoogle Scholar
Bull, Martin J. 2016. “Introduction: Open Access in the Social and Political Sciences: Threat or Opportunity?European Political Science 15 (2): 151–57.CrossRefGoogle Scholar
Elman, Colin, Kapiszewski, Diana, and Lupia, Arthur. 2018. “Transparent Social Inquiry: Implications for Political Science.” Annual Review of Political Science 21 (1): 2947.CrossRefGoogle Scholar
Engzell, Per, and Rohrer, Julia M.. 2020. “Improving Social Science: Lessons from the Open Science Movement.” PS: Political Science & Politics. doi: 10.1017/S1049096520000967.CrossRefGoogle Scholar
Fecher, Benedikt, and Friesike, Sascha. 2014. “Open Science: One Term, Five Schools of Thought.” In Opening Science: The Evolving Guide on How the Internet Is Changing Research, Collaboration and Scholarly Publishing, ed. Bartling, Sönke and Friesike, Sascha, 1747. Cham, Switzerland: Springer.CrossRefGoogle Scholar
Franco, Annie, Malhotra, Neil, and Simonovits, Gabor. 2014. “Publication Bias in the Social Sciences: Unlocking the File Drawer.” Science 345 (6203): 1502–5.CrossRefGoogle ScholarPubMed
Janz, Nicole, and Freese, Jeremy. 2020. “Replicate Others as You Would Like to Be Replicated Yourself.” PS: Political Science & Politics. doi: 10.1017/S1049096520000943.CrossRefGoogle Scholar
Kapiszewski, Diana, and Karcher, Sebastian. 2020. “Transparency in Practice in Qualitative Research.” PS: Political Science & Politics. doi: 10.1017/S1049096520000955.CrossRefGoogle Scholar
Longino, Helen E. 1990. Science as Social Knowledge: Values and Objectivity in Scientific Inquiry. Princeton, NJ: Princeton University Press.CrossRefGoogle Scholar
Lupia, Arthur. 2020. “Practical and Ethical Reasons for Pursuing a More Open Science.” PS: Political Science & Politics. doi: 10.1017/S1049096520000979.CrossRefGoogle Scholar
Lupia, Arthur, and Elman, Colin. 2014. “Openness in Political Science: Data Access and Research Transparency.” PS: Political Science & Politics 47 (1): 1942.Google Scholar
Merton, Robert K. 1973. “The Normative Structure of Science.” In The Sociology of Science: Theoretical and Empirical Investigations, ed. Storer, Norman W., 267–78. Chicago: University of Chicago Press.Google Scholar
Moravcsik, Andrew. 2014. “Transparency: The Revolution in Qualitative Research.” PS: Political Science & Politics 47 (1): 4853.Google Scholar
Oreskes, Naomi. 2019. Why Trust Science? Princeton, NJ: Princeton University Press.Google Scholar
Rohlfing, Ingo, Königshofen, Lea, Krenzer, Susanne, Schwalbach, Jan, and Bekmuratovna R, Ayjeren. 2020. “A Reproduction Analysis of 106 Articles Using Qualitative Comparative Analysis, 2016–2018.” PS: Political Science & Politics. doi: 10.1017/S1049096520001717.CrossRefGoogle Scholar
Rose, Gillian. 1997. “Situating Knowledges: Positionality, Reflexivities, and Other Tactics.” Progress in Human Geography 21 (3): 305–20.CrossRefGoogle Scholar
Solomon, Miriam. 2001. Social Empiricism. Cambridge, MA: MIT Press.CrossRefGoogle Scholar
Women’s Caucus for Political Science. 2017. The Women’s Caucus for Political Science 2017 Statement on DA-RT. https://dialogueondartdotorg.files.wordpress.com/2017/09/wcps-dart-2017.pdf (accessed June 16, 2020).Google Scholar
Wuttke, Alexander. 2019. “Why Too Many Political Science Findings Cannot Be Trusted and What We Can Do about It: A Review of Meta-Scientific Research and a Call for Academic Reform.” Politische Vierteljahresschrift 60 (1): 119.CrossRefGoogle Scholar