2.1 Ethics in language assessment (Isbell and Kim)

Conflicts of interest are widely acknowledged as a concern in research ethics (e.g., Steneck, Reference Steneck2007) and, in testing, researcher independence is highly valued when evaluating validity (Kane, Reference Kane2013). This presentation reported on the initial findings from a systematic review of high-stakes English proficiency test validation studies, with a focus on studies reported in five leading language testing journals over the last five years (2016–2021). Areas of focus in this presentation included conflict of interest (COI) disclosure and broad trends in research methodology, amongst others. The systematic review revealed that 65% of the validation studies included an identifiable COI statement, and that of the 67 studies with a potential conflict, only three disclosed these COIs. Regarding trends in research methodology, we found that research conducted or otherwise supported by a test developer (in contrast to independently conducted research) featured larger sample sizes and more frequently focused on aspects of validity related to the evaluation of responses, generalization of test scores, and extrapolation of test scores to performance in real-world domains. This presentation concluded with a set of recommendations for journals, authors, test developers, professional organizations, and government organizations for improving accuracy and transparency in COI reporting.

2.2 Editorial ethics in applied linguistics (Gass and Plonsky)

Journal editing is a task that is often laden with ethical dimensions. These tensions stem in part from the different stakeholders involved which include authors and reviewers, of course, but also the discipline as a whole as well as the journal publisher. Given that journal editing is often a high-stakes activity and journal editors are in a position to shape and give direction to a discipline, it is surprising that there is little in the way of training to help editors navigate some of the potential conundrums they inevitably face. Further complicating this role and the ethically-charged decisions editors make is the lack of transparency typically involved in the editorial/review process.

In the presentations we gave at NAU and UU, we sought to view editing through the lens of ethics in many of the decisions we have made as editors. We problematized some of the personal experiences we have encountered as editors of Studies in Second Language Acquisition related to, among other issues, (disputed) authorship, (non-)anonymity, excessive self-citation, non-collegiality by reviewers, and handling submissions by friends and colleagues, to name a few. We concluded by laying out what we see as several paths forward which we feel may lead toward more equitable and less ethically fraught editorial practice including a call for editor training and more research on editorial ethics.

2.3 Moral dilemmas in open access publication (Andringa)

The field of AL seems to be witnessing a gradual transition towards open access (OA) publication and instrument sharing, which stand to increase scientific and societal impact of research, transparency, research quality, and equal access. However, there are also factors that impede our efforts to move towards fully open and transparent scientific research practices. This presentation demonstrated that the choice of journals for publishing articles also involves moral dilemmas. One such moral dilemma pertains to OA journals that are not freely available to authors, which directly impedes equity/equal access. Journals following Gold and Hybrid models, for example, which implement author processing fees to access content, necessarily exclude authors on economic grounds owing to high author processing charges. These journals also tend to be journals of high prestige, which is determined by factors such as perceived quality, name recognition, editorial board, and journal impact factor. Publishing in prestigious journals is generally important for career advancement, making it far more attractive to researchers than OA journals. This presentation concluded with a call on academic societies and advanced career scholars to support and launch Diamond OA publication venues.

2.4 Questionable research practices (Larsson, Plonsky, Sterling, Kytö, Yaw, and Wood)

Questionable research practices (QRPs) are often viewed as the “murky waters” of research ethics and cover practices that fall between ideal behavior and absolute misconduct (e.g., falsification). Whether intentional or not, researchers make choices that may lead them to engage in QRPs (Isbell et al., Reference Isbell, Brown, Chan, Derrick, Ghanem, Gutiérrez Arvizu, Schnur, Zhang and Plonsky2022); for example, with limited space in a manuscript, a researcher might opt to leave out some detailed information in the methods section. Other examples include employing instruments without a strong validity argument and HARKing (hypothesizing after results are known). We developed a field-specific taxonomy (k = 58) for the quantitative humanities using the Delphi method (see Plonsky et al., Reference Plonsky, Larsson, Sterling, Kytö, Yaw, Wood, DeCosta, Rabie-Ahmed and Cinagliasubmitted; Sterling et al., Reference Sterling, Plonsky, Larsson, Kytö and Yaw2023). We then surveyed researchers (n = 230) active in the U.S. and Sweden to investigate (1) what types of QRPs and related ethical considerations they have done and (2) the perceived severity of these QRPs (see Larsson et al., Reference Larsson, Plonsky, Sterling, Kytö, Yaw and Woodunder review). The results showed that 95.7% of the respondents reported having engaged in one or more of the practices listed, with giving the same presentation at multiple conferences being the most common one (in 1–20% of their projects in the past five years). There was a strong negative correlation between frequency and severity.

2.5 Past and present of research ethics training (Wood, Yaw, and Sterling)

This session examined research ethics training literature from the 1980s to 2022, using historical trends and materials to jumpstart discussions about current training practices. By tracing the development of research ethics training literature, an expansion can be noted in focus from ethical guidelines for informed consent in the 1980s to research methods textbooks with dedicated ethics chapters in the 2000s to exploration of micro-ethical decision points and QRPs in the 2010s and beyond (see Yaw et al., Reference Yaw, Plonsky, Larsson, Sterling and Kytö2023).

A survey of the current state of research ethics training in AL revealed three key themes: (1) a focus on research ethics pertaining to informed consent, (2) recurring reference to select decision points in the research process (e.g., selection of the appropriate research design), and (3) ethics presented in single sections or chapters of textbooks. Based on these themes, we provide recommendations for research ethics training materials development: expand ‘research ethics’ to include ethical gray-zone issues, be thorough and explicit in discussing these issues in our materials, and incorporate research ethics as a recurring theme throughout courses and textbooks (see Wood et al., Reference Wood, Sterling, Larsson, Plonsky, Kytö and Yawunder review).

To conclude the session, attendees reflected on personal experiences teaching research methods courses, including the research ethics training materials they use and those they would like to have available.

2.6 Creating training materials (Sterling)

Training materials for research ethics often include case scenarios that contain short summaries of an event along with an ethical complication. These scenarios allow scholars to consider ethical dilemmas within a safe space without having to personally experience them, provide interesting talking points for classroom discussions, and prompt critical thinking and future planning on ways to manage ethical challenges. However, scenarios rarely include solutions, and if they do, from a limited number of voices. Instead, they rely on the reader to evaluate the evidence in the case and learn the intended lesson. For this interactive session, we created a set of five scenarios (link below) that included one or more QRPs from our Delphi study (see Plonsky et al., Reference Plonsky, Larsson, Sterling, Kytö, Yaw, Wood, DeCosta, Rabie-Ahmed and Cinagliasubmitted; Sterling et al., Reference Sterling, Plonsky, Larsson, Kytö and Yaw2023). Audience members read each prompt and then worked in groups to discuss issues related to the case and possible solutions they would offer. We additionally asked participants to submit short write-ups on how they would advise a person facing the issues included in each scenario. These responses were shared as part of our team's online training toolkit for researchers in the quantitative humanities to provide improved self-study and in-class training on QRPs.

2.7 Future directions in ethics training (Hancock, Gass, Isbell, Larsson, Plonsky, and Sterling)

In the NAU symposium, a lively panel discussion was held on navigating the many ethical decision points throughout the quantitative research process. Three key takeaways emerged.

First, quantitative research ethics is not binary: research, researchers, and research-specific decisions are not either ethical or unethical, nor do we benefit from such thinking. The research process requires myriad decisions and a willingness to recognize and wade into their grey areas, thoughtfully, transparently, collaboratively, and nonjudgmentally.

Second, much ethics training happens through faculty deliberately mentoring their graduate students, fostering a culture of researcher reflectivity and reflexivity. This includes explicit instruction as well as actively modeling the process of continuously interrogating the critical decisions and assumptions we make throughout the quantitative research process.

Third, research ethics goes beyond the key decision points in each individual research project, for which faculty mentors, those teaching research methodology, and indeed our professional societies are active in developing and offering guidance. The research process also includes dissemination of that research – manuscript reviewing, journal editing, and publishing more broadly – where little formal guidance currently exists for navigating ethical decision points and where training developed and provided by our professional societies could be quite beneficial.