Morelli, Potosky, Arthur, and Tippins (2017) are correct in calling for more conceptual models explicitly linking technology to industrial-organizational (I-O) psychology. As these authors note, in the absence of models and theories of technology to guide the research and practice of I-O psychology, the field runs the risk of chasing the impacts of specific technological innovations and devices rather than guiding organizations on best practices regarding the use of technology. Building theories and models that directly involve technology and placing them within individual psychological and larger organizational processes provides researchers with a way to stay ahead of the fast pace of technological innovation and anticipate its effects on measurement and prediction. Moreover, there are aspects to the use of technology that I-O psychologists are uniquely qualified to consider, including legal considerations (e.g., accessibility concerns), ethical questions (e.g., access in disadvantaged communities), practical concerns (e.g., user and target reactions), and measurement issues (e.g., construct irrelevant variance). In this commentary, we present two main points of consideration that demonstrate how I-O psychologists might use and create technology to improve assessment. First, we argue that technology can improve the measurement of psychological variables if we critically consider how technology can positively influence various parts of response behavior. Additionally, we encourage future research to consider the effects of technology in I-O psychology more comprehensively by extending the emphasis on psychological processes beyond cognition and behavior to include affect and motivation.

Cucina, Walmsley, Gast, Martin, and Curtin (2017) started an important dialogue about survey key driver analysis (SKDA). We believe that promoting more useful and valid ways to understand survey data is critical not only for the organizations we serve, but also for advancing the relevancy of our field. We use the terms useful and valid quite intentionally. “Useful” is driven by our practitioner side, but “valid” is driven by our science side. It is the science that often sets industrial and organizational (I-O) psychology apart from other fields. But in some ways, it also holds us back from being timely and relevant. Overall, we believe that the focal article erred too much on the science side.

In their focal article, Tett, Hundley, and Christiansen (2017) stated in multiple places that if there are good reasons to expect moderating effect(s), the application of an overall validity generalization (VG) analysis (meta-analysis) is “moot,” “irrelevant,” “minimally useful,” and “a misrepresentation of the data.” They used multiple examples and, in particular, a hypothetical example about the relationship between agreeableness and job performance. Four noteworthy problems with the above statements, other similar statements elsewhere in Tett et al.’s article, and their underlying assumptions are discussed below along with alternative perspectives.

The Society for Industrial and Organizational Psychology (SIOP) and its Foundation have increasingly focused on linking science to practice, and to that end have partnered with Society of Human Resource Management (SHRM) and its Foundation to create the Human Resource Management (HRM) Impact Award for firms demonstrating great practices in evidence-based human capital and management practices. Jack in the Box, the 2016 winner of this award, was so honored because of its integration of surveys and analytic models with leadership practices to understand, predict, and manage the right people drivers of important business outcomes. Instead of a research paper, we have chosen to share their story through conversational interview questions with the various stakeholders who contributed to the journey leading to the award and, in so doing, provide key lessons for others wishing to increase the impact of human capital in their organizations.

Tett, Hundley, and Christiansen (2017) make a compelling case against meta-analyses that focus on mean effect sizes (e.g., rxy and ρ) while largely disregarding the precision of the estimate and true score variance. This is a reasonable point, but meta-analyses that myopically focus on mean effects at the expense of variance are not examples of validity generalization (VG)—they are examples of bad meta-analyses. VG and situational specificity (SS) fall along a continuum, and claims about generalization are confined to the research question and the type of generalization one is seeking (e.g., directional generalization, magnitude generalization). What Tett et al. (2017) successfully debunk is an extreme position along the generalization continuum significantly beyond the tenets of VG that few, if any, in the research community hold. The position they argue against is essentially a fixed-effects assumption, which runs counter to VG. Describing VG in this way is akin to describing SS as a position that completely ignores sampling error and treats every between-sample difference in effect size as true score variance. Both are strawmen that were knocked down decades ago (Schmidt et al., 1985). There is great value in debating whether a researcher should or can argue for generalization, but this debate must start with (a) an accurate portrayal of VG, (b) a discussion of different forms of generalization, and (c) the costs of trying to establish universal thresholds for VG.

I am concerned about industrial and organizational (I-O) psychology's relevance to the gig economy, defined here as the broad trends toward technology-based platform work. This sort of work happens on apps like Uber (where the app connects drivers and riders) and sites like MTurk (where human intelligence tasks, or HITs, are advertised to workers on behalf of requesters). We carry on with I-O research and practice as if technology comprises only things (e.g., phones, websites, platforms) that we use to assess applicants and complete work. However, technology has much more radically restructured work as we know it, to happen in a much more piecemeal, on-demand fashion, reviving debates about worker classification and changing the reality of work for many workers (Sundararajan, 2016). Instead of studying technology as a thing we use, it's critical that we “zoom out” to see and adapt our field to this bigger picture of trends towards a gig economy. Rather than a phone being used to check work email or complete pre-hire assessments, technology and work are inseparable. For example, working on MTurk requires constant Internet access (Brawley, Pury, Switzer, & Saylors, 2017; Ma, Khansa, & Hou, 2016). Alarmingly, some researchers describe these workers as precarious (Spretizer, Cameron, & Garrett, 2017), dependent on an extremely flexible (a label that is perhaps euphemistic for unreliable) source of work. Although it's unlikely that all workers consider their “gig” a full time job or otherwise necessary income, at least some workers do: An estimated 10–40% of MTurk workers consider themselves serious gig workers (Brawley & Pury, 2016). Total numbers for the broader gig economy are only growing, with recent tax-based estimates including 34% of the US workforce now and up to 43% within 3 years (Gillespie, 2017). It appears we're seeing some trends in work reverse and return to piece work (e.g., a ride on Uber, a HIT on MTurk) as if we've simply digitized the assembly line (Davis, 2016). Over time, these trends could accelerate, and we could potentially see total elimination of work (Morrison, 2017).

The lengthy and complex focal article by Tett, Hundley, and Christiansen (2017) is based on a fundamental misunderstanding of the nature of validity generalization (VG): It is based on the assumption that what is generalized in VG is the estimated value of mean rho ($\bar{\rho}$). This erroneous assumption is stated repeatedly throughout the article. A conclusion of validity generalization does not imply that $\bar{\rho}$ is identical across all situations. If VG is present, most, if not all, validities in the validity distribution are positive and useful even if there is some variation in that distribution. What is generalized is the entire distribution of rho ($\bar{\rho}$), not just the estimated $\bar{\rho}$ or any other specific value of validity included in the distribution. This distribution is described by its mean ($\bar{\rho}$) and standard deviation (SDρ). A helpful concept based on these parameters (assuming ρ is normally distributed) is the credibility interval, which reflects the range where most of the values of ρ can be found. The lower end of the 80% credibility interval (the 90% credibility value, CV = $\bar{\rho}$ – 1.28 × SDρ) is used to facilitate understanding of this distribution by indicating the statistical “worst case” for validity, for practitioners using VG. Validity has an estimated 90% chance of lying above this value. This concept has long been recognized in the literature (see Hunter & Hunter, 1984, for an example; see also Schmidt, Law, Hunter, Rothstein, Pearlman, & McDaniel, 1993, and hundreds of VG articles that have appeared in the literature over the past 40 years since the invention of psychometric meta-analysis as a means of examining VG [Schmidt & Hunter, 1977]). The $\bar{\rho}$ is the value in the distribution with the highest likelihood of occurring (although often by only a small amount), but it is the whole distribution that is generalized. Tett et al. (2017) state that some meta-analysis articles claim that they are generalizing only $\bar{\rho}$. If true, this is inappropriate. Because $\bar{\rho}$ has the highest likelihood in the ρ distribution, discussion often focuses on that value as a matter of convenience, but $\bar{\rho}$ is not what is generalized in VG. What is generalized is the conclusion that there is validity throughout the credibility interval. The false assumption that it is $\bar{\rho}$ and not the ρ distribution as a whole that is generalized in VG is the basis for the Tett et al. article and is its Achilles heel. In this commentary, we examine the target article's basic arguments and point out errors and omissions that led Tett et al. to falsely conclude that VG is a “myth.”

Morelli, Potosky, Arthur, and Tippins (2017) articulate a strong need for industrial and organizational (I-O) psychologists to develop a more theory-based understanding of the role of technology in employee selection and assessment. We agree with their concerns but argue that this issue should include examination of how technology impacts training also. Researchers have noted that training is increasingly important for firms, and technology-enhanced training can improve learning and transfer (Ford & Meyer, 2013). However, the arguments that the authors make about the need for a theory-driven approach for examining the impact of technology on selection applies to training outcomes as well. Although considerable evidence exists that workplace training is effective and that technology can impact the success of training, there has been less theory-driven research exploring how technology can enhance or detract from training success. Researchers have already identified several variables related to technology that promote learning, but one variable that remains consistently unexplored in the organizational literature is immersion. This research is particularly important given how increasingly accessible immersive technology, such as virtual reality (VR), is becoming. Thus, we argue that as virtual training environments become more widely available, the variable of “immersion” in training environments is a particularly important one that warrants additional research.

With this issue of the Industrial and Organizational Psychology: Perspectives on Science and Practice (IOP) journal, the Practice Forum concludes its first full year of operation. The Practice Forum published its first manuscript in Volume 9, Issue 4, and since then, three additional papers have been published.

Tett, Hundley, and Christiansen (2017) argue that the concept of validity generalization in meta-analysis is a myth, as the variability of the effect size appears to decrease with increasing moderator specificity such that the level of precision needed to deem an estimate “generalizable” is actually reached at levels of situational specificity that are so high as to (paradoxically) refute an inference of generalizability. This notion highlights the need to move away from claiming that effects are either “generalizable” or “situationally specific” and instead look more critically and less dichotomously at degrees of generalizability, or effect size variability.

Traditional change management approaches that focus on linear models and top-down control have proved less than adequate in addressing organizational change within the complexity and speed of today's unprecedented change. Researchers have suggested that by developing greater workforce agility, companies may be better positioned to manage or moderate rapid change and use this capability as a competitive advantage. Complementing current strategies with a different approach to managing change focused on individual agility and resilience may be a first step. This article focuses on the development, validation, and practical application of an employee agility and resilience measurement scale as part of a program in support of an alternative approach to managing organizational change. Results indicate that focusing on individual agility and resilience can prepare employees to handle uncertainty more successfully by adapting to change quicker and managing stress more effectively.