Accepted Wisdom for Assessing Senior Leaders

Assessment experts understand that strong implementations are based on a solid understanding of the context for targeted roles and jobs, clear ties to the constructs being assessed, reliable and valid assessment measures, clear communication to participants, and involvement of stakeholders across the organization. Accepted wisdom in each of these areas is summarized below.

The Emerging Role of Context

The role of context in understanding the demands of leadership is becoming recognized as an essential aspect of assessment. As a result, organizational stakeholders as well as participants are likely to value contextualized assessment more than generalized assessment. Contextualized assessment can take many forms. Common examples include the use of real job content within an assessment center or high-fidelity video of common leadership scenarios within situational judgment tests. The use of these tools raises the question of how contextual variables should best be factored into leadership assessment and whether the gains in authenticity are worth the tradeoffs that come with increasing levels of specificity. It is possible that taking this perspective too far may diminish our ability to understand leadership capabilities that generalize across situations.

Given the increasing importance of context for understanding and assessing leaders, there is a need to improve the consistency and precision with which we describe and use contextual variables. We should develop taxonomic structures for context that mirror existing taxonomies of leadership behaviors. Such a structure should consider factors at the environmental, organizational, group, and individual levels. In fact, several scholars are vigorously pursuing the development of such taxonomic structures (e.g., Parrigon, Woo, Tay, & Wang, Reference Parrigon, Woo, Tay and Wang2017). No doubt, cross-disciplinary collaborations with fields such as strategic management that are strongly focused on context will greatly facilitate work in this arena.

There are several implications of increasing the focus on context for understanding leadership behavior and designing effective assessment processes. First, the external landscape is continuously evolving. Several broad environmental shifts have a bearing on leadership assessment, such as simultaneous globalization and erosion of traditional borders, and technology advancement that leads to democratization of information, new business models, and unpredictability and volatility in the external environment. Second, the level of analysis of primary interest is usually at the person level, with person-specific description or prediction as the central goal of most leadership assessment. So, what organizations want to know about leaders often comes down to very specific questions, such as how a given individual will perform in a crisis or against a challenge the business is facing. However, these competing forces create a dilemma when assessments are designed: how to address questions of readiness for specific future roles within the context that will exist for the role and acknowledge that context is constantly shifting, thus creating a need to consider leadership capability across a range of likely leadership demands.

Evolving Criteria for Leaders

Just as the context for leadership has become more challenging, so too have the criteria that define successful leadership. Accepted wisdom dictates that organizations should first determine how they are defining an effective leader—both at the individual level as well as at the organizational level—before configuring an appropriate assessment. What do successful leaders do, what makes them distinct from average leaders, and are these factors consistent across different environments? Have the requirements for leadership changed or expanded as the work environment has become more complex? When identifying relevant criteria for evaluating leaders, organizations tend to consider how well the individual fits with the management team and how engaged and committed individuals are in the organization and their intentions to stay. At the individual level, effective leadership is often operationalized by considering judgment and decisiveness, interpersonal skills, and how well individual and team behaviors align with organizational culture and the strategic goals of the business. Given the emphasis on context fit described above, leadership criteria now often include how one responds to the extreme situational characteristics such as VUCA in the modern work context.

From an organizational perspective, leaders are often evaluated by variables such as the performance of their business unit and the engagement levels of the people in their portion of the organization. Here again the rapidly changing environment for leadership may impact the criteria for success. These criteria are likely to vary over time, and this complexity impacts both the effectiveness of assessment and the process of validation when they are used as criteria.

Increasingly research and practice related to leadership are recognizing the value of understanding leadership capability as an emergent quality of a team of leaders, not simply as a function of an individual leader's actions (Day & Dragoni, Reference Day and Dragoni2015). This shared capacity of leadership, which involves looking at the operation and makeup of the full leadership team, may be a better fit in complex organizations, but the work to determine how to best factor a team-level of analysis into assessment and development programs is just beginning. If leadership success is conceptualized at the team level, and the composition and operating climate of the team affects what actions are required for the individual leaders on the team, can assessment processes be designed with enough flexibility to provide an adequate sense of how an individual leader will contribute to the team? When viewed as a team-level construct, the concept of equifinality becomes paramount when defining leadership success at the individual level. In contrast to the dilemma posed above related to context, the definition of leadership at the team level may push the situation-specific versus context-free pendulum back toward generalizable leadership capabilities that will enhance the fit of an individual leader with a range of other leaders who will operate as a team. This emphasis is further justified because the make-up and dynamics of a future team may be difficult to predict.

Moving Beyond Competencies: Measuring Leadership Potential

The emphasis on individual traits as antecedents of leadership has shifted over time in our field (Lord, Day, Zaccaro, Avolio, & Eagly, Reference Lord, Day, Zaccaro, Avolio and Eagly2017). Current research and practice related to leadership may place an overemphasis on competencies, and the development and use of assessments may need to focus beyond them for several reasons. When considering near term roles (within a year or two), it seems reasonable for organizations to focus on competencies to make judgments about readiness, but a broader range of variables is likely needed for making projections about the potential to grow into senior leadership roles over longer spans of time (i.e., over many years or even decades). This suggests that assessments go beyond measuring the skills and competencies observed within the current role to effectively address one's potential, including factors that may underlie the ability to learn and adapt to future demands that are not possible to adequately define at the time of the assessment. Current models of leadership potential tend to emphasize a range of personality, values, and cognitive variables that may contribute to a growth orientation (e.g., Church & Silzer, Reference Church and Silzer2014; Paese, Smith, & Byham, Reference Paese, Smith and Byham2016).

Herein lies a challenge for those who design programs intended to identify leadership potential and accelerate leadership development: Conceptualizing leadership potential through the same lens as leadership readiness for selection or promotion into a new role (where criteria are defined and people are sought who match), even with a focus on different variables, can lead to exclusionary practices that anoint some leaders as having potential and others as lacking. Readiness assessments can be more precise and behavioral because the target role should be definable and within reach. Assessments designed to identify potential may need to be used less strictly because the prediction of the “potential to grow” toward unknown targets, over a broad and unspecified period of time, is inherently less certain, and leaders can take many paths to acquire the capabilities they need for future roles.

Some of the traditional assumptions that serve as the foundation for assessments used for selection may not apply when assessing leadership potential. If one assumes that leadership potential is a construct upon which individuals vary and that selection methods can be applied to identify those who have a lot of it, you run the risk of excluding talented candidates from the development activities that could help them the most. Instead of assuming potential is that trait to be measured, a broader conceptualization may lead to more inclusive practices. For example, the notion of a “development trajectory” might underlie the application of assessments and development activities, where individuals are considered at various points on a path of developmental progress over time (Day & Sin, Reference Day and Sin2011). The slope of progress may rise or fall over time depending on a combination of individual characteristics and environmental opportunities present at a given point. Judgments regarding leadership potential are then considered in terms of where individuals are on their paths and the speed and direction of development. Leadership assessment can occur at various points along the path with a focus on how to best influence an individual's trajectory. The direction and speed of progress are then the primary outcomes to be predicted by measures of potential. This view raises the responsibility of the organization to actively manage potential across the span of leadership development and places emphasis on the skills of managers who must be exceptional developers of leadership talent within their organizations. In contrast, the traditional view simply classifies individuals as high potential and enrolls them in a development program run by the talent management function, thereby simplifying the problem and shifting the responsibility to a different part of the organization.

At an organizational level, accepted wisdom suggests that leadership bench strength should reflect more than a count of developed leaders who are “ready now” to move on to the next level. Rather, the idea of bench strength should be broadened to include the organization's overall capacity for leadership—the aggregated, organization-level capability. Assessment information could then be combined with information about the organizational context to conduct both scenario planning (what to prepare for) and succession planning (who should be prepared).

Influencing Stakeholders and Consumers of Assessment

It seems likely that the effectiveness and longevity of assessment programs deployed at any level, but especially at senior levels, are more affected by the organizational support for the program than by the specific design features of the program. As much as this fact may be recognized by long-tenured practitioners, the steps for improving the success rates of an implementation are too often overlooked. Several elements to consider when seeking support from senior stakeholders who often serve as sponsors for (or barriers to) assessment work inside their organizations were identified by participants at the symposium.

At the outset, it is important to state a clear case for the program and the active use of its outputs. Sustaining an integrated assessment and development process requires that the purpose and value of the tools are made clear to the stakeholders at various levels of leadership to which they apply. It is helpful for assessments to be engaging, challenging, and realistic for the target level; they should yield specific, actionable feedback that can be incorporated into individual development plans. The organizational infrastructure and support for the use of program outputs is stronger when the purpose, content, and use of the assessment program are well aligned.

It is also advantageous to educate stakeholders about the difference between common practices and proven practices. Senior executives may not understand why they should have a vested interest in seeing that good science is applied within an assessment process (cf. McPhail & Stelly, Reference McPhail, Stelly, Scott and Reynolds2010), nor are they necessarily able to distinguish good from bad scientific foundations. Executive stakeholders are seldom aware of or persuaded by academic or even practitioner-based publications, but they may be receptive to the input if packaged appropriately. Unless assessment experts educate executive stakeholders, their attention will be more likely directed to fads and benchmarked practices of untested origin. This bias can be a practical barrier to doing sound applied work, including gathering the data necessary for a successful implementation.

Accepted Wisdom for Assessment-Driven Development Processes

Assessment processes that motivate senior leaders to change and grow should provide these leaders with feedback that clarifies improvements needed to enhance their effectiveness, help them identify the most important development goals to pursue, and hold them accountable for following through on efforts to improve. The ideal assessment-driven development process starts with an assessment report tailored to a senior leader audience—one that conveys in a straightforward manner the dimensions assessed, why these dimensions matter, and how the leader performs on each one. A coach with expertise in the measures and familiarity with the individual's context guides the leader through a process of digesting and making sense of the results, and then setting specific goals for improvement and action plans for implementing those goals. The next level of management is a key participant in this development planning process that typically yields one to three goals that the leader is not only personally energized to pursue but will clearly benefit the organization. The leader and the leader's boss regularly seek feedback from others about the leader's progress, and both are held accountable for achieving visible improvement.

Yet even high-quality assessment, feedback, and development planning processes will not maximize change and growth if the targeted developmental experiences themselves are not powerful and engaging. On-the-job stretch experiences supported by developmental relationships and formal development programs should make up the bulk of the learning opportunities after assessment. Stretch assignments (via special projects, temporary assignments, or job moves) are chosen to provide new challenges and ample opportunity to practice needed skills. Appropriate people are recruited to serve in specific roles to support the leader with current developmental goals (e.g., a role model who can demonstrate targeted skills and share advice, a coach who can help change an ingrained behavior, or a supportive colleague who can encourage the leader). Individuals can also pursue formal development programs that deliver relevant knowledge, additional practice and feedback in a safe environment, and ample opportunities to connect with and learn from other leaders facing similar challenges.

Using Assessment Data to Make Decisions and to Develop

Organizations typically keep processes used to inform decision making about senior leaders (i.e., promotion) independent from those used to provide feedback to leaders for their development. This separation can be the result of different groups being responsible for different HR tasks (e.g., selection, performance management, leader development) and tailoring assessments for their specific tasks. For example, assessments in the selection context are designed to yield overall scores and profiles that predict specific criteria, whereas assessments in the development context are designed to yield in-depth feedback about behaviors and their impact. The separation is also shaped by the understanding that data collected for one purpose might not be useful or appropriate for another. For example, data on stable traits like intelligence are viewed as valuable in the selection context but less so in the development context. Or 360-feedback data collected from coworkers may be more honest when the coworkers know that the data will be used for development rather than organizational decision making.

Yet there is a growing interest in designing integrated assessment processes that can be used for multiple purposes—particularly at senior levels. One driver is efficiency, with a focus on minimizing the time needed from senior leaders and from assessment professionals. Another is the desire to maximize the return on investments made to develop high quality assessments. There is also a movement toward more integrated talent management processes that produce a coherent, shared understanding of individual leaders and their capabilities. This integrated approach contributes to aligning individual and organizational resources to support and accelerate senior leader development. Finally, for some organizations the integration is driven by culture shifts toward increased transparency or by government-mandated protections of individual privacy, both of which compel organizations to share with an individual data collected about that individual, thus making assessment data available to the individual for his or her own development.

Integrated assessment processes are not without their dilemmas. A central concern is the impact of providing assessment results to individuals and subsequent use of those assessments in organizational decision making. For example, debriefing an assessment center exercise with participants can make explicit the behavior that would have led to higher ratings. This, in turn, makes the rating protocols more public and shareable with others who use that knowledge to gain higher ratings when they are assessed. Might doing so not invalidate or compromise the assessment? Another concern is the orientation of individuals as they engage in assessments for decision making versus development. In what ways might impression management tendencies in selection contexts make it difficult to pinpoint development needs? If coworkers are providing input to the assessment, how might the dual purpose of the assessment impact their honesty or willingness to provide critical data?

Talent management professionals who are moving toward a more integrated assessment process combining decision making and leader development are introducing practices for dealing with the dilemmas (Church, Del Giudice, & Margulies, Reference Church, Del Giudice and Margulies2017). For example, they emphasize the importance of being transparent about who has access to what assessment information, documenting and sharing the specifics of what type of data summaries will be shared with whom and for what purpose. Clearly, this arena needs research—both research that examines the validity of concerns raised about integrated process and research that investigates how the benefits of such processes can be maximized and the downsides mitigated.

Partnering With Line Managers to Develop Leaders

Studies abound that show leader shortages across organizations and concerns about both the quality of current leaders and those in the pipeline (DDI & The Conference Board, 2014; DDI, The Conference Board, & EY, 2018). If much is known about assessing and developing senior leaders, then why do so many organizations experience leadership shortages? Why are assessment and development efforts not driving changes in the quality of leaders ready to take on senior roles?

One explanation is a version of the knowing–doing gap. Knowledge about assessment-driven development does not always lead to organizational actions consistent with that knowledge for several reasons. First, if executives rely more on their subjective evaluations of leaders than on data from objective assessments, it is unlikely that the assessment data will be used to identify appropriate development goals. Second, in building a leader development system, talent management professionals may rely too heavily on formal programs and executive coaching because such interventions are under their control and are often what stakeholders expect. To extend the development system to also include stretch assignments and informal coaching by bosses and peers, talent management professionals have to share accountability for outcomes with line managers, many of whom may be less committed to and savvy about development. Closing the knowing–doing gap requires influencing managers throughout the organization: educating them about how formal assessments improve the quality of the data that inform decisions about leaders, enhancing their awareness of human biases in making judgments about others, exploring their resistance to using assessment data, and making the politics of choosing individuals for high-stakes positions discussable. Organizations also need to develop their managers’ capacity and energy for developing others.

Another explanation for leadership shortages is the dynamic nature of senior leader roles. As organizations reinvent themselves in the face of volatile external environments and changing employee expectations, the knowledge and capabilities needed at senior levels grow broader and more complex. These challenges call for even greater intrapersonal maturity, capacity to work collaboratively with diverse others, intersystemic perspective, and the agility to respond to unpredictable demands (Bunker, Hall, & Kram, Reference Bunker, Hall and Kram2010). Thus, the bar for those deemed “ready” keeps rising. In this context, readiness cannot be perceived as an end state but rather an ongoing process of constant preparation. Leadership development systems that grow effective cadres of senior leaders not only deliver assessment and development processes for employees throughout their careers but are also able to fill unanticipated gaps by accelerating development for particular leaders.

The most advanced leader development systems rely on more than formal processes; they rely on an organizational culture where it is common for leaders to coach other leaders, to hold each other accountable for building a strong pipeline of leaders at all levels, and to give one another feedback (McCauley, Kanaga, & Lafferty, Reference McCauley, Kanaga, Lafferty, Van Velsor, McCauley and Ruderman2010). The field would benefit from organization-level research that examines these advanced systems more closely, and that sheds light on how organizations mature their assessment and development practices toward an integrated system embedded in a development culture.

Accepted Wisdom for Validating Assessment Tools

It is easy to get enthralled by the appeal of complex assessments for evaluating senior leaders; however, the fundamental criteria for evaluating these measures are their reliability and validity. Practitioners need to continue to emphasize to organizations the importance of demonstrating the psychometric quality of assessment practices and the value of adhering to professional and legal standards and guidelines. Accepted wisdom dictates taking a broader perspective when thinking about the definition and conceptualization of validity in this context. The challenge is to determine what criteria to use in the validation study, assuming there is sufficient sample size to conduct such a study (McPhail & Jeanneret, Reference McPhail, Jeanneret and Schmitt2012). Clearly, assessments should be linked to the outcomes of effective leadership (e.g., business results), not just to the leader's displayed behaviors. Other factors to consider include employee retention, staff morale, and organizational climate.

When designing a validity study, researchers should start with the end in mind, with hopes to capture criteria that will reflect the impact the leader may have on the business. The organization should have procedures in place to continuously improve the assessments to ensure relevance, and there should be a conscious effort to balance effectively the interests of multiple stakeholders regarding when, how, and what to assess. The resulting information should be packaged so that it is easily understood by the relevant consumers (e.g., organization, individual, and team). That said, it is difficult to determine how best to represent information collected in situations with very small sample sizes for stakeholder consumption.

Given the small number of vacancies at the most senior-level positions, organizations often implement individual assessments to evaluate the applicant pool (Silzer & Jeanneret, Reference Silzer and Jeanneret2011). These high-touch, deep-dive approaches are aimed at offering rich, tailored insights into a person's capabilities, competencies, readiness, and potential for leadership roles. However, sample size constraints make it difficult to evaluate the effectiveness and validity of these assessments. Certainly, accepted wisdom dictates that the tools provide information on job-relevant capabilities and align with business needs, similar to large-scale assessments. Although it may be difficult to demonstrate the operational reliability and criterion-related validity of these individual assessments, researchers and practitioners should be able to use content validity evidence to support the use of assessments, particularly in situations where sample size is an issue. Under some circumstances, evidence from consortium studies and qualitative research techniques may provide useful evidence that builds our understanding of the effectiveness of specific assessment approaches.

Strengthen Evaluation Research on Current Practices

Senior leader assessments tend to rely on high-touch methods that many practitioners have convinced themselves are required to assess accurately a leader's potential and readiness for higher level leadership roles. There is a long track record of the effectiveness of assessment centers and a huge individual assessment practice that justifies its value both in terms of assessment but also in terms of providing meaningful developmental feedback (McPhail & Jeanneret, Reference McPhail, Jeanneret and Schmitt2012). Yet, the field needs more concrete information comparing these comprehensive assessments with those that have a nimbler delivery mode while considering the accuracy of assessment results. Does an organization reap sufficient benefit from comprehensive assessments to justify their cost, time, and other potentially obtrusive effects?

In terms of research associated with traditional assessment centers, practitioners would benefit from additional information and research regarding data integration, a central feature of this assessment methodology. We understand dimensional judgments, but we need more research on integration and overall assessor decision making, including how to integrate individual scores into recommendations (e.g., the impact of adding ratings of dimensional importance) and how assessors might best bridge the gap between their conclusions and the development activities that best address weak competencies. Furthermore, additional work is needed to delineate the most critical role for human judgment in the assessment process and where mechanical aggregation is better suited to the task.

As noted earlier, a key research arena is the potential impact of context. When are situationally specific assessments better at predicting outcomes than measures that focus primarily on the generalizable aspects of leadership? For example, there are situations specific to some job contexts that rarely occur, but when they do they are critical (e.g., data breaches, manufacturing accidents). In these instances, an organization may prefer custom measures. To what degree does information collected from these situation-specific assessments generalize to other leadership situations or are more generic measures better suited for prediction across the range of situations leaders will eventually face on the job?

In a global economy with organizations having a presence in multiple countries, questions about the impact of local and cultural context arise (Ryan & Tippins, Reference Ryan, Tippins, Scott and Reynolds2010). What is the potential impact of the local context on the scoring and interpretation of the assessment results? How do cultural differences and dynamics affect the process of engaging participants when using assessments on a global level?

More rigorous evaluation of assessment-for-development practices is also critical. Does the impact of assessment feedback vary widely based upon the assessment criteria and methods employed? Which assessment practices provide the most insight for senior leaders? What are the keys to ensuring assessments add value when developing senior leaders? What is the impact of using the same assessment results for development and decision making? What system-level factors influence the efficacy of assessment-for-development practices? What are the best evaluation criteria? Moreover, the field would benefit from more rigorous evaluation studies of growth from development interventions based on theory and that yield validity scores for different methods of development.

Addressing these research questions, of course, requires data that are often limited by either access or sample size constraints. Practitioners are more likely to have large data sets but often lack permission to publish and share results based on those data for many reasons, including concerns about legal exposure and the loss of competitive advantage associated with the disclosure of confidential intellectual property. Additionally, practice-based research does not always appeal to top journals. Scholars with access to publication outlets often have limited access to data resources; therefore, what appears in print may at times be based upon small and less generalizable samples. Such issues pose a threat to evidence-based practice and stand to sustain or widen the science–practice gap.

The path forward is likely to be dependent on collaboration among organizations to share data, perspectives, and lessons learned. Although the practice of data sharing may be difficult to put into practice, many researchers and practitioners alike are persistent in their calls for this level of collaboration, increasing the likelihood of it coming to fruition.

Emerging Concepts and Technology

Current conceptualizations of leadership have been shifting from conventional, “exclusive” definitions of leadership that categorize individuals as either leaders or followers toward broader, more inclusive definitions that emphasize leadership as a collective enterprise that can be shared by many individuals within the organization (Day & Dragoni, Reference Day and Dragoni2015; Yammarino, Salas, Serban, Shirreffs, & Shuffler, Reference Yammarino, Salas, Serban, Shirreffs and Shuffler2012). Therefore, researchers should explore the concept of collective leadership capacity at senior levels and identify appropriate criteria for evaluating it. What properties of lateral relationships and social networks characterize effective shared leadership, and how are these properties best assessed? How do leadership teams develop shared mental models and develop their collective capacity to lead? How can the impact of individual leaders be disentangled from their joint impact as a collective? From an even broader perspective, we should investigate and define the concept of community-based leadership, differentiating it from other forms of leadership.

A useful dialogue among researchers and practitioners would be to evaluate whether leadership assessment models should incorporate issues of social justice and fairness in the workplace (e.g., income inequality, fairness in selection, equitable pay, and gender-based harassment). Should “compassion capital” be added to the leader scorecard, with the expectation that management should do more to take care of and advocate for their employees as they work to meet shareholder expectations? Viewed even more broadly, this question can be expanded to an organization's efforts related to corporate social responsibility in general. Opportunities to shape organizational progress can extend far beyond issues of leadership readiness to drive business success into questions of how an organization influences social equality and global health and wellness.

Finally, the digital revolution is generating new tools for assessing individuals and predicting work performance (Chamorro-Premuzic, Winsborough, Sherman, & Hogan, Reference Chamorro-Premuzic, Winsborough, Sherman and Hogan2016). For example, personality can be assessed via social media posts, computers can discern emotional states from voice and facial patterns, and text analysis can predict who is more likely to emerge as a group leader. The digital revolution is also making it easier for organizations to link sources of employee data (e.g., job performance metrics, engagement scores, workplace experiences), creating “big data” that can be mined for better prediction of outcomes important to the organization. What potential do these new tools hold for the assessment and development of senior leaders or for the validation of instrumentation? Assessment professionals need to be at the forefront of exploring ways available data from these nontraditional sources can inform leadership assessment practices. We also need to recognize that future leaders will be more comfortable with modes of measuring leadership capabilities completely outside of our assessment comfort zone (e.g., sole reliance on computer-based assessments void of any human interaction).