WHAT IS THE DELPHI METHOD?

The Delphi method was originally developed to harness the wisdom of a group of experts in a field to determine an anticipated outcome. Usually, it requires an anonymous method to iteratively refine the answer to a question. Key elements of a Delphi method include the assembly of a large and diverse group of experts who anonymously provide their opinions, a mechanism for a structured data collection and feedback, and a facilitator who collects the data and reports them back to the experts for reconsideration and recalibration. In its original implementation by the RAND Air Force corporation of America in the 1950s, this was done via paper-based questionnaires and surveys, but today most of these techniques use some form of an electronic method of data collection and reporting.

The Delphi technique was developed by the RAND Corporation to effectively gather experts’ opinions to predict the probability of enemy attacks in wartime.Reference Jones and Hunter¹ Since then, the Delphi has been used extensively in multiple fields of study, including medicine, nursing, business, public policy, science and technology, and medical education. In medical education, where decisions must often be made with limited information, the consensus methods, including the Delphi, are used extensively for the purposes of developing or renewing curricula, developing assessment tool items, defining competencies, and more recently to determine entrustable professional activities (EPAs).Reference Humphrey-Murto, Varpio and Wood² In this issue of the journal, the modified Delphi has been used to derive essential steps for a novel technical skill (bougie-assisted cricothyroidotomy) and for educational resource development (top EM papers to teach junior trainees).Reference Dharamsi, Gray and Hicks^3, Reference Bazak, Sherbino, Upadhye and Chan⁴ These studies provide excellent examples of how the Delphi can be used to collect expert judgements to arrive at a decision that is arguably considered more credible and defensible than a singular opinion.

STRENGTHS AND WEAKNESSES OF THE DELPHI METHOD

Recent studies have demonstrated the variability in how the Delphi method has been applied. Both studies in this issue of CJEM use a modified Delphi, yet there is no clear consensus on the true definition of the “modified Delphi.” This lack of clear definition has led to considerable confusion and substantial variation in the quality of reporting of Delphi studies. For example, a recent review demonstrated that one-third of medical education Delphi studies failed to report that a literature review on the topic of interest had been conducted, and over half failed to report key aspects such as what background information was provided to participants; the response rates for each round; what formal feedback of group rating was shared between rounds; a statement that anonymity was maintained; and a clear definition of consensus.Reference Humphrey-Murto, Varpio and Wood²

One of the drawbacks of the Delphi method is that the underlying assumption that decision-making in groups leads to wider perspectives and, ultimately, better decisions has not been definitely proven. Critics have suggested that consensus methods generate nothing more than collective ignorance or simply reflect a formalized form of peer pressure.Reference Jones and Hunter¹ In the event of multiple rounds, attrition in participation can lead to a false consensus, because those with dissenting views drop out.

Despite these pitfalls, consensus methods have many potential benefits when rigorously conducted and reported. Table 1 lists potential strengths and weaknesses of the Delphi.

Table 1.

Potential strengths and weaknesses of the Delphi method

CAN WE DO BETTER?

Although modified Delphi methods have their benefits, inconsistent definitions and poor reporting can lead to a lack of scientific credibility. Lack of clarity in the reporting of procedures and methodological choices associated with modified Delphi studies can prevent readers from effectively appraising and interpreting findings. In response to this, many “how to” guidelines have been published.Reference Sinha, Smyth and Williamson^6–Reference Diamond, Grant and Feldman⁸ For a quick reference, we have adapted a list of best practices in Box 1 of this paper.

In addition to rigorously designing, conducting, and reporting Delphi studies, there is a need for further research to understand best practices for defining consensus, identifying participants, exploring the nature of consensus building, determining outcomes, and how to select a consensus method.Reference Humphrey-Murto, Varpio and Wood²

Medical educators have identified Delphi methods as a useful way to approach educational problems that require synthesizing expert opinion to make decisions. The work of the Dharamsi and Bazak teams is a helpful example of different applications of this method and, although generally provides rigorous reporting, could be improved upon. For example, some readers may have desired more information regarding the expert participants in the Bazak study,Reference Bazak, Sherbino, Upadhye and Chan⁴ and the inclusion of otolaryngology or trauma surgeon participants in the Dharamsi article might have provided a more holistic understanding of key steps in this procedure.Reference Dharamsi, Gray and Hicks³

EM researchers and educators who wish to apply this method to consensus decision-making are encouraged to use a systematic approach to study design and implementation, with careful attention to and reporting of published recommendations to ensure methodological rigor. Although word count limitations may provide a barrier to detailed descriptions of the methods, authors can reference method papers in the citations, or submit more detailed methods for consideration for publications as supplemental material. Methodological rigor and transparent reporting are essential to assure readers that the consensus results are applicable to their environment, and to translate expert opinion into practice.