INTRODUCTION
Innovation in medical education is becoming an increasingly common form of scholarship.Reference Boyer 1 , Reference Glassick 2 Emergency medicine (EM) has contributed immensely to this field, and our specialty is poised to continue to leading in the future.Reference Yarris and Deiorio 3 In the face of increased public accountability, reduced working hours, competency-based medical education (CBME),Reference Frank, Snell and Cate 4 and new educational delivery models, modern medical education is in a state of transition. 5 Simultaneously, the number of outlets for the publication of innovative work is increasing. Although innovations were historically shoe-horned into standard research report formats in the past, many medical education and EM journals now have specific sections or categories for innovation reports. 6 , 7 One prominent example is the Brief Educational Report section in the CJEM. Reference Sherbino 8 Despite this increase in outlets for dissemination of innovative work, there is no clear consensus statement or set of guidelines to inform educators hoping to publish papers about their medical education innovations.
To innovate is defined in the Oxford Dictionaries as: “[To] make changes in something established, especially by introducing new methods, ideas, or products.” 9 Innovations in medical education are often developed to improve a program or to solve a problem. Examples include curricular improvements, development of assessment tools, or creation of faculty development programs.Reference Blanchard, Nagler and Artino 10 BarrowsReference Barrows 11 points out that many publications of educational innovations have historically taken the form of descriptions of the innovations with an evaluation of learner or teaching satisfaction or an assessment of learner knowledge acquisition immediately post-implementation. He argues that these restricted evaluations are of “limited usefulness and are subject to bias and error, and they provide no information as to whether the information or skills acquired will be recalled and applied.”Reference Barrows 11 To be publishable, manuscripts must deal with topics that matter to the teachers, administrators, and scholars in the medical education community.Reference McGaghie and Webster 12 If so, this begs the question: What are the key features of a well-written educational innovation report?
The realm of scholarship, publication, and career advancement has been referred to as an ‘alien culture’ to junior clinician educators, with many finding scholarly writing and dissemination strategies foreign and unfamiliar.Reference McGaghie and Webster 12 Many innovations in medical education remain as improvements at their site of development only, and they are never successfully disseminated.Reference Barrows 11 , Reference Kanter 13 For our specialty to benefit from innovative work, innovators must describe and disseminate their work so that others can learn from, adopt, or build on these new techniques.Reference Barrows 11 Published innovations move the work beyond local change, to benefit many more trainees, physicians, educators, administrators, and the public.Reference Blanchard, Nagler and Artino 10 These new and improved ideas are key drivers of progress in medical education.Reference Kanter 13 Because scholarly publication is often a key performance marker for academic health professionals, publishing is additionally advantageous for career development.Reference Blanchard, Nagler and Artino 10
It was the goal of this study to derive a set of evidence-based features of high-quality innovation reports in EM to assist clinician educators hoping to publish innovation studies and help current educators assess innovation papers for quality.
METHODS
Database search
A scoping review, as described by Arksey and O’Malley,Reference Arksey and O’Malley 14 was undertaken to collate existing literature on how to publish an innovation report in medical education. The MEDLINE, EMBASE, and ERIC databases were searched along with Google Scholar on January 21, 2016, using “and/or” combinations of variations of the following keywords: “innovation,” “advancement,” “medical education,” “publication,” “guide,” “manual,” “instruction,” “improvement,” “dissemination,” and “how-to.” A librarian was consulted to assist with the terms and databases selected. We chose these databases because they captured the bulk of published material relating to medical education and medical education innovation. All searches were limited to English-language papers but were not limited in publication date. Titles were reviewed initially by two investigators independently (AKH, CH). Articles were included for subsequent review at each stage if the reviewing investigator felt the title (or abstract) indicated that the content of the article would in any way help answer the question: What are the key features of a high-quality medical education innovation report? Innovation reports themselves were excluded. A calibration step was undertaken after each of 10 titles to discuss rationale for inclusion. At 50 titles, agreement was perfect (κ=1). We hypothesize that perfect agreement was a result of broad inclusion criteria and substantial discussion between investigators prior to initiation of the title review. Subsequent titles and abstracts were reviewed by one of the two investigators.
Hand search
Anticipating the paucity of journal articles that we would identify in our database search, we also performed a hand search, looking for other guidelines or instructions for the publication of education innovations. The websites, publication guidelines, guidelines for authors, and submission portals of PubMed-indexed journals that focused on EM and Medical Education were searched for potentially relevant information along with a selection of pediatric EM and simulation-focused journals chosen by the study investigators by consensus based on perceived relevance. The titles of all MEDLINE-indexed journals in EM and medical education were reviewed by one investigator (CH). Several were excluded from the hand search based on an alternative focus (e.g., critical care), journal inactivity, or duplication. A complete list of journals searched is included as a supplemental appendix. Additionally, the reference lists of the included full-text articles from the previously mentioned database searches and hand search were reviewed for articles warranting inclusion per the criteria mentioned previously.
Analysis
Two investigators (AKH, CH) independently reviewed the selected full-text articles, including text, diagrams, figures, and tables to extract quality indicators. A master list of quality indicators was compiled, and a thematic analysis was performed by a single investigator (AKH) to group and reduce this list. This investigator is a board-certified EM physician who has completed a Masters of Medical Education degree and has experience in the successful publication of education innovations and peer-review for medical education and EM journals. The thematic analysis followed the analysis technique (i.e., charting and thematic analysis) described by Arksey and O’Malley.Reference Arksey and O’Malley 14 To enhance the rigour of this analysis, three other investigators (CH, TMC, FB) reviewed and edited the groupings and list to generate a final list by consensus.
Demographics of publication venues
We sought to establish a list of venues for the publication of education innovations, which may be usable by authors hoping to publish education innovations in EM. Concurrent with the process of literature review and hand searching, a list of venues was compiled by the reviewing authors (AKH, CH). Medical education journals were included if they published a section or type of publication specifically relating to innovations. EM journals were included if they contained either a Medical Education section, or a specific section or publication category for educational innovations.
RESULTS
Database search
The MEDLINE search yielded 419 citations. We reviewed the titles of each citation for relevance or duplication, and eliminated 389 citations. These citations underwent an abstract review, which then excluded an additional 26 citations, leaving 4 citations. The search of EMBASE yielded 584 citations. We reviewed the titles of each citation for relevance or duplication, eliminating 558 citations. Abstract review excluded an additional 20 citations, leaving 6 citations. The search of the ERIC database yielded 283 citations. We reviewed the titles of each citation for relevance or duplication, which eliminated 270 citations. All remaining excluded citations were eliminated by abstract review. The Google Scholar search yielded 2,130,000 results. As presented in the search strategy by Chan et al.,Reference Chan, Wallner and Swoboda 15 the abstracts for the top 500 search results were reviewed. Eight abstracts were deemed relevant. These citations underwent an abstract review, which excluded an additional six citations, leaving two citations. As outlined in Figure 1, unique articles of 1,786 were identified via database searches in total, and 10 were retained for full text review. The hand search yielded an additional 27 articles, and reference searching yielded 1 article. In total, we performed a full-text review of 38 articles and excluded 24 records based on irrelevance as determined by a consensus of the reviewing investigators (AKH, CH).
Flow diagram for literature review.
Thematic analysis
Table 1 outlines the complete list of themes and quality indicators for an innovation report derived from the thematic analysis of quality indicators found in the literature. The sources of the items are indicated.
Features of a high-quality innovation report
Demographics of publication venues
As summarized in Table 2, we identified multiple potential venues for the publication of education innovations. Whereas innovations are published often as original research articles in EM and medical education journals, some journals have a dedicated section or category dedicated to innovations, or indicate in their instructions for authors that educational innovations are encouraged. Blanchard et al.Reference Blanchard, Nagler and Artino 10 provided an excellent summary of venues in medical education journals. Within EM-specific journals, there are a few noteworthy examples. For example, CJEM has a section titled Brief Educational Report,Reference Sherbino 8 which publishes articles on innovative educational advances in EM. Additionally, a new sister journal of Academic Emergency Medicine titled, Academic Emergency Medicine Education and Training,Reference Kline and Promes 25 calls for authors to submit pieces on innovations in a number of areas (e.g., curriculum development, instructional strategies, validity evidence development, assessment, quality improvement, and faculty development).
Innovation report publication venues
* Extracted from Blanchard et al. 7 , Reference Blanchard, Nagler and Artino 10
† Self-reported impact factor as of October 2016.
‡ Not considering submissions as of October 2016.
DISCUSSION
Whereas evaluating and disseminating work about innovations is not new, the concept of the innovation report is relatively novel. As such, there is limited guidance in the literature about markers of quality in innovation reports. Without consensus-based standards, each journal has provided a unique set of guidelines for this type of publication. We performed a scoping review to develop a list of recommendations, advice, and guidelines for the publication of education innovations. We have extracted key items from these resources to establish a set of quality markers for innovation reports.
The themes established in our analysis parallel Kern’s six-step curricular development framework: 1) problem identification, 2) needs assessment of targeted learners, 3) goals and objectives, 4) educational strategies, 5) implementation, and 6) evaluation and feedback.Reference Kern, Thomas and Howard 26 Key features such as problem identification, needs assessment, and implementation are quite similar between Kern’s framework and our list. Our list expands on these concepts with an emphasis on transparency and reproducibility in the development and implementation process, the imperative of building on existing literature and theory, and providing a more detailed process of evaluation, including reflective practice. The additional theme of reporting and dissemination highlights that this list is not just about key features of an innovation, but rather a high-quality innovation report.
A review of the quality markers suggests that an ideal innovation report is written to share ideas that address a pressing need or existing problem that is of broad interest. Educational innovation and the scholarship around it create a community of scholars where the interaction works to stimulate ideas and move the field forward. Dissemination serves to benefit individuals in the form of career development and the community of educators by driving progress in medical education. For publication in academic journals, innovations need to be novel and transferrable to other contexts. It is important for such reports to paint a clear picture of the context of the problem and the solution. Ideally, innovations are grounded in education theory/frameworks and evaluated in some capacity beyond satisfaction and knowledge acquisition. For those looking for guidance in evaluating their innovations, there are several potential frameworks that may be used for innovation or program evaluation, including the Kirkpatrick Model,Reference Kirkpatrick 27 McGaghie 3Ts translational framework,Reference McGaghie 28 and Logic Model.Reference Wholey 29 Finally, a thoughtful analysis of the impact or potential impact of the innovation and a description of how it fits in the “big picture” should be included. Overall, KanterReference Kanter 13 describes the ideal innovation report as one that provides insight that fosters a cycle of progressive thinking and leads to further innovation and development.
We hope that the list generated by our scoping review will be of specific value to junior EM educators. The heterogeneity of innovation reports, variable requirements of journals, and the relative novelty of this category of manuscript have all contributed to an absence of, or confusion relating to, guidance for publishing. The value of this list is in its clarity and generalizability, being applicable to innovation reports across multiple fields and specific journals. With this list, derived from the best available literature, junior educators can perform strong innovation research and produce subsequent manuscripts. In the future, this list may be refined, built-upon, and consolidated via a consensus process with education-research methodologists, knowledge translation specialists, and journal editors to create a set of formal guidelines similar to PRISMAReference Moher, Liberati and Tetzlaff 30 and CONSORTReference Schulz, Altman and Moher 31 publications.
LIMITATIONS
There are multiple important limitations in our work. Firstly, there is the difficulty defining the medical education innovations type of publication. Innovations are sometimes published as original research using traditional quantitative or qualitative methodology. As such, there is overlap between what may now be termed an innovation report and other publication types. For the purposes of our study, we focused on brief or summarized, educational innovation reports, as exemplified by the Brief Educational Report section in CJEM. Reference Sherbino 8 It is important to note that this list of quality markers is an aggregate list of markers derived from multiple papers, editorials, and guidelines for authors, and does not necessarily correspond with the contributor guidelines for the Brief Educational Report.
There are also limitations of our literature review. We chose a scoping review as a way to capture key concepts related to quality in innovation reports. We felt that a systematic review would be inappropriate, as such reviews aim to answer narrow and specific questions.Reference Arksey and O’Malley 14 We recognized that our question would be difficult to answer given a perceived paucity of existing literature and restriction to English-language papers, so we added a more detailed hand search of journals and electronic guidelines to find additional relevant contributions. Interestingly, our hand search yielded the same number of articles as the traditional database search. This could be an indicator of an overly specific or inadequate database search. Most of the documents found in the hand search were “guidelines for authors” or “aims and scope” submission information from Web-based documents, which are not typically found via systematic database searches of the peer-reviewed literature.
An additional limitation of the literature review was our use of only one of two investigators to review most of the titles and abstracts. This decision was informed by an initial perfect agreement in the review of the first 50 titles. Including additional reviewers might have increased the number of articles included; however, the effect of this is hopefully minimized by the very broad inclusion criteria (single question), with a goal of inclusion rather than exclusion. It can also be noted that over half of the articles were excluded after a full-text review, giving additional evidence that the search was reasonably inclusive. Despite this, there is the potential for introduction of bias without a double-review of all titles.
CONCLUSIONS
The publication and dissemination of innovations are critical for the collective benefit of the community of medical educators and trainees. Our study has attempted to summarize the existing material providing guidance for the publication of medical education innovations and present a list of quality markers for innovation reports. We anticipate that this list could be used by medical education innovators as a set of features to guide the development and publication of medical education innovations in EM. The next steps include consolidating and prioritizing our list of findings to create formal reporting guidelines for innovation reports in medical education.
Competing interests: None declared.
Supplementary Material
To view supplementary material for this article, please visit https://doi.org/10.1017/cem.2017.28
