Hostname: page-component-77f85d65b8-v2srd Total loading time: 0 Render date: 2026-03-27T18:32:29.627Z Has data issue: false hasContentIssue false
  • English
  • Français

A writer’s guide to education scholarship: Quantitative methodologies for medical education research (part 1)

Published online by Cambridge University Press:  26 April 2017

Brent Thoma ,
Paola Camorlinga ,
Teresa M. Chan Open the ORCID record for Teresa M. Chan [Opens in a new window] ,
Andrew Koch Hall ,
Aleisha Murnaghan  and
Jonathan Sherbino
Brent Thoma*
Affiliation:
Department of Emergency Medicine, University of Saskatchewan, Saskatoon, SK
Paola Camorlinga
Affiliation:
Department of Emergency Medicine, University of British Columbia, Vancouver, BC
Teresa M. Chan
Affiliation:
Division of Emergency Medicine, Department of Medicine, McMaster University, Hamilton, ON
Andrew Koch Hall
Affiliation:
Department of Emergency Medicine, Queen’s University, Kingston, ON
Aleisha Murnaghan
Affiliation:
Department of Emergency Medicine, University of Ottawa, Ottawa, ON.
Jonathan Sherbino
Affiliation:
Division of Emergency Medicine, Department of Medicine, McMaster University, Hamilton, ON
*
Correspondence to: Dr. Brent Thoma, Room 2646, Box 16, 103 Hospital Drive, Saskatoon, SK, S7N 0W8; Email: brent.thoma@usask.ca

Abstract

Background

Quantitative research is one of the many research methods used to help educators advance their understanding of questions in medical education. However, little research has been done on how to succeed in publishing in this area.

Objective

We conducted a scoping review to identify key recommendations and reporting guidelines for quantitative educational research and scholarship.

Methods

Medline, ERIC, and Google Scholar were searched for English-language articles published between 2006 and January 2016 using the search terms, “research design,” “quantitative,” “quantitative methods,” and “medical education.” A hand search was completed for additional references during the full-text review. Titles/abstracts were reviewed by two authors (BT, PC) and included if they focused on quantitative research in medical education and outlined reporting guidelines, or provided recommendations on conducting quantitative research. One hundred articles were reviewed in parallel with the first 30 used for calibration and the subsequent 70 to calculate Cohen’s kappa coefficient. Two reviewers (BT, PC) conducted a full text review and extracted recommendations and reporting guidelines. A simple thematic analysis summarized the extracted recommendations.

Results

Sixty-one articles were reviewed in full, and 157 recommendations were extracted. The thematic analysis identified 86 items, 14 categories, and 3 themes. Fourteen quality evaluation tools and reporting guidelines were found.

Discussion

This paper provides guidance for junior researchers in the form of key quality markers and reporting guidelines. We hope that quantitative researchers in medical education will be informed by the results and that further work will be done to refine the list of recommendations.

Résumé

Introduction

La recherche quantitative est l’une des nombreuses méthodes de recherche utilisées pour aider les éducateurs à mieux comprendre les questions en matière d’enseignement de la médecine. Toutefois, peu de recherche a été faite sur la manière de s’y prendre pour réussir à publier des articles dans le domaine. Aussi avons-nous mené un examen de la portée afin de relever les principales recommandations et les grandes lignes directrices sur l’établissement de rapports en ce qui concerne la recherche quantitative en enseignement et la production de travaux scientifiques.

Méthodes

Les auteurs ont compulsé les bases de données Medline, ERIC et Google Scholar à la recherche d’articles publiés en anglais, entre 2006 et janvier 2016 à l’aide des termes research design, quantitative, quantitative methods et medical education. Une recherche manuelle de références complémentaires a aussi été effectuée durant l’examen des textes en version intégrale. Les titres et les résumés analytiques ont fait l’objet d’un examen par deux auteurs (BT, PC), et les articles ont été retenus s’ils portaient principalement sur la recherche quantitative en enseignement de la médecine et s’ils présentaient des lignes directrices sur l’établissement de rapports ou s’ils faisaient état de recommandations sur la manière d’effectuer de la recherche quantitative. Cent articles ont été examinés en parallèle, dont les 30 premiers aux fins d’étalonnage, et les 70 autres, aux fins du calcul de la valeur kappa de Cohen. Deux examinateurs (BT, PC) ont procédé à l’examen des textes en version intégrale, puis en ont dégagé les recommandations et les lignes directrices sur l’établissement de rapports. C’est à l’aide d’une simple analyse thématique que les recommandations ont ensuite été résumées.

Résultats

Les auteurs ont procédé à l’examen de 61 articles en version intégrale, et dégagé 157 recommandations. L’analyse thématique a permis de diviser la matière en 3 thèmes, en 14 catégories et en 86 éléments. Il s’est dégagé finalement de la recherche 14 outils d’évaluation de la qualité et lignes directrices sur l’établissement de rapports.

Discussion

L’article offre aux jeunes chercheurs un guide présentant les principaux marqueurs de qualité et les grandes lignes directrices sur l’établissement de rapports. Nous espérons que les résultats obtenus sauront guider les futurs spécialistes de la recherche quantitative en enseignement de la médecine et que d’autres travaux de recherche seront entrepris afin d’étoffer la liste des recommandations.

Keywords

academic writingeducation scholarshippublishingquantitative research

Information

Type
Resident Issues
Information
Canadian Journal of Emergency Medicine , Volume 20 , Issue 1 , January 2018 , pp. 125 - 131
Copyright
Copyright © Canadian Association of Emergency Physicians 2017 

INTRODUCTION

In 2013 the Canadian Association of Emergency Physicians (CAEP) conducted an inaugural academic symposium to develop a list of recommendations to promote educational scholarship among emergency physicians in Canada.Reference Bandiera, LeBlanc and Regehr 1 - Reference Bhanji, Cheng and Frank 3 The resulting manuscripts emphasized the importance of educational scholarship among emergency physicians in Canada and outlined the systemic and institutional supports necessary to develop clinician educators. Subsequently, this was identified as an area of leadership for emergency physicians.Reference Gordon 4 , Reference Sinclair, Worthington and Joubert 5 Despite this success, it is challenging to begin conducting research in medical education.

Most residents and physicians in emergency medicine have familiarity with the critical appraisal of quantitative diagnostic and therapeutic trials. However, translating this familiarity into research success remains challenging.Reference Perry, Snider and Artz 6 This is especially true in medical education, where biomedical methodologies are often proposed to study questions that are more appropriately investigated using techniques developed in the social sciences.Reference Schuwirth and van der Vleuten 7 The perception that medical education research is “easy” relative to clinical research also persists,Reference Schuwirth and van der Vleuten 7 despite the availability of numerous masters and PhD level programs focused on preparing trainees to conduct rigorous medical education research.

Herein, as part of a series of writer’s guides aimed at residents and junior scholars, we describe a scoping reviewReference Arksey and O’Malley 8 of the literature that outlines a list of recommendations for conducting and publishing quantitative educational research. In addition, we present a list of guidelines that establish the reporting and writing standards relevant to this field. We hope that this writer’s guide will aid junior investigators as they conduct quantitative research in medical education.

METHODS

Search methodology

The authors conducted a focused literature review of three databases. An initial search using the MEDLINE database was restricted to English-language articles published from 2006 through January 2016 using “and/or” combinations of the following search terms: “research design,” “quantitative,” “quantitative methods,” and “medical education.” A second search using ERIC database was performed of English language articles published from 2006 through January 2016 using “and/or” combinations of the following search terms: “research design,” “quantitative,” “quantitative methods,” and “medical education.” A final search was conducted using Google Scholar, which was limited to English-language papers from 2006-2016 using the keywords “medical education” and “quantitative research.” These articles were amalgamated in Mendeley, and duplicates were removed.

Inclusion criteria

Articles focused on quantitative research in medical education that outlined reporting guidelines, discussed the evaluation of quality in quantitative research, or provided advice on conducting quantitative research. Articles that focused on other forms of education scholarship, mixed methodologies, or did not address any of the study questions were excluded.

Article review

The results of the literature search were uploaded into Mendeley for analysis. Two authors (BT, PC) developed and performed the literature search and conducted the title and abstract review. The first 30 articles were reviewed in parallel to calibrate the reviewers. The subsequent 70 articles were reviewed separately, but in parallel. A kappa value was calculated to quantify inter-rater reliability. The remaining articles were then divided between the two authors (BT, PC) and reviewed by a single reviewer. A hand search of the reference lists of the included articles was conducted to identify those that were not found in the literature search.

A full text review of the included articles was conducted by two authors (BT, PC) with disagreements resolved through a discussion resulting in consensus. During the full text review, any quality evaluation criteria or reporting guidelines mentioned in the papers were abstracted along with recommendations for the conduct of quantitative medical education research.

Analysis

During the full text review, a list of quality markers was developed by two reviewers (BT, PC) by extracting recommendations mentioned in text, figures, charts, diagrams, pictures, or tables. These recommendations were annotated and organized by article from which they were extracted. Two authors (BT, PC) performed a simple thematic analysis refining and aggregating the recommendations into a final common list. Items deemed alike were amalgamated into a single descriptive statement (Table 1). Simultaneously, during the full text review, reporting guidelines and quality assessment tools that were relevant to quantitative research and were mentioned in the text, figures, charts, diagrams, pictures, or tables were extracted and compiled.

Table 1 Results of the qualitative analysis of literature-based recommendations for the performance of quantitative medical education research

RESULTS

Scoping review

The results of the literature review are outlined in Figure 1. The database search identified 356 articles (270 from Medline and 88 from ERIC with 2 duplicates), of which 46 were included on title/abstract review. The Google Scholar search conducted on January 18, 2016 returned 895,000 results. Of the 250 which were reviewed, 7 were included on title/abstract review. An additional 8 articles were included from a review of the references of the included studies.

Figure 1 Flow diagram for systematic review.

The title/abstract review of articles 31 to 100 resulted in concordance on 97.1% (68/70) of the items with a kappa of 0.935 (95% CI=0.847-1.0). The full text review excluded an additional 32 articles; 157 recommendations were extracted from the remaining 29 articles.

Thematic analysis

The simple thematic analysis of the recommendations for quantitative medical education research resulted in 3 themes (study design, writing, and submission), 14 categories, and 86 items (see Table 1).

Quality evaluation tools and reporting guidelines

Table 2 outlines the fourteen quality evaluation tools and reporting guidelines that were found during the full text review, which were applicable to quantitative medical education research.

Table 2 Summary of reporting guidelines and quality assessment tools relevant to quantitative medical education research

*AEM Score = Academic Emergency Medicine Score, MERSQI = Medical Education Research Study Quality Index.

DISCUSSION

Our scoping reviewReference Arksey and O’Malley 8 identified 86 recommendations for conducting and 14 tools/guidelines for reporting high-quality medical education research using quantitative methods. The recommendations found in the literature were congruent; however, their usefulness will be limited by the high number of items that were found. Although the qualitative analysis decreased the gross number of recommendations substantively, 86 may still be too many to consider. Regardless, we believe that reporting the complete list will be helpful to both inform future research and allow junior researchers to identify those items most pertinent to them. Moving forward, we will aim to triage the recommendations to determine which are the most important so that those can be focused on. Multiple methods of achieving consensus among diverse populations have been proposed and could be applied to identify the most important recommendations.Reference Hsu and Sandford 23

A significant number of quality evaluation criteria and reporting guidelines were found that were applicable to quantitative medical education research. Whereas some of these tools are quite broad,Reference Lo, Mertz and Loeb 13 covering quantitative research in general, others are extremely specific, outlining specifically how to report quantitative medical education research using a specific methodology.Reference Cohen, Mcgaghie and Wayne 16 Because these guidelines outline the criteria for identifying and producing high-quality research, they should serve as a starting point for junior medical education scholars who are designing quantitative studies. We anticipate that these scholars will know what type of quantitative research that they are conducting and quickly identify the most relevant tools for their work using Table 2.

Although this work is likely to provide a helpful starting point for junior researchers, the complexity of medical education research is such that no advice or sets of guidelines can replace formal training in research methodologies nor advice from experts in the field.Reference Perry, Snider and Artz 6 , Reference Sherbino 24 Indeed, as the depth and scope of quantitative research in medical education continue to grow, increasing amounts of expertise and/or collaboration with skilled partners will be needed to produce high-quality, publication-worthy scholarship.Reference Sherbino 24

LIMITATIONS

Although our literature review returned a reasonable number of articles for review, the restrictiveness of the search could be criticized. We believe that the databases and search terms that were selected were appropriate for the topic of interest and, given that medical education research has advanced substantially in the past decade, restricting the review to articles published within the last 10 years allowed more modern and applicable lists of recommendations and reporting guidelines to be identified. However, in limiting the search to the English language, we likely missed literature that could have added to our findings.

CONCLUSION

In this paper, we attempted to provide guidance for junior researchers and authors in the form of key recommendations and reporting guidelines. We achieved this through a scoping review of the literature and thematic analysis, with a focus on quantitative educational research and scholarship. We believe that these results will assist junior researchers who heed the recommendations and guide them to key tools for their type of research.

Competing interests: None declared.

References

REFERENCES

1. Bandiera, G, LeBlanc, C, Regehr, G, et al. Education scholarship in emergency medicine part 2: supporting and developing scholars. CJEM 2014;15(S1):S6-S12, doi: 10.2310/8000.2014.141455.Google Scholar
2. Sherbino, J, van Melle, E, Bandiera, G, et al. Education scholarship in emergency medicine part 1: innovating and improving teaching and learning. CJEM 2014;15(S1):S1-S5, 10.2310/8000.2014.141454.Google Scholar
3. Bhanji, F, Cheng, A, Frank, JR, et al. Education scholarship in emergency medicine part 3: a “how-to” guide. CJEM 2014;16(Suppl 1):S13-S18.CrossRefGoogle Scholar
4. Gordon, JA. Reflections on the consensus process: a leadership role for emergency medicine in educational scholarship and practice across health care. Acad Emerg Med 2012;19(12):1333-1335, 10.1111/acem.12020.Google Scholar
5. Sinclair, D, Worthington, JR, Joubert, G, et al. CAEP 2015 Academic Symposium: leadership within the emergency medicine academic community and beyond. CJEM 2016;18(S1):S1-S9, 10.1017/cem.2016.9.Google Scholar
6. Perry, JJ, Snider, EC, Artz, DJ, et al. CAEP 2014 Academic Symposium: how to make research succeed in your emergency department: how to develop and train career researchers in emergency medicine. CJEM 2015;17(3):334-343, 10.1017/cem.2015.63.Google Scholar
7. Schuwirth, LWT, van der Vleuten, CPM. Medical education: challenges for educationalists. BMJ 2006;333:544-546.Google Scholar
8. Arksey, H, O’Malley, L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005;8(1):19-32, 10.1080/1364557032000119616.Google Scholar
9. Beckman, TJ, Cook, DA. Developing scholarly projects in education: a primer for medical teachers. Med Teach 2007;29(2-3):210-218.Google Scholar
10. Farrell, SE, Kuhn, GJ, Coates, WC, et al. Critical appraisal of emergency medicine education research: the best publications of 2013. Acad Emerg Med 2014;21(11):1274-1283.Google Scholar
11. Reed, DA, Beckman, TJ, Wright, SM, et al. Predictive validity evidence for medical education research study quality instrument scores: quality of submissions to JGIM’s Medical Education Special Issue. J Gen Intern Med 2008;23(7):903-907.CrossRefGoogle ScholarPubMed
12. Jarlais, D, Don, C. Improving the reporting quality of nonrandomized evaluations of behaviorial and public health interventions: the TREND statement. Am J Public Health 2004;94(3):361-366, 10.1111/j.1360-0443.2004.00785.x.Google Scholar
13. Lo, CK-L, Mertz, D, Loeb, M. Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments. BMC Med Res Methodol 2014;14(45):1-5, 10.1186/1471-2288-14-45.Google Scholar
14. Sullivan, GM. Deconstructing quality in education research. J Grad Med Educ 2011;3(2):121-124, 10.4300/JGME-D-11-00083.1.Google Scholar
15. Kirkpatrick, DL. Evaluating training programs, 3rd ed. San Francisco: Berrett-Koehler; 1994.Google Scholar
16. Cohen, ER, Mcgaghie, WC, Wayne, DB, et al. Recommendations for Reporting Mastery Education Research in Medicine (ReMERM). Acad Med 2015;90(11):1-6, 10.1097/ACM.0000000000000933.Google Scholar
17. von Elm, E, Altman, DG, Egger, M, et al. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Int J Surg 2014;12(12):1495-1499, 10.1016/j.ijsu.2014.07.013.CrossRefGoogle ScholarPubMed
18. Education Group for Guidelines on Evaluation. Guidelines for evaluating papers on educational interventions. BMJ 1999;318(7193):1265-1267.Google Scholar
19. Cook, DA, Beckman, TJ, Bordage, G. Quality of reporting of experimental studies in medical education: a systematic review. Med Educ 2007;41(8):737-745.CrossRefGoogle ScholarPubMed
20. Schulz, KF, Altman, DG, Moher, D. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Med 2010;340:c322.Google Scholar
21. Hoffmann, TC, Glasziou, PP, Boutron, I, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014;348:g1687, 10.1136/bmj.g1687.Google Scholar
22. Phillips, AC, Lewis, LK, McEvoy, MP, et al. A Delphi survey to determine how educational interventions for evidence-based practice should be reported: stage 2 of the development of a reporting guideline. BMC Med Educ 2014;14(1):159, 10.1186/1472-6920-14-159.CrossRefGoogle ScholarPubMed
23. Hsu, C, Sandford, B. The Delphi technique: making sense of consensus. Pract Assessment Res Eval 2007;12(10):1-8, 10.1016/S0169-2070(99)00018-7.Google Scholar
24. Sherbino, J. Education scholarship and its impact on emergency medicine education. West J Emerg Med 2015;16(6):804-809, 10.5811/westjem.2015.9.27355.Google Scholar
Figure 0

Table 1 Results of the qualitative analysis of literature-based recommendations for the performance of quantitative medical education research

Figure 1

Figure 1 Flow diagram for systematic review.

Figure 2

Table 2 Summary of reporting guidelines and quality assessment tools relevant to quantitative medical education research