Skip to main content Accessibility help

Peak and ceiling effects in final-product analysis of mastoidectomy performance

  • N West (a1), L Konge (a2), P Cayé-Thomasen (a1), M S Sørensen (a1) and S A W Andersen (a1)...



Virtual reality surgical simulation of mastoidectomy is a promising training tool for novices. Final-product analysis for assessing novice mastoidectomy performance could be limited by a peak or ceiling effect. These may be countered by simulator-integrated tutoring.


Twenty-two participants completed a single session of self-directed practice of the mastoidectomy procedure in a virtual reality simulator. Participants were randomised for additional simulator-integrated tutoring. Performances were assessed at 10-minute intervals using final-product analysis.


In all, 45.5 per cent of participants peaked before the 60-minute time limit. None of the participants achieved the maximum score, suggesting a ceiling effect. The tutored group performed better than the non-tutored group but tutoring did not eliminate the peak or ceiling effects.


Timing and adequate instruction is important when using final-product analysis to assess novice mastoidectomy performance. Improved real-time feedback and tutoring could address the limitations of final product based assessment.


Corresponding author

Address for correspondence: S A W Andersen, Department of Otorhinolaryngology – Head & Neck Surgery, Rigshospitalet, Blegdamsvej 9, DK-2100 Copenhagen Ø, Denmark E-mail:


Hide All
1.George, AP, De, R.Review of temporal bone dissection teaching: how it was, is and will be. J Laryngol Otol 2010;124:119–25
2.Mills, R, Lee, P.Surgical skills training in middle-ear surgery. J Laryngol Otol 2003;117:159–63
3.Wiet, GJ, Stredney, D, Sessanna, D, Bryan, JA, Welling, DB, Schmalbrock, P.Virtual temporal bone dissection: an interactive surgical simulator. Otolaryngol Head Neck Surg 2002;127:7983
4.Rhienmora, P, Haddawy, P, Khanal, P, Suebnukarn, S, Dailey, MN.A virtual reality simulator for teaching and evaluating dental procedures. Methods Inf Med 2010;49:396405
5.Ungi, T, Sargent, D, Moult, E, Lasso, A, Pinter, C, McGraw, RC et al. Perk Tutor: an open-source training platform for ultrasound-guided needle insertions. IEEE Trans Biomed Eng 2012;59:3475–81
6.Sewell, C, Morris, D, Blevins, N, Dutta, S, Agrawal, S, Barbagli, F et al. Providing metrics and performance feedback in a surgical simulator. Comput Aided Surg 2008;13:6381
7.Zirkle, M, Taplin, MA, Anthony, R, Dubrowski, A.Objective assessment of temporal bone drilling skills. Ann Otol Rhinol Laryngol 2007;116:793–8
8.Laeeq, K, Bhatti, NI, Carey, JP, Della Santina, CC, Limb, CJ, Niparko, JK et al. Pilot testing of an assessment tool for competency in mastoidectomy. Laryngoscope 2009;119:2402–10
9.Butler, NN, Wiet, GJ.Reliability of the Welling scale (WS1) for rating temporal bone dissection performance. Laryngoscope 2007;117:1803–8
10.Martin, JA, Regehr, G, Reznick, R, Macrae, H, Murnaghan, J, Hutchinson, C et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997;84:273–8
11.Szalay, D, MacRae, H, Regehr, G, Reznick, R.Using operative outcome to assess technical skill. Am J Surg 2000;180:234–7
12.Andersen, SA, Cayé-Thomasen, P, Sølvsten, Sørensen M.Mastoidectomy performance assessment of virtual simulation training using final-product analysis. Laryngoscope 2015;125:431–5
13.Wan, D, Wiet, GJ, Welling, DB, Kerwin, T, Stredney, D.Creating a cross-institutional grading scale for temporal bone dissection. Laryngoscope 2010;120:1422–7
14.Munz, Y, Moorthy, K, Bann, S, Shah, J, Ivanova, S, Darzi, SA.Ceiling effect in technical skills of surgical residents. Am J Surg 2004;188:294300
15.Moorthy, K, Munz, Y, Adams, S, Pandey, V, Darzi, A.A human factors analysis of technical and team skills among surgical trainees during procedural simulations in a simulated operating theatre. Ann Surg 2005;242:631–9
16.Visible Ear Simulator. In: [15 December 2014]
17.Sorensen, MS, Mosegaard, J, Trier, P.The visible ear simulator: a public PC application for GPU-accelerated haptic 3D simulation of ear surgery based on the visible ear data. Otol Neurotol 2009;30:484–7
18.Trier, P, Noe, , Sørensen, MS, Mosegaard, J.The visible ear surgery simulator. Stud Health Technol Inform 2008;132:523–5
19.Bech, B, Lönn, L, Falkenberg, M, Bartholdy, NJ, Räder, SB, Schroeder, TV et al. Construct validity and reliability of structured assessment of endovascular expertise in a simulated setting. Eur J Vasc Endovasc Surg 2011;42:539–48
20.Reznick, RK.Teaching and testing technical skills. Am J Surg 1993;165:358361
21.Bjurström, JM, Konge, L, Lehnert, P, Krogh, CL, Hansen, HJ, Petersen, RH et al. Simulation-based training for thoracoscopy. Sim Healthcare 2013;8:317323
22.Kruglikova, I, Grantcharov, TP, Drewes, AM, Funch-Jensen, P.The impact of constructive feedback on training in gastrointestinal endoscopy using high-fidelity Virtual-Reality simulation: a randomised controlled trial. Gut 2010;59:181–5
23.Brydges, R, Carnahan, H, Safir, O, Dubrowski, A.How effective is self-guided learning of clinical technical skills? It's all about process. Med Educ 2009;43:507515
24.Brydges, R, Nair, P, Ma, I, Shanks, D, Hatala, R.Directed self-regulated learning versus instructor-regulated learning in simulation training. Med Educ 2012;46:648656
25.Fann, JI, Caffarelli, AD, Georgette, G, Howard, SK, Gaba, DM, Youngblood, P et al. Improvement in coronary anastomosis with cardiac surgery simulation. J Thorac Cardiovasc Surg 2008;136:1486–91
26.Kerwin, T, Wiet, G, Stredney, D, Shen, H-W.Automatic scoring of virtual mastoidectomies using expert examples. Int J Comput Assist Radiol Surg 2012;7:111


Peak and ceiling effects in final-product analysis of mastoidectomy performance

  • N West (a1), L Konge (a2), P Cayé-Thomasen (a1), M S Sørensen (a1) and S A W Andersen (a1)...


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed