The Postgraduate Medical Education and Training Board (PMETB) is now responsible for all aspects of postgraduate medical education and assessment. Modernising Medical Careers (MMC) has set out a career structure in which doctors will be appointed to structured training programmes. Selection is only at entry to the training grade when a national training number (NTN) is awarded; trainees should progress through the programme to completion and the award of their certificate of completion of training (CCT).

Selection methods

The new training programmes gave rise to new thinking about methods for selection and recruitment into training posts. The Chief Medical Officer stated that ‘reform must take account of weak selection and appointment procedures: these are not standardised and are frequently not informed by key competencies’ (Reference DonaldsonDonaldson, 2002). A single application and selection method for all medical specialties was introduced by MMC and conducted through the online Medical Training Application Service (MTAS). In this system a trainee was required to submit a single application stating their specialty and geographic preferences within a 2-week window in late January into early February 2007. For 2007 and probably for the next few years the transitional arrangements mean that existing specialist trainees without an NTN (i.e. those in the current senior house officer grade) must apply for entry to training. The selection procedure adopted by MTAS was based on that developed for general practice and uses a competency based model (Reference Patterson, Lane and FergusonPatterson et al, 2001). There is evidence that this system has good predictive validity (Reference Patterson, Ferguson and NorfolkPatterson et al, 2005), i.e. trainees selected into general practice by this method appear to do well in their training. For psychiatry, person specifications were set out for each level of entry. The application included personal details, clinical and academic achievement, questions on commitment to specialty, including information on what ‘extras’ the junior had sought to further their cause, and specialty-specific questions. For psychiatry the latter were concerned with communication, problem-solving and probity. When filling in the application candidates were asked to provide examples of when they had demonstrated certain skills and behaviours. The answers to these questions were scored independently by markers using agreed criteria. Trainees were able to view references, which were structured around identified competences in order to improve the reliability of the process (Reference Patterson, Lane and FergusonPatterson et al, 2001). Interviews were scheduled on a national basis.

Unit of application

The unit of application is equivalent to a deanery or strategic health authority. The tasks at the unit of application level were to shortlist and interview. Consultants and lay members for panels were trained according to a prescribed schedule, including briefing on the process, and equality and diversity training. Longlisting was to be performed by MTAS and deanery staff, leaving the appointed selectors to shortlist applications, scoring against a set schedule. Selectors were masked to the candidate's identity, examination and employment history. The interview itself was designed at each unit of application using a ‘multi-station’ method and was structured against the persons specifications.

West Midlands experience

In the West Midlands plans were made to appoint for approximately 190 vacancies, with 1200 applications and 360 interviews anticipated. No longlisting occurred. The authors prepared a calibration exercise for each of the 4 days of shortlisting, involving a series of ‘mock applications’. During shortlisting, quality control was ensured by reviewing a number of ‘marked’ applications. Completed forms and scores were submitted to deanery staff, who entered data onto MTAS. Interview dates were then offered to high-scoring candidates. The team shortlisting were unaware if candidates were international medical graduates requiring work permits, and their applications were scored in the same fashion as candidates from the UK and/or European Economic Area.

Interviews consisted of three stations, each lasting 10 min with two interviewers at each station. The stations were a portfolio review, a review of the candidate's training and curriculum vitae, and a clinical question station concerned with commitment to specialty (the question at ST2 and above required some knowledge of psychiatry) and probity. Although station interviews require more planning and expense than conventional interview methods they appear to be fairer (Reference Smith, Dunstan and WilesSmith et al, 2006). Following the initial 5 days of interviews (round 1a) the Department of Health/Academy of Royal Colleges Review Panel decided upon a further round of interviews (1b) for those candidates who had placed the West Midlands as first choice but had not received an interview; thus a further 213 candidates were interviewed over 2 days. Immediately before these interviews, following a series of problems that included breaches of security, the Secretary of State announced that MTAS was abandoned and that deaneries would assume responsibility for all aspects of arranging the interview process. Round 2 interviews are currently being organised to fill vacancies through the West Midlands School of Psychiatry in partnership with a lead mental health trust.

Potential benefits and what went well

The MTAS system was a centralised and standardised selection process with the potential for being efficient, reliable and valid. In the West Midlands the School of Psychiatry was able to estimate approximately how many applications there would be and shortlist them by the deadline. From across the West Midlands an average of 26 consultants and 5 lay people were recruited for each day of shortlisting, and for each day of interviews 15 consultants and 3 lay people were recruited. Nationally in all specialties MTAS received 30 000 applications for 22 000 jobs and these were processed on time; a considerable achievement. The interview process ran well and was perceived as being valid and fair by those organising it; informal feedback suggested it was well received by candidates.

What could have been done better?

The selection process has been a bruising and demoralising experience with an enormous amount of anxiety generated both among trainees, fearful of future employment opportunities, and among their seniors, concerned for the future of specialty training in the UK. Of the 82 trainees on the All Birmingham Psychiatric Rotation, 21 failed to secure any offers of interviews during the first round. The change in the selection process was overly ambitious and implementation should have been staged. For instance, the new system could have been implemented only for recruitment into the ST1 grade in 2007 or for one specialty. This would have allowed time for feedback and further development of MTAS. Each of the issues of fairness, reliability and validity need to be reconsidered. The specialty schools must be more closely involved at every stage, for example if longlisting had taken place this would have ensured that only eligible applications were shortlisted.

The process must be seen by the applicants to be fair; for instance it is unfair to shortlist candidates for interview only to subsequently tell them that they would be ineligible to take up a post because of the work permit regulations. The application form used for specialist training was unproven for the specialties outside general practice. The application form did not have ‘correct answers’ (even ‘hard’ issues such as publications were open to interpretation). Several internet sites offered to help write application forms for candidates for a fee. Where the ratio of applicants to interviews was of the magnitude described it is obvious that a few points’ difference in the marking will have been highly significant but the reliability of the process was insufficient.

Shortlisters were unaware of candidates’ previous experience, meaning that candidates may have been mismatched to the level of specialist training to which they had applied. All aspects of the validity of the process have to be tested, including predictive validity, which can only be judged when candidates are followed up during their training. The Department of Health have now acknowledged that the shortlisting process was a failure.

The future

The experience of this round needs to be considered carefully and decisions made about future rounds. Any selection process should incorporate the results of the considerable body of evidence available on the validity and utility of different selection methods (Reference Schmidt and HunterSchmidt & Hunter, 1998). The selection process for general practice now includes an examination including multiple choice questions (MCQs) with assessment of medical knowledge and problem-solving. One of the advantages of MCQs is that they are perceived as fair (Reference McCoubrieMcCoubrie, 2004). We suggest that a valid and reliable method for the future should include an application form that takes account of academic record and prior work experience, a knowledge-based test and a structured interview.

The extent of the long-term damage to the medical profession of the selection process is not yet clear. The redesign of the selection process based on the lessons of the first year must start now if it is to be ready for 2008. The Department of Health has announced an urgent independent review of MTAS/MMC to be chaired by Sir John Tooke. All those who were involved in the process will hope that this review will address the flaws that we have highlighted.

Declaration of interest

N.B. was responsible at deanery level for recruitment into the specialty training grade in psychiatry in the West Midlands. He also works for the Royal College of Psychiatrists on Modernising Medical Careers. C.V. and A.P. helped plan and implement shortlisting and interviewing for specialty training grade recruitment in psychiatry in the West Midlands.