Medical audit : basic principles and current methods

The aim of the MSc is to emphasise the multidisciplinary origin of psychiatry by teaching in an inte grated fashion and to engender a critical approach to research and practice. As with the Diploma, the Masters follows the pattern for such Degrees in the University ofKeele. This will include six additional teaching modules attended over one or two years and a one week research methods course. A 10,000 word dissertation will be prepared on an agreed topic, supervised by a member of the Department, with an associated research protocol. There will be two three-hour essay papers and an oral examination. The written examinations will include the presen tation of research material for critical evaluation, as well as questions covering sub-specialty topics. An oral examination centres on the dissertation and other clinical material. The University also has a Master's Degree by thesis and we currently have a

examination if felt appropriate. Successful candi dates will be awarded a Diploma by the University and may then apply for extension to the Masters' Degree.

MSc in General Psychiatry
The aim of the MSc is to emphasise the multidisciplinary origin of psychiatry by teaching in an inte grated fashion and to engender a critical approach to research and practice. As with the Diploma, the Masters follows the pattern for such Degrees in the University ofKeele. This will include six additional teaching modules attended over one or two years and a one week research methods course. A 10,000 word dissertation will be prepared on an agreed topic, supervised by a member of the Department, with an associated research protocol. There will be two three-hour essay papers and an oral examination. The written examinations will include the presen tation of research material for critical evaluation, as well as questions covering sub-specialty topics. An oral examination centres on the dissertation and other clinical material. The University also has a Master's Degree by thesis and we currently have a number of external and internal students registered for such a degree. This may be extended to a PhD.

Senior registrar training and beyond
The University of Keele has Masters in Medical Ethics and Medical Social Anthropology which could be suitable for senior registrars or consultants. We have had discussions with several departments on Masters' degrees providing courses relevant to the higher training and sub-specialisation senior regis trar level. The first of these, Brain, Behaviour and Development, a Masters in Clinical and Develop mental Neuropsychiatry, is now being planned. This will provide teaching, probably in three day Units over one year, with related course work and followed by a supervised research project leading over one or more years to a thesis, oral and written examinations. A legitimate sibling of this will be a Diploma/Masters on Social and Cultural Influences in Psychiatry and preliminary discussions on the possibility of such a course are taking place. Senior Lecturer posts are to be advertised in 1990/91 and these planned develop ments depend on successful appointments to these posts.
Finally, the presence of the Centre for Health Planning and Management at Keele offers the possi bility of developing modules and other courses on management, audit, quality assurance and resource utilisation.

Comment
The Keele Department of Psychiatry is still growing and we are aware both of our youth and the special responsibilities and privileges of being part of a Post graduate School without formal undergraduate medical teaching commitments (lest we forget, all academics in the School are subject to a three or five yearly academic performance reviews). This paper has outlined current and planned teaching. It remains to be seen whether or not our 'products' will have a 'market' in 1990spostgraduate psychiatry. the spotlight by the NHS Review, the idea that doc tors should evaluate their practice as a routine is not new (SCOPME, 1989).
In this paper the basic principles of audit and the ways in which these have been applied will be illus trated with examples drawn from general medicine. This field more than any other shares with psychiatry certain problems which impede the development of effective audit, and the means by which they have been addressed may be of interest.

Principles of medical audit
Medical audit essentially consists of a sequence of separate activities, some of which are easier to accomplish than others, linked to form a loop which can be repeated as necessary. This is usually known as the "audit cycle" (Fowkes, 1982).There is no particu lar need to begin at any given point in the cycle although most audits start at the least difficult stage, the observation of current practice.
Observation of practice involves the collection of detailed information about the care of certain patients. It would clearly be impracticable to record systematically every relevant fact about even a limited sample of patients, so certain criteria must be chosen in advance. This choice may be made on the grounds of clinical importance or ease of access to the item of data selected.
The commonest source of information is the patient's medical record, but pathology reports or responses to specially designed questionnaires may also be used (Emerson et al, 1989). The storage of clinical information on computer provides an excel lent source, provided the individual records are detailed enough.
The next stage of the audit process is the compari son of this information to what was expected or desired. A standard need not be explicitly defined at first, because all doctors have their own internal ones which will make an immediate judgement, especially of other colleagues! In order to progress to the next stage of the cycle however, there needs to be agreement on common standards of practice.
Standards may be set nationally by a panel of experts or as a result of a consensus conference (Vang, 1986). Alternatively, they may be worked out by the participants in an audit with no outside help. Their academic correctness is much less relevant than their acceptability and relevance to the local situation.
The final stage of the audit process, and the most important, is the taking of action to address the differences identified in the comparison of observed and expected. The type of action required will be determined by the nature of the difference. It is commonly assumed that this will be a change in clini cal behaviour, but successful audits often involve other types of action, such as the redefinition of a Audit in practice policy or redistribution of resources (Dixon, 1989). The standard of technical care itself is usually adequate, as this is the part of the total process most conscious to the doctor.
Having completed one round of the cycle, the next step is to return to the first stage again, observation of current practice, if only to confirm that the action taken had its desired effect.

Methods of audit
So much for the theory of audit; what evidence is there that it works in practice, and how does one get started in a world where even getting a set of casenotes for each patient may be difficult?
The simplest method is the review of a random sample of casenotes by a colleague not directly involved with the delivery of care in that particular case. The Royal College of Physicians has begun to undertake such a system of audit as part of its reaccreditation of training posts (RCP, 1989). The ad vantages of this method are its simplicity and speed. Although someone must obtain the notes and review them before an audit meeting, no other preparation is required. The disadvantage is that the insights which can be drawn from such a nonspecific observation are limited, and repetition may lead to boredom.
The most common discovery from this process is a disappointing standard of notekeeping, and this can be expected to improve merely as a result of under taking the audit. There is a danger of the burden of criticism falling disproportionately upon the junior staff, and this may bring out other issues for dis cussion, adding to the value of the audit if they are tackled in a positive manner (Van't Hoff, 1989).
An easy refinement of random casenote review is "topic based audit". More effort is required to select several records bearing the same diagnosis or pro cedure rather than a simple random series, but the peer review process is more likely to be able to draw general conclusions about the management of the condition under consideration.
It is good practice to keep a written record of the conclusions reached, and it may be possible to trans late these into concise "guidelines" for distribution to junior staff with the aim of modifying the manage ment of the condition in future. A fresh look at the "topic" by the audit of a further series of notes after a suitable interval will hopefully show respect for the guidelines. Successful completion of the audit cycle using this method has been demonstrated for the appropriate use of diagnostic tests (Fowkes et al, 1986).
A further extension of this method is "criterionbased audit". To save on doctors' time, and allow a larger sample of notes of a particular topic to be considered, the peer review meeting defines a number of criteria and standards for each that an "audit assistant" can check each case against. Only those cases which "fall out" of this filtering process need be considered in detail by the peer review process. Thus the audit meeting can concentrate its attention on atypical cases only, rather than reviewing a general sample of all cases ofthat topic.
This method has been widely used in the USA, with varying enthusiasm (Nelson, 1976). Although superficially attractive, it tends to cast a punitive aspect over the audit process, stigmatising variance rather than promoting overall quality in a positive way.
All the audit methods so far considered tend to focus upon the process of medical care. A different approach is "outcome audit". The emphasis is placed on defining and measuring criteria of outcome, such as patient satisfaction or quality of life indicators (Shaw, 1989). Expected standards or targets can be set as for process measures, and plans implemented to seek to narrow the differences. This is the most sophisticated and valid type of medical audit, but has corresponding difficulties. Patients have to be followed up after discharge, and measurement tools need to provide good response rates. A comprehensive approach is to base audit upon "intermediate outcomes" which can be determined at or soon after discharge.
Finally, mention should be made of "information based audit", that is the review of aggregated activity and financial data, including in some cases crude out come measurements such as mortality within hospi tal (Kind, 1988). The power of this method lies in the ability of computers to handle large amounts of numerical data with ease. Hospitals can be compared with each other, and trends in time examined. This can highlight aspects which require more detailed analysis, but the process is limited by the accuracy and completeness of the original data, and an inability to adjust satisfactorily for variations in casemix (Charny, 1988).
In conclusion, a variety of methods of medical audit have been developed. No single method is both robust to criticism and simple to implement, but a start can be made with little effort. The potential rewards are demonstrable self regulation for doctors and a higher quality of care for their patients.

Audit
Audit was defined in the White Paper Working for Patients as the systematic, critical analysis of the qual ity of medical care, including the procedures used for diagnosis and treatment, the use of resources, and the resulting outcome and quality of life for the patients. Medical audit is not a new thing since it represents an activity which we were commanded to carry out by Hippocrates. Medical audit should be doctor led. It involves the setting and review of standards and it should lead to improved quality of care (SCOPME, 1989). The report by the Standing Committee for Post graduate Medical Education (SCOPME) points out that continued medical education has not been enough to improve standards. Feedback is essential. Every doctor must be involved in the monitoring (selfevaluation) of performance against professional standards.
The activity of audit can be seen as a cycle, first to set standards, then to observe the practice and compare with standards and then to implement change moving on once more to setting perhaps new standards, and once more observing practice and comparing that with the new standards.
Every doctor must be involved in audit but at the same time it is important to use doctors' time efficiently, thus auditing audit. It has already been found that the collection of vast amounts of data with little idea of how these are to be handled is ineffective. In addition, many computer systems have been devised to assist audit but none proved to be infallible or efficacious in every case. It has been found well worthwhile to have audit assistants, par ticularly for district audit or for more complex work. Any method that collects data is only reliable if all the data required is indeed collected. There must be something in it for the person who is expected labor iously to type in the information. The Bloomsbury system, under the direction of Dr Jonathan Seeker-Walker, has terminals on every ward and working place. The benefit is the production of routine letters and discharge summaries as well as easy access to statistics. A very helpful addition to the team is a computer literate audit assistant who can act as troubleshooter going at once to the help of someone having difficulty with their terminal.
The three main components of audit are: resources or input; process; and outcome.
In addition, audit can be carried out at at least six different levels.
(1) First, at the lowest level, audit can be done within clinical teams, looking, for example, at discharge letters, use of PRN medication, untoward events. (2) Audit between clinicians and between clinical teams. This can take the form of peer review when doctors look at the work of other doc tors. Case conferences are familiar to every one but peer review should be based on a randomly chosen selection of cases rather than one chosen by the clinicians concerned for discussion. There is some argument about whether or not all grades of doctors should be involved. Sometimes through lack of confi dence, the senior doctors feel that they should exclude juniors. Nonetheless it has been found in a number of situations that involving everyone is effective and not a destructive or frightening ordeal. Confidentiality of this sort of audit is essential respecting both confiden tiality of patients and the confidentiality of doctors. The results of audit may be published if these conditions are met. (3) The third level is within the hospital and could compare describing practices between different teams, units etc., or waiting lists, the way in which out-patients are conducted etc. Untoward events, e.g. suicides, acts of serious aggression, and failed appointments can also be audited at this and at the higher levels. (4) Audit can take place between units in differ ent geographical areas or between hospitals with a comparison of results of modes of treatment or sometimes with different levels of resources.
Audil in practice (5) At regional level the care of less usual cases can be considered. Although not rare, regional appraisal of treatment of anorexia nervosa has taken place in Scotland and all over the country regional comparisons of vascular or even heart transplant surgery, are being subjected to audit, with important practical findings. (6) Finally, on a national basis there are the College Approval Visits, the activities of the HAS and the National Development Team. A problem which is particularly difficult in psy chiatry is that of outcome measures. In individual cases clinical rating, such as of symptoms, can be supplemented by looking at the needs met. But inves tigation of needs must be done at the outset in order to be able to measure this as part of the outcome.
Follow-up performance at school, in the case of a child, or at work, in the case of an adult, or coping behaviour in everyday life etc., can be used as an outcome measure. It is important not to forget family and to look at the satisfaction which must include patient, family members and perhaps referrers.
There are distinct problems in assessing "customer" satisfaction. Should an adolescent's or his/her parents' satisfaction be recorded as they may be far from agreeing with each other.
Consumer satisfaction should be considered under the heading of process. Secondly, the process, which includes the procedures of investigation, diagnosis and treatment must be investigated for their personal cost to the patient. It is necessary to consider pain, discomfort or perhaps loss of personal dignity. And once again the cost to the family could be looked at under headings such as 'time spent', or 'money spent' or perhaps 'not earned', and 'acceptability for the family and of the culture'.
Most of what has just been described under process could come under the heading of quality assurance.
There are many unsolved problems left: (1) Perhaps the major priority for psychiatrists concerns timing of outcome measures since most of our patients have disorders going over considerable periods of time and where relapse, for example even after treatment of a comparatively straightforward problem, such as a simple phobia is all too common.
(2) There are few reliable outcome measures and most psychiatric patients have multiple problems. Should global or specific ratings be used in our outcome measures? (3) The concept of needs is helpful and recording of unmet needs should be part of outcome assessment. (4) In psychiatry, much more than other branches of medicine, there are difficult problems of family relationships. It is known, for example, that resilience of families bears more relation to outcome in child psychiatry than treatment methods used. We are also familiar with family measures such as expressed emotion and the connection that these have with relapse in schizophrenia. (5) Quality of life is a big area, but already there is considerable research on this, some of it complex and highly mathematical. Nonethe less, it bears looking at from a simple point of view by each and every doctor considering the day to day work they do. (6) Australian psychiatrists have produced some information about model treatments under the title of 'the quality assurance project'.
The aim is to produce a consensus opinion on what is the best way to treat a particular problem. There may be several ways, but certainly there should be some generally accepted standards of what should be avail able. In child psychiatry a start is to be made on a model for investigation and treatment of autism. Another problem on which model treatment can be considered is the treatment of behavioural and neurological sequelae following head injury to find out what could be available in each district. Before a start can be made setting standards it is first necessary to look at what is already on offer, then appraise these existing services and review the literature before setting standards on devising a model. (7) Finally, relative costs, but here it should be stressed that the purpose of audit is to improve treament. It is not primarily to assist budgeting and not to underlie the drawing up of contrasts. ANNGATH Registrar

Helpful reading on audit
Following the papers there was an extended dis cussion during which the following main themes emerged.
( I) To what use might managers put the data pro duced? Might managers seek to encourage cheaper and quicker treatment alternatives of only short term benefit, disregarding longer term consequences for patients and psychi atric services; might managers fail to make appropriate allowances for local deficiencies in services and peculiarities of catchment population if any nationally produced and prescriptive guidelines ever emerged? Vari ous participants stressed that medical audit is an educational tool; the data are collected by doctors for doctors. The usefulness of audit for managers is of only secondary concern; data required for service planning would have to take into account the whole range of treat ment and its providers, in the community as well as in hospital, not just the simple, and perhaps over-simplified, issues reviewed in the early stages of medical audit. However, while looking at only part of a service would inevitably provide a very distorted picture, nevertheless such a picture might be of value if common problems or deficiencies regularly appeared in many parts of the country. Such observations would have great importance for national service planning. (2) Is it to take too narrow a viewif hospital based doctors review only the quality of hospital practice? What are the views of primary carers, general practitioners, social workers?
The opinion of such individuals is likely to be determined by their style of practice and their expectations. Ought there to be some mech anism by which primary carers can audit hospital practice and vice versa? Each would then better appreciate strengths and weak nesses of the others' approach. (3) We must take care not to generate "an insti tutionalised system of virtue " -best results being obtained in those who do not need the treatment provided. There must be some righting mechanisms for "tidy nice units".
On the other hand a consensus view must not be permitted to emerge which sets too low a common denominator of level and quality of service, which encourages a defensive view of practice and militates against trying something new.