Hostname: page-component-8448b6f56d-c47g7 Total loading time: 0 Render date: 2024-04-20T08:29:22.169Z Has data issue: false hasContentIssue false

Usefulness of reader feedback on the Royal College of Psychiatrists' public information leaflets

Published online by Cambridge University Press:  02 January 2018

Martin Briscoe*
Devon Partnership NHS Trust Royal College of Psychiatrists Public Education Editorial Board
Simon Briscoe
Lancashire Teaching Hospitals NHS Foundation Trust
Philip Timms
Royal College of Psychiatrists Public Education Editorial Board South London and Maudsley NHS Foundation Trust
Ros Ramsay
Royal College of Psychiatrists Public Education Editorial Board South London and Maudsley NHS Foundation Trust
Dr M. Briscoe (
Rights & Permissions [Opens in a new window]


Aims and method

To describe the process for reader feedback on the Royal College of Psychiatrists' online public information leaflets, to report the findings of a retrospective analysis of feedback received over a 14-month period, and to discuss the value of feedback, particularly in relation to the Information Standard quality mark introduced by the Department of Health.


We received 38 700 completed feedback forms during the period under analysis. We derived scores from the feedback forms, which enabled us to identify those that should be prioritised for review. Written comments from readers highlighted specific areas of the leaflets that required further work.

Clinical implications

The development of our public mental health information can be guided using feedback from our readers.

Original Papers
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution (CC-BY) license (, which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright © Royal College of Psychiatrists, 2011

Providing patient information has become increasingly recognised as an important part of clinical practice. Without it, informed choice about treatments is not possible. Unfortunately, much patient information is written in complex language and is poorly presented. Reference Coulter, Ellins, Swain, Clarke, Heron and Rasul1

In an attempt to improve patient information, the Department of Health recently established the Information Standard quality mark ( This mark signposts trustworthy information. It is awarded to organisations after assessing their editorial and review processes, and comparing a small sample of their leaflets against a modified 29-item version of the International Patient Decision Aids Standards Instrument checklist. Reference Elwyn, O'Connor, Stacey, Volk, Edwards and Coulter2 The Royal College of Psychiatrists was involved in the pilot phase of this project and received its accreditation in 2009.

The College website ( has hosted patient information resources for over 10 years. The website has become a leading source of public mental health information in the UK. We know this because the leaflets are ranked highly by Google, the most popular search engine in the UK. A search of mental health keywords such as ‘depression’, ‘anorexia’ and ‘schizophrenia’ returns the leaflets in the top five search results. In addition, at the time of writing, the website is ranked higher in terms of popularity than many of the leading main mental health charity websites in the UK, including Mind, Sane, Rethink, Depression Alliance and the Mental Health Foundation (

The public information section currently has 113 mental health information leaflets, with over 150 translations into Arabic, Bengali, Persian, Urdu, Chinese, Hindi, Greek, Spanish, French, Russian, Polish, Gujarati, Punjabi and Welsh. The 113 leaflets are divided into four categories:

  1. Help is at Hand leaflets, which contain up to 12 pages of in-depth information about mental health conditions;

  2. Keyfacts leaflets, which are briefer, two-page overviews of the same conditions;

  3. Changing Minds leaflets, which are designed to challenge stigma;

  4. Mental Health and Growing Up (MHGU) leaflets, which cover mental health problems in young people.

The College produced paper leaflets for over 10 years before the website was developed, but it had difficulty evaluating the leaflets and gathering feedback. The website provided an opportunity to find out what our readers thought of the leaflets and to collate feedback to guide their further development. Against this background, this paper reviews the feedback received from our readers between 1 September 2008 and 31 October 2009.


Readers are offered the opportunity to complete a feedback form at the end of each leaflet. There are three parts to the form:

  1. information about the reader

  2. free-text comments

  3. a five-point scale to rate the readability, usefulness, respectfulness and design of the leaflet (Fig. 1).

    Fig 1 Example of feedback form.

The feedback forms are emailed to the College and the results automatically collated and stored in a database.


Website visits

During the period under analysis there were 3 378 000 visits to the College website by 2 247 000 people from more than 222 countries. The highest visiting countries can be seen in Table 1. The country of origin could not be identified for 20 000 visitors (approximately 0.5%).

Table 1 Website visits by country of origin

Country Visits, n % of total visits
UK 2 500 000 73
USA 257 000 7
Canada 80 000 2
Ireland 80 000 2
Australia 77 000 2
India 30 000 1
Iran 24 000 0.5
Other (including unknown) 330 000 12.5

On average each visitor looked at three pages on the website, resulting in about 10 million page views. The public information section is the most popular area of the website, accounting for about 40% of these visits.

Feedback forms

We received 37 800 feedback forms - approximately one form for every 80 visitors to a leaflet. Forms that were completed only partially were excluded from the analysis. Leaflets with fewer than 20 completed feedback forms were excluded to ensure that average scores for each leaflet were based on a significant number of responses. We also excluded feedback on foreign-language leaflets.

Table 2 shows the overall number of leaflets in each category; the number of leaflets with 20 or more replies included in the analysis; the number of feedback forms received in each category; and the number of complete feedback forms included in the analysis. The final column gives the overall average leaflet score for each category. This is calculated by averaging the scores for the four modalities (readable, useful, respectful, well-designed) in the feedback form. The modalities are measured on a scale of 1 (lowest) to 5 (highest).

Table 2 Numbers of responses and feedback scores for each leaflet category

Leaflet category Leaflets, n Leaflets included in analysis, n Feedback forms received, n Feedback forms included in analysis, n Average score (out of 5)
Main 52 51 26 355 17 797 4.55
Keyfacts 16 13 1830 1222 4.54
Changing Minds 7 7 1968 1546 4.48
Mental Health and Growing Up 38 33 3943 2237 4.38

These figures show that the main leaflets are by far the most popular. This is probably because they are ranked very highly on Google. During the period under analysis, the most popular leaflet was on cognitive-behavioural therapy (CBT), attracting over 385 000 readers. During this time, Google ranked this leaflet in first place on a search for ‘CBT’ or ‘cognitive behavioural therapy’. The second most popular leaflet was on antidepressants, which was also ranked in first place by Google.

The overall average scores for each leaflet category are high (4.38-4.55 out of a maximum of 5), suggesting that the leaflets are well received.


The age breakdown of the people submitting feedback forms can be seen inFig. 2 (4% of responders did not indicate their age). As might be expected, the MHGU leaflets attract young readers: 20% are aged 16 years or under. The Changing Minds leaflets also attract young readers: 28% are aged 16 years or younger and 29% are aged 17-24 years. This could be because the leaflets are short and written in a direct and provocative style.

Fig 2 Age of readers.

Overall, 25-38% of readers describe themselves as patients, about 15% as health professionals and 15% as students. The remainder are carers, relatives or friends of a patient.

Individual leaflets

To measure the popularity of individual leaflets within the four categories, we ranked each leaflet according to its average score.Table 3 shows the results for the four highest scoring and four lowest scoring leaflets.

Table 3 Average scores for highest and lowest performing leaflets

Leaflet Leaflet category Feedback forms, n Mean leaflet score (mean category score)
Highest scoring Post-traumatic stress disorder Main 1108 4.75 (4.55)
Obsessive-compulsive disorder Main 287 4.75 (4.55)
Bipolar disorder Keyfacts 115 4.74 (4.54)
Physical activity Main 136 4.73 (4.55)
Lowest scoring Anorexia and bulimia Changing Minds 968 4.03 (4.48)
Alcohol and depression Main 280 3.72 (4.55)
Drugs and alcohol Mental Health and Growing Up 58 3.72 (4.38)

The lowest scoring leaflets are about issues that attract controversy in the public domain. This controversy may partly account for their higher than average share of negative free-text feedback, as described in the next section. The low scores encouraged us to revisit these titles to see whether we can improve them.

Analysis of free-text feedback

We also undertook a simple qualitative analysis of the free-text responses for a number of leaflets. For the purposes of this paper, we focused on the two highest scoring main leaflets (‘post-traumatic stress disorder (PTSD)’ and ‘obsessive-compulsive disorder (OCD)’) and the two lowest scoring main leaflets (‘Cannabis and mental health’ and ‘Alcohol and depression’). For the highest scoring leaflets, 54% provided a simple commendation such as ‘great leaflet’; 33% provided some autobiographical information about their problem; 7% asked a question; 6% gave a constructive suggestion; 4% said they intended to use the leaflet in their work; and about 2.5% left a negative or abusive comment. The lowest scoring leaflets were just as likely as the highest scoring leaflets to receive commendations but more than twice as likely to receive a negative comment (6% v. 2.5%).

The constructive criticisms are particularly useful in helping us to identify specific ways in which to improve the leaflets (Box 1).


The introduction of the Information Standard quality mark should lead to an improvement in the reliability of patient information among participating organisations. However, it does not necessarily follow that leaflets conforming to this standard will be well received by the public. The only way to assess the reception of a leaflet is to ask the readership.

The quantitative feedback enables us to rank the leaflets and to determine which leaflets need more attention. The qualitative feedback provides constructive advice, suggestions and challenges that help us to identify specific areas for rewriting or fine-tuning. Because the Spearman's correlation between the modalities on the feedback form is high (0.72-0.96), we could simplify the form and use a single score, such as the five-star rating system used by In the future we would like to display this rating alongside the leaflets so that readers can see at a glance how previous readers have rated them.

Our current findings suggest that reader feedback provides invaluable guidance about the substance and presentation of our public mental health information. This feedback complements the Information Standard quality mark, which is focused mainly on editorial, review and production processes.

Box 1 Examples of commendation, constructive and negative responses

Alcohol and depression leaflet

  1. Commendation: ‘I strongly agree with everything you have commented on and have taken many points into consideration. I really hope I can learn from this leaflet.’

  2. Constructive: ‘My only criticism of your leaflet is that it does not address the social pressure to drink which is ingrained in British culture. I'd love to go out for “a” drink but in practice I end up drinking up to 8 units in one sitting.’

  3. Constructive: ‘Could use some pictures for a friendlier feel.’

  4. Negative: ‘This information is appauling (sic) and of terrible quality. Alcoholics know what is wrong and why they need SOLUTIONS.’

Cannabis and mental health leaflet

  1. Commendation: ‘This was really really useful and opened my eyes to a lot of things I didn't realise.’

  2. Constructive: ‘More subheadings would be useful for young people - perhaps even illustrations. It has quite a high reading age.’

  3. Constructive: ‘It would need to have more visual diagrams and probably less slang (which can quickly get out of date).’

  4. Negative: ‘Negative and typical of official sources.’


Declaration of interest

M.B., P.T. and R.R. are involved in the production and development of the College's online and printed mental health information leaflets.


1 Coulter, A, Ellins, J, Swain, D, Clarke, A, Heron, P, Rasul, F, et al. Assessing the Quality of Information to Support People in Making Decisions About Their Health and Healthcare. Picker Institute Europe, 2006.Google Scholar
2 Elwyn, G, O'Connor, A, Stacey, D, Volk, R, Edwards, A, Coulter, A, et al. Developing a quality criteria framework for patient decision aids: online international Delphi consensus process. BMJ 2006; 333: 417.CrossRefGoogle ScholarPubMed
Figure 0

Fig 1 Example of feedback form.

Figure 1

Table 1 Website visits by country of origin

Figure 2

Table 2 Numbers of responses and feedback scores for each leaflet category

Figure 3

Fig 2 Age of readers.

Figure 4

Table 3 Average scores for highest and lowest performing leaflets

Submit a response


No eLetters have been published for this article.