Eighteen per cent of all internet users have sought mental health-related information (Reference Powell and ClarkePowell & Clarke, 2006) and they are more likely to search for information on depression than on any other condition (Reference TaylorTaylor, 1999). However, information available on the internet has been reported to be of variable quality (Reference BowerBower, 1996). Therefore, clinicians may have a role in helping the public find better quality websites.
Website sponsorship by organisations such as drug companies may be one factor contributing to disseminating ‘poor’ quality information (Reference Christensen, Griffiths and MedwayChristensen et al, 2000) and the public needs guidance on where to find reliable information on health issues. International initiatives such as DISCERN attempt to apply quality control to internet-based health information (Reference Charnock, Shepperd and NeedhamCharnock et al, 1999; Reference Gagliardi and JadadGagliardi & Jadad, 2002). However, DISCERN and other currently available instruments are complex and unwieldy, a user-friendly rating tool is needed.
Method
Websites on depression available on the internet at the time of the research were identified using the single search term ‘depression’ on multiple search engines. The UK and other English language websites were targeted to get an international perspective. Information available on the SearchEngineWatch.com website identifies the most commonly used search engines and eight of them were used for this research.
It is rare for people to look beyond the first ten results of a query (Reference Silverstein, Henzinger and MaraisSilverstein et al, 1998), thus our analysis was limited to the first ten sites listed by each of the eight search engines. The sites were excluded if the website: had no depression content; focused more on other forms of affective illness; was not in English; was linked solely to another search engine; or focused on the non-adult population (i.e. those under 16 years of age).
Two key documents were analysed for common factors suitable for converting into a simple measuring scale: the National Institute for Health and Clinical Excellence (NICE) draft document (NICE, 2001) and the World Health Organization Mental Health Checklist (WHO, 2001). From these, a ten-item scale was developed (Table 1). Each item attracts one mark and therefore all items are given equal weighting. The items address illness domains - aetiology, symptoms, treatment and prognosis. Summation of item marks gives a total score (C tot); the maximum possible score is 10 and the minimum is 0.
Depression-Website Content Checklist of key points on websites on depression
| Areas of interest | Illness domain | Items | Score |
|---|---|---|---|
| Nature of illness | Aetiology/symptomatology | 1. Low mood | 1 |
| 2. Loss of interests | 1 | ||
| 3. Decreased energy levels | 1 | ||
| Course of illness | Prognosis | 4. Duration | 1 |
| Treatment options | Treatment | 5. Antidepressants | 1 |
| 6. CBT/IPT | 1 | ||
| 7. Problem-solving | 1 | ||
| Treatment problems | Treatment | 8. Side-effects | 1 |
| Warning signs | Prognosis | 9. Suicidal thoughts/plans | 1 |
| Treatment | 10. Consider referral to specialist mental health services, e.g. psychiatrist for further risk assessment | 1 |
The quality of websites on depression was then assessed using both the Depression-Website Content Checklist and DISCERN - a previously validated instrument used to judge the quality of written healthcare information best applied by ‘experienced users’ (Reference Charnock, Shepperd and NeedhamCharnock et al, 1999). The DISCERN instrument generates a total score (D tot, maximum 75) for 15 key criteria. Scores were calculated for each website.
In addition to their individual quality, websites were compared according to two grouping variables, that is sponsored (yes) and non-sponsored (no). Evidence of financial support for the development and/or content of each of the websites was sought. A statement of such support on any of a website's pages resulted in allocation of that website to the ‘yes’ group, otherwise it was allocated to the ‘no’ group; the Mann-Whitney U-test (the non-parametric equivalent of a t-test) was used to compare the websites grouped in this way. They were also grouped according to organisation type - public organisations (governmental/professional bodies and charities) and private organisations (drug companies and individuals) - and analyses of their respective websites were performed.
Validating the Depression-Website Content Checklist
As DISCERN had been previously validated, correlation between it and the Depression-Website Content Checklist was measured to ascertain concurrent validity. Both are ordinal scales, hence Spearman's rank correlation coefficient was used. Internal scale consistency was measured using Cronbach's α. Each website was rated by a second-rater (S. M.) oblivious to previous scores and interrater reliability for the Depression-Website Content Checklist was measured using the intraclass correlation coefficient (ICC).
Results
We identified 68 different website addresses by analysing the first ten sites listed at each of the eight search engines when using the search term ‘depression’. In total, 39 websites were excluded according to previously stated criteria. The main reason for exclusion was that the site did not contain any depression content. There were 29 websites remaining for further investigation. However, several of those websites linked to the same website; for example, three addresses linked to the website for the National Institute for Mental Health. We were finally left with 23 websites to be further analysed (Table 2). Examination of the websites using the two instruments showed them to be of variable quality (range 2 - 10 on the Depression-Website Content Checklist, 24 - 74 on DISCERN).
Details of websites (sponsorship and nature of organisation) with Depression-Website Content Checklist total scores (C tot) and DISCERN total scores (D tot)
Identifying factors that appear to predict quality
Comparison of the sponsored and non-sponsored websites is illustrated in Table 3.
Descriptive and statistical analyses of websites on depression classified by absence/presence of sponsorship and type of organisation (public/private), using DWCC total score (C tot) and DISCERN total score (D tot)
| Grouping variable | DWCC total score (C tot) | DISCERN total score (D tot) |
|---|---|---|
| No sponsorship (n=15) | 9.00 (2–10) | 58.00 (24–73) |
| Sponsorship (n=8) | 5.50 (4–8) | 36.50 (31–48) |
| U | 25.500 | 26.500 |
| P | 0.024* | 0.030* |
| Public organisation (n=9) | 10.00 (3–10) | 64.00 (30–73) |
| Private organisation (n=14) | 6.50 (2–10) | 40.00 (24–61) |
| U | 24.000 | 22.500 |
| P | 0.013* | 0.011* |
Non-sponsored websites have higher median scores than sponsored websites when the two ratings C tot and D tot were used (Table 3). Observed differences are significant at the 0.05 level (C tot;Mann-Whitney U=25.500; P=0.024).
Comparison of the websites when grouped according to whether they belonged to public or private organisations is illustrated in Table 3. Public organisations have higher median scores for C tot and D tot. Observed differences are significant at the 0.05 level (C tot;Mann-Whitney U=24.00; P=0.013).
Assessment of the Depression-Website Content Checklist
There was a high degree of positive correlation between the Depression-Website Content Checklist total score for each website and the corresponding DISCERN total score (Spearmans r=0.935, significant at 0.01 level, two-tailed). Cronbach's α=0.857 for the Depression-Website Content Checklist and α=0.935 for the DISCERN tool. These values (α>0.8) indicate a high level of positive correlation between each of the items on both scales. The intraclass correlation coefficient was 0.85 (CI 95% 0.66-0.94) indicating very good agreement between raters when using the Depression-Website Content Checklist to assess the quality of websites.
Discussion
We aimed to assess the quality of information on depression available to lay internet users. Although other authors (Reference Griffiths and ChristensenGriffiths & Christensen, 2005) have also examined the quality of websites on depression, they were not previously ranked according to their popularity and the multiple search engines had not been employed. Our study demonstrates two things: first, that public, non-sponsored information on depression is of better quality than private, sponsored information available on the internet; and second, a simple user-friendly rating instrument (Depression-Website Content Checklist) can be used to assess the quality of websites.
Overall the quality of websites we analysed was extremely variable. The large proportion of websites excluded in this study (45/68, i.e. >65%) indicates the quality of information on depression on many of the most commonly accessed websites is rather poor. Many websites initially identified by search engines as containing information on depression did not actually have such information, which shows that individuals may be directed to a number of irrelevant sites. Several sites linked to the same provider, for instance the Royal College of Psychiatrists or the National Institute for Mental Health. This indicates that users may access the same information in different ways, but lack of clarity in accessing better quality information may ensue.
For websites with depression content it was possible to identify factors associated with ‘good’ quality. Our data indicate that the absence of sponsorship is associated with better quality information. Statistically significant differences were found when non-sponsored sites were compared with sponsored ones. Analysis of the content of the eight sponsored websites showed that all were provided by drug companies. Possible bias in favour of a drug company's own advocated treatment has been noted elsewhere (Reference Wahlbeck and AdamsWahlbeck & Adams, 1999). One might suggest that the poorer quality information on these websites compared with others is related to businesses wishing to promote their own products. This is not a concern of non-sponsored website providers. Such organisations include the National Health Service in the UK and the National Institute for Mental Health in the USA, which both aim to provide comprehensive information on health. These sites achieved the highest ranking scores on DISCERN and Depression-Website Content Checklist.
Other authors (Reference Griffiths and ChristensenGriffiths & Christensen, 2005) have also found non-sponsored sites to be of higher quality. Further research would help clarify to what extent findings from the UK apply across the rest of the English-speaking world. It is reasonable to assume that users are likely to access both national/regional and international websites.
In order to simplify the assessment of quality of internet information on depression we developed a simple-to-use Depression-Website Content Checklist that we also evaluated. Apart from its high degree of utility the instrument has good internal consistency (Cronbach's α>0.8 is comparable with a rating instrument suitable for making clinical decisions). It remains to be seen, however, how valid and reliable such simple instruments are.
As far as validity is concerned, both the Depression-Website Content Checklist and DISCERN have been shown to provide concurrent results. Moreover, as the Checklist's content was taken from the World Health Organization, its construct and face validity are also significant. It also possesses a high interrater reliability such that different users of the instrument are likely to reach the same conclusions about the quality of any given website. Since a certain degree of experience is needed to use DISCERN, some clinicians may prefer to use the Depression-Website Content Checklist.
In future this methodology might be employed to develop a similar scale for assessing internet information on other major mental illness such as schizophrenia. It would be relatively simple to devise item scores for website information on other illnesses, based around aetiology, symptoms, treatment and prognosis.
The instrument developed in our study concentrates on identifying the quality of information on depression on each of the websites we analysed. We defined the comprehensiveness of a website by the extent to which it displays the ten items of illness information that make up the Checklist. However, there may be other things about the websites such as their layout and design, which make them more, or less, helpful. A poorly designed website may make the information difficult to read and absorb, therefore reducing clarity. These factors were not considered in this study (errors of commission). Rather, we have been able to accurately identify those websites on depression that contained good quality information by checking how they score on the ten-item checklist.
We conclude that healthcare professionals have a key role in directing service users to good quality (public, non-sponsored) websites and the Depression-Website Content Checklist may help them do that.
Limitations
A small number of websites on depression were analysed and investigations focused on sites accessible by UK users. One of the authors (P.F.-L.) undertook the search for relevant websites on depression and rated them in turn, which means that the analyses are prone to rater bias. Also, we only looked at whether the items of the rating instrument were present/absent in each of the sites and did not examine the rest of the site content. Therefore, although the information we gathered is accurate and measured by our instrument, other aspects of the site (e.g. layout and design) were not analysed.
Declaration of interest
None.
Open access
eLetters
No eLetters have been published for this article.