Conceptualizing or Theorizing PPI

Good reporting is ideally based on clear concepts and definitions. While there are some helpful definitions of involvement, the conceptualization or theorization of PPI has generally been poor. There have been some attempts to develop conceptual or theoretical frameworks, but there is no overall conceptual model of PPI impact that captures the essence of the concept and has been empirically tested. Such models can be very helpful because they can provide a blueprint for evaluation, identifying key areas for assessment. The often poor reporting of definitions and the lack of an agreed conceptual framework or model for PPI was problematical in assessing the conceptual equivalence of studies, that is, the extent to which they were examining the same concept (Reference Brett, Staniszewska and Mockford4;Reference Mockford, Staniszewska and Griffiths15). The lack of conceptual frameworks or models also made it difficult when attempting to assess impact of PPI. Both the PIRICOM and PAPIRIS reviews adopted a broad approach and regarded impact as the reporting of any information concerned with the difference patient and public involvement has made to any aspects of research, researchers, patients and the public, users, organizations, or to service delivery, development, or evaluation, to ensure all potentially relevant information was included. The lack of a clear conceptual or theoretical underpinning for the reporting of impact may explain why studies have adopted a variety of approaches, epitomizing the phrase “letting a thousand flowers bloom.” While this provides a fascinating insight into the richness and diversity of patient and public involvement, it hinders consistent and more formal evaluation of impact.

Reporting of Methods

Studies of patient and public involvement varied enormously in the data provided about the PPI activity, the level of PPI adopted (consultative, collaborative, or user-led), the stages at which the activity occurred (whether one stage or multiple stages), and the research design used. A key difficulty was that methods were often described in very little detail and often inconsistently with varying information. It was rare to have a full description of participants, both service users and researchers. In addition, authors rarely identified the level or levels of involvement (consultation, collaboration or user-led) they used, leaving it to the reader to form an assessment. The stages at which PPI occurred in a study were also poorly reported, and it was often unclear whether PPI had occurred at one point or throughout a study.

Content Validity of Studies

A key difficulty with many studies of PPI was that they rarely assess the content validity of their investigation. Content validity is an assessment of the extent to which a study or an instrument captures all the relevant dimensions or aspects of a concept (Reference Streiner and Norman23). Ideally the assessment of content validity would be based on some conceptual or theoretical understanding of a concept. For example, when Ware and Sherbourne (Reference Ware and Sherbourne25) developed the Short-Form 36 (SF-36) questionnaire they stated that health comprised eight concepts including physical function, role physical, bodily pain, mental health, role emotional, social function, general health, and vitality. Thus, to ensure content validity, any generic patient reported outcome measure (PROM) should measure all eight concepts. This conceptual framework provides guidance on which dimensions of health all generic patient reported outcome measures should include, and offers a basis for the assessment of content validity. While content validity is a standard part of evaluating PROMs, the concept could also be transferred to studies of patient and public involvement. PPI studies rarely report the extent to which they have considered all the potentially relevant dimensions or aspects of PPI, particularly important when evaluating impact. Using the concept of content validity in the field of PPI could enable more consistent reporting of all relevant aspects of a study and so enhance future evaluation of the full spectrum of possible impacts.

Reporting of Context and Process

The PIRICOM review, in particular, through collaboration with patients and the public, identified the importance of context and process in the interpretation of PPI impact (Reference Brett, Staniszewska and Mockford4). Context refers to the conditions required for PPI to have an impact. For example, the appropriate support and training, the appropriate funding, positive attitudes toward PPI, and appropriate time allocation might be important in a particular situation. Process refers to the methods used to undertake the involvement (see Figure 1), such as level of involvement and the stages of the research process where involvement occurs. While some studies did describe context and process information, although in varying detail, it was rarely linked to any interpretation of impact. The importance of context and process suggests that PPI should be viewed as a complex intervention that requires multi-layered reporting.

Figure 1. The complexity of PPI impact evaluation.

The Status of Impact

While PIRICOM and PAPIRIS identified a wide range of studies that reported on various PPI activities, relatively few papers included an evaluation of the impact of involvement as a primary aim, possibly because studies focused on assessing the effectiveness of an intervention. This may indicate the low status accorded to PPI and the difference it makes and may explain why less emphasis has been placed on consistent reporting.

Locating the Evidence of Impact

In searching for and selecting papers for the PIRICOM and PAPIRIS review, it became apparent that there were some significant challenges in developing a sensitive and specific search strategy, which identified relevant papers, while excluding irrelevant ones. A key difficulty was the lack of MeSH (Medical Subject Headings) for PPI, which if they existed, could offer an efficient way of finding relevant literature. Instead authors have used a wide variety of keywords, often in an inconsistent way, which limited the utility of keyword searching and made the development of a search string very complex. A further difficulty related to using the title and abstract information as a way of identifying relevant studies, a common approach in reviewing and helpful in sorting through large numbers of studies. Sifting PPI studies by title and abstract was not a useful approach as information about impact was not always mentioned in the title or abstract, but could appear in another part of the paper, such as the results or discussion. As a result, entire papers had to be read for a robust assessment of relevance, which had a significant impact on the time needed for the review of papers. Such difficulties have important implications for future syntheses of evidence as the same difficulties will remain unless there is more emphasis placed on appropriate reporting in the title or abstract and appropriate use of keywords.

Locating Impact Information in a Paper

In addition to difficulties locating and selecting relevant papers, the PIRICOM and PAPIRIS studies also highlighted important difficulties locating information about impact within papers. Information about impact could appear anywhere in a paper and there was no consistent place where authors reported impact, and which a reviewer could consistently use during data extraction. Rarely was there a full and consistent appraisal of impact data with information about impact appearing in all the appropriate sections of a paper, (for example as an aim, in the methods section, or in the results, discussion, and conclusion) to allow a full appraisal of the difference PPI made in the study. More commonly, information about impact appeared as a one-off comment, which could occur anywhere in the paper. This inconsistency in reporting meant that papers had to be read in full for the purposes of data extraction, a very inefficient way of identifying and extracting relevant information which normally relies on key data appearing in expected parts of the paper.

In addition to the difficulties of finding the evidence, there was a range of issues that inhibited the appraisal of information about impact which relates to the nature of the PPI evidence base more broadly, including the poor conceptualization and theorization of impact. These issues need to be addressed as part of strengthening this evidence base alongside the development of better reporting and are briefly considered in the conclusion.

The Level of Detail About Impact

Both the PIRICOM and PAPIRIS reviews attempted to extract data on impact or the difference PPI made to research or to a health service. Any form of data was sought, including qualitative descriptions or quantitative measurements. Of the studies that considered impact, most reported impact using short descriptions. These were often very limited, lacking in detail and with no consistent structure. Descriptions of impact usually took the form of one or two lines of text and were not always described as a PPI impact, meaning that the researcher had to decide that this data represented impact. During data extraction, the identification of longer sections of text reporting impact were like “nuggets of gold.” In the absence of formal and structured forms of reporting, the most helpful types of report in the PAPIRIS study were the small number of case studies, often contained in longer reports, which were time-consuming to read and not always peer-reviewed.

While helpful in providing some insight and enabling the identification of broad groupings of impact (impact on research, researchers, service users, community, policy, funders, journals), overall the level of reporting was inadequate if the reader's aim was to understand properly the difference that patient and public involvement had made, to use this information to develop a more coherent conceptual or theoretical model of impact, or to use such information to develop a bank of items for the development of an instrument that includes all relevant dimensions of impact. In many ways, the nature of current patient and public involvement reporting could be compared with a scenario where a researcher reports the findings of a randomized controlled trial (RCT) by stating that “a particular intervention has worked and patients feel better,” and there is no further information. Such poor reporting would obviously impede a rigorous assessment of the effectiveness of an intervention. Such difficulties in RCT reporting prompted the development of CONSORT guidance to help improve the quality of reporting (Reference Moher, Hopewell and Schultz16;Reference Moher, Schulz and Altman17).

Positive and Negative Impacts

In reporting the impact of PPI most studies report positive impacts, with negative impacts more rarely considered. This may reflect an implicit assumption that PPI is a good and worthwhile activity, or that negative impacts have been politically more difficult to report. In ensuring content validity, papers should report both positive and negative impacts and also evidence of no impact, to enable a comprehensive understanding of the full breadth of PPI impact in a particular study. The need for such a comprehensive approach to reporting positive and negative impacts is particularly important for the development of robust instruments to measure impact.

Quality of PPI

Current approaches to assessing the quality of research focus on evaluating the study design but do not usually include PPI as an element of quality. There was no formal way of evaluating the quality of PPI or the quality of PPI impact reporting when the PIRICOM and PAPIRIS studies were conducted. In many respects, this might reflect the lack of critical evaluation which pervades much of the PPI evidence-base. More recently, guidance on evaluating the quality of PPI has emerged and may help strengthen quality assessment in this area (Reference Wright, Foster, Amir, Elliott and Wilson26). This guidance aims to help reader's assess the quality of published studies, researchers to develop effective strategies for user engagement, and funding bodies to establish principles of effective user involvement. It relies on good quality reporting, which the GRIPP checklist will seek to encourage.

Capture and Measurement of PPI

The main ways in which PPI impact is represented is through short descriptions. No standard formats exist for describing or capturing these impacts, and so they tend to vary in content, structure, and presentation. While other areas, such as patient experiences or patient-reported outcome measures have developed instruments which, with varying degrees of success, measure the concept of interest, PPI does not have a pool of robust, well developed instruments to measure PPI impact (Reference Ware and Sherbourne25). Robust measurement of the extent of PPI impact could provide additional information that could enable a greater understanding of what works, for whom and in what circumstances. The application of psychometrically derived methods of measurement has much to offer PPI in developing robust instruments to measure the impact, context, and process.

Economic Aspects

In exploring the impacts of patient and public involvement, it is important to consider all potential impacts, including economic impacts, particularly if forming a judgment about whether a particular involvement activity is cost-effective. In the PIRICOM and in the PAPIRIS systematic reviews, there was no evidence of any economic modeling of costs or benefits, with only a very small number of papers mentioning costs of particular patient and public involvement activities. It is important that, in future theorizing of involvement, economic impacts are considered alongside forms of impact as part of a broader development of the patient and public involvement evidence-base.

A Linked Body of Research

The PIRICOM (Reference Brett, Staniszewska and Mockford4) and PAPIRIS (Reference Mockford, Staniszewska and Griffiths15) systematic reviews both identified a large number of diverse studies that have examined different aspects of patient and public involvement. However, this diversity poses several difficulties when attempting to synthesize different types of information about patient and public involvement including impact. Many studies simply report the results of a project, not always linking their study to a much broader body of work, or clearly outlining how their work moves thinking forward conceptually or methodologically. We risk reinventing the wheel many more times unless greater efforts are placed on developing a coherently linked area of work. This provides significant challenges for researchers in stating explicitly how their study adds to the body of knowledge to enable the conceptual and methodological developments to be clearly visible.

Reflective Evaluation and Interpretation

The ability to critique and to reflectively evaluate and interpret research underpins the development of conceptual and methodological thinking. While such reflective thinking is present in the field of PPI, it tends to be found in opinion or reflective articles and less often in studies reporting the results of a project trying to establish the impact of PPI. For example, when studies presented their findings in relation to impact, the short descriptions (which formed the main way in which such data were captured) were rarely accompanied by any critical reflection. There was rarely any attempt to interpret or explain any relationship with context and process in evaluating impact. As a result, there is an important need to develop this critical reflective capacity in the reporting of impact, to enable the reader to critically assess a study and judge its merit and contribution to our broader understanding of patient and public involvement.

Development of the GRIPP Checklist

This paper has reported the narrative synthesis of the key issues that emerged from the PIRICOM and PAPIRIS systematic reviews and were primarily concerned with the poor quality of reporting within PPI studies. To develop the GRIPP checklist in this follow-up study, the research team carefully considered each issue in relation to several criteria: (i) whether the information was important to report within a paper that included some level of PPI, (ii) whether it would contribute to enhancing the evidence-base of PPI reporting more generally, and (iii) where the information should be reported to create greater transparency and so enhance the ease of future synthesis.

Criteria iii was useful for considering where information about an aspect of PPI should be reported within the structure of a paper, to enhance the quality of reporting. These deliberations helped to structure the checklist according to the key sections usually expected within a paper. The aim was to create a checklist that was logically structured and could be easily used by authors in writing their PPI papers and reports, editors and peer reviewers in reviewing manuscripts for publication, and readers in critically appraising published articles and reports.