The National Institute for Health and Care Excellence (NICE) strategic review of its public involvement offer included a survey with stakeholders to explore how NICE can continue to deliver high quality, meaningful public involvement in a rapidly-changing environment.

NICE staff, committee lay members, and an external academic ran the project and designed an online survey. The survey was open for two weeks. A purposive sample, recruited through various communication channels, was invited to participate. The sample comprised: (i) external individuals involved in NICE work, (ii) NICE committee and Board members, (iii) NICE staff. The survey included qualitative and quantitative questions, covering the ‘who’, ‘when’, ‘how’ and ‘what’ of NICE's public involvement approaches.

The survey yielded 684 responses, which were stratified by stakeholder type. Overall the responses indicated that: (i) the suggested stages for involvement are all important, but on a sliding scale: ‘defining outcomes guidance should consider’ is most important, and ‘helping committee chair recruitment’ is least important; (ii) different perspectives are needed such as individual treatment or care decisions should incorporate views of directly affected people, and population-based public health decisions need the views of citizens. Quality improvement suggestions included: (i) seeking feedback on people's experiences of care, using clear, structured approaches including focus groups, interviews, surveys, social media; (ii) increasing communications about NICE's work, specifically about involvement opportunities and use of patient evidence; (iii) using data on people's experiences equally with academic evidence; (iv) providing education and training on involvement to NICE staff and the general public; and, (v) partnership working with other organizations to enhance engagement. A focus group with key stakeholders used the survey findings to shape the subsequent public consultation document.

There was consensus that public involvement is necessary throughout guidance development; however, the type of person involved and nature of participation should vary across the development stages. Project challenges included managing diametrically opposing views, and the associated implications for engagement.

The National Institute of Health and Care Excellence (NICE) assesses the efficacy and safety of interventional procedures for use in the National Health Service (NHS). Since 2006, NICE's Public Involvement Programme (PIP) has obtained ‘patient commentary’ to inform committee decisions, using a questionnaire asking patients about their experience of the procedure including benefits, disadvantages and side effects. Commentary is considered by the committee alongside other evidence. The PIP has piloted a project to: capture the impact of the patient commentary on the committee's decision-making; explore patterns of impact; and identify criteria that indicate when patient commentary may not be required.

The pilot included all interventional procedures guidance started between February 2016 and February 2017. Committee members’ views were captured using a form completed whenever patient commentary was considered. Responses were anonymized, entered into an electronic system, analyzed, and correlated against ‘committee comments’ in the published guidance. After twelve months, there was an unrepresentatively narrow spread of conditions, and most topics were updating previously published guidance rather than novel topics. The pilot was therefore extended by six months.

Patient commentary commonly had an impact on decision-making; however, no discernible patterns have yet been identified, nor criteria for when it may not be required. Key findings were: (i) patient commentary is equally useful for guidance updates as novel guidance, and (ii) interpretation and assessment of ‘impact’ varied across committee members but the majority agreed it reinforced the other evidence.

Patient commentary has a measurable impact on committee decision-making. Very occasionally it provides new evidence and routinely provides reassurance that the published evidence is substantiated by real-world patient opinion. Measuring the impact of commentary seems to have raised its profile, with more committee comments about patient issues included in guidance during the pilot than in preceding years. The project needs to be extended to identify which procedures are least likely to benefit from patient commentary and why.

The National Institute for Health and Care Excellence (NICE) has a formal policy stating patients, carers and citizens are involved throughout each Health Technology Assessment (HTA). One key way patient/carer organizations are involved is by nominating patient experts to participate in appraisal committee meetings.

A NICE 2014 report (1) on Patient Experts experiences identified a need to routinely survey Patient Experts. This has been ongoing since October 2015. This study highlights key findings, including new recommendations and whether previous concerns have been addressed.

We refined the 2012 survey to be routinely sent to all patient experts that attended a NICE technology appraisal committee meeting. Between October 2015 and December 2016 this online survey was sent to eighty-eight patient experts. After analysis the findings were compared to the previous report to identify whether concerns have been addressed and whether new recommendations should be considered.

There was a response rate of 47 percent (n = 41). Quantitative results and qualitative quotes demonstrate that patient experience varies widely. Key findings from the new data revealed that patient experts feel supported by the Public Involvement Programme, however would welcome more opportunities to speak. Notable improvements since 2012 include favourable opinions of support documents and the Chair more regularly introducing themselves to the patient expert before the meeting. Some experts still find the paperwork cumbersome and find the meetings very technical.

Progress has been made since 2012, but further improvements could facilitate even more effective patient involvement. We continue to routinely measure experiences to identify and address evolving issues. Some tensions remain between NICE's remit and processes, and patients expectations of these. The findings, although specific to NICE, hopefully can feed into other patient involvement developments in the wider HTA ecosystem.

Patient evidence is submitted to the National Institute for Health and Care Excellence (NICE) by patient organizations and individual patient experts. Previously NICE developed a new patient organization evidence submissions template, based on the international HTAi patient submission template for medicines (1). The NICE template was reviewed by surveying committee members and also patient organizations who had used the submission template. The findings were presented at HTAi 2016.

The limitation of that review was the low response rate from patient organizations. The key recommendation was to extend the survey to include a larger number of patient organizations. These local findings are an opportunity to contribute to the global Health Technology Assessment (HTA) ecosystem.

A project group was convened consisting of NICE staff, a committee lay member and a patient organization representative. Together we reassessed the suitability of the previous feedback survey. This was then sent out to patient groups who had completed the submission template from July 2014 to November 2016. Additionally, public involvement staff telephoned selected patient organizations to increase the feedback response rate and gain greater understanding. The anonymized results were shared with patients involved in NICE who helped interpret the results from a patient organization's perspective.

Key findings are that patient organizations find:

• • the template clear

• • it was easiest to provide information about living with the condition

• • it was hardest to give information on equality issues and research evidence.

They would also like a submission guide, and to receive feedback on their submissions.

Although it was difficult to obtain feedback from the patient organizations on the submission template, the depth of information provided by them was fundamental to updating the template and producing a supporting guide.

This feedback on the local English needs can be used when evaluating the international submission template to form a greater part of the HTA ecosystem.