Theme 1: Many Decisions and Judgments Are Necessary during Model Development

The focus groups highlighted that a large number of decisions and judgments are made by modelers during the process of model development, especially related to relevance and appropriateness to the decision problem.

“So if you set up five criteria [for selection of parameter estimates] and then you end up judging what you choose on the basis of six criteria, only two of which were of the five you started out with, then I think it tells you something about how much judgement is required in the process..”(Modeller)

Some participants expressed concerns regarding the difference in judgments made by different modelers to represent the same part of reality: participants were more comfortable if the judgments relating to what should be considered relevant to a particular decision problem did not rest solely with the individual developing the model but rather as a joint task between modelers, decision makers, health professionals, and other stakeholders. Failure to reflect conflicting views between alternative stakeholders may lead to the development of models which represent a contextually naïve and uninformed basis for decision making.

“There is also an issue of whose judgement it is - is it down to the modeller to make that judgement or should it be in conjunction with other people in the team or a clinical expert?” (Statistician)

Planning was regarded as a key element for model development. This could take the form of a protocol or project plans that clearly state the methodology for the model, information needs and key roles and responsibilities for all team members. Most respondents considered that having the evidence requirements set before model implementation allowed the whole team to identify potentially useful information and help to keep track of changes made during the process.

“I think that it would be useful for us to map out a list of things that you might consider . . . pathways in this particular way or problem structuring methods may have some role in identifying what should be included in a model . . .” (Modeller)

The modelers in the focus groups discussed the use of existing models and indicated that the main reason they used these for model development was to critique them so that they can avoid repeating the same mistakes and in turn produce a better model.

“I don't review models for their results, I don't review them to find things that I like – I review them to find things to avoid” (Modeller)

Although the use of existing models was considered to be potentially useful, the modelers suggested that they should be used with caution and should not be relied upon without considerable scrutiny, as the appropriateness or credibility of an existing model may be questionable and there may exist a gap between the decision problem that the model was developed to address and the current decision problem under consideration.

“Well, you have to be a bit careful with that, don't you? Sometimes you trace it back and you find somebody just thought that number up about twenty years ago, and everyone's used it and they’ve all done that since.” (Modeller)

There was variability between participants concerning their approaches to conceptual model development. Respondents discussed the use of several approaches including documenting proposed model structures, developing mock-up models in Microsoft Excel, developing sketches of potential structures, and producing written interpretations of the evidence. The participants all agreed that having a draft model or conceptual model would help to develop a common understanding of the evidence requirements among those involved in the model development. It would help to ensure that health professionals understood how the model would capture the impact of the interventions under consideration on costs and health outcomes, would ensure that the proposed model was clinically relevant and met the needs of the decision maker, and provide an explicit platform for considering and debating alternative model structures and other model development decisions before implementation.

“You’ve got to sort of get a feel of what the model is doing, not just looking at a whole lot of numbers on the spread sheet. So that's why I’m a big believer in a back-of-an-envelope version of the model, which forces you to really understand what's going on in the model.” (Modeller)

Furthermore, the participants expressed that where possible, alternative model development choices should be tested to assess their impact upon the model results.

“.. my view of structural uncertainty is it's where you’re equivocal about whether one set of assumptions is superior to another. . .I wouldn't expect someone to build 10 different models, but I’d expect them to consider what else they could have.” (Modeller)

There was a broad view among the respondents that the use of conceptual models and the use of existing models both have a part to play in model development and that the many decisions and judgments needed during the model development process should involve the whole team.

Theme 2: Searching Methods Appropriate to Modeling Are Required

One common concern raised was the need for appropriate searching methods to ensure the retrieval of relevant evidence for the model. Cost-effectiveness models have multiple information needs requiring different types of evidence drawn from multiple disparate information sources. The participants considered it to be difficult to capture the information needs in a single search query. Exhaustive search methods such as those used in systematic reviews may not be feasible.

“. . .it's relatively easy to find cost studies, it's relatively easy to find RCTs, it's not so easy to find adverse events-so I think maybe hints about how difficult it is to search for something like this in Medline ..might be helpful.” (Information specialist)

During the interviews, respondents discussed the difficulties in finding evidence and the different approaches used for seeking information, including formal Medline and other database searching, contacting experts in the field, searching registries and administrative or routine data sources, snowballing references, following leads, semantic technology, text databases and focused searching. However, the majority of the information retrieval processes are not explicit.

“In terms of parameters, we frequently have parameters that are only ever reported incidentally, things like unusual or irregular adverse events, ..- how do you find them? I don't know any systematic way that could do that. It's pot luck really,..” (Modeller)

“. . . I’m sure that some of the things I’ve had to resort to to obtain particular estimates much more resemble investigative journalism than having anything to do with what I’m trained to do”(Modeller)

The need for guidance and advice was expressed regarding factors that impact on the way evidence is identified, for example, what search techniques might be used to maximize the rate of return of potentially relevant evidence and to ensure appropriate steps have been taken to make the process systematic, reproducible and transparent.

“.. the identification of evidence is any information seeking process, which includes searching on Medline .. phoning people up or having a group of clinicians who advise .., you draw information from all of them and you use that information as evidence to support or justify or make decisions about the model to give it credibility. And as soon as you start to say, .. we just need guidance on searching, it suddenly excludes a lot of information seeking processes that are absolutely knitted in to the model development process” (Information specialist)

These findings show that evidence for models is identified from a variety of sources and searching methods differ from those used for systematic reviews of clinical effectiveness.

Theme 3: Methods for the Selection and Rapid Review of Evidence Are Needed

The participants identified several reviewing related issues such as: selection and prioritization of data, methods for reviewing, the use of hierarchies of evidence and assessment of evidence. These are covered in considerably more detail elsewhere (Reference Kaltenthaler, Tappenden and Paisley12).

The processes for selecting and prioritizing evidence used to inform parameter estimates need consideration according to the participants who also suggested that additional attention should be given to the reporting of parameters where the decision regarding the optimal selection of particular evidence sources was considered equivocal. The participants considered that it was important to prioritize parameters and focus reviewing resources on those elements of the model that were most likely to impact on model results, bearing in mind that the importance of parameters is subject to change during the course of the modeling process.

Owing to the time and resource constraints within the HTA process, all participants considered rapid review methods potentially useful to identify and select evidence. As the use of rapid review methods risk missing relevant information the participants considered it essential that methods were reported transparently, including the potential limitations of the chosen methods.

According to the participants, there is a wide range of types of evidence used to populate models and hierarchies of evidence sources, as suggested by Coyle et al. (Reference Coyle, Lee, Cooper, Shemilt, Mugford, Vale, Marsh and Donaldson18), may be useful as a means of judging the quality of individual parameter estimates and aid the study selection process. To incorporate the quality of individual studies into the selection process, one participant suggested that the Grading of Recommendations Assessment, Development and Evaluation (GRADE) (Reference Shemilt, Mugford and Vale19) system may be a potentially useful framework for rating the quality of evidence from all potential sources of data components that may be used to populate model parameters.

With regard to study selection and assessment of evidence, relevance or applicability could be assessed first to speed up the evidence selection process.

“.. actually if you look at applicability or relevance first, in a time constrained scenario, often that fatally rules out a huge bunch of stuff that you don't need to look at.” (Health economist)

In summary, reviewing resources should be focused on the most important model parameters. Relevance should be considered initially in evidence selection.

Theme 4: Communication among Team Members Is Essential

An important, recurrent and cross-cutting theme on which there was clear common agreement among focus group participants was that to have a credible and robust model effective communication across the whole project team was crucial. The team might include modelers, clinical experts, decision makers, information specialists, systematic reviewers, and others. The group agreed that all parties involved should have a common understanding of the model development process which could be achieved by recruiting clinical experts early on in the process and involving them in the development of the conceptual model, writing protocols, highlighting evidence needs, and having regular meetings with the whole team, presenting and discussing the model. Meetings should use nontechnical language easily understood by clinical experts and other team members. The sharing of internal draft reports was also considered important.

“. . .always having clinical experts, and not just one who is local . . ., but a panel who are available .. very early. You .. involve them in the model development process, which again will also be a fairly whole-team endeavour initially, so that the people who are doing the systematic review of effectiveness studies will be party to most of those conversations about the conceptual modelling of what the model structure might eventually look like.” (Modeller)

Engaging with experts and other researchers can serve as a face validity check to ensure that important information has not been missed, that the most appropriate parameter estimates are used and the opportunity for errors is reduced according to the participants.

“this is another reason why it's important to have, as well as clinical experts, .. other people who are researchers or have worked in the area, so at least after you’ve made some of these choices, then you can do .. reality checks and say, .. is there anything we’ve missed, are there any other recent or grey literature studies that might provide an alternative estimate . . .?”(Statistician)

An issue raised during the focus groups was the lack of communication and understanding between modelers and information specialists. This inadvertently led to the development of ineffective search strategies. The participants considered it important that modelers engage with the information specialist and keep them updated with the process of model development and information needs so that an effective and focused search strategy could be developed.

“the ..ability to go back to the information scientists and say this is more specifically what I’m looking for, but likewise also once the preliminary modelling results come back, .. to say we don't need to do this search anymore because the parameter that we thought it was going to inform is not going to affect the way this particular model works . . .” (Modeller)

All participants considered regular communication across the whole project team useful to ensure a common understanding and avoid mistakes.

Theme 5: Previous Modeler Experience and Clinical Input Are both Important

Input from clinical experts was considered to be crucial for developing the model structure to form an understanding of the decision problem.

“When we are presented with a new disease area, often the first time . . .. You are missing things all the way down the line. If you had. . .somebody else's expertise .. to draw on, you could make a much better job of it.” (Modeller)

In addition, the opinions of clinicians and a wider group of people who are involved in caring for patients were considered to be essential, as they can help provide parameter estimates or identify alternative sources of evidence (including unpublished literature). Furthermore, clinical experts and other researchers can serve as a reality check to ensure that important information has not been missed and that the estimates used are appropriate.

“one of our . . . clinical experts. . . pointed out that viral resistance was not included in our model. And we’d reviewed every single model of prophylaxis – none of those included it, and we’d reviewed the clinical evidence base, and that wasn't there ..and it completely changed the results.” (Modeller)

The group thought that the modeler's previous knowledge and experience of the disease area were extremely important, particularly in a time and resource constrained scenario.

“If you’ve been working in that field, and built up a network in it, then you are a long way ahead, and you have what you need.” (Modeller)

All participants believed that using clinical experts ensured an understanding of the decision problem and serves as a reality check. Previous experience of modelers in particular disease areas was also considered to be useful.

Theme 6: Reporting of Methods Should be Transparent

The focus group participants considered transparency to be an important issue. A view was expressed that there was rarely sufficient time to ensure that the methods used were reported adequately.

“I think transparency is one of the big things that is missing, isn't it? In the whole of modelling, anyway, there's the transparency issue.” (Modeller)

“. . .even if you have enough time to do that as an analyst you might not have enough time to write it up in a way that would make it more transparent.”(Modeller)

The group reflected that a comprehensive account of every evidence source used in the modeling process would be very time consuming and potentially difficult to read. Brief summary tables of the main inputs and sources of information could be used with the possibility of more detailed information presented in appendices.

The participants believed that it is important that decisions and judgments are clearly documented to ensure the credibility of the model including the following: search strategies, selection and justification of studies and model parameters; structural assumptions, decisions relating to the prioritization of key information needs, acknowledgment of limitations and potential biases and documentation of any deviations from the study protocol.

“It's more like the thought process you went through to get there, and what did you (do and what) did you decide not to do..” (Modeller)

Several participants believed that the language used in reporting model development and results should be clearly understood by health professionals and other individuals involved in the process. The model could be represented in both diagrammatic and textual forms using nontechnical, nonmathematical language.

“Again the language to describe that sort of thing would have to be very careful . . ..it needs to be more in terms of principles and things to consider than anything too prescriptive.” (Modeller)

In terms of model credibility it was emphasized that the modeler needs to cater for the target audience and there is a tension between accuracy and credibility.

“..but you know that the clinical audience . . . will expect to see that factor, because the literature has been going on about it for years, So I think we are playing lots of tensions to do with meeting the right audience, trading off accuracy versus credibility.” (Modeller)

The participants agreed that decisions made during the modeling process, especially potentially controversial ones, need to be transparently reported and summary tables may be useful. Key findings are presented in Table 3.

Table 3. Summary of key findings