In a recently published review of supplementary search methods, we proposed that researchers could usefully record time taken to search and present outcome values in similar way to existing studies, to facilitate generalisability of outcomes, where appropriate. We also discuss the idea of linking literature search effectiveness to study value. In this vignette, we discuss which outcomes we believe are important to measure and why. We discuss this in the context of the review of supplementary search methods and using a recently submitted evaluation of contacting study authors for context.

In a recently completed systematic review, we contacted eighty-two study authors to ask three questions. We aimed to measure the following outcomes when contacting study authors: Effectiveness - determined as number of contacts compared to number of replies; Efficiency - i) time to make contact and ii) time between contact and reply. We determined this in hours, minutes and seconds, in line with other studies; Cost - determined by comparing the efficiency of contacting authors with the effectiveness; and Value - determined by reading and comparing the published studies with the replies received to see if any unique data were identified.

Effectiveness: thirty-eight answers were received from eighty-two possible contacts. Efficiency: In total, author contact took six hours, fifty-four minutes and twenty-five seconds across thirty-nine weeks. Replies were received across zero to thirty-nine days (median fourteen days). Cost: Cost for staff time was GBP 80.33 (EUR 91.20) or GBP 2.11 (EUR 2.40) per e-mail reply received. Value: We were able to identify value in author replies for each of the questions asked.

In a recently published review of supplementary search methods, and a linked evaluation of the effectiveness of contacting study authors, we suggest outcomes that should be measured to determine effectiveness of literature search methods. We conclude that measuring these outcomes demonstrate both effectiveness and value.

High quality evidence for test accuracy can be scarce. We assessed the test accuracy of two tests (Actim Partus and PartoSure) for the prediction of preterm birth. Twenty published full-text papers were included whilst conference abstracts were excluded. Since systematic reviews of diagnostic tests on other topics may need to rely on data from conference abstracts, we test whether the findings of our review would change with conference abstracts included.

Conference citations previously excluded (n=108) were re-screened for inclusion using the following criteria: i) the diagnostic test was Actim Partus or PartoSure ii) test accuracy data of preterm delivery within seven days was reported iii) the population was women with signs/symptoms of preterm labor with intact membranes. Relevant test accuracy data were extracted and used to calculate sensitivity and specificity. Pooled sensitivity and specificity for each test were run using data from full-text papers and conference abstracts combined. These values were compared with the pooled sensitivities and specificities produced for the systematic review using full-text papers only.

Preliminary pooled sensitivities of the sixteen full-text Actim Partus studies and sixteen full-texts and two abstracts were 0.77 (95% confidence interval (CI) 0.68, 0.83) and 0.76 (95% CI 0.69, 0.83) respectively whilst pooled specificities were 0.81 (95% CI 0.76, 0.85).and 0.80 (95% CI 0.75, 0.84) respectively. Preliminary, pooled sensitivities of the four full-text PartoSure studies and four full-texts and three abstracts were 0.83 (95% CI 0.61, 0.94) and 0.82 (95% CI 0.65, 0.92), respectively, whilst pooled specificities were 0.95 (95% CI 0.89, 0.98) and 0.96 (95% CI 0.94, 0.97), respectively.

Our findings suggest that the test accuracy results would not alter substantially with the inclusion of conference abstracts. However, work is ongoing to investigate how the assessment of heterogeneity and risk of bias across studies would alter given the difficulties associated with limited methodological reporting from conference abstracts.