3.1.1 #1 Information source selection

A primary application of CiteSource involves facilitating data-driven information source selection within the review process. Through the systematic tagging of search results with ‘source’ metadata, CiteSource enables researchers to determine the degree of citation overlap and uniqueness among information sources. This analysis illustrates which sources have the greatest potential for review versus those that may be considered excessive or potentially insignificant due to substantial duplication. We broadly consider ‘Information sources’ to encompass contributions from distinct methodologies, such as citation chasing, in effect, treating a single paper as a source of information. Consequently, by integrating the assessment of both information sources and methodological approaches, researchers can attain a holistic visualization of the collective performance of their comprehensive search strategy.

As an example of how CiteSource can be applied to guide the selection of information sources, we can consider the following scenario. A research team is investigating the ecological consequences of acoustical disturbances originating from pile driving activities in marine settings. The selection of appropriate information repositories may require the inclusion of discipline-specific databases, for instance, ProQuest’s Aquatic Sciences and Fisheries Abstracts (ASFA) and Oceanic Abstracts (OA), as well as multidisciplinary platforms, including Clarivate’s Web of Science (specifically, the Science Citation Index Expanded—SCIE), Elsevier’s Scopus, and Digital Science’s Dimensions. Furthermore, the team believes that academic theses and dissertations may be valuable and decides to include specialized databases such as ProQuest’s Open Access Theses and Dissertations (OATD). To initially evaluate the potential coverage and overlap of these sources, a relatively broad ‘naive’ search string is developed to capture a wide array of potentially relevant records. This string (Figure 1; Web of Science syntax) was subsequently adapted and applied across the six information sources. The results were then uploaded to CiteSource for comparative analysis, with each set of records distinguished by their assigned ‘source’ metadata.

Figure 1 Example Search String is a simpler approach combining the three PEO concepts within each search field. This figure shows strings in Web of Science syntax (visualized in https://www.medsyntax.org).Reference Hair, Bahor, Macleod, Liao and Sena ⁴² String syntax was adapted to meet the syntax requirements for each database. Full strings can be found in the supplementary file.

Once automated deduplication is complete, the team can quickly assess the overlap between each source. In this example, the heatmap (Figure 2) clearly shows us that all 77 records that were found in Oceanic Abstracts (OA) were also found in Aquatic Sciences and Fisheries Abstracts (ASFA). This overlap is explained by the fact that Proquest’s Oceanic Abstracts is included within the ASFA database while maintaining its own scope and the ability to search the database independently. While this overlap was included by design in this example, it provides an example of how CiteSource can be used to identify sources that may not provide unique content. In this case, the team (who may or may not have been knowledgeable about the ASFA/OA relationship) can use these data to make a data-informed decision to drop Oceanic Abstracts as a source. Simultaneously, recognizing a highly unique source like OATD might prompt consideration of exploring additional specialized repositories for theses and dissertations to ensure comprehensive coverage of this document type.

Figure 2 A heatmap showing the six sets of records uploaded and the number of overlapping records between each set. In this example, all 77 records from Ocean Abstracts (OA) are found in Aquatic Sciences and Fisheries Abstracts (ASFA), highlighted here in orange. The plot also shows that none of the 16 records from ProQuest’s Open Access Theses and Dissertations (OATD) were found in any of the other sources, highlighted here in green.

This example highlights CiteSource’s capacity to support data-driven information source selection, as illustrated by the potential findings of redundancy between ASFA and Oceanic Abstracts and the novel records contributed by OATD. CiteSource facilitates the strategic inclusion of sources offering unique value and the exclusion of those providing largely redundant content, thereby optimizing the efficiency and effectiveness of evidence-gathering efforts from the outset.

3.1.2 #2 Search string development

Search strings can range in complexity based on several factors, including the research topic, scope, database/platforms being searched, etc. Testing and iterating search strings can be time-consuming, due to differences in syntax across platforms and the need to compare the results of various techniques. CiteSource assists users in systematically evaluating and refining search strings. This process involves not only optimizing retrieval to enhance sensitivity and precision but also identifying and rectifying potential critical errors in syntax or logic,Reference Salvador-Olivan, Marco-Cuenca and Arquero-Aviles ⁹ all while speeding up the iterative analysis. By enabling the distinct tagging of search result sets (e.g., by string version or database) and the subsequent analysis of overlapping and unique records, users can quickly assess multiple iterations of a search string. Examples of such assessments include evaluating:

• • the impact of including or excluding terms

• • variations in syntax

• • different combinations of strings

• • different approaches to Boolean architecture

• • the use of different search fields

• • the effectiveness of capturing benchmark articles

To demonstrate CiteSource’s application in search string development and optimization, this example examines different approaches to Boolean architecture for a simple search string. Due to differences in the way database platforms function, it is often useful to compare search techniques, including variations in Boolean nesting (i.e., the construction of parentheticals), metadata field selection, and verifying correct use of syntax. For example, some database platforms do not allow users to combine the results of multiple strings, nest Boolean logic, or combine field codes to search across multiple fields at once; consequently, simplified search strategies are often required to accommodate these limitations. In this example, our team is once again examining the impact of pile driving on marine life, utilizing a search string that represents three concepts: marine/underwater environment (population), pile driving (exposure), and noise/sound (outcome). Two search strings (Figure 3) were developed with their Boolean architecture intentionally designed to yield different results, illustrating CiteSource’s capability to facilitate analysis of such variations. The first version, used in the prior scenario, is designed for broad compatibility across most platforms, reflecting a common necessity due to database constraints. The second string involves nested Boolean logic, representing a more robust and strategic approach that can be used in databases like Web of Science. Here, Web of Science, Dimensions, and Scopus have been selected to demonstrate how CiteSource can compare search results across different string approaches and database platforms; while observing string variations might not always necessitate multiple platforms, applying this method here serves to further showcase CiteSource’s capabilities in cross-database analysis.

Figure 3 Two different search strings using different approaches to Boolean architecture are shown. String Version #1 is a simpler approach combining the three PEO concepts within each search field. String Version #2 shows a more complex approach with nested Boolean phrases for a more comprehensive search. This figure shows strings in Web of Science syntax (visualized in https://www.medsyntax.org). String syntax was adapted to meet the syntax requirements for each database. Full strings can be found in the supplementary file.

After importing the search result files for each string version and database, metadata tags are assigned within CiteSource to differentiate each distinct set of results (Figure 4). CiteSource streamlines tagging and helps reduce errors by optionally auto-filling the ‘source’ field based on the file name; it also defaults the ‘label’ field to ‘search’, the most common initial designation. Both the source and label fields can be easily edited as necessary. Once citation files are tagged, we perform automated and manual deduplication (Figure 5). To efficiently resolve these flagged pairs, CiteSource’s ‘Individual Record Table’ (Figure 6) is particularly useful, as it displays comprehensive bibliographic details for every record, and critically, provides direct links (e.g., via DOI or URL) to the original source documents when available. By examining the manual deduplication table in combination with the full details, users can quickly access the source articles and accurately determine if the flagged items are true duplicates, even in cases of inconsistent or incomplete metadata.

Figure 4 After uploading files, users have the ability to tag each accordingly. In this case, six uploaded .ris files are tagged according to the information source and string. For efficiency, CiteSource auto-fills the source field according to the file name and automatically adds ‘search’ for each label, as this is the most common tag.

Figure 5 Manual deduplication table. The three rows highlighted in blue have been selected as duplicates. Columns show metadata for each record identified as a potential duplicate. Green boxes are added here to show records in pairs #3 and #4 are separate conference papers. Red boxes are added here to highlight incorrect metadata.

Figure 6 Individual record table is shown with two records that are identified in pair #1 from Figure 5. Used in conjunction with the manual deduplication table, records can be quickly checked by linking directly to the item. In this case, record ‘Reyff 2012a’ is correct, while metadata from 2012b incorrectly uses the book series title.

CiteSource utilizes the UpsetR packageReference Lex, Gehlenborg, Strobelt, Vuillemot and Pfister ⁴³ to visualize overlap across sources (Figure 7). This plot offers a powerful visualization for comparing the retrieval of different search string versions across multiple databases. Each row represents an information source, defined by the user with the ‘source’ field. Columns show the number of records that overlap across these sources. In this example, examining the intersections reveals that String Version #2 identified 17 records not captured by String Version #1(blue boxes indicate records found exclusively by String Version #2). The Detailed Record Table (Figure 8) is another way users can evaluate this overlap. The table shows the breakdown of each source, its overall record contribution, and the percentage of unique record contributions. In this case, there were only 11 unique records, 5 each from Dimensions and Scopus, and a single unique record from Web of Science.

Figure 7 The upset plot visualization can be used to assess the number of overlapping and unique records across each search string version and across databases (WoS=Web of Science; DM=Dimensions). This upset plot shows 17 additional records being found using Search String Version #2; blue boxes are added to highlight these records. Of the 17 additional records, 11 are unique to one of the three sources; red boxes highlight these records.

Figure 8 The Detailed Record Table provides information on the number of imported records, distinct records, unique records, duplicate records, the percentage of records contributed by each source, the percentage of unique records contributed by each source, and the percentage of each source that was unique.

3.1.3 #3 Benchmark testing

One way to understand a search string’s effectiveness is through benchmark testing. By starting with a set of articles that meet the stated inclusion criteria, researchers can gauge the sensitivity and effectiveness of search strings, identify potential changes to a search string to improve sensitivity, and understand the limitations of the search.Reference Lagisz, Yang, Young and Nakagawa ⁴⁴ These benchmarking functionalities are particularly useful in fields where certain key studies are considered foundational. Ensuring these foundational studies are captured by search strategies is critical for the credibility and completeness of a review. Here, we demonstrate how CiteSource can assist researchers when benchmark testing the results of searches.

Continuing with the example above, we have hypothetically selected 20 articles as a benchmark set (designated ‘BM’) based on prior knowledge of the field and examining previous related reviews. In this case, the Upset plot (Figure 9 – similar to Figure 7 but now including the benchmark set as a distinct set) shows us the performance of our search strategies against these key articles. The plot reveals that ten of the 20 benchmarking articles (designated BM) were captured across the three databases by both String Version #1 and String Version #2. Looking at the remaining overlap, we can see that String Version #2 found three additional benchmark articles that String Version #1 missed, and that two benchmarking articles were not found in any database with either string. This information, used in combination with the record-level table (Figure 10), can be used to quickly review why these benchmark articles were missed, whether due to missing/incorrect terms/syntax, or whether these records were not indexed in the information sources.

Figure 9 This upset plot shows an additional three benchmarking articles being found in Version #2 and two benchmarking articles that are not found. Blue boxes are added to highlight these records.

Figure 10 The individual record table facilitates the quick review of findings. In this example, we see the two benchmarking articles that were not found across the three sources using both string versions.

3.2.1 #4 Analyzing the impact of information sources & methods

To illustrate how CiteSource can be used to analyze the impact of various sources and methods, we can return to the research team investigating acoustical disturbances from pile driving. Once the team completed their searches using their optimized string across Web of Science, Scopus, and Dimensions, they supplemented their search with a call for papers from experts in the field, as well as by applying forward and backward citation chasing on two seminal review articles. At title/abstract screening, the team includes 100 articles, and after full-text review, the resulting final set contains 55 articles. To analyze the impact of each source and method, the team can now upload all sets of records and tag them accordingly using the source and label fields (Figure 11). After automated and manual deduplication is complete, the team can generate the Phase Analysis plot (Figure 12), which provides a visual breakdown of each source’s unique/overlapping record count as well as a count of these records over the screening phases. Looking at the plot, it’s easy to see the performance of each information source and method. Citation searching, for example, while initially providing a large number of unique articles, contributed no unique records after title and abstract screening. In fact, this method was by far the least precise, as only five of the 192 distinct records it produced were included in the final synthesis. The plot also shows that while the call for papers yielded a small number of results, this method of discovery contributed the largest number of unique records to the final synthesis. Finally, we can see that both Scopus and Dimensions outperformed Web of Science in the number of unique records that they contributed to the final synthesis. The data are further detailed in the Precision/Sensitivity Table (Figure 13), providing a full picture of performance. Looking at the table, we can see that despite the lower number of unique articles included in the final synthesis, Web of Science had a slightly higher precision.

Figure 11 After uploading files to assess impact, users remove source tags for any files that represent records that have been screened and included in the final synthesis. Labels can be changed from ‘search’ to ‘screened’ or ‘final’. In this example, forward and backward citation chasing are combined as ‘CiteSearch’ in the ‘source’ field.

Figure 12 The phase analysis bar plot shows the number of unique and duplicate records across each source and screening phase. Each facet contains the data for an individual source. In this example, the call for papers yielded the lowest number of records, but had a significant number of unique articles that were included in the final synthesis.

Figure 13 The precision/sensitivity table shows each source/method’s calculated precision and sensitivity. These calculations are based on the number of distinct records each source/method contributed, as well as the number of records from each source that were included in the final record set.

In practical terms, this impact assessment functionality can lead to more strategic decision-making in resource selection and method application. This quantification of contribution and overlap builds upon methods such as the Search Summary Table (SST) devised by Alison Bethel, which offers a structured overview of search results, indicating which searches found included references and providing sensitivity and precision calculations for search methods.Reference Bethel, Rogers and Abbott ⁴⁸ For instance, if certain methods consistently yield high-quality results that pass through all screening phases, they might be prioritized in future reviews or updates to the review. Conversely, methods or sources that contribute little value can be de-emphasized or excluded. This functionality enhances transparency about where and how included studies were found, facilitates a data-driven approach for future search method decisions, and deepens the understanding of the true impact of different resources and search methods. By providing insights into the performance and impact of each resource or method, CiteSource ensures that systematic reviews are both comprehensive and methodologically sound, supporting more robust and credible evidence synthesis.