A. Defining the Process

Modern, high-throughput genomic testing comprises a series of steps performed within or at the direction of the laboratory. First, the biospecimen is collected, stabilized or preserved, and transported to the laboratory in an identified fashion. DNA is then extracted from the patient's biospecimen, and the laboratory applies special-purpose chemical reagents to create a library, which is a collection of short DNA fragments that can be analyzed simultaneously in parallel. Three analytical phases then follow, known as the “bioinformatics pipeline,” involving the use of instrumentation (sequencing analyzers and physical devices), consumables and supplies (e.g., chemical reagents), analytical software algorithms, and skilled human personnel.

• Primary analysis, which involves raw data generation and base calling, uses instruments and software to read the sequence of nucleotides in the various DNA fragments and to assess the quality of the readings.^{Reference Celesti, Goldfeder and Moorthie24} The current output of this process is generally a FASTQ data file that records the line-up of nucleotides in the tested fragments.

• Secondary analysis uses software to map these fragmentary readings onto a human reference genome, probabilistically aligning the fragments back into order and using the software to filter and clean the data and call variants; that is, identify specific locations where the tested individual's genome differs from the human reference genome.²⁵ The current outputs of this process are generally a binary alignment map (BAM) file or compressed alignment (CRAM) file,^{Reference Hsi-Yang26} in which the fragmentary readings are reassembled into a model of the person's tested DNA, and a variant call file (VCF) that summarizes the identified genetic variants.

• Tertiary analysis involves annotating the variants with available information about the potential clinical significance of those variants, prioritizing the variants through filtration using various annotation fields, and interpreting a subset of the variants to prepare a test report.²⁷ The interpretive process typically requires a mix of interpretive software and expert human judgment.

In addition to the software required for these three analytical steps, genomic testing laboratories also depend on process-related software — for example, laboratory information management systems that help orchestrate workflows, guide laboratory technicians and automated systems in using the right reagents in the right way, track specimens and results, and generate quality control data.²⁸

Some of the software in the genomic testing bioinformatics pipeline is a component of instrumentation that the laboratories purchase (“embedded software”). However, some laboratories rely, at least in part, on software sold separately as an accessory to their instrumentation, software developed in-house, or software supplied by external software vendors or cloud-based service providers (“stand-alone software”). The level of regulatory oversight software receives may vary, depending on its provenance, as discussed later in this article. Whatever its origin, all of the analytical and process-related software requires appropriate verification and validation,²⁹ and the complex algorithms embodied therein require validation and testing. Data scientists, clinicians, commentators, and regulators are only beginning to identify the requirements necessary for responsible clinical implementation of complex algorithms, which may but do not always include machine learning algorithms and deep learning neural networks that are now involved in sequencing.^{Reference Vollmer, Wiens and Schrag30} Definitions of quality are not yet widely agreed on for medical algorithms.

B. Analytical Validity Challenges

In the context of genomic sequencing, analytical validity is particularly challenging because of the sheer scale of the genome (3 billion haploid base-pairs or 6 billion per individual). Current methodological limitations preclude analytic validation of every potentially meaningful variant, and certain regions of the genome are known to be extremely challenging to measure accurately with currently available laboratory methods.^{Reference Pant31} Most NGS produces short reads and then aligns them to a reference genome. Using ordinary methods and algorithms, however, NGS may not be able to detect differences in the copy number of a gene or gene segment because it might align all copies to the same place. This technology may also fail to detect major rearrangements because it simply aligns short segments to the reference and does not put them in the order in which they actually appear in the patient's genome. Sequencing algorithms may filter out short rearranged segments.^{Reference Wheeler32} Consequently, no extant analytical approach will reliably detect and map every variant in a single test. Moreover, whole genome or whole exome sequencing may be more accurate for some variants, while more focused sequencing (of a single gene or a few genes) will work better for others. Genomic sequencing may also result in the identification of many “novel” results, that is, variations that have not been previously observed. Novel findings need to be confirmed using a second (orthogonal) sequencing method to determine whether the variation is actually present or is instead an artifact resulting from technical error,³³ whether inherent in the method or caused by the operator. These many limitations of the current sequencing technologies and processes make it critical to identify the strengths and weaknesses of different analytic approaches^{Reference Scheuner34} and to specify areas of potential uncertainty for each, because clinicians and patients need to be able to assess the accuracy of the particular variants on which they are basing decisions about care.

At present, the best indicator of a laboratory's ability to analyze a DNA sequence accurately is comparison to a standardized reference sample developed based on sequencing and analysis of a significant number of other genomes. A number of organizations are working to develop suitable reference materials for clinical laboratories to use in method development, test validation, internal quality control, assay calibration, and proficiency testing,^{Reference Zook and Hardwick35} but many more reference materials are needed.

Bioinformatic algorithms that call bases, map bases to reference sequences, and interpret sequence data can be highly technical. While it is clear that complex analytical problems can be solved correctly using different mathematical and computational methods, bioinformatic systems also need to be validated and tested extensively so their performance parameters are well understood and can be communicated to users.³⁶

C. Clinical Validity Challenges

In the context of genomic sequencing, the number of variants that can be identified vastly exceeds the number for which clinical significance has been established,^{Reference Dewey37} a gap that will persist even as our understanding of the impact of variants individually and in combination continues to grow. Moreover, clinical significance resides along a continuum, so that geneticists classify variants into one of five categories: pathogenic, likely pathogenic, uncertain significance, likely benign, and benign.^{Reference Nykamp38} If a variant with strong, reliable evidence of pathogenicity (e.g., co-segregation with disease status within large families and/or functionally tested in a valid assay) is identified in a patient who already has cardinal signs and symptoms of the related disease, then the clinical significance of the variant is typically not in doubt.

Even where a variant has strong, reliable evidence of pathogenicity, the impact of such a variant in the context of a specific patient can be uncertain. Even single-gene diseases may manifest a variety of possible phenotypes; for example, patients with sickle cell disease, the prototypical disease caused by a single base-pair change, can have a range of possible symptoms, including painful crises, acute chest syndrome, overwhelming sepsis, and/or strokes.^{Reference Pecker, Little, Meier, Abraham, Fasano and Beutler39} Furthermore, single-gene disorders frequently vary in their “penetrance,” that is, the proportion of individuals with a pathogenic variant who exhibit clinical symptoms. For example, many genes contribute to the development of cardiac arrhythmias. However, some individuals who carry one of these dominant mutations do not exhibit any symptoms.^{Reference Miko40} Incomplete penetrance complicates the ability to predict disease in currently asymptomatic individuals, as some individuals who carry a pathogenic mutation may never become ill.⁴¹ Further complicating clinical interpretation of genetic information is the fact that our understanding of a variant's clinical significance can shift over time, as new scientific evidence accumulates. Consequently, a variant suspected to increase risk today may turn out to be benign, while a variant believed to be unrelated to disease risk may turn out to increase risk in certain contexts.^{Reference SoRelle, Taber and David42} At any one time, experts may have differing opinions about how best to analyze the genome or to interpret the probable phenotypic effect of particular variants.

Computational algorithms for interpreting genomic data are essential tools for establishing clinical validity, but as in the case of analytic validity, challenges abound. Each algorithm may have unique filters and cutoffs. For instance, many algorithms filter in variants that are rarely seen in a population, but the algorithms can use different cut-offs for what counts as rare. Each algorithm also embodies its developers' views on how best to predict the pathogenicity or non-pathogenicity of a variant. Many users may not be able check the output of these algorithms by manually analyzing the clinical significance of variants. In any event, as sequencing becomes more common, the sheer volume of data may preclude most manual assessments.

Analysis algorithms search databases for reports of disease associated with the variants of interest in a clinical sequence. Unfortunately, many genomic databases contain mistakes or misleadingly incomplete information.^{Reference Coovadia43} In addition, databases at present do not adequately represent population diversity^{Reference Popejoy, Fullerton and Manrai44} and may or may not include unaffected as well as symptomatic individuals. These gaps can lead to interpretive error, and consequently to misjudgments regarding the clinical validity and significance of a variant.^{Reference Kohane45} As discussed in further detail below, FDA is working to address these technical gaps through guidance.

Finally, difficulties assessing clinical validity also arise at the level of the gene and not the variant. Whether a given gene is associated with a given disease can be perplexing. In a recent example, it was found that standard panels for testing of Brugada Syndrome included a high number of genes for which there was insufficient evidence that they were causally related to the condition.^{Reference Hosseini46}

Thus, genomic approaches face all the challenges that can bedevil the interpretation of single genes. But genomics also permits the study of more complex diseases, which may be influenced by the effects of numerous genes often in a particular pathway and numerous potential variants of variable penetrance or expression,^{Reference Vihinen47} leading to a variety of symptoms. Genomic risk scores — which combine the individual, often small, effects of tens or hundreds of different genes — are currently being developed^{Reference Natarajan48} and marketed.^{Reference Ray49} At present, however, genomic risk scores are rife with interpretive uncertainty, raising questions about their value for diagnosis, much less for prediction of future disease state in an asymptomatic person.^{Reference Hunter, Drazen, Rosenberg and Curtis50} Creating a massive matrix of computer-searchable medical phenotypes associated with genotypes, as well as a wealth of environmental and other data from large numbers of individuals (as initiatives such as NIH's All of Us Research Project seek to do) may ultimately help to provide more insights into the clinical impact of most variants.⁵¹

D. Clinical Utility Challenges

Finally, clinical utility is more difficult to define and assess in the context of genomic sequencing than in interpreting the significance single-gene variants. Genomic approaches clearly can be useful in guiding clinical management for individuals with a family history suggesting a dominant disorder that could be attributable to any one of several genes^{Reference Ma52} and for people who have an undiagnosed disease that appears to have a genetic contribution.^{Reference Splinter53} While these uses of sequencing fall easily within a narrow definition of clinical utility, they raise a number of questions that are beyond the scope of this paper, including deciding whether to pursue genome-based versus more focused approaches as well as whether to search for and return secondary findings. In the absence of a clinical indication for sequencing, however, the likelihood that a person will receive results that will improve long-term health depends in part on how much of the genome is being analyzed, the nature of the diseases or conditions being assessed, whether the diseases would otherwise be detected and adequately treated, competing morbidities, and the strength of the evidence regarding the efficacy of prevention or intervention.

There have been some efforts to assess the utility of genomic screening when the individual does not have a pertinent family history or current symptoms. For example, investigators at Geisinger Health System sought to understand whether genomic screening could identify previously undiagnosed patients with familial hypercholesterolemia (FH). They analyzed genomic sequencing and electronic health record data from 50,726 individuals from the Geisinger Health System to understand the prevalence and clinical impact of FH variants in this clinical cohort. They identified 229 individuals carrying one of the FH variants. They determined that only 24% of these carriers would have met the criteria for probable or definite FH diagnosis in the absence of variant identification, although most of the carriers, whether or not identified as having FH, were receiving lipid-lowering treatment. The study concluded that genomic data can augment the detection of individuals with FH and lead to identifying patients who would have been missed by using only clinical criteria in electronic health record (EHR) screening.^{Reference Abul54} In the eMERGE network, which examines variants in approximately 100 genes (including the ACMG list of 59 genes as well as a number of other genes) in approximately 25,000 individuals, preliminary data show that somewhere between 3-5 percent of the participants, some of whom had previously been diagnosed, had returnable results.^{Reference Gibbs, Rehm and Reuter55}

Notwithstanding the previous examples, few studies to date have offered genome-based screening to adults in the absence of a clear clinical indication. This makes it difficult to determine whether and under what circumstances routine genome screening may improve patient outcomes, especially in light of the potential for overdiagnosis^{Reference Elmore and Welch56} and iatrogenic costs and harms from non-indicated interventions.^{Reference Khoury, Meagher, Berg, Narod, Sopik, Cybulski and Vassy57} The likelihood of net benefit will depend not only on which genes are interrogated and the strength of the geno-type-phenotype correlation⁵⁸ but also on the ability of providers, many of whom are not genetic specialists, correctly to interpret and act on the results.^{Reference Arora59} Some commentators have raised questions about whether primary care providers are willing and prepared to handle information from genomic screening, particularly in the absence of clinical decision support (CDS) tools, availability of genetics specialists for referral, and insurance coverage for the initial consultation and follow-up.^{Reference Christensen and Pet60} A 2017 pilot study sought to describe the effect on clinical care and outcomes of adding whole genome sequencing to a standardized family history assessment in primary care. The study concluded that adding whole genome sequencing (WGS) to primary care reveals new molecular findings of uncertain clinical utility, and that while non-geneticist providers may manage some genetic results appropriately, in other cases the information may prompt additional clinical actions of unclear value.⁶¹

Notwithstanding uncertainty around the clinical utility of genomic information, consumer interest in obtaining genetic testing outside the clinical setting as well as enrollment of individuals in longitudinal WGS research suggests that some people value receiving the information and perceive it as meaningful for themselves or their children.^{Reference Robinson and Genetti62} Recent studies have sought to assess the views of those who have taken part in WGS research and to evaluate whether their a priori expectations were met. One randomized study surveyed 202 primary care and cardiology patients in the Med-Seq study before and up to six months after receiving either WGS and family health history (FHH) or family history information alone. The study found that decisional regret overall was low in both groups, but that those who did not receive WGS information were more likely to report at least some regret about participating in the study. Participants who received both FHH and WGS information were much more likely to report that study results had provided new information with some level of personal or clinical utility. For example, they were over seven times more likely to report that study results had yielded accurate identification of disease risk and more than twice as likely to report that results had or would influence their medical treatment. Yet the majority of those who were found to have a pathogenic variant had no evidence of disease even after extensive clinical investigation. At the same time, they expected a higher level of benefits from the study than were actually achieved and also were more likely to report receiving too much information (particularly in the primary care group), which led the authors to recommend tempering patients' expectations.^{Reference Roberts63}

Another study, involving 29 HealthSeq healthy participants who completed six-month follow-up, sought to gauge the psychological and behavioral impact of receiving WGS results at various time points. The study found that most patients had positive emotional reactions to receiving their results, although a few expressed negative reactions.^{Reference Sanderson64} Of the seven participants who received pathogenic or likely pathogenic rare disease variant results, two were concerned about and acted on the information. Two participants who had APOE e4/e4 variants (indicating increased Alzheimer disease risk) reported being concerned, while one participant who had APOE e4/e3 variants (associated with more modestly increased risk) was confused about the result. The data also showed that among those who reported distress from the information, such distress had largely subsided by the six-month assessment. The study concluded that currently neither the benefits nor harms of personal genome sequencing are significant for most individuals, but that there may be important exceptions warranting further investigation and that the impact of returning WGS results on a larger scale remains to be seen.

A. Regulation of Clinical Laboratories

B. Government Regulation of Clinical Utility

CMS does not regulate the clinical utility of tests under CLIA.¹⁵¹ The simplistic account of FDA regulation is that FDA also does not require evidence of clinical utility for genomic tests. The reality is more nuanced. First, FDA assesses the safety and effectiveness of tests relative to the manufacturer's intended use for the test. If the manufacture states a clinical intended use (e.g., “This test is useful for diagnosing cystic fibrosis”), as opposed to stating a purely analytic use (e.g., “This test accurately detects the presence or absence of specific CFTR genetic variants”), then the test's intended use implicitly asserts clinical utility. When FDA confirms that the test is safe and effective for a clinical intended use, FDA in effect is confirming that the test has its stated clinical utility. Second, FDA has authority to oversee not just the analytical and clinical performance of tests — that is, their analytical and clinical validity — but also the labeling of tests.¹⁵² FDA “want[s] to make sure that what a manufacturer is saying about their test in their labeling, in their materials about the test is truthful and accurate.”¹⁵³ There is no requirement for test manufacturers to make any claims of clinical utility in their labeling, and a test can be brought to market with purely analytical claims. But if a manufacturer does elect to make any claims about clinical utility in a test's labeling, FDA will require evidence to support those claims.¹⁵⁴ This is part of FDA's statutory mandate to ensure that drug and device labeling shall not be “false or misleading in any particular.”¹⁵⁵

FDA recognizes, however, that clinical utility is, to a large degree, a medical practice issue. Clinical utility addresses whether a test (and subsequent interventions taken based on the result) leads to better clinical management or an improved health outcome among people with a positive test result.¹⁵⁶ No matter how accurately a test does its job (e.g., detecting cancer), the test will have no clinical utility if the clinician orders an inappropriate course of treatment after receiving the test result. For this reason, ensuring the clinical utility of tests is largely the province of state medical practice regulators and tort law. Professional societies also offer guidance.¹⁵⁷ FDA's role is generally confined to ensuring that any assertions test manufacturers make about clinical utility are supported by sound evidence.^{Reference Jillson158}

For many people, genomic tests are clinically available as a practical matter only if they are covered by insurance. Importantly, both government and commercial payers evaluate clinical utility, usually in terms of the impact of test results on the patient's health outcome, as part of determining whether to cover clinical genomic testing.^{Reference Deverka and Dreyfus159} In light of the rapidly evolving but still incomplete understanding of the clinical utility of particular variants, payers are more likely to pay for focused genetic tests with clearly documented clinical impact than for broad-based genomic tests in which the clinical implications of individual variants vary widely.¹⁶⁰ The local contractors that administer Medicare payments in specific regions of the country have significant discretion in determining whether to cover diagnostic testing, including genomic sequencing. The same is true of private payers, which often but not always, look to Medicare coverage policies in deciding whether to cover a particular test.^{Reference Phillips161} In 2018, CMS issued a national coverage decision stating that coverage of NGS for tumor profiling is required only where there is an FDA-approved companion diagnostic linked to the testing,¹⁶² but the agency is reconsidering its decision in light of significant stakeholder objections.¹⁶³ Pressure is also growing for payers to provide coverage with evidence development in which patients are enrolled in clinical trials or registries to inform future reimbursement policy.^{Reference Eisenberg and Varmus164} Private payers often follow the CMS coverage determinations but, because private insurance is a matter of contract between private parties, they are not bound to do so. As a result, private payer coverage can be inconsistent from one payer to the next and can change over time.

To the extent a patient is willing and able to pay for testing directly, however, the ability to obtain genomic testing is limited only by a physician's willingness to order it, which may reflect at least in part the clinician's assessment of its value, and the laboratory's willingness to provide it. Some states allow direct-access testing, in which patients can order their own laboratory tests without having to go through a physician, although not all states allow this. In states that allow direct-access testing, physicians are less able to serve as a check on the use of tests with uncertain clinical utility, and patients' access to tests is limited only by their pocketbooks.¹⁶⁵

C. The Role of Non-Governmental Entities in Ensuring Genomic Testing Quality

Many non-governmental entities, such as the Association for Molecular Pathology (AMP),^{Reference Schrijver166} CAP, and the American College of Medical Genetics and Genomics (ACMG), have issued standards and guidance documents designed to promote the quality of clinical genomic tests. One major topic has been improving the interpretation of sequence variants,^{Reference Roy, Richards and Kearney167} particularly in the area of oncology.^{Reference Jennings and Li168} The CAP issued broad-based laboratory standards for next-generation sequencing¹⁶⁹ and has a number of standing committees on issues related to molecular pathology and genomics.¹⁷⁰ The ACMG has issued a statement about the need for genomic data sharing to promote quality.¹⁷¹ As befits an organization focused on clinical care, a number of its statements are directed more toward clinical utility and practice.^{Reference David172} These guidelines, and the steps that lead to their development,¹⁷³ play a crucial role in shaping practice. While these documents are not directly enforceable, their impact can be increased if they are adopted by payers as a condition of coverage and reimbursement.

A. Findings

1. Several barriers at times challenge laboratories' ability to deliver high quality genomic tests, including:

   • a. The clinical evidence base, while developing rapidly, is still incomplete.

   • b. Current testing methods are less accurate in some parts of the genome and for certain types of variants.

   • c. Interpretation of variants can be challenging, especially for individuals who are not of northern European ancestry due to a lack of diversity in individuals tested to date.

2. Quality is a joint effort involving laboratories, expert input (including from physicians and genetic counselors, laboratorians, and bioinformaticists), industry, professional societies, patient advocacy groups, patients themselves, and regulators. Cooperative and consensus-based approaches should be developed to advance quality (see, for example, the “Collaborative Community” concept being discussed by FDA).¹⁸⁶

   • a. Professional and technical standards can advance quality when developed in a transparent, evidence-based, and rigorous fashion that includes all stakeholders and that includes and addresses the interests of patients.

3. Stakeholders, including patients, need certainty and clarity on:

   • a. The jurisdiction of each relevant oversight agency.

   • b. Requirements for demonstrating quality.

   • c. Regulatory processes.

4. The same activity should be regulated (or not regulated) in the same way.

   • a. Genetic and genomic tests can be developed either by the medical device industry (i.e., test kits) or by laboratories (i.e., LDTs), but, in the current scheme, are often regulated differently depending on this distinction.

   • b. Patients, physicians, and other stakeholders do not care whether a test is a laboratory developed test or a test kit or what type of entity developed or performed the test. Patients, physicians, and others do and should care about whether the test is reasonably safe and effective for its claimed indications, and they should be aware when a test is being used outside its appropriate indications in ways that may lead the results and their interpretations to be misleading or even inaccurate.

5. The Centers for Medicare & Medicaid Services in their enforcement of CLIA, along with accreditation bodies such as College of American Pathologists and state regulators in CLIA-exempt states, play a major role in overseeing laboratory operations to ensure analytic verification and validity but fail at times to update their regulations or guidance to take account of new laboratory methods and the complexity of modern testing technologies such as genomic testing.

6. The Food and Drug Administration regulates both analytic and clinical validity of tests in their intended uses, although the scope of its jurisdiction has been challenged by some.

   • a. With respect to test kits manufactured by third parties and sold to clinical laboratories, FDA's medical device requirements generally address analytical validity and clinical validity as set forth in the intended use of the test and may sometimes touch on clinical utility depending again on the intended use claimed by the manufacturer. The “indication-blind” nature of genomic testing — the fact that it generates vast amounts of data that could be put to any number of uses beyond the problem that led to testing in the first instance — poses a major challenge for FDA's traditional oversight processes.

   • b. The FDA traditionally has exercised enforcement discretion over LDTs, and its recent attempts to regulate these tests have been spotty and inconsistent. Their actions have been met with opposition by clinical laboratories and other stakeholders and have led to some confusion and uncertainty on the part of stakeholders, including patients.

   • c. Regulation of software is essential to the quality of genome-scale tests and to interpreting the clinical significance of genomic test data and is a major challenge that FDA has only begun to deal with. It has been widely assumed — but is far from clear — that FDA is the appropriate regulator for all phases of the bioinformatics pipeline, some phases of which (e.g., variant interpretation) may be more in the nature of medical practice regulation than product regulation.

   • d. Clinical utility is generally addressed in the practice of medicine and in payer decision making. Although tests are not required to make claims about clinical utility as a condition for entering the market, FDA can regulate such claims if test developers choose to make them.

B. Recommendations

• 1. Relevant regulatory agencies need to develop/maintain expertise in genetic and genomic test development and implementation.

• Cooperative and consensus-based standards should be developed to advance quality.

  • 2. Regulators should have an efficient process for identifying, recognizing, and encouraging the adoption of such standards.

    • i. One of the FDA's current strategic priorities is to establish “collaborative communities” to work toward common objectives in device regulation.¹⁸⁷ Such communities might serve as a nucleus or model for cooperative standards development as well as provide input to inform regulatory actions taken by the agencies.

• 3. CLIA should be modernized and harmonized with broader quality systems and modern terminology.

  • a. Relevant agencies and scientific stakeholders should work to generate more samples for proficiency testing.

  • b. Laboratories need to have robust preanalytical processes and standards, including robust purchasing controls, processes for collecting and processing samples, and corrective and prevention action processes.

  • c. Bioinformatics pipelines need to be reviewed appropriately.

  • d. CMS needs to develop a genetic and genomics testing specialty.

    • i. In the interim, CMS should designate which professional organization standards and guidelines, such as the CAP Molecular Pathology Checklist, are recognized as the most authoritative sources with respect to particular aspects of genomic testing.

• 4. FDA needs to continue to pursue improvements in several arenas.

  • a. Regulation needs to be risk-based, taking into account the characteristics of tests and the clinical context in which they will be used for their intended purpose.

  • b. FDA should work collaboratively with other oversight bodies, including state medical practice regulators and professional organizations, to discourage inappropriate uses of genomic tests for unintended purposes that may be unsubstantiated or unsafe.

  • c. Regulatory processes or standards need to be implemented to assess all phases of software development for use in genomic testing, and to ensure uniform, consistent oversight of this software regardless of whether it is embedded or stand-alone and whatever its provenance. These processes need to ensure that the strengths and weaknesses of software are transparent, both for regulators and user groups such as physicians and laboratorians who rely on software systems. This includes transparency about the limitations of software algorithms, transparent access to non-proprietary databases and useful descriptions of the limitations of proprietary databases on which the software relies, and transparent business practices that foster open discussion about software characteristics and performance.

    • i. Standards for developing and testing algorithms need to be developed.

    • ii. Several groups, including FDA, CDC, AMP, and ACMG have been working to develop standards or other regulatory and non-regulatory approaches to assess the validity of bioinformatics systems. Consensus approaches should be adopted, and software developers should be required to demonstrate their adherence.

    • iii. It would be helpful if scientists, regulatory professionals, and engineers involved in designing and marketing relevant algorithms and software encouraged their professional organizations to develop the expertise necessary to collaborate in proposing and implementing quality approaches.

  • d. To the extent that FDA chooses to rely on standards stated as special controls, the agency must develop strategies to ensure consideration of the interests of all stakeholders including patients in their development. FDA also needs to make sure that appropriate special controls are implemented in a timely fashion.

  • e. FDA needs to develop a pathway that relies more on post-market surveillance for rare diseases and breakthrough tests that meet an unmet clinical need or that represent a clinically significant advance over current technology. These tests should have analytical validity before being used clinically and while clinical validity is being confirmed in the post-market space. Using this approach requires application of sound methods for demonstrating clinical validity in the post-market context and a commitment by regulators to ensure that high quality post-market studies are conducted in a timely fashion.

• 5. The genomics community and other stakeholders need to continue to improve genetic and genomic variant databases and their use in variant interpretation.

• 6. Limitations of test accuracy and of interpretation of clinical validity need to be clearly communicated to clinicians and patients.

• 7. Payment systems should recognize and reward quality. Payers should use coverage with evidence development in areas where evidence is not yet sufficiently strong for optimal decision making.

• 8. CLIA and FDA should work jointly to develop a public, searchable database for use by clinicians, patients, and other stakeholders that displays all information about the regulatory status of genomic tests and devices.