Clinical Epidemiological Quality in Molecular Genetic Research: Title and subTitle BreakThe Need for Methodological Standards

Rapid progress in molecular biologic techniques has revolutionized the field of genetics and pushed the frontier to complex, everyday phenomena such as hypertension,¹ coronary artery disease,^{2 - 6} diabetes mellitus,⁷ and schizophrenia.⁸ Efforts to unlock the genetic code and to determine "the master blueprint"⁹ for individual persons have received added impetus from the Human Genome Project^{10 - 11} and from the potential for profit.¹² The anticipated health benefits of genetic diagnosis and therapy are countered by new social, ethical, and legal concerns regarding discrimination in insurance and work,^{13 - 14} the possibility of a new eugenics,^{15 - 18} and the potential for personal and family distress.¹⁹ The recent report of the National Institutes of Health Task Force on Genetic Testing²⁰ highlights both the challenges involved in developing and evaluating genetic tests and the perils of not doing so appropriately and thoroughly.

The role of genetic factors in complex, multifactorial conditions has not been easy to establish.¹² Examples of problems are the difficulty confirming^{21 - 26} the proposed associations between the DD allele of the angiotensin-converting enzyme and coronary artery disease,^{2 ,27 - 28} the subsequent doubts about initial findings of links between the D4 allele of the dopamine receptor and alcoholism,^{29 - 30} the flawed research designs⁸ and inability to replicate findings³¹ for genetic factors in schizophrenia, and the absence of confirmation, despite substantial media publicity,³² for the so-called novelty-seeking (or bungee-jumping) gene.³³ Many of these problems can be traced at least in part to basic methodological flaws, including lack of consideration of potential confounders such as ethnicity,^{29 - 30} failure to delineate case and control groups adequately,²⁹ and so-called definition drift in which subsequent studies substantially alter the definition for the condition of interest.³⁴

Because evaluations of previous reports on new clinical technologies that do not involve molecular methods have often revealed major methodological flaws,^{35 - 44} we wondered about the quality of clinical epidemiology in molecular studies. Prominent examples of sophisticated new tests that were widely adopted before adequate evaluation and subsequently found to perform disappointingly include the carcinoembryonic antigen test,^{43 ,45} the dexamethasone suppression test for depression,⁴⁴ the indirect immunofluorescence assay for Lyme disease,⁴⁶ and the iodine 125–labeled fibrinogen scans for deep venous thrombosis.⁴⁷ From earlier this century, the Wasserman test for syphilis is a striking example of the deleterious effects that false-positive results can have on both medical status and human life.⁴⁸

Our aim was to examine the quality of published clinical literature on molecular genetic findings using a set of methodological standards, based on established principles of clinical epidemiology, which can be used for designing, reporting, or evaluating molecular genetic research. We assumed that certain principles of clinical epidemiological science are needed to produce reliable and valid clinical information^{35 - 44 ,49 - 56} and that the errors of inaccurate, irreproducible, or inadequately evaluated tests can have serious consequences.^{50 ,57} We decided to evaluate all publications on molecular genetic analysis in 4 prominent clinical journals, but to focus our evaluation on those articles reporting association studies that can be regarded as epidemiological. Although some of these articles would be early studies, collecting data several steps away from actual clinical application, we believed that basic principles of clinical epidemiology should not be ignored merely because an early or preliminary study reports a novel finding.

We performed a manual search of original research studies published during 1995 in 4 prominent general clinical journals: Lancet, New England Journal of Medicine, JAMA, and BMJ. We retrieved all articles in which human genetic material was analyzed at a molecular level, and we did not limit the search to any particular laboratory technique or clinical condition. Furthermore, since genetic is not synonymous with inherited, we included articles dealing with either inherited or somatic mutations. We also included articles evaluating the influence of normal genetic polymorphisms, referring collectively to mutations, polymorphisms, or any other genetic finding under study as genetic abnormalities. The search revealed 83 articles of which 36 were from Lancet,^{25 ,58 - 92} 34 were from New England Journal of Medicine,^{24 ,93 - 125} 8 were from JAMA,^{126 - 133} and 5 were from BMJ.^{134 - 138}

Our first task was to classify the 83 eligible articles into logical, easily interpretable categories to facilitate the application of clinical epidemiological principles to the appropriate articles. Several axes were considered for the classification, including the type of molecular biologic technique (eg, polymerase chain reaction vs Southern blotting), the study design (eg, case-control vs kindred for linkage analysis), and the type of genetic abnormality under study (eg, single or multiple genes, inherited or acquired abnormality). The most cogent attribute on which to base the classification, however, seemed to be the presence of an inferential conclusion about cause and effect.

Articles that reach inferential conclusions, with statements pertaining to cause and risk, are generally structured as analytic contrasts between a study group and a control group, and follow either a cohort or a case-control design. Most of these articles are called association studies and report the association of a genetic abnormality with a clinical condition; some articles, however, report on genetic abnormalities as markers for clinical use in diagnosis, prognosis, or therapy. Articles that are not inferential are descriptive, and generally report on a study group alone, without a control group; descriptive articles include studies that describe the prevalence of a genetic lesion, that describe the distribution of clinical findings for a given genetic lesion, or that describe the distribution of genetic findings for a given clinical condition. For the purposes of this report, we include as descriptive those articles that are structured as linkage studies, in which kindreds are analyzed to localize a putative gene to a region on a chromosome. Although a particular study could have more than 1 focus, a primary classification was always assigned to what seemed to be the main intent of the study, based on claims made in the title and abstract (or text, when necessary).

The methodological standards for the conduct and reporting of studies were based on existing clinical epidemiological principles,^{35 ,139} which have been developed and used in previous evaluations of clinical technologies. The standards for the current report were developed with the intent of focusing on inferential studies; these studies can be regarded as epidemiological in nature and subject to the same kinds of problems that can prejudice the soundness of the conclusions in nongenetic studies of association that make causal inferences. Mindful of the complexity and variety of the articles under examination, we set the criteria for meeting the standards at a lenient level. The methodological standards include reproducibility, objectivity, delineation of cases, adequacy of spectrum in case group, delineation of comparison group, adequacy of comparison group, and quantitative summary. The 7 standards are described below.

Reproducibility. Despite impressive molecular accomplishments, the results of DNA tests regularly appear in electrophoretic "bar codes" that require visual inspection and interpretation, with the attendant challenges of observer variability.¹³⁹ A recent study¹⁴⁰ demonstrated poor reliability for different laboratories in the molecular analysis of genetic material. We therefore required that reproducibility be considered and mentioned. Although requiring the authors to provide a direct comment that the molecular genetic method was either tested in the cited study or previously shown to be reproducible in the laboratory where the work was done, we did not insist on either a formal presentation of the results or a specific method for appraising reproducibility. We accepted either repetition of the test in a pertinent sample of specimens, or confirmation of the test with another procedure. In an example of a satisfactory study,²⁴ each sample found to have the angiotensin-converting enzyme DD genotype was subjected to a second, independent polymerase chain reaction amplification. An unsatisfactory study¹²⁹ of the prognostic significance of apolipoprotein E did not mention any attempt to verify or check the reliability of the DNA analysis.

Objectivity. To ensure objectivity of interpretation, the person doing a measurement should be blinded (or masked) to pertinent characteristics of the patient or hypothesis being tested. When appropriate, the blinding may keep either the patient's clinical assessors from knowing the genetic results, or the genetic analysts from knowing the patients' clinical status. To satisfy this standard, an appropriate method of blinding should have been used and reported in each study. In a successful example,¹⁰⁰ genetic susceptibility to asthma was studied with each gel being read by observers who were unaware of the corresponding clinical status. A study on the association of the angiotensinogen gene and coronary heart disease, on the other hand, contained no mention of any blinding.²⁵

Delineation of Cases. The group chosen as cases of the condition under investigation should be described in enough detail so that other investigators could assemble a similar group in similar circumstances. This description should include the criteria for diagnosing the selected condition and a simple indication of the sampling frame (pertinent demographic details and source for the patients). For leniency in scoring, we did not demand that studies report the calendar time or time interval of sampling, or the sampling method (eg, random, consecutive, or other), and we accepted an easily obtainable reference if it seemed to offer an adequate description of the case group. An excellent article, however, would give an outline of the condition, place, and time of the cited study. Specific diagnostic criteria are seldom needed for histological or gross morphologic abnormalities, but should be cited if a particular diagnosis depends on a specifically demarcated value (such as weight in obesity) or on a combination of components (as in myotonic dystrophy). Again, for leniency, we did not require a statement of the rules for combining components. For example, a study of type 2 diabetes mellitus could cite the diagnostic criteria of the World Health Organization. An example passing the standard was a study of coagulation factor V and myocardial infarction, stroke, and venous thrombosis.¹⁰⁸ The investigators provided clear criteria for the diagnosis of each case condition and described the origin of the cohort. An example failing the standard was a study of the β₃-adrenergic receptor gene and type 2 diabetes mellitus, in which no criteria were stated for the diagnosis of diabetes (eg, published standards vs type of therapy used).¹¹³

Adequacy of Spectrum in Case Group. The spectrum of cases should be suitable to support the claim of the article. For example, results from a narrow portion of a condition's clinical spectrum (eg, cases of only mild severity, or with only metastatic or debilitating disease) do not usually apply to the full spectrum. Although citation of results for pertinent subgroups would be desirable, we were willing to accept undifferentiated results for a broad spectrum of patients. If the investigated condition is rare enough that a broad spectrum of patients is difficult to obtain, even a single case may sometimes suffice to demonstrate the spectrum. For our analyses, however, we excluded articles with fewer than 10 case subjects. In a successful example, we gave credit to an article²⁵ that studied genetic associations with heart disease and included subjects with a wide range of disease severity. In an example of a study failing this standard, an article⁵⁸ on spina bifida did not indicate the spectrum of examined disease despite a general conclusion about neural tube defects.

Delineation of Comparison Group. The comparison group should also be described in enough detail to allow its recapitulation by other investigators. The description should include the main clinical criteria for inclusion in the comparison group and the sampling frame from which the group was chosen. For leniency, however, we did not insist that the time frame or sampling method be cited. A single description of the sampling frame could suffice if a single group was chosen and then divided into case and comparison groups, according to the presence of a genetic abnormality; but if the single description is inadequate, neither group will satisfy delineation of cases and delineation of comparison group standards. As with the delineation of the case group, we accepted an article that offered a satisfactory account of the comparison group, although a brief summary description should preferably be reported for the comparison group even if a reference is provided. A passing article¹³⁶ provided a detailed description of the source and diagnostic criteria for the comparison groups in a study of apolipoprotein E and Alzheimer disease. An example failing this standard was an article⁶⁴ on IgA nephropathy, which listed no source or criteria for the healthy controls.

Adequacy of Comparison Group. The comparison group should include pertinent challenging conditions, such as a related or similar ailment that might plausibly share an etiology, or that should be suitably differentiated from the main condition. For leniency in scoring, this criterion could be satisfied by members from 1 or 2 challenging groups rather than from a full spectrum of challenges. In studies claiming to identify risk factors, we accepted healthy or community controls, but appropriate challenge groups were needed for studies of diagnosis or claims about etiology or pathogenesis. The standard was satisfied in a study⁹⁴ of Gilbert syndrome, which included both healthy controls and patients with Crigler-Najjar syndrome type 2. The standard was not satisfied in a study of idiopathic short stature, which contained healthy controls but no other forms of short stature.⁹⁷

Quantitative Summary of Results. The main claim made in comparisons of 2 or more groups should receive a quantitative summary of the results. The summary should indicate both the magnitude of the difference and a statistical expression of numerical stability (eg, a confidence interval or P value) for the possible role of chance in the results. If the compared numbers were provided, we did not insist on a specific index of contrast, such as a risk ratio. An example of successful citation contained odds ratios and 95% confidence intervals for a study of the angiotensinogen gene and risk of coronary artery disease.²⁵ An unsatisfactory article reporting on the frequency of a gene potentially involved in the cause of Gilbert syndrome gave no statistical summary for the comparison of allele frequency in patients vs healthy controls.⁹⁴

To avoid excessively complicated evaluation rules, we applied the 7 standards only to articles that had inferential conclusions and that studied at least 10 patients. We excluded articles reporting on fewer than 10 patients because standards such as the adequacy of case spectrum and quantitative summary of results are more difficult to apply to individual case reports and small case series. In addition, although some standards appeared to be potentially applicable to some of the articles classified as descriptive, we do not report the results for those articles here. Finally, some studies (8/83) did not include a control group but made what seemed to be possibly overstated claims of association or causation; although these articles probably contained defective inferences, we classified them as descriptive for this report and did not apply the standards to them.

To reduce and then measure interobserver variability in the application of the standards, 20 of the 83 articles were randomly selected and then independently scored for the methodological standards by each of the 3 authors. After disagreements were resolved in conference, the standards were revised and clarified. When 10 more articles were then randomly selected from the remaining 63 and independently evaluated by 2 of the authors (S.T.B., J.C.) the overall percentage of agreement was 92%, with κ=0.83, reflecting excellent agreement.¹⁴¹

Table 1 shows the number of articles in each category of classification; Table 2 lists the clinical conditions examined in the 83 articles. The largest proportion of articles (43/83 [52%]) had inferential conclusions, with most being association studies aimed at determining the cause or risk for a particular condition. Six of the 43 articles were also inferential but aimed at clinical decisions, such as diagnosis, prognosis, or therapy. Of the 43 inferential articles, 40 studied 10 or more patients. The remaining articles (40/83 [48%]) were primarily descriptive but included some not classified as association studies that nevertheless made inferential claims (ie, 4 articles based on linkage analysis and 8 articles that made a causal claim but did not include a comparison group).

Table 3 shows the results for individual methodological standards applied to the 40 inferential articles that studied 10 or more patients, and Table 4 shows the overall results for the 7 standards in these articles. Most articles did not pass the standards for reproducibility (n=25, 62.5%) or objectivity (n=27, 67.5%), and substantial proportions also did not pass the standards for delineation of case group (n=9, 22.5%) or comparison group (n=12, 30%). Most articles, however, did pass the standards for adequacy of case group (n=35, 87.5%) and comparison group (n=35, 87.5%), and for quantitative summary of results (n=36, 90%). Of the 40 articles analyzed, 5 (12.5%) passed all 7 standards and 10 (25%) failed only 1 standard; the remaining 25 (62.5%), however, failed 2 or more standards, with 9 (22.5%) of these failing 4 or 5 standards.

The results indicate that a substantial proportion of articles involving molecular genetic research may contain flaws in basic elements of research design and reporting. The importance of this finding is emphasized by the recent report²⁰ by the National Institutes of Health Task Force on Genetic Testing, which urged a more careful and systematic approach to the development of genetic tests. The methodological standards used in the current study are based on well-accepted fundamental scientific principles, applicable in any type of clinical investigation involving human subjects and diseases, and have the strong face validity of scientific "common sense." For example, anyone using a test would want to know its reproducibility in yielding the same result if applied twice to the same specimen, and objective interpretations are needed to avoid the potential for bias whenever data receive human interpretations, whether for clinical diagnoses or for reading a gel.

Giving an adequate description of the groups under study and justifying the clinical spectra of the chosen conditions are perhaps the most fundamental steps in any act of clinical research. For example, differences in composition of groups may be crucial if a gene abnormality occurs more frequently in one ethnic group than another; and inadequate spectra may undermine conclusions about the relationship of a gene and a particular clinical condition. Similarly, certain inferences rest on demonstrations that an abnormality often present in one group is often absent in others, and that the difference between the groups is statistically stable.

The neglect of fundamental methodological principles has important clinical consequences and can lead to contradictions and even retractions in published work.^{142 - 143} An example of conflicting results was found even in the limited number of studies we examined: 2 articles^{72 ,94} published in different journals reached different conclusions about the genetic basis of Gilbert syndrome. In addition, with the extraordinary pace of development in genetic research and the pressure to publish results quickly, quality may be sacrificed in favor of rapid dissemination. For example, some of the articles we evaluated can be described as preliminary reports, and lacked methodological rigor. Even early articles that hasten to report a new finding, however, can give suitable attention to the quality and reliability of the research. More careful scientific methods can spare other investigators the wasted effort of attempting to replicate spurious findings.

We were deliberately lenient in the requirements for meeting the standards. For example, we did not set specifications for the particular manner in which reproducibility should be checked or quantitative summaries reported. One reason for the leniency was the extraordinary complexity of many of the articles, complexity that makes reliable classification and grading a challenging task. Nevertheless, despite the relatively "low height of the bar," many articles failed to meet the standards. A count of the number of standards failed, however, does not necessarily determine either the overall quality of the work or the validity of its results because 1 major flaw could potentially undermine all the conclusions, or, conversely, a series of minor flaws might not affect the conclusions.

Since critiquing decisions of journal editors is not the aim of this study, we did not try to correlate the quality of the articles with the sites of publication. The 4 journals used for this study were chosen because they are the most widely read, influential general clinical journals in the English language, and are particularly likely to publish work that is carefully reviewed and clinically pertinent. We do not know what might have been found for articles published in other journals, which may place a greater emphasis on the publication of molecular genetic studies than the journals chosen for this report.

We also did not try to determine whether the cited flaws were produced by research design or because of authors' omissions or editorial excisions. Although flaws such as an inadequate spectrum of cases or controls seem due to research design, the source is more difficult to assign for omitted details such as reproducibility or objectivity of tests, or delineation of groups. For example, journals that vigorously attempt to limit the length of articles may delete material that other journals would publish, and this may be 1 explanation for the high failure rate for the standards referring to reproducibility and objectivity. Regardless of where or how each problem arises, however, its unsatisfactory management will prevent other investigators from replicating or suitably interpreting the findings.

We conclude that the 7 standards listed here are often not satisfied in published reports of molecular genetic findings. Although not everyone may agree with each of our decisions, we believe the standards themselves have strong face validity and are important to ensuring the reliability of future genetic research. The standards can be used by investigators in planning and reporting genetic research, and by reviewers, editors, and readers who evaluate the reports. Unless applied with analogous principles of clinical epidemiological science, the molecular science that has produced magnificent genetic advances may also lead to an epidemic of harm from spurious associations, unnecessary fears, and unfulfilled promises of benefit.