How to Use an Article About Genetic Association: Title and subTitle BreakB: Are the Results of the Study Valid?

A 55-year-old man with a family history of dementia is inquiring about genetic testing for Alzheimer disease, in particular testing for APOE. Armed with knowledge of the basic genetic concepts outlined in the introductory article of this 3-part series,¹ you return to your electronic medical reference discussion. Of the studies they cite, you focus on the largest study (n = 6852), with longest follow-up (up to 9 years), representing a general, community-based population aged 55 years and older,² and using the stronger of the candidate gene study designs (cohort rather than case-control). The authors report a relative risk of 2.1 (95% confidence interval, 1.7-2.7) for dementia in APOE e4 (e for epsilon) heterozygotes and 7.8 (95% confidence interval, 5.1-11.9) for APOE e4 homozygotes compared with e3/e3 individuals.

Initial epidemiologic studies addressing a novel association tend to overestimate the magnitude of association,³ possibly as a result of publication bias (studies addressing previously unreported associations are published only if they show significant results), and this phenomenon is even more frequent in genetic association studies.⁴ This highlights the importance of examining the validity of such studies,^{5 - 9} the focus of this article.

We adopt the same framework as previous users' guides:

This article deals with the first of these questions; the final article in the series will address the latter 2. Box 1 provides a summary of our guides and Box 2 provides a glossary of genetic terms.

Allele

One of several variants of a gene, usually referring to a specific site within the gene

Genetic heterogeneity

A situation in which a particular phenotype may result from more than one genetic variant

Genetic marker

A specific genetic variant known to be associated with a recognizable trait

Genome-wide association (GWA) study

A study that evaluates association of genetic variation with outcomes or traits of interest by using 100 000 to 1 000 000 or more markers across the genome

Genotype

The genetic constitution of an individual, either overall or at a specific gene

Haplotype

Alleles that tend to occur together on the same chromosome due to single-nucleotide polymorphisms (SNPs) being in proximity and therefore inherited together

Hardy-Weinberg equilibrium (HWE)

A situation in which a defined population displays constant genotype frequencies from generation to generation, and those genotype frequencies can be calculated from the allele frequencies based on the HWE formula

Heterozygous

An individual is heterozygous at a gene location if (s)he has 2 different alleles (one on the maternal chromosome, one on the paternal) at that location

Homozygous

An individual is homozygous at a gene location if (s)he has 2 identical alleles at that location

Phenotype

The observable characteristics of a cell or organism, usually being the result of the product coded by a gene (genotype)

Polymorphism

The existence of 2 or more variants of a gene, occurring in a population

Population stratification

Describes the situation in which a population may be composed of multiple subgroups of different ethnicity; case and control group differences in the mix can confound the comparison and lead to spurious genetic associations

Similar to traditional prognostic or etiologic studies, genetic association may use cohort or case-control designs.^{10 - 13} Cohort studies sample a group of people (eg, older individuals) who vary in their genetic characteristics (eg, APOE e2/e2, e2/e3, e2/e4) and follow them forward in time to determine who has the outcome of interest (eg, Alzheimer dementia). In case-control studies, investigators choose affected individuals (case patients, eg, those with Alzheimer dementia) and a sample of unaffected individuals from the same underlying population and determine the genetic characteristics of the individuals in each of the 2 groups.

Case-control studies in traditional epidemiology are subject to a number of potential biases, many of which are less of a concern in genetic studies. In contrast to most environmental exposures, the genetic “exposure” does not vary with age or calendar year, there is no recall bias and no choice of exposure made by the participant, and the exposure is not influenced by disease (or treatment). The case-control design also facilitates large sample sizes and therefore power, which is particularly important for detecting potentially small genetic effects. Our discussion will focus on validity issues of particular relevance for genetic studies.

In the absence of a standardized definition of the disease or trait of interest, investigators may run association analyses with varying definitions and report only the most significant findings, resulting in spurious associations.¹⁴ On the other hand, what appears at first glance to be a single disease entity may in fact consist of many genetically separate but clinically similar diseases, a situation called genetic heterogeneity. In this situation, including diseases with different genetic etiologies may dilute or obscure a true association.

Even if the disease definition is well standardized, it is important to ask whether the disease phenotype has been appropriately measured during the study. Misclassification (here, categorizing people as having dementia when they do not or vice versa) may affect the strength of the genetic association. If the misclassification is a result of random error, the association will be diluted. If misclassification errors are influenced by previous knowledge of the genotype of each individual, eg, if APOE genotype influences the diagnosis of dementia, then the genetic effect may be overestimated. Thus, individuals conducting the phenotyping should be blind to the genotyping result (and vice versa).

In our clinical scenario, because different etiologies of late dementia are likely to have different genetic determinants, researchers who do not separate individuals with Alzheimer disease from those with vascular dementia (common) and Lewy body dementia (rare) may fail to establish genetic links. Slooter et al² separate Alzheimer from vascular dementia and use widely accepted definitions. Moreover, the investigators made meticulous efforts to minimize misclassification caused by measurement error by using a panel of several tests and by blinding appropriately.

As we have pointed out, some common variables that, in traditional epidemiologic studies, can cause bias as a result of an association with the condition of interest and misdistribution in exposed and unexposed populations (we call such variables confounders) are less likely to introduce bias in genetic epidemiology. Genetic studies, however, may yield misleading results if their disease and nondisease populations include a different ethnic/racial mix; this particular form of confounding is referred to as population stratification. The problem occurs if the likelihood of developing the condition of interest varies with ancestry. If ancestry groups also happen to differ in allele frequency of genetic polymorphisms unrelated to the condition of interest, the result will be spurious associations.

Most association studies of unrelated individuals try to avoid this problem by using populations that are homogeneous in terms of ancestry. Self-reporting will usually suffice at least for populations of European ancestry,^{15 - 17} although there are rare examples, such as genes that regulate susceptibility to lactose intolerance, in which there is marked variation. To address such possibilities, a number of techniques have been developed to check for differences in the potential mix of ancestries and, if differences are found, to make corrections; these corrections use self-reported ethnicity, family-based controls, or statistical techniques termed genomic control to test for patterns in unlinked markers.^{18 - 19} For example, a spurious association between the CYP3A4-V polymorphism and prostate cancer in blacks disappeared when results were adjusted for additional genetic markers associated with ancestry in the population studied.²⁰

Ancestry is not the only potential confounder that may compromise the validity of a genetic association study. For example, 2 genome-wide association (GWA) studies showed an association between type 2 diabetes and a single-nucleotide polymorphism (SNP) in the FTO (fat mass and obesity associated) gene.^{21 - 22} These studies selected diabetic patients and controls irrespective of their body mass index (BMI); another study that matched diabetic patients and controls on BMI showed no association. Thus, although the study accurately identified the association between diabetes and the particular SNP, the causal association is probably between the candidate allele and BMI regulation/obesity, not type 2 diabetes.

Readers should consider whether diseased and nondiseased groups were similar with respect to other important characteristics that are likely to be genetically determined and associated with the outcome of interest. Alternatively, they may determine whether the investigators adjusted for such characteristics.

Returning to the clinical scenario, one might imagine that ancestry and alcoholism are characteristics that are both genetically influenced and that would be associated with Alzheimer dementia. Slooter et al² recruited their entire cohort from among the white population of the Netherlands, which is likely a homogeneous group with little genetic variability; this is verified by results from a recent GWA study from the same cohort.²³ They did not, however, consider alcohol history.

Genotyping error is a threat to the validity of genetic association studies. Genotyping may go wrong if there is a problem with the biological material (the samples) or with the application of the molecular technique that is used to call alleles.

The biological material that provides the source for genotyping may differ between diseased and nondiseased participants in ways that lead to inaccuracies in genotyping. For example, in a GWA study for type 2 diabetes, blood stored in 1958 provided the basis for genotyping nondiseased individuals, whereas blood drawn more recently was used for genotyping diseased individuals. The older blood resulted in genotyping errors²⁴ that led to some false-positive SNP associations.

Genotyping error may occur even when disease and nondisease samples are drawn and stored in identical ways. Although laboratory-based methods and DNA information may have the cachet of being absolute, these data are subject to error in the same way as traditional epidemiologic information. Genotyping error rates vary widely, from less than 1% up to 30%,²⁵ and rates of up to a few percent are not uncommon in even the best studies.^{26 - 28} Genome-wide association studies should aim to minimize genotyping error rates. Another useful piece of information is the “call rate” of genotyping, ie, the proportion of samples in which the genotyping provides an unambiguous reading. If this proportion is not high, then information is lost. In many studies, investigators decide to avoid analyzing SNPs in which the call rate is less than 90% or even less than 95%. Even high call rates can, however, fail to prevent bias if specific genotypes have lower call rates than others, eg, heterozygotes are more likely to get ambiguous readings or false readings than homozygotes.

These sources of error are most easily detected by the researchers using the raw data; it is impossible for a reader to identify them from the limited data usually reported in an article. A reader may, however, seek a description of how samples were handled, what genotyping method was used, whether any quality checks were implemented, whether any rules were established to say when the genotyping results would be considered valid, and the extent of missing data.

Returning to our clinical scenario, Slooter et al² refer to an earlier article from their team for genotyping details²⁹ ; in this article, they state that genotyping was performed independently and in triplicate and without knowledge of the outcome status. They also state that their original cohort had 7983 persons, and they had to exclude 14% of the participants (n = 1131) because APOE genotype could not be determined. There is no mention about whether this loss may have been related to underlying genotype or to Alzheimer disease, but at face value, it seems unlikely. Although the method was not specified, given the prospective cohort design, one may assume that samples were stored in similar conditions regardless of the subsequent development of dementia.

Failure to observe Hardy-Weinberg equilibrium (HWE) is one way of detecting possible genotyping error, although it is nonspecific and may be insensitive.^{30 - 32} Investigators typically conduct statistical tests to check whether the observed genotype frequencies are consistent with HWE; P<.05 is the usual threshold for declaring Hardy-Weinberg “disequilibrium.”³³ However, with simultaneous testing of a large number of possible associations, as in GWA studies, it is expected that 5% of SNPs will violate HWE simply because of multiple testing. In this setting, investigators may use more stringent P-value thresholds. Empirical studies suggest that disequilibrium is common and many articles do not explicitly acknowledge this^{34 - 35} ; as discussed in the first article, there are many reasons for disequilibrium (eg, inbreeding) aside from bias or error.

Therefore, readers should look for evidence that the investigators have tested for HWE and raise their level of skepticism about the results if they have not. Given that erroneous reports of HWE occur, they may even check for HWE themselves by using a simple freely available statistical program (Box 3). For a cohort study, HWE should be tested in the whole study population, whereas for a case-control study, it should be tested in the controls because these are supposedly representative of the general population.

Readers can check whether the data at a biallelic single-nucleotide polymorphism (SNP) are consistent with Hardy-Weinberg equilibrium (HWE) by inserting the numbers in each genotype group into an online program.³⁶ For example, an article may report that among 100 controls, there are 80 homozygote wild types, 12 heterozygotes, and 2 homozygous variants. The program calculates the expected distribution among the 3 genotype groups, the χ² value, and the corresponding P value. There are limitations to the hypothesis testing, whether done by the authors or the online program. Most HWE tests are weak because most sample sizes are small, and thus the likelihood of a false negative because of inadequate power is high. On the other hand, with very large sample sizes, the tests can detect very small deviations from HWE that are of no importance. In the setting of genome-wide association studies, a large number of SNPs are expected to have nominally significant deviations from HWE. For example, with 500 000 tested SNPs, 25 000 of them may have P<.05 on HWE testing by chance alone. Therefore, in GWA studies far more strict thresholds are appropriate to identify worrisome HWE deviation.

In our scenario, Slooter et al² found that their study population did observe HWE (P = .45 in a well-powered study of n = 6852). Given that this is a 3-allele system, we are not able to use the online program to check HWE.

One of the main reasons for false-positive results is inadequate attention to the problem of multiple comparisons. The scenario of an experiment testing 100 SNPs for association with a disease outcome in which no real association exists illustrates the magnitude of the problem. If the threshold P value of .05 is left unchanged, then the chance of finding an apparent but spurious positive association in this scenario can be calculated as ((1−(1−.05)¹⁰⁰)×100), or 99.4%. The easiest method to correct for this problem of multiple comparisons is the Bonferroni method, in which the threshold P value is divided by the number of tests. In this example, the P value would be set at .05/100, or .0005. This is, however, probably overly conservative and stringent, and authors have suggested many other methods^{17 ,37 - 40} (Box 4). This potential for false-positive results also makes genetic association studies particularly susceptible to publication bias, in which initially strongly positive results find their way into publication more easily, whereas studies with negative results take longer to get published.⁴⁴ Such bias is not corrected by simply accounting for multiple comparisons.

The Bonferroni correction is overly conservative and stringent, and there have been many suggestions for other methods. Two of the more popular ones include the following.

False-discovery rate calculations estimate the proportion of associations that are seemingly “discovered” (pass some required threshold of evidence) but are nevertheless expected to be false positives. The Benjamini-Hochberg method is used when loci (or single-nucleotide polymorphisms) are independent,⁴¹ whereas the Benjamini-Lui method is applied when there is correlation or linkage disequilibrium between loci.⁴² Both methods work on ranking the P values of the associations within one study and adjusting that P value by its position in the ranking list.

The false-report probability rate similarly states how likely an association is to be false if it emerges with a given level of statistical significance, given the power of the study and the perceived prior odds of an association being true.¹⁷ The developers of this method have constructed a user-friendly spreadsheet to allow easy calculations.⁴³

In GWA studies, in which more than 500 000 SNPs are tested simultaneously, the multiple comparison problem takes on a magnitude never imagined in traditional epidemiology. To avoid false-positive results, a consensus seems to be forming that for such large-scale studies, a P value in the range of 5×10⁻⁸ (as opposed to the usual 5×10⁻²) should be considered the threshold for claiming what is called “genome-wide” significance.^{45 - 46} Increasingly, full results from GWA studies are publicly available, providing further insurance against publication bias.

In our scenario, Slooter et al² have not adjusted their results for multiple comparisons. They test only the APOE polymorphism (although they address 3 outcomes, myocardial infarction, stroke, and Alzheimer disease). They reasonably consider theirs a hypothesis-testing rather than hypothesis-generating study.

Any users' guide—whether for diagnosis, therapy, prognosis, or harm—could include a validity criterion demanding replication. Although we have not included this criterion in considering other sorts of individual studies, the multiple comparison problem and the forces that lead to differential publication of positive results suggest that, here, it is particularly important. Until results are replicated in similar populations, one should interpret them with caution.^{8 ,47}

Most of the genetic associations between SNPs and complex diseases are small (much smaller than the odds ratios >2.0 observed for apoE e2/e3/e4),⁴⁸ and therefore even sizeable studies may fail to detect underlying associations.⁴⁹ Therefore, given that most individual studies are not large enough to detect these small effect sizes, typically, GWA studies pick the SNPs that have the lowest P values and test them in additional replication samples (either other GWA studies or focused studies targeting only the specific SNPs) to increase sample size and power until the cumulative results pass genome-wide significance or similar thresholds. Even more teams may then continue to try to replicate these associations, and all these data become essential in judging the credibility of these associations.

Therefore, just as we suggest that clinicians interested in issues of therapy, diagnosis, prognosis, and harm first seek a systematic review, so also do we suggest that they do the same for genetic associations.^{50 - 51} The Human Genome Epidemiology Network (HuGE Net) group is emerging as the Cochrane equivalent for genetic association studies. The HuGE Net Web site lists many of the meta-analyses performed to date^{52 - 53} and also hosts the HuGE Navigator, where one can determine what single studies, GWA studies, meta-analyses, and synopses are available.^{54 - 55} Another possible aid in searching for previous genetic association studies is the genetic association database maintained by the National Institutes of Health.⁵⁶

A MEDLINE search using apoE and dementia as search terms and restricted to English and meta-analysis, or a search on the HuGE Navigator, leads to 2 meta-analyses in the general population^{57 - 58} and a Web site collating all the Alzheimer genetic association studies as an all-encompassing synopsis.⁵⁹ The meta-analyses demonstrate that results for the APOE e2/e3/e4 polymorphism are largely consistent across studies. This is probably the exception even among well-replicated genetic associations, and it reflects the fact that the apoE-dementia association is much stronger than almost any other associations recorded to date.

Slooter et al² meet the crucial validity criteria:

Given that we are satisfied with the validity of the study, we continue our critical appraisal. In the next article, we will discuss how to interpret results of genetic association studies and how to apply this information in the context of patient care.