Claims of Sex Differences:  An Empirical Assessment in Genetic Associations FREE

Sex is a factor that has been invoked extensively in the past as a modulator of effects in clinical research. However, empirical data from randomized trials suggest that many claimed subgroup differences based on sex have been spurious and led to serious misconceptions.¹ For example, aspirin was believed to be ineffective in secondary prevention of stroke in women for more than 10 years based on an underpowered subgroup analysis.²

In the human genome era, for many common diseases, published research has often considered that some common gene variants may have different effects in men vs women. Many diseases or traits with strong genetic backgrounds have different prevalence in the 2 sexes. For example, autoimmune diseases, endocrinopathies, and longevity are more common in women, while coronary artery disease, ischemic stroke, and high cholesterol levels are more common in men.³ These observations do not necessarily mean that a specific gene variant should also have a different effect in men vs women. For most phenotypes, many common gene variants are likely to be responsible for determining susceptibility to disease.⁴ Among autosomal variants, only some of them, if any, may interact with sex. However, given that sex information is always readily available in such studies, it is easy to test whether it influences genetic effects. Eventually, a large number of claims are made for sex differences. However, are these claims justified and valid?

Herein, we describe empirically a large sample of prominently claimed sex differences for genetic effects. We evaluated whether these claims were methodologically robust or were made based on selected and/or suboptimal analyses and with insufficient or spurious documentation. We also examined whether claims that seemingly had optimal internal validity had been corroborated by any additional studies.

We aimed to assemble a sample of studies that claimed sex subgroup differences in gene-disease associations and in which the claim was so prominent that it was mentioned even in the title of the article. Sex subgroup differences are a very common theme in the epidemiological literature. Assembling all of them or even a systematic fraction thereof would be prohibitive. Any electronic search would yield only a modest sample of the thousands of articles evaluating sex subgroup differences, since these are often listed as secondary or tertiary results. Conversely, by focusing on the title, we would favor the selection of most prominent perceived subgroup differences between sexes. These studies are the ones in which the authors (and apparently also the peer reviewers and editors) are most confident of the strength of the observed sex subgroup differences.

We searched PubMed through July 6, 2007. PubMed is considered highly representative and inclusive of genetic epidemiology studies.⁵ We used a search strategy that would have high specificity for assembling a convenient sample of eligible studies: polymorphism* [ti] AND (gender [ti] OR sex [ti]). We perused titles and abstracts and, if in doubt, also the full text, for eligibility. We considered eligible all retrieved association studies that claimed different genetic effects across sexes of 1 or more gene variants for any human disease or phenotype. We considered both biallelic and multiallelic markers (including haplotypes) and both binary and continuous phenotypes and traits.

We excluded non–English-language studies; studies evaluating only 1 sex; studies in which sex was treated only as an independent predictor of disease; studies that did not address any association of the investigated genetic variant with a disease or trait; studies not involving humans; and studies in which the authors did not claim any sex difference. Eligible studies were included regardless of the extent of the quantitative information that they provided to support their claims. Eligibility assessment was performed by 2 independent evaluators (N.A.P. and A.T.). Discrepancies were resolved by consensus.

Two evaluators independently extracted data, with a third evaluator arbitrating their discrepancies. We extracted the following data from each eligible study: first author, journal of publication, year, total sample size, percentage of women, gene(s) and variant(s) for which sex differences were claimed, and disease/phenotype(s) thereof.

For each pair of gene variant and phenotype for which the investigators claimed a sex difference, we recorded the exact phrasing of the claim and any allusion that the difference was tested based on a priori plans or as part of post hoc analyses; the type of genetic variant (eg, biallelic, multiallelic single-locus, haplotypes of many variants); and the type of phenotype (ie, binary outcome or continuous trait). Especially for biallelic variants, we recorded if a specific genetic contrast (recessive, dominant, additive, allele-based, model-free, or other/unclear) was used. We also recorded (per sex) the presented effect sizes and measures of uncertainty thereof or raw data that could be used to verify the presence of each claimed sex difference.

For each claim of sex difference, we recorded whether it was based on an analysis of the entire study sample or a subset thereof. If the latter, we recorded the definition of the subset and whether any a priori justification was provided for the subset selection. We specified whether subsets were defined based on genetic information for the gene variant of interest (eg, comparison of AA vs aa homozygotes without consideration of Aa heterozygotes; comparison of haplotypes 1 vs 4 without consideration of haplotypes 2 and 3), genetic information for some other gene variant (eg, selection of individuals who have some specific genotype for another gene marker), other patient characteristics (eg, age, ethnic or racial descent), other considerations, or combinations of the above.

For each claim of sex difference, we also recorded whether it was appropriately documented, insufficiently documented, or spurious. For appropriately documented claims, 3 criteria were required. First, the article had to address a genetic effect that was based on the same genetic contrast in both sexes. Second, it did not compare different subsets in the 2 sexes (eg, old men vs young women). Third, it needed to either report a nominally statistically significant (defined as a P value threshold of .05) test that examined sex-gene interaction or the interaction had to be very obvious because the presented confidence intervals of effects for each sex were not overlapping. Insufficiently documented claims were those in which only the first 2 criteria were fulfilled. Spurious claims failed either one or both of the first 2 criteria.

Insufficient documentation does not mean that the claim is necessarily inappropriate and wrong. An insufficiently documented claim may be correct (a significant interaction may exist after all) or wrong (a significant interaction may not exist), but this is not clear from the analyses performed or the way the claim is made in the article.

For each sex claim that was not spurious and where suitable information was available, we tried to perform calculations to test whether there was indeed a nominally statistically significant sex subgroup difference in the effect sizes (sex-gene interaction).

Whenever a dominant, recessive, or allele-based model was implied, we used for each sex the natural logarithm of the odds ratio (binary outcomes) or the absolute difference (continuous outcomes) and their variances. Whenever effect sizes were not reported for both sexes, we tried to estimate them from the presented information on published 2 × 2 tables and means per sex. Whenever variances were not reported, we tried to estimate them from the presented information on confidence intervals, standard deviation, or standard error of the mean and number of observations. We then calculated the z score as a ratio where the nominator is the difference of normalized effects and the denominator is the square root of the variance of the difference. Whenever an additive model was implied, we fitted trend models for each sex and used the resulting coefficients and respective standard errors to calculate the z score. Finally, whenever a model-free approach was implied, we used an analysis of variance with a sex-genotype interaction term.

All these analyses were performed using the data that pertain to the same subjects for which the sex subgroup difference was originally claimed by the authors. Thus, if the sex claim had been made for the entire study population, we examined sex-gene interaction in the entire population; if the claim had been made for a subset, we examined sex-gene interaction in that same subset. When both unadjusted and adjusted estimates of effect were available in full detail, we used the unadjusted estimates, except when both unadjusted and adjusted estimates were reported as different claims. In this case, both were used if possible.

Even if a sex-gene interaction has been presented with optimal statistical and analytical support, it is not certain that it truly exists. Genetic effects are subject to an extensive multiplicity of testing and are also susceptible to diverse errors and biases.⁶ Therefore, replication by additional independent studies is considered essential for reinforcing the credibility of genetic effects.⁷ We examined whether proposed sex-gene interactions in our sample had indeed been evaluated by additional studies and, if so, whether the results of these studies agreed with the proposed interactions.

We focused on claims of sex differences that met all of the following criteria: their analyses were stated to have relied on a priori considerations, raw data were provided, sex-gene interaction was nominally statistically significant at the P = .05 level on our reanalysis of the data, and either the whole study sample had been analyzed or a subset had been selected based on a priori considerations. These claims were the ones that apparently had the best possible internal validity.

We perused in detail each of the articles reporting such claims and recorded whether they had cited any previous studies investigating the same sex-gene interaction. Additionally, we searched the ISI Web of Knowledge for articles that cited the articles meeting the criteria mentioned above. We then retrieved these cited and citing studies and examined whether they had evaluated the same sex-gene interaction and, if so, what was found.

We acknowledge that our search strategy may not have been 100% sensitive to find all studies that tested these interactions. However, there is no documented reliable strategy to retrieve all such articles, and interactions are probably often buried in text or not reported at all, especially if not nominally significant (P>.05) and “noteworthy.” However, prior supporting studies are very likely to have been cited in these articles when the interactions were the main theme and even were part of the article's title. Similarly, subsequent studies that did find the same interaction would be likely to cite an article in which the main theme had been that same interaction. Overall, our strategy probably favors the retrieval of studies that agreed rather than disagreed with the proposed interactions.

Analyses were conducted in Intercooled Stata, version 8.2 (Stata Corp, College Station, Texas). P values are 2-tailed.

The electronic search yielded 215 citations; 138 were excluded on close scrutiny: 5 were non–English-language papers, 34 studies evaluated only 1 sex, 11 used sex as an independent predictor, 28 were not association studies, 54 were nonhuman studies, 4 had no claim of sex difference, and we could not find the full articles for 2 citations. Seventy-seven articles^{8- 5859- 84} were eligible and contained a total of 432 distinct claims of sex subgroup differences (median = 4 [interquartile range {IQR}, 2-7] claims per article).

These studies were published between 1994 and 2007 in 63 different peer-reviewed journals with a median impact factor of 3.868 (IQR, 2.826-5.699). The median sample size of these studies was 560 (IQR, 274-921) and the median proportion of women was 49% (IQR, 44%-53%). Sixty-three different genes were implicated across the 432 claims. The claims pertained to a wide variety of diseases and phenotypes, the most common being hepatitis C virus infection (n = 32 claims), high-density lipoprotein cholesterol (n = 26), lung cancer (n = 21), type 2 diabetes mellitus (n = 19), multiple sclerosis (n = 18), hypertension (n = 14), low-density lipoprotein cholesterol (n = 13), Paget disease of bone (n = 12), and diabetic nephropathy in type 1 diabetes (n = 10) (Table 1).

Of the 432 claims, 286 (66.2%) were reported as being based on a priori stated comparison of the sexes and 68 (15.7%) were acknowledged to be post hoc analyses; in the other 78 (18.1%), the analysis plan was unclear.

The variant of interest was of a biallelic locus in 328 claims, a multiallelic single locus in 44, multilocus haplotypes in 45, and multiple polymorphisms without haplotype construction in 15. The phenotype was binary in 212 claims, continuous in 218, and categorical in another 2 (cross-tabulations and other supplementary data are available at http://www.dhe.med.uoi.gr/sup_mat.php).

Among the biallelic variants, a specific genetic contrast was used in 227 claims (recessive in 33, dominant in 95, additive in 19, allele-based in 16, and model-free [all 3 genotypes considered separately] in 64). The remaining 101 were based on other, more peculiar genetic contrasts (wild-type homozygous vs variant homozygous, n = 30; heterozygous vs variant homozygous, n = 8; heterozygous vs wild-type homozygous, n = 13; and other considerations, n = 50).

The entire sample size was used in 210 claims (48.6%) and the other 222 used subsets of the study population. The subset selection was based on genetic information for the gene variant of interest in 60 claims, on genetic information for some other gene variant in 2, on other patient characteristics in 101, and on combinations of the above in 59. The selection of the subset was reported to have been determined a priori in 98 claims (44.1%), it was acknowledged to have been done post hoc in 43 claims (19.4%), and the timing of the selection was unclear in 81 claims (36.5%).

Appropriate documentation was evident in only 55 claims (12.7%). This includes 49 claims that performed statistical testing for interaction and another 6 for which no formal testing was shown, but the sex-specific 95% confidence intervals of the genetic effects were readily presented and did not overlap. Forty-four (44/49 [89.8%]) of the provided P values were between .01 and .05. The smallest P value reported was .00008.

A total of 303 claims had insufficient documentation. In 81 claims, the investigators said that a statistically significant effect was found in one sex but not in the other, and both effects were in the same direction. In 46 claims, they stated that there was a statistically significant effect in one sex but not in the other, and the point estimates were in opposite directions. In 16 claims, they stated that a larger or more statistically significant sex effect was seen in one sex than in the other, and the effects were nominally statistically significant in both sexes. In 107 claims, a statistically significant effect was shown in one sex, but there was no information on statistical significance in the other sex. Finally, 53 claims reported a statistically significant effect in one sex and no statistically significant effect in the other, but the direction was not specified in either sex. For all of these situations, the way the claim was made cannot ensure whether sex-gene interaction is nominally statistically significant or not. Illustrative examples are shown in Table 2. Among the claims with insufficient documentation, 9 also reported some formal gene-sex interaction testing that was nevertheless not statistically significant at the P = .05 level (P value range, .088-.983).

A total of 74 claims were spurious. The reasons for spurious claims are shown in Table 3. The most common reasons were the comparison of male vs female cases with a selected genotype, ignoring other genotypes (ie, no genetic contrast; n = 28 claims), and the comparison of male cases directly with female cases, ignoring controls (n = 8 claims). A wide variety of other comparisons were also invoked (Table 3).

Data were available for the reanalysis of 188 claims (30 appropriately documented and 158 insufficiently documented). Overall, 105 of the 188 claims (55.9%) were not nominally statistically significant based on our calculations. Illustratively, the effect sizes and 95% confidence intervals are shown in pairs for male and female participants for the non–statistically significant claims for which the effect sizes are odds ratios (n = 44) (Figure 1 and Figure 2). Eighty-three of the 188 reanalyzed claims were nominally statistically significant, but the majority (n = 44) had modest P values between .01 and .05, 25 had P values between .001 and .01, and only 14 had P values less than .001 for the interaction.

In 30 claims, the original investigators had provided appropriate statistical documentation and raw data were also available for us to retest gene-sex interaction. Of these, in 23 claims the sex difference was statistically significant in both the reported and calculated P values, while in the other 7 we could not replicate the alluded statistical significance. Six of these used multivariate analyses, while we could calculate only unadjusted estimates based on given raw data. In the remaining claim, the authors reported to have used a Breslow-Day test of homogeneity with P = .05. Our reanalysis yielded P = .059, while our Breslow-Day recalculation gave P = .054.

Of the 432 claims, only 37 claims (in 17 articles) were stated to rely on a priori considerations, had raw data available that documented that they were indeed nominally statistically significant, and had analyses performed on the whole study sample. We found any kind of corroboration history for only 3 of them. One sex difference had already been described (same trend) in 2 previous studies. For another claim, a previous study had found no effect while a subsequent one replicated the interaction. The third claim had already been described in a previous study but in the opposite direction; however, 2 previous studies reported no effect (Table 4).

Another 23 claims (in 12 articles) were stated to rely on a priori consideration, had raw data documenting a nominally statistically significant sex-gene interaction, and had analyses performed on a sample subset that was also justified a priori. Of these, 2 claims were made in an article that had at least 1 referenced study showing no statistically significant results for the respective sex-gene claim, and 1 had at least 1 reference that had shown results in the opposite direction. We found no subsequent articles replicating any of these 3 claims (Table 4).

We have empirically evaluated observational studies claiming to have found sex-related differences in genetic effects for common diseases and traits. Claims covered a variety of genes and outcomes. Most authors stated that these analyses had been conceived a priori. Nevertheless, the majority of these claims were insufficiently documented or spurious, and reporting of statistical interaction tests was rare. When we reanalyzed the available data, more than half of the tested gene-sex interactions failed to reach nominal statistical significance at the P = .05 level, and most of those that did reach significance had very modest P values. Even among the claims that seemingly had the best internal validity, corroboration in other studies was very rare.

Subgroup comparisons have been evaluated previously, mostly in the clinical trials literature.^93- 95 To our knowledge, no such assessment exists for genetic epidemiology, although genetic determinants of common traits represent an exploding clinical research literature.⁵ In the clinical trials literature, subgroup analyses with sex or other variables have been a common strategy used to find and report statistically significant results. Some authors have argued for the need to perform and transparently report subgroup analyses, in particular those related to sex.⁹⁶ However, the vast majority of claimed subgroup differences are likely to be chance findings.^97- 98 Notorious chance findings were illustrated years ago in the analysis of the ISIS-2 results based on the signs of the zodiac,⁹⁹ and the principle has been further elaborated recently in simulations of subgroup analyses that highlight the major threat of false-positive results in such endeavors.^100- 102 In the analysis of sex-specific effects in genetic associations, investigators very often seem to fall into classic traps. The most typical error is to claim nominal statistical significance for 1 of the 2 sexes when the difference in the effects between the 2 sexes is not beyond chance. Simulations show that a significant effect in 1 subgroup only may be a very common occurrence even in studies of modest sample size, occurring in 7% to 64% of analyses in 1 simulation study.^100- 101 The majority of such claims are expected to be false-positive results.

Another major problem is the lack of power to detect interactions of effects with sex in these studies. Brookes et al have estimated that a study with 80% power for the overall effect has only 29% power to detect an interaction effect of the same magnitude.¹⁰¹ Meaningful pursuit of interactions may require almost a 10-fold increase in the sample size compared with the samples needed to document main effects.^{100,103- 104} At the same time, these genetic association studies are already more than 10-fold smaller than what would be required to pursue even main effects, based on what is known of the plausible size of effects for common genetic variants.¹⁰⁵ Most of the main effects proposed in the last decade have not been replicated.¹⁰⁶ Under these circumstances, a nominally statistically significant interaction test at the P = .05 threshold probably has very low positive predictive value for the presence of a true interaction.

Some limitations should be addressed. We used a sampling strategy that was systematic but was also heavily driven by convenience. The sampling tends to select more prominently claimed sex differences, and one may assume that these are likely to have better-than-average internal validity and better-than-average chances of external corroboration. However, this cannot be formally proven. Sex is a patient characteristic available in virtually all genetic association studies, so our sample of articles is probably only the “tip of the iceberg.” It is impractical to find and evaluate all sex comparisons, even in a circumscribed sample of the literature. Selective reporting of subgroup and secondary analyses is an increasingly recognized bias that may lead to a preponderance of “positive” findings in the published literature.^107- 109 At a minimum, the studies that we evaluated are probably among the ones in which authors were most certain about some, if not all, of the sex claims that they presented in their results; otherwise, they would not have drawn attention to the claims in the titles of their articles.

We should also acknowledge that for some claims, especially the ones that were first made most recently, corroboration may not yet have been performed but may be performed in the future. However, genetic epidemiology is a quickly moving field, and replication efforts are currently typically performed very rapidly.

The issues addressed herein focus on gene-sex interaction, but their implications probably extend to any kind of subgroup analysis in genetics. Genetic epidemiology is a field that often invites subgroup analyses, not only by sex, but also by age, racial/ethnic descent, other polymorphisms, diet, lifestyle, and other exposures.^110- 111 Similar caution may be needed in the analysis, reporting, and interpretation of all of these postulated effect modifications.

We hope that our empirical evaluation will help sensitize clinicians, geneticists, epidemiologists, and statisticians who are pursuing subgroup analyses by sex or other subgroups on genetic associations. The pursuit of gene-sex interactions should not be necessarily abandoned. Ideally, sex differences should be based on a priori, clearly defined, and adequately powered subgroups. Post hoc, discovery-based analyses are also of interest, but their post hoc character should be clearly stated in the manuscript. Both a priori and post hoc claims should be documented by interaction tests and proper consideration of the multiplicity of comparisons involved. Even then, results should be explained with caution and should be replicated by several other studies before being accepted as likely modifications of genetic or other risks.