Within-Person Across-Neuropsychological Test Variability and Incident Dementia FREE

Developing strategies to improve the prediction and diagnosis of dementia has paramount therapeutic and public health implications.¹ Neuropsychological tests are used to evaluate the diagnosis of dementia,^2- 3 the transitional stages that precede it, such as mild cognitive impairment,^4- 5 and cognitive changes over time.⁶ Level of performance on tests of memory^7- 8 and executive function^9- 10 has been reported to predict future dementia, and memory declines more rapidly up to 7 years prior to diagnosis of dementia.^11- 12

When neuropsychological tests are used for diagnostic purposes, an individual's level of performance on specific tests is measured against healthy normative samples to determine cognitive impairment.^13- 14 However, this approach does not take into account intraindividual variability in cognitive function. The taxonomy of intraindividual variability, considered as inconsistency in cognitive performance within a person, includes the following definitions: (1) variability on the same cognitive task across multiple assessments over a long period (such variability may be high in healthy individuals across the life span¹⁵); (2) variability on repeated trials of a single cognitive task administered on 1 occasion or over short periods (cross-sectional studies show that this form of variability is increased in aging¹⁶ and dementia,^17- 18 but has shown variable results in individuals with mild cognitive impairment^19- 20); and (3) variability in performance across neuropsychological tests administered in a single session (such variability is expected in the normal population,^21- 22 including older adults without dementia,²³ reflecting the individual's relative cognitive strengths and weaknesses). This study focused on within-person across-neuropsychological test variability because it can be estimated directly from standardized clinical neuropsychological procedures.

Increased within-person across-neuropsychological test variability has been related to poor cognitive function in normal older adults in cross-sectional studies.²³ Accordingly, we examined whether increased within-person across-neuropsychological test variability was associated with future dementia after controlling for level of performance on individual neuropsychological tests. We also assessed whether variability improved the sensitivity and specificity of a model predicting development of dementia 1 year later.

Participants were enrolled in the Einstein Aging Study (EAS), a longitudinal study of aging and dementia located at the Albert Einstein College of Medicine in Bronx County, New York. The study design, recruitment, and follow-up methods have been previously described.^24- 25 Briefly, the EAS used telephone-based screening procedures to recruit and follow-up a community-based cohort since 1993. The primary aim of the EAS was to identify risk factors for dementia. Eligibility criteria required that participants be aged 70 years or older, reside in the Bronx, and speak English. Exclusion criteria included severe audiovisual disturbances that would interfere with completion of neuropsychological tests, inability to ambulate even with a walking aid or in a wheelchair, and institutionalization. Potential participants older than 70 years from the Centers for Medicare & Medicaid Services population lists of Medicare-eligible individuals were first contacted by letter, then by telephone, explaining the purpose of the study. The telephone interview included verbal consent, a brief medical history questionnaire, and telephone-based cognitive screening tests.²⁴ Following the interview, an age-stratified sample of individuals who matched on a computerized randomization procedure was invited for further evaluation at the medical center. This procedure was implemented to ensure that the individuals included in the EAS represented the Bronx population at that age stratum, and that those who agreed to participate were not different from nonresponders in terms of key demographic characteristics. Written informed consent was obtained at clinic visits according to study protocols and approved by the Committee on Clinical Investigation (institutional review board of the Albert Einstein College of Medicine).

A total of 1797 participants were enrolled in the EAS between October 1993 and December 2007. Participants seen for the baseline visit only (n = 750), prevalent dementia cases (n = 72), and those with missing information (n = 78) were excluded. Hence, a total of 897 participants were included in this investigation. Participants had follow-up visits every 12 to 18 months, at which they underwent detailed neurological and neuropsychological evaluations.

To address the issue of diagnostic circularity, we derived an algorithmic diagnosis of dementia that was independent of the neuropsychological test performance. Two conditions had to be met. First, participants had to make 8 or more errors on the Blessed Information Memory Concentration test (BIMC; best score: 0 errors and worst possible score: 32 errors).²⁶ This test has high test-retest reliability (0.86) and correlates well with the pathology of Alzheimer disease.^27- 28 (The BIMC test is used to screen for cognitive impairment on the telephone interview at baseline and the in-person interviews in the EAS, but the scores are not used when assigning dementia diagnosis in consensus case conferences.²⁴) Second, impairments on basic activities on the Lawton-Brody activities of daily living,²⁹ secondary to cognitive impairment, had to be documented using previously described procedures.³⁰ Incident dementia using the above algorithmic procedures was used as the main outcome measure in the primary analyses reported herein. Agreement between the algorithmic and clinical diagnosis of dementia was 96% in this cohort.

The EAS participants suspected to have dementia received a complete diagnostic workup as previously described for this cohort.³¹ Diagnoses of dementia were assigned according to the criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition,³² at case conferences attended by at least 1 study neurologist, a neuropsychologist, and a geriatric nurse or social worker. Alzheimer disease was diagnosed according to the criteria for probable disease detailed by the National Institute of Neurological and Communicative Disorders and Stroke and the Alzheimer Disease and Related Disorders Association.³ The State of California Alzheimer Disease Diagnostic and Treatment Centers criteria was used to assign diagnoses of probable, possible, or mixed-vascular dementia.³³ In individuals diagnosed with dementia, neuroimaging was used to help allocate the diagnosis of probable Alzheimer disease or probable vascular dementia. We have reported good agreement between clinical diagnoses of Alzheimer disease,³⁴ vascular dementia,³¹ and dementia with Lewy bodies,³⁵ and pathological findings in our study. For the purpose of this analysis, clinical diagnosis of dementia subtypes was used only in secondary analyses to explore whether associations with variability varied as a function of diseases processes. Hence, only pure subtypes (Alzheimer and vascular) but not mixed were considered in the exploratory analyses.

The tests included in this battery have been validated for use in the aging population in our and other aging studies.^8,36- 37 Our recent studies reveal that factor analysis of the neuropsychological test battery consistently yielded 3 empirically derived and statistically orthogonal cognitive domains including verbal IQ, attention/executive function, and memory.^38- 39 For the purpose of this study, we identified 3 tests that represented the 3 cognitive domains mentioned above, and were available for all 897 participants included in this study. The Vocabulary (total score) subtest of the Wechsler Adult Intelligence Scale Revised (WAIS-R⁴⁰), considered a hold test in that it is not sensitive to the effect of aging and age-related diseases, represented the verbal IQ domain. The Digit Symbol Substitution subtest of the WAIS-R⁴⁰ is commonly used to assess attention and executive function. The Free and Cued Selective Reminding Test (FCSRT; free recall) is commonly used to assess verbal memory and is sensitive to dementia.⁴¹

The method used herein to estimate within-person across-test variability was described in other studies as well.^23,42 Assumptions of normality were met for the distribution of scores on the FCSRT, WAIS-R Vocabulary, and WAIS-R Digit Symbol Substitution tests. The raw scores of each test were z transformed on the basis of the distribution of the entire sample (n = 897). Then, the z-transformed test scores were used to calculate within-person variability across the 3 tests using this equation. Variability = 

where Z_ik is the kth cognitive test score for ith individual. Here, k = 1, . . . , K, K = 3 (FCSRT, WAIS-R Vocabulary, and WAIS-R Digit Symbol Substitution) and

is the individual's mean z-transformed score based on the 3 tests.

Consistent with our previous studies,^38- 39,43 dichotomous rating (present or absent) of diabetes, chronic heart failure, arthritis, hypertension, depression, stroke, Parkinson disease, chronic obstructive pulmonary disease, angina, and myocardial infarction was used to calculate an illness index summary score (range, 0-10). Medical history was obtained from multiple sources including significant others and family physicians when available. The study physician obtained the medical history during the neurological examination and trained research assistants conducted the structured clinical interview. Smoking was ascertained but did not contribute to variability in incident dementia in this study and was not included in the analyses reported herein.

We performed 3 Cox proportional hazards regression analyses to assess the relationship between neuropsychological tests and dementia. First, we examined whether level of performance on the neuropsychological tests at baseline predicted the development of dementia. Second, we examined the association between within-person across-neuropsychological test variability and the development of dementia. Third, we assessed whether within-person across-neuropsychological variability provided incremental prediction of incident dementia above and beyond what was predicted by the absolute level of performance on the neuropsychological tests by controlling for performance differences on the WAIS-R Vocabulary, FCSRT, and WAIS-R Digit Symbol Substitution. Individuals who died before developing dementia were censored at the time of their last study visit when their status as not having dementia was determined.

To assess the amount of variability of development of incident dementia explained by the models, we compared the goodness of fit of nested Cox proportional hazards models using the partial likelihood ratio test.⁴⁴ We compared a model with the 3 individual neuropsychological tests with a model that added within-person across-neuropsychological test variability. Finally, using the McNemar test we examined whether within-person across-neuropsychological test variability significantly improved sensitivity for the prediction of new cases of dementia within 1 year.

We used age as the time scale because it is considered more appropriate than follow-up time in cohort studies.⁴⁵ When age serves as the time scale, the hazard function can be directly interpreted as the age-specific incidence function and age is accounted for in the nonparametric term of the hazard function, providing a more flexible and effective control of age.⁴⁵ Time to event was from age at the baseline assessment accounting for the left truncation at study inclusion to age at which diagnosis of dementia was ascertained or to final study contact for participants without dementia. In addition, all multivariate analyses were stratified by sex and controlled for education and medical illness. For all primary analyses, incident dementia was defined using the algorithmic procedures described earlier. Proportional hazards assumptions of the models were examined analytically and graphically and were met.

Finally, Cox proportional hazards regression analysis with age as the time scale was used to examine whether within-person across-neuropsychological test variability predicted incident dementia subtypes (eg, vascular and Alzheimer) defined by consensus diagnosis as previously described. Given the low number of incident cases of Alzheimer dementia and vascular dementia, as well as concerns for potential diagnostic circularity, these analyses were considered exploratory. All analyses were performed using SAS statistical software version 9.1 (SAS Institute Inc, Cary, North Carolina) and S-Plus version 8.0 (Insightful Corp, Seattle, Washington). The significance level was set at .05 and all tests were 2-sided.

Participants were community residents who were relatively healthy and had normal global mental functioning based on BIMC scores that were within the normal range (Table). Of the 897 participants, there were 61 cases of incident dementia (6.8%) (defined by the algorithmic procedure described earlier) identified during the follow-up period (mean [SD], 3.3 [2.4] years). On the basis of the consensus clinical diagnostic procedures, 47 participants developed incident dementia of the Alzheimer type and 18 participants developed incident vascular dementia. During the study, 128 individuals died, as expected for the age of this cohort. Of these, 18 had developed incident dementia.

All Cox proportional hazards regression models used age as the time scale and were adjusted for sex, education, and medical illness. In the first model, higher scores on the FCSRT (hazard ratio [HR] for 1-point difference on the test, 0.87 [95% confidence interval {CI}, 0.84-0.91]; P < .001) and WAIS-R Digit Symbol Substitution (HR for 1-point difference on the test, 0.97 [95% CI, 0.94-1.00]; P = .04) were associated with a lower risk of developing dementia. Level of performance on the WAIS-R Vocabulary test was not associated with future dementia (HR for 1-point difference on the test, 1.01 [95% CI, 0.98-1.03]; P = .67).

The second Cox proportional hazards regression analysis also used age as the time scale and adjusted for sex, education, and medical illness. Within-person across-neuropsychological test variability was significantly associated with incident dementia defined algorithmically (HR for 1-point difference in variability, 3.93 [95% CI, 2.04-7.56]; P < .001). Risk of dementia increased as a function of greater within-person variability.

The third Cox proportional hazards regression analysis controlled for each of the neuropsychological test scores to determine the independent association of variability with incident dementia. The model used age as the time scale and adjusted for sex, education, and medical illness. Similar to the previous analysis, the results revealed that higher scores on the FCSRT (HR for 1-point difference on the test, 0.89 [95% CI, 0.85-0.92]; P < .001) and WAIS-R Digit Symbol Substitution (HR for 1-point difference on the test, 0.97 [95% CI, 0.94-1.00]; P = .04) were associated with lower risk of incident dementia, and the WAIS-R Vocabulary test was not associated with it (HR for 1-point difference on the test, 1.00 [95% CI, 0.98-1.03]; P = .81). Importantly, even after controlling for individual neuropsychological test scores, within-person across-neuropsychological test variability remained significantly associated with incident dementia (HR for 1-point difference in variability, 2.10 [95% CI, 1.04-4.23]; P = .03). The HR of variability was lower after adjusting for performance level on the individual tests, as expected, but the correlations between variability and performance on the neuropsychological tests were relatively low (FCSRT, r = −0.21, P < .001; WAIS-R Digit Symbol Substitution, r = −0.12, P < .001; WAIS-R Vocabulary, r = −0.18, P < .001), suggesting that these measures were not collinear. The lower, but significant HR for within-person across-neuropsychological test variability in the third compared with the second Cox analysis, along with HRs for level of performance on the individual neuropsychological tests that were similar in the 2 analyses, are evidence that part, but not all, of the effect of variability is mediated through changes in levels of cognitive performance.

We used the partial likelihood ratio test⁴⁴ to examine whether adding within-person across-neuropsychological test variability to neuropsychological test performance improved prediction of dementia compared with using neuropsychological test performance alone. The results were significant (log partial likelihood ratio with and without variability, respectively, −200.5 and −202.6; χ²₁ = 4.19, P = .04), indicating that prediction of incident dementia was improved by adding within-person across-neuropsychological test variability. We also examined whether within-person across-neuropsychological test variability improved the sensitivity for the prediction of dementia within 1 year. In a model that included the 3 individual neuropsychological tests and controlled for age, sex, education, and disease illness, a score combining the coefficients of all covariates in the model that resulted in 80% specificity for detecting dementia yielded 83% sensitivity for predicting dementia within 1 year. Including within-person across-neuropsychological test variability in this model, maintaining specificity at 80%, significantly increased the sensitivity for predicting dementia within 1 year to 88% (McNemar test, P = .01).

Kaplan-Meier curves were used to delineate the cumulative risk of incident dementia using age as the time scale. Within-person across-neuropsychological test variability dichotomized into the highest (ie, worst) quartile vs the rest revealed that (consistent with the previous results) higher variability was associated with increased risk of incident dementia (HR, 2.25; 95% CI, 1.32-3.86) (Figure, A). Additionally, Kaplan-Meier curves for the FCSRT (HR, 4.77; 95% CI, 2.80-8.10), WAIS-R Digit Symbol Substitution (HR, 2.43; 95% CI, 1.44-4.09), and WAIS-R Vocabulary tests (HR, 1.51; 95% CI, 0.88-2.57) dichotomized into the lowest (ie, worst) quartile vs the rest are illustrated in the Figure, B-D, respectively.

Cox proportional hazards regression analyses with age serving as the time scale, adjusting for sex, education, and illness index, were used to examine whether associations between variability and clinical diagnosis of dementia using consensus case conference procedures varied as a function of dementia subtype. Variability predicted development of both incident dementia of the Alzheimer type (n = 47; HR, 3.63 [95% CI, 1.79-7.37]), and incident vascular dementia (n = 18; HR, 5.26 [95% CI, 1.70-16.30]). The HR for Alzheimer disease after adjusting for neuropsychological test results was 1.68 (95% CI, 0.80-3.55; P = .17); these results are comparable with dementia overall but power was limited to demonstrate a significant difference. The disease processes underlying pure subtypes of Alzheimer dementia and vascular dementias are different. Thus, it appears that variability is sensitive to dementia irrespective of disease subtype.

Our assessment of intraindividual variability addresses within-person differences in cognitive function and its use as a marker of pathology.⁴⁶ The taxonomy of intraindividual variability offers several operational definitions for this construct.⁴⁷ To our knowledge, this is the first study to demonstrate that within-person across-neuropsychological test variability is associated with incident dementia in a population-based cohort aged 70 years or older, even after adjusting for level of performance on each individual neuropsychological test.

The premise guiding the study was that variability or inconsistency in performance across neuropsychological tests would increase in the preclinical stages of dementia compared with normal aging. This, in part, is attributed to the decline in memory function years before the diagnosis of dementia.¹¹ As opposed to summary scores or intraindividual indices that estimate function within 1 cognitive domain, within-person across-neuropsychological test variability may be conceptualized as a single representation of variability across multiple domains subserved by several cortical regions and networks. In this context, this form of cognitive variability may be considered a signature of decline in cerebral integrity in the early stages of dementing illnesses.

In choosing the individual tests used to calculate within-person across-neuropsychological test variability, we aimed to maximize the number of participants, represent different cognitive domains, and increase generalizability by including a few easily administered tests that are commonly used in other studies and by clinicians assessing cognitive impairment in elderly individuals. Within-person across-neuropsychological test variability can be computed using tests other than those used in this study. Further, this form of cognitive variability can be estimated using standard and widely used clinical and neuropsychological assessment procedures that are typically given in 1 testing session. Hence, the potential clinical utility of this aspect of cognitive function is quite appealing because it requires no changes to standard assessment procedures in aging studies or assessment of cognitive disorder in clinic settings. It is noteworthy that variability on subdomains of the Mini-Mental State Examination⁴⁸ was related to short-term decline on a global measure of cognitive function in a small cohort of centenarians.⁴² Hence, examining across-cognitive domain variability in screening measures, including those currently used to support the diagnosis of mild cognitive impairments,⁴⁹ is of interest.

Research and theories concerning within-person variability on single measures, although relatively recent, suggest that increased inconsistency represents impaired top-down executive control processes⁵⁰ subserved by frontal regions.^51- 52 However, at present, little is known about within-person across-cognitive test variability, the theoretical and neuroanatomical basis for this putative construct. Our findings suggest that even though substantial in older adults without dementia,²³ within-person across-neuropsychological test variability may be pathological when a certain threshold is exceeded. It is noteworthy that variability was sensitive to both Alzheimer and vascular dementia subtypes, which vary in terms of their etiology and cognitive profile.^33,53- 54 Therefore, as a signature of early decline in global cerebral integrity, variability may capture the summation of differing sensitivities of brain regions and networks to various disease processes as opposed to estimating the effect of disease on a single brain region. The relatively low correlations between variability and the level of performance on the individual neuropsychological tests appear to support this notion.

It is of further interest to examine whether and the extent to which different aspects of intraindividual variability in cognitive functions are related in terms of the theory, underlying mechanism, and utility in predicting outcomes of interest. For instance, research suggests that individuals are not consistently diagnosed with mild cognitive impairment across multiple sessions.^55- 56 Although some fluctuations may be attributed to error in measurement and limitations of this construct, it would be of interest to examine the relationship between across-testing and within-testing session variability and whether the latter improves understanding and prediction of transitional stages in cognitive aging.

The limitations of the study should be considered. First, the sample, although representative of the Bronx, consisted of volunteers who resided in the community and who were relatively healthy and willing to travel to the medical center. Hence, the generalizability of the findings may potentially be limited. Second, the diagnosis of incident dementia was based on an algorithmic procedure that was independent of neuropsychological test scores and consensus diagnosis. Hence, diagnostic circularity is less likely to confound the findings. However, to examine longitudinal associations between this form of variability and clinical diagnosis of dementia, within-person across-neuropsychological test variability ideally should be derived from tests that are not part of the clinical diagnostic procedures. Third, assignment of subtypes of dementia is fallible. Although the diagnoses were made according to standardized criteria, some misclassification is inevitable. Further, differential associations between variability and dementia subtypes may be demonstrated, perhaps when using neuropsychological tests that are more sensitive to either vascular or Alzheimer diseases.^33,53- 54 Fourth, attrition is a major issue of concern in any longitudinal study. A significant number of individuals enrolled in the EAS were not eligible to participate in this investigation because they had only the baseline evaluation and were awaiting the next yearly visit. However, the subsample included in this study was not different from the entire EAS cohort in terms of age, sex distribution, and level of education (data available from author on request). Nonetheless, the potential for selection bias should be considered. We emphasize that a single variability measure cannot replace a complete neuropsychological examination, nor do we advocate that 3 neuropsychological tests are sufficient to provide adequate assessment of an individual's cognitive function. Instead, we propose that measures of within-person across-test variability be viewed as complementary to standard assessment procedures that are used to predict future risk of dementia. Finally, our results were identified in a single population; these results should be validated in a new population and their value in discriminating between populations that will and will not develop dementia should be defined before intraperson across-neuropsychological test variability is used routinely in clinical settings.

In summary, within-person across-neuropsychological test variability was associated with development of dementia independently of performance of the neuropsychological tests. This finding needs to be replicated in different populations before it is applied in a clinical setting.