Cost-effectiveness of 3 Methods to Enhance the Sensitivity of Papanicolaou Testing FREE

Screening with the Papanicolaou (Pap) test is widely regarded as a cost-effective but imperfect approach to the prevention of cervical cancer.^1- 3 New technologies improve the sensitivity (true-positive rate or TPR) of Pap testing. The AutoPap 300 QC (NeoPath Inc, Redmond, Wash) and Papnet (Neuromedical Systems Inc, Suffern, NY) systems combine automated microscopy and computerized analysis to reduce screening error—the failure to identify abnormalities on the slide—by detecting abnormal cells overlooked on initial examination. The ThinPrep 2000 (Cytyc-Sands, Boxborough, Mass) is a semiautomated, liquid-based slide preparation system. It filters noncellular material before depositing cells in a thin layer on the slide. ThinPrep may reduce screening error as well as sampling error, the failure to obtain a representative sample of cells on the slide. All 3 technologies increase the incremental cost per slide screened.⁴ We explore the role for these technologies by assessing their cost-effectiveness (CE) in the prevention of cervical cancer morbidity and mortality.

We searched MEDLINE for articles published between January 1987 and December 1997 by text word for ThinPrep, monolayer, AutoPap, and Papnet and by subject for neural network combined with cervix neoplasms. We hand searched Diagnostic Cytopathology, Modern Pathology, and Acta Cytologica for the same years and obtained unpublished articles from the manufacturers of the 3 technologies. We included any article if it (1) reported the number and cytologic results of all slides, (2) reported on the Food and Drug Administration–approved use of 1 of the technologies for cervical screening, (3) used biopsy or review of discrepant results by a panel of at least 3 cytopathology professionals to validate all positive cytologic findings,^5- 6 and (4) included slides with a validated diagnosis of low-grade squamous intraepithelial lesion (LSIL) and slides with more severe diagnoses.

Screening with the conventional Pap smear and the ThinPrep slide is followed by rescreening of 10% of all slides initially screened as within normal limits (WNL). AutoPap-assisted rescreening uses a statistical classifier algorithm to select for microscopic rescreening a subset of WNL slides (typically 10%-20%) that are most likely to display abnormalities.^7- 8 With Papnet-assisted rescreening, cytotechnologists review computerized images of all WNL slides and select a proportion (typically 20%-30%) of these WNL slides for manual microscopic rescreening.^9- 13

We combined the estimates of the proportional increase in TPR associated with each technology over conventional Pap screening or rescreening in the selected studies, weighted by the number of slides in each study with a validated finding of LSIL or more severe abnormality (LSIL+). Both methods of validation may bias results in favor of the new technologies.^14- 15

To calculate the TPR of a screening strategy (primary screening and rescreening) that includes 1 of the technologies, we used the following methods. For ThinPrep screening, sensitivity was calculated by applying the estimated proportional increase in TPR to a baseline estimate of TPR of 80% for conventional Pap screening.^1- 2,16 For conventional Pap screening and ThinPrep screening, the overall TPR was based on initial screening plus 10% random rescreening, assuming statistical independence and the same sensitivity as the initial screen. For AutoPap- and Papnet-assisted rescreening, the overall TPR was calculated by applying the estimated TPR of the technique to the WNL slides remaining after first screening with manual microscopy (TPR, 80%). A proportional 5% increase in the TPR of initial screening from a technology means that the TPR of initial screening would increase to 84%. When followed by rescreening of 10% of WNL slides (TPR, 80%), the overall TPR of screening would rise to 85.3%. Our model posits a maximum TPR of any screening strategy (combination of initial screening and rescreening) of 97%.¹

The published literature contains insufficient evidence to determine whether the effect of each technology on the TPR to LSIL+ findings is lower or higher than the effect on the TPR to specific grades of abnormality, such as carcinoma in situ.

Because there was no well-validated information about specificity (true negative rate) of screening,^5- 6 we assumed that the technologies do not affect specificity. This assumption is reasonable: cytotechnologists and pathologists review all positive slides, giving them an opportunity to detect false positives, regardless of the screening technology. Furthermore, the use of a stringent definition of abnormality (LSIL+) tends to minimize the false-positive rate.

We used the societal perspective to estimate costs, including all direct costs associated with the screening strategy, whether paid by patient, insurer, or another party.¹⁷ We obtained information on the cost per slide screened from peer-reviewed articles, the manufacturers, publicly available financial reports, and a survey of pathology laboratories in northern California. In each case, estimates of the increased cost per slide processed are based on the most commonly cited costs, including the processing costs per slide, the labor cost of rescreening, and the baseline cost of obtaining and screening a Pap smear. We did not include the costs of training or capital costs, which account for less than $0.25 of the incremental cost per smear for all 3 technologies when the relevant equipment is used at full capacity.

Estimates of the marginal cost per slide for each of the technologies vary widely.^4,18- 22 Responses from laboratories in our survey and financial reports suggest that the additional disposable cost is $9.75 per ThinPrep slide compared with conventional Pap testing. The additional cost per WNL slide of using AutoPap-assisted rescreening is about $5, of which $4.25 covers the cost of processing a WNL slide with the AutoPap system. We assumed that laboratories would select 20% of slides for manual rescreening, adding about $0.75 per slide initially screened. Information from the manufacturer places Papnet's additional cost at $10. Shipping to a central facility and processing charges cost $8.50 per WNL slide. Viewing the Papnet-generated computer images of all WNL slides combined with manual microscopic review of a subset (assumed to be 20%) adds another $1.50 in incremental labor costs per slide initially screened.

Costs of care are based on Medicare figures from Eddy's original model¹ updated to 1996 dollars using the consumer price index.²³ Although these costs are similar to those used in other analyses,²⁴ newer practices, such as loop electrosurgical excision, may generate lower costs for the treatment of noninvasive lesions. All costs and benefits are discounted at 3% (range for sensitivity analyses, 0%-5%) per annum.¹⁷

We integrated estimates of TPR and cost into a 9-state time-varying transition state model developed by Eddy^1,25- 27 to calculate the lifetime costs and health effects associated with different screening strategies in a theoretical cohort of women who begin screening at 20 years of age. In the absence of screening, their life expectancy is 78.27 years. In the model, unscreened women have about a 2.5% (range for sensitivity analyses, 1.2%-5.0%) lifetime chance of developing cervical cancer and a 1.2% (range for sensitivity analyses, 0.6%-1.8%) chance of dying from the disease.²⁸ The probability of dying from cancer is a function of the probabilities of developing precancerous lesions, identifying abnormalities or cancer through screening or clinical examination, successfully treating diagnosed abnormalities and cancer, and dying from other causes. These probabilities change with the stage of cancer at diagnosis and with the age of the woman. Mortality has declined since Eddy developed the model. Thus, by overestimating the mortality of cervical cancer, we overestimate the health effects of the new technologies.

The model assumes that all cancers develop from preinvasive lesions that may regress spontaneously and that the majority of cancers (80%-95%) develop from a long preinvasive stage. The specific cytologic classifications of preinvasive lesions used in Eddy's original model are similar to those of the currently used Bethesda system.²⁹ Using the Bethesda system changes the results of the model little.

All CE ratios are incremental, expressed as the incremental cost in 1996 US dollars per incremental year of life saved (YLS) by using a technology. Interventions with relatively low CE ratios represent good values.

The baseline screening strategy is conventional Pap testing with 10% rescreening of all WNL slides. Before calculating incremental CE ratios, we exclude screening strategies that are strictly dominated, ie, that are both less effective and more costly than at least 1 other strategy. We also eliminate those alternatives that have a higher incremental CE ratio than a more effective option, ie, that fail by extended dominance.³⁰ Rational decision makers will never choose an option excluded under extended dominance. They will choose a more expensive alternative that yields a lower or equivalent CE ratio with greater total health benefit.^31- 32Table 1 and Table 2 list major parameters and assumptions used in the model and the range of values used for the sensitivity analysis.

Of the nearly 200 articles identified, 12 met the inclusion criteria. The characteristics of the screening populations, study methods, and TPR of conventional Pap screening varied across studies.

No studies of the ThinPrep 2000 system provided biopsy results or panel adjudication for all cytologic results of LSIL+ findings. Studies with validation by biopsy, colposcopy, or both for a portion of slides with a positive result reported a percentage increase in TPR to LSIL+ findings of 0.1% to 10%.^33- 34 Studies that provided results by biopsy, colposcopy, or both for all positive cytologic results report that earlier versions of the ThinPrep system increased the TPR of screening by 9.4% to 20.9%. The mean proportional increase across these studies in the TPR of initial screening was 14.9%,^35- 37 leading to a TPR of initial screening of 91.9% and an overall TPR of screening and rescreening of 92.6%.

Less information is available on the TPR of the AutoPap 300 QC system. None of the studies of AutoPap had many LSIL+ slides. The strongest evidence comes from studies in which a panel of 3 independent pathologists confirmed positive cytologic results. In 5 of 6 laboratories the system selected between 33.3% (1/3 cases) and 100% (1/1 case) of all LSIL+ slides at less than 20% review rate across sites.³⁸ We limited our estimates to the use of AutoPap to select exactly 20% of WNL slides for review, because this was the most effective reported setting. When set to select exactly 20% of slides for review, AutoPap identified 77% of the LSIL+ slides at 1 laboratory.⁸ We estimated that a screening strategy that includes initial Pap smear screening with a TPR of 80% followed by AutoPap-assisted rescreening with a TPR of 77% achieves an overall TPR of 95.4%.

Seven distinct studies with validation by biopsy, colposcopy, or both^{9- 10,39- 43} showed that the TPR of Papnet to LSIL+ slides ranges between 19.6% and 100%.^42- 43 Based on 4 studies with comparable design, we estimate that the average TPR of rescreening with Papnet is 85.9%.^10,39- 41 When Papnet-assisted rescreening with a TPR of 85.9% is added to conventional Pap screening (TPR of 80%), the overall TPR exceeds 97%.

Table 3 and Figure 1 show the costs and outcomes resulting from the use of each technology at different screening frequencies. The CE of moving from 1 screening strategy to a more expensive alternative is the difference in cost divided by the difference in life expectancy associated with the 2 strategies. In the Figure 1, life expectancy increases from bottom to top, and costs increase from left to right. Thus, the difference in costs between 2 strategies is the horizontal difference between 2 points; the difference in outcomes is the vertical distance between them. Consequently, the CE ratio is the reciprocal of the slope of the line connecting the 2 screening strategies under comparison. The slope of the line is steeper when the net gain in health per dollar is greater.

All 3 technologies improve outcomes and increase costs compared with conventional Pap screening. All are more cost-effective when used as part of screening every 3 or 4 years than when used as part of annual screening. As Table 3 shows, with quadrennial screening, life expectancy per woman screened increases by between 1.2 and 1.6 days over conventional Pap testing, depending on the technology used, at an incremental cost of $30 to $62 per woman. With annual screening, the increase in life expectancy is only 5 to 7 hours, at a cost of $134 to $257.

Strategies using ThinPrep produce less health benefit at greater cost than (are dominated by) strategies combining conventional Pap testing with AutoPap-assisted rescreening. If the frequency of screening is fixed, the CE ratio of choosing AutoPap in comparison with triennial conventional Pap testing is about $16,000 per YLS. For annual screening, the CE ratio for AutoPap exceeds $166,000 per YLS.

The incremental cost per YLS of using Papnet instead of AutoPap is much higher and increases approximately from $147,000 with triennial screening to $1,100,000 with annual screening.

If frequency of screening can change, AutoPap- and Papnet-assisted rescreening can dominate conventional Pap testing conducted at shorter intervals. For example, Figure 1 shows that triennial AutoPap- and Papnet-assisted rescreening produce more life years at lower cost than biennial conventional Pap testing.

The sensitivity analysis explores the consequences of variation in 3 sets of assumptions when the screening frequency is fixed: characteristics of the populations screened; the cost and TPR of conventional Pap testing; and the cost and TPR of screening assisted by the new technologies.

For a wide range of assumptions about the population being screened, as shown in Table 4, AutoPap dominates ThinPrep. Furthermore, at most screening frequencies, Papnet has a higher CE ratio than AutoPap. Changes in the risk of developing cervical cancer change the CE ratios substantially; halving the risk of developing cervical cancer doubles the CE ratio while doubling the risk roughly halves the CE ratio. Other changes in the population screened have less effect. Quadrupling the proportion of rapidly progressing cancers to 20% slightly reduces the cost per YLS. If screening is initiated at a later age than the 20 years assumed in the base case, the CE ratios of conventional Pap testing and the technological enhancements decline, because the prevalence of cytologic abnormalities rises with age. The CE ratios become somewhat higher when screening starts near the age of 65 years, when the cancers detected tend to be at later stages.

The estimated CE of the new technologies improves when the TPR of conventional Pap testing worsens. As Table 4 shows, compared with conventional Pap testing with a TPR of 70%, AutoPap would cost approximately $9000 per YLS with triennial screening and $103,000 per YLS with annual screening. The CE ratio of Papnet compared with AutoPap also drops by about half to $72,000 per YLS with triennial screening and $591,000 per YLS with annual screening. If the TPR of conventional screening were as low as 50%, AutoPap-assisted rescreening would dominate conventional Pap testing, while the approximate cost per YLS with Papnet would fall to $26,000 with triennial screening and to $228,000 with annual screening.

Table 4 also shows that changes in either the cost or the TPR of any of the 3 technologies can change their relative CE. The performance of AutoPap and Papnet is similar; with modest changes in cost and TPR, either rescreening technology could dominate the other.

Larger changes would be required to make ThinPrep cost-effective. If ThinPrep increased the TPR by 50% more than the increase assumed in our base case, it would cost about $28,000 per YLS as part of triennial screening, compared with AutoPap-assisted screening. If the incremental increase in TPR were 25% more than in the base case and the incremental increase in cost were 50% less, ThinPrep would no longer be dominated. With these assumptions, ThinPrep would cost about $15,000 per YLS when compared with triennial conventional Pap testing. The cost per YLS would rise to about $154,000 if incorporated in annual screening. If ThinPrep were the only alternative to conventional Pap testing, its cost per YLS compared with conventional screening would be about $37,000 with triennial screening and $370,000 with annual screening. In contrast, if the incremental increase in TPR from AutoPap were 25% more than assumed in the base case, it would dominate all other screening options. If the costs of all technologies except ThinPrep increase by 50%, AutoPap still dominates ThinPrep.

If 1 of the technologies were more sensitive to more severe abnormality (eg, carcinoma in situ), its CE would improve relative to the other technologies. The effect of such a hypothetical difference, however, decreases as the frequency of screening increases. Similarly, if the technologies affect different sources of screening failure, the effect on CE ratios diminishes as screening becomes more frequent.

Changes in the discount rate, along with changes in the costs of conventional Pap testing, of biopsy and treatment, and of other tests and treatments that affect all technologies equally have little effect on the estimated CE of the new technologies relative to one another. For example, a 75% reduction in the cost of treating carcinoma in situ and lower-grade abnormalities lowers the CE ratios for AutoPap and Papnet across all screening frequencies by only 1% to 3%. Doubling the cost of diagnostic follow-up for false-positive results increases the cost per YLS by roughly 10%.

Based on current estimates of their costs and accuracy, the incremental CE ratios of AutoPap- and Papnet-assisted rescreening are comparable to those of other common interventions, when used for screening every 3 or 4 years or less frequently. For example, when compared with no screening, annual clinical breast examination and mammography in women aged 55 to 65 years costs about $45,000 per YLS.⁴⁴ Screening for high blood pressure once with a sphygmomanometer in women aged 40 years costs about $32,000 per YLS.⁴⁵ Furthermore, use of these technologies in screening may provide greater gains in life expectancy at lower cost than more frequent conventional screening.

Our estimates are subject to uncertainty because the literature on the effectiveness of the 3 technologies studied here is incomplete and sometimes contradictory. Although some studies reported that each technology can identify virtually all abnormal slides, the highest-quality studies suggest that the technologies increase the TPR by a modest amount, especially in a laboratory that is already highly accurate. Our findings are insensitive to all but very large changes (about 50% increase or decrease from baseline values) in the estimated TPR and cost of each technology. However, such large changes in TPR are within the range of values reported in the literature.

Our base case assumes low costs for each of the technologies and generally underestimates their cost per additional YLS. Even a strategy excluded under strict or extended dominance in the base case might appear cost-effective under a plausible alternative set of costs. In principle, the technologies could reduce medicolegal costs by preventing malpractice lawsuits. If so, their CE relative to conventional Pap testing will improve.

Unlike previously published articles, our study estimates the cost per YLS of 3 technologies. Research by O'Leary and colleagues¹³ suggests that Papnet is not cost-effective compared with 100% manual rescreening of all WNL slides. An editorial by Hutchinson⁴ evaluated the cost per additional abnormality detected with Papnet and AutoPap and found that neither was cost-effective compared with a range of manual rescreening techniques.⁴ Neither article addresses the effects of the technologies on final health outcomes such as YLS. They are consistent with work by Benneyan, Kaminsky, and colleagues^46- 48 and by Raab,⁴⁹ which suggests that rapid rescreening by cytotechnologists and pathologists will typically be cost-effective compared with the current practice of rescreening 10% of WNL slides and to the use of AutoPap and Papnet. The results of a comparison of Papnet and conventional Pap testing are similar to those reported here.⁵⁰

These findings may change as new evidence becomes available on the TPR of the technologies, especially if they differ in their ability to classify the stage of abnormality correctly. The findings may also change as the technologies improve, as they are used in novel ways, or as their material or processing costs change. The ThinPrep system has been combined with a hybrid DNA capture mechanism for detecting types of human papillomavirus associated with increased risk for cervical cancer.^51- 52 The CE of using this approach as part of the triage of abnormal smears is unknown⁵³ but is 1 of the subjects of a large clinical trial. The Papnet system has been tested as a tool to triage slides initially classified as atypical squamous cells of undetermined significance.⁵⁴ The AutoPap system has recently been approved for initial (primary) screening of Pap smears. The results of our analysis do not apply to these novel uses.

Technological enhancements to an already highly effective screening test may not be cost-effective compared with other common screening interventions. If added to annual screening, the 3 technologies have little effect on life expectancy. The major barrier to prevention of cervical cancer is not the accuracy of the Pap test, but the failure to be screened at all.⁵⁵ These technological improvements in the Pap test can be cost-effective when used as part of less frequent screening. However, if their high cost deters participation in cervical cancer screening programs, they will not reduce the toll of the disease.