PAPNET-Assisted Rescreening of Cervical Smears:  Cost and Accuracy Compared With a 100% Manual Rescreening Strategy FREE

ALTHOUGH the Papanicolaou (Pap) smear has contributed to a significant decrease in cervical cancer mortality, conventional cytologic examination misses a significant number of premalignant lesions in which diagnostic cells are present on the slide. Among the tools that are available for reducing the false-negative rate are automated rescreening devices that have been approved recently by the Food and Drug Administration.^1- 4

The cost-effectiveness of automated rescreening devices is likely to depend on other laboratory costs, the prevalence of significant lesions within the population screened, and the false-negative rate of the laboratory. Among the alternatives to machine-assisted rescreening is manual rescreening of slides previously diagnosed as negative. The advantage to manual rescreening is timeliness and possibly lower costs; the disadvantage is a potentially lower detection rate, particularly if computer-assisted rescreening relies on features that are not ordinarily appreciated by cytotechnologists.

In this article we report the results of a study that assessed the effectiveness of a computer-assisted rescreening system (PAPNET, Neuromedical Systems Inc, Suffern, NY) in identifying cellular abnormalities in Pap smears that had been previously diagnosed as "within normal limits" or "benign cellular changes" after both primary screening and a second manual rescreening. The study allows us to compare the cost and effectiveness of PAPNET-assisted rescreening with that of manual rescreening, because it allows us to identify cases that were missed by a second manual screen, to calculate the cost of identifying these additional cases, and to compare this cost with that associated with completely manual rescreening.

The Armed Forces Institute of Pathology receives and screens approximately 40000 cases per year from 8 to 12 Air Force hospitals and clinics located throughout the United States and Japan. Pap smears are taken from active duty and retired military personnel, and from eligible wives and daughters. We identified 5478 cases among those diagnosed in 1994 and 1995 that had been interpreted as "within normal limits" or "benign cellular changes" on both primary screening and a 10% random rescreen. These cases were sequential and included Pap smears from women of all ages (Table 1). Approximately 95% of all cases examined by the laboratory are diagnosed as within normal limits (87%), infection (3%), or reactive changes (5%). Approximately 2% of our cases are diagnosed as low- or high-grade squamous intraepithelial lesion (SIL); from 2% to 3% are diagnosed as atypical squamous cells of undetermined significance (ASCUS), and the rest are classified as unsatisfactory. Invasive carcinomas are identified only rarely.

Slides were imaged using the PAPNET system, and the digitized images of the 128 most "abnormal" regions were reviewed by 1 of 4 individuals (3 cytotechnologists, 1 cytopathologist) who had been trained by Neuromedical Systems. Slides deemed appropriate for review, based on the Neuromedical Systems criteria, were manually rescreened for a third time. If the smear was diagnosed as "atypical squamous/glandular cells of undetermined significance" or more abnormal, the case was further reviewed by a consensus panel consisting of 3 pathologists and 3 cytotechnologists and a consensus opinion was achieved.

Of the 5478 Pap smears imaged using the PAPNET system, 3864 (71%) were triaged as negative without further microscopic review. Of the remaining 1614 cases selected for microscopic review, 1166 were identified because no definitive endocervical component was identified within the PAPNET images. Microscopic review revealed that 257 (22%) of these cases actually demonstrated endocervical cells elsewhere on the slide (typically the edge), while 909 did not.

The remaining 448 cases that underwent manual review did so because 128 tiles were not available for review, only a scant squamous component was present, or uninterpretable or atypical-appearing cells were present on the PAPNET display. In 11 of these cases, the reviewer believed that previously undiagnosed abnormal cells might be present, and the case was further reviewed microscopically by the consensus panel. Of these 11 cases, 5 were classified as atypical squamous cells of undetermined significance (ASCUS) and 1 as atypical glandular cells of undetermined significance (AGUS). In no case did observers universally favor either a reactive or a preneoplastic origin for the cells giving rise to the diagnosis. In 1 case (that reclassified as AGUS), the patient has had 2 subsequent Pap smears demonstrating low-grade squamous intraepithelial neoplasia (LSIL); 2 patients whose smears were reclassified as ASCUS have demonstrated a single normal Pap smear subsequent to the slide entered into this investigation. We have not been able to obtain follow-up for the remaining 3 patients; their clinical courses remain unknown.

Manual screening was found to take approximately 3 times longer than PAPNET-assisted rescreening in those cases in which manual rescreening was not required (approximately two thirds of cases), yielding an average savings of 2 minutes per case for PAPNET-assisted rescreening.

The clinical utility of a rescreening procedure depends on both its efficacy and cost. PAPNET testing is expensive; our study was designed to determine whether the increased yield of "abnormal" diagnoses obtained using PAPNET for rescreening justifies this cost. For purposes of this analysis, we have assumed that all abnormal cases identified by manual rescreening alone would also be identified using PAPNET-assisted rescreening. Our study demonstrates that PAPNET-assisted rescreening identifies a small number of abnormal smears (0.11%) that are not identified on a single manual rescreening. This result is similar to that reported by Ashfaq et al,⁵ and is substantially lower than that reported by other investigators.^6- 15 The differences are readily explained by the study design. Most other studies have been conducted on samples with a relatively high percentage of abnormal specimens, such as known false negatives,^6- 7,13,16 cases originally diagnosed as ASCUS,¹² slides that had been enriched in "abnormal" diagnoses,⁸ or cases that had not been previously screened.⁹ Our study was conducted on a sample expected to have a particularly low fraction of false negatives—slides that had already undergone a negative rescreen. In contrast to all the above studies, therefore, ours strictly addressed the question of whether PAPNET-assisted rescreening identified abnormal smears that had not been diagnosed as abnormal on the basis of a primary screen plus a manual rescreen.

While the value of detecting high-grade, or even low-grade squamous intraepithelial lesions is not questioned, the value of ASCUS and AGUS diagnoses in guiding patient treatment has not been established. For this reason, it is difficult to conclude on the basis of our results that there is any clinical value to PAPNET-assisted rescreening. Given the low number of additional abnormal cases that we identified using the PAPNET system, cost-effectiveness analysis is difficult. For the following analysis, we assume that the PAPNET-assisted rescreen might have been expected to identify 2 additional cases of LSIL (which it did not, but which might be expected on the basis of 6 ASCUS diagnoses and other reports in the literature).

We assume in this cost analysis that the total cost associated with manual rescreening is the labor cost for the cytotechnologist (C_rescreen). The cost (C_PAPNET) for PAPNET-assisted rescreening is shown in the 2 equations below:

where C_system consists of the fees charged by Neuromedical Systems, Inc, for digitization and analysis of the slides, C_review is the cost for cytotechnologist review of the PAPNET images, and C*_rescreen is the cost for rescreening those slides that are identified as potentially abnormal during the cytotechnologist review of PAPNET images.

where p_rescreen is the probability that a given slide will be selected for manual rescreening based on review of the PAPNET images. (Based on the data presented above, we have assumed p_rescreen = 0.295 in our analyses.)

A cost analysis for our laboratory has identified the cytotechnologist cost for either initial screening or manual rescreening (C_rescreen) to be approximately $3 per slide screened, a figure also used by Kaminsky et al¹⁷ in their analysis of rescreening strategies. We use this cost for completely rescreening a slide, whether as the sole rescreening strategy or if the slide was identified as potentially abnormal on the basis of PAPNET images. C*_rescreen is thus $0.88. Based on the time analysis above, cytotechnologist cost for initial PAPNET examination (ie, review of stored images, or C_review) is one third this cost ($1). The PAPNET system charges (C_system) were $7.50 per slide, for a total cost of PAPNET-assisted rescreening (C_PAPNET) of $9.38 per slide. The marginal cost of PAPNET-assisted rescreening over that of 100% manual rescreening is $6.38 per slide. Based on our data above, the use of PAPNET will yield 1 more ASCUS or AGUS diagnosis than will manual rescreening for every 913 cases rescreened, at a total additional cost of $5825. The use of PAPNET might be expected to yield 1 more LSIL diagnosis than will manual rescreening for every 2739 cases rescreened, at an additional cost of $17475. If the costs quoted in advertisements for the PAPNET system ($40 per case) are used in this analysis, the marginal cost of PAPNET over 100% rescreening is 913×$37, or $33781 per case of ASCUS or AGUS identified, and 2739×$37, or $101343 per expected case of LSIL identified. These marginal costs are somewhat higher than those estimated by Hutchinson¹⁸ for PAPNET-assisted rescreening, apparently reflecting the fact that PAPNET-assisted rescreening was less effective in our study than assumed by Hutchinson.

Although Hutchinson's analysis has been criticized for relying on an inappropriate previously published data set,¹⁹ our study nevertheless substantiates the general conclusions regarding the cost-effectiveness of PAPNET-associated rescreening.

We may compare the costs of this rescreening strategy with the costs of the routine rescreening program used in our laboratory. Our program consists of 100% rescreening of patients identified as "high risk"²⁰ because of a history of previously positive Pap smears or because the smears were obtained in colposcopy clinic, and 10% "random rescreening." Cases that were identified as abnormal on initial rescreening were subjected to consensus review of 5 cytotechnologists and pathologists. During 1996, rescreening of approximately 4970 cases identified 3 LSIL diagnoses, 3 AGUS diagnoses, and 11 ASCUS diagnoses, at a cost per LSIL diagnosis of $4970, and a cost per ASCUS or AGUS diagnosis of $1065. These costs are also higher than those estimated by Hutchinson,¹⁸ reflecting our identification by rescreening of a smaller fraction of "missed" cases than assumed in Hutchinson's analysis.

Reduction of cervical cancer mortality can be achieved by a number of different approaches.¹ An attractive alternative to rescreening is to implement more effective screening programs, ie, performing Pap smears on women who do not currently get them. As noted above, our laboratory identifies approximately 2 cases of SIL and 2 to 3 cases of ASCUS for every 100 Pap smears examined. Based on an office visit cost of $135²¹ and our laboratory's charge of $10 per Pap smear, SIL can be identified for $7250 per case by screening populations similar to ours. Our laboratory's population consists of women with a relatively high degree of access to care. Waugh et al²² have shown that a population-based cervical screening program aimed at women who have not previously had a Pap smear cost approximately £5780 (US$9826 [approx]) per life saved. Schwarz et al²³ have demonstrated that patients who have had negative Pap smears reported within the previous 4 years were more likely to have negative Pap smear results than were women who had never been screened. We thus infer that strategies aimed at increasing the number of women screened, particularly among those who have not had a Pap smear, are likely to be much more cost-effective than computer-assisted rescreening at reducing cervical cancer morbidity and mortality.

The relative utility of rescreening, in turn, will be strongly influenced by the prevalence of disease in the screened population, by the sensitivity of initial screening, and by the cost associated with rescreening.²⁴ Clearly, large reductions in the cost of PAPNET-assisted rescreening would contribute substantially to its cost-effectiveness. In addition, cost-effectiveness will be higher for laboratories that screen populations having a higher prevalence of cervical disease and for laboratories that have a higher false-negative rate on initial screening.

Since absence of diagnostic cells in the smear is a more frequent cause of missed cases than is screening error,²⁵ and because additional screening errors may occur because of sample thickness, air-drying artifacts, and other controllable factors, educational efforts that teach more effective sampling of the cervix may also be more cost-effective than any rescreening strategy, whether manual or automated.

In summary, PAPNET-assisted rescreening identifies a few more cases of ASCUS than does manual rescreening, but at a relatively high cost that must be carefully considered given its expected efficacy in reducing cervical cancer mortality.