0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Toward Optimal Laboratory Use |

PAPNET-Assisted Rescreening of Cervical Smears:  Cost and Accuracy Compared With a 100% Manual Rescreening Strategy FREE

Timothy J. O'Leary, MD, PhD; Miguel Tellado, MD; Sally-Beth Buckner, SCT, (ASCP); Izzat S. Ali, CT, (ASCP); Angelica Stevens, CT, (ASCP); Curtis W. Ollayos, MD
[+] Author Affiliations

From the Department of Cellular Pathology, Armed Forces Institute of Pathology, Washington, DC.


Toward Optimal Laboratory Use section editor: George D. Lundberg, MD, Editor, JAMA.


JAMA. 1998;279(3):235-237. doi:10.1001/jama.279.3.235.
Text Size: A A A
Published online

Context.— The Food and Drug Administration has recently approved several devices that use computerized image analysis to rescreen Papanicolaou (Pap) smears that have already been examined by cytotechnologists. Physicians and laboratories must decide whether the utility of these devices justifies the cost.

Objective.— To determine the effectiveness and cost of PAPNET-assisted rescreening in identifying cervical abnormalities not identified by manual rescreening.

Design.— PAPNET-assisted rescreening of 5478 Pap smears obtained in 1994 and 1995 previously identified as "within normal limits" or "benign changes" on both initial and random screening.

Patients.— Female service members and dependents aged 12 to 88 years.

Setting.— Air Force clinics in the United States and Japan.

Intervention.— Rescreening of Pap smears by PAPNET, followed by reevaluation of abnormal smears by the consensus panel, consisting of 3 cytotechnologists and 3 pathologists.

Main Outcome Measures.— Proportion of Pap smears initially screened as normal identified as abnormal by both PAPNET and consensus panel; costs of rescreening.

Results.— PAPNET screening identified 1614 (29%) slides requiring additional microscopic review. On further review, 448 (8% of total) had possibly abnormal cells. Ultimately, 11 of these cases were reviewed by the consensus panel for potentially atypical cells. Of these 11 cases, 5 were reclassified as atypical squamous cells of undetermined significance (ASCUS) and 1 as atypical glandular cells of undetermined significance (AGUS). No additional squamous intraepithelial neoplasia (SIL) was identified in these smears; the patient with a diagnosis of AGUS on rescreening was diagnosed as having a low-grade SIL (LSIL) on follow-up. Costs were $5825 to $33781 for each additional ASCUS or AGUS diagnosis. A cost of $17475 to $101343 is expected for each case of LSIL identified by PAPNET-assisted rescreening and not by traditional manual rescreening.

Conclusions.— PAPNET-assisted rescreening identified a few more cases of ASCUS than did manual rescreening, but at a relatively high cost. The costs of rescreening should be carefully compared with the expected efficacy in reducing cervical cancer mortality.

ALTHOUGH the Papanicolaou (Pap) smear has contributed to a significant decrease in cervical cancer mortality, conventional cytologic examination misses a significant number of premalignant lesions in which diagnostic cells are present on the slide. Among the tools that are available for reducing the false-negative rate are automated rescreening devices that have been approved recently by the Food and Drug Administration.14

The cost-effectiveness of automated rescreening devices is likely to depend on other laboratory costs, the prevalence of significant lesions within the population screened, and the false-negative rate of the laboratory. Among the alternatives to machine-assisted rescreening is manual rescreening of slides previously diagnosed as negative. The advantage to manual rescreening is timeliness and possibly lower costs; the disadvantage is a potentially lower detection rate, particularly if computer-assisted rescreening relies on features that are not ordinarily appreciated by cytotechnologists.

In this article we report the results of a study that assessed the effectiveness of a computer-assisted rescreening system (PAPNET, Neuromedical Systems Inc, Suffern, NY) in identifying cellular abnormalities in Pap smears that had been previously diagnosed as "within normal limits" or "benign cellular changes" after both primary screening and a second manual rescreening. The study allows us to compare the cost and effectiveness of PAPNET-assisted rescreening with that of manual rescreening, because it allows us to identify cases that were missed by a second manual screen, to calculate the cost of identifying these additional cases, and to compare this cost with that associated with completely manual rescreening.

The Armed Forces Institute of Pathology receives and screens approximately 40000 cases per year from 8 to 12 Air Force hospitals and clinics located throughout the United States and Japan. Pap smears are taken from active duty and retired military personnel, and from eligible wives and daughters. We identified 5478 cases among those diagnosed in 1994 and 1995 that had been interpreted as "within normal limits" or "benign cellular changes" on both primary screening and a 10% random rescreen. These cases were sequential and included Pap smears from women of all ages (Table 1). Approximately 95% of all cases examined by the laboratory are diagnosed as within normal limits (87%), infection (3%), or reactive changes (5%). Approximately 2% of our cases are diagnosed as low- or high-grade squamous intraepithelial lesion (SIL); from 2% to 3% are diagnosed as atypical squamous cells of undetermined significance (ASCUS), and the rest are classified as unsatisfactory. Invasive carcinomas are identified only rarely.

Table Graphic Jump LocationAge Distribution of Women Whose Smears Were Included in Study (n=5478)

Slides were imaged using the PAPNET system, and the digitized images of the 128 most "abnormal" regions were reviewed by 1 of 4 individuals (3 cytotechnologists, 1 cytopathologist) who had been trained by Neuromedical Systems. Slides deemed appropriate for review, based on the Neuromedical Systems criteria, were manually rescreened for a third time. If the smear was diagnosed as "atypical squamous/glandular cells of undetermined significance" or more abnormal, the case was further reviewed by a consensus panel consisting of 3 pathologists and 3 cytotechnologists and a consensus opinion was achieved.

Of the 5478 Pap smears imaged using the PAPNET system, 3864 (71%) were triaged as negative without further microscopic review. Of the remaining 1614 cases selected for microscopic review, 1166 were identified because no definitive endocervical component was identified within the PAPNET images. Microscopic review revealed that 257 (22%) of these cases actually demonstrated endocervical cells elsewhere on the slide (typically the edge), while 909 did not.

The remaining 448 cases that underwent manual review did so because 128 tiles were not available for review, only a scant squamous component was present, or uninterpretable or atypical-appearing cells were present on the PAPNET display. In 11 of these cases, the reviewer believed that previously undiagnosed abnormal cells might be present, and the case was further reviewed microscopically by the consensus panel. Of these 11 cases, 5 were classified as atypical squamous cells of undetermined significance (ASCUS) and 1 as atypical glandular cells of undetermined significance (AGUS). In no case did observers universally favor either a reactive or a preneoplastic origin for the cells giving rise to the diagnosis. In 1 case (that reclassified as AGUS), the patient has had 2 subsequent Pap smears demonstrating low-grade squamous intraepithelial neoplasia (LSIL); 2 patients whose smears were reclassified as ASCUS have demonstrated a single normal Pap smear subsequent to the slide entered into this investigation. We have not been able to obtain follow-up for the remaining 3 patients; their clinical courses remain unknown.

Manual screening was found to take approximately 3 times longer than PAPNET-assisted rescreening in those cases in which manual rescreening was not required (approximately two thirds of cases), yielding an average savings of 2 minutes per case for PAPNET-assisted rescreening.

The clinical utility of a rescreening procedure depends on both its efficacy and cost. PAPNET testing is expensive; our study was designed to determine whether the increased yield of "abnormal" diagnoses obtained using PAPNET for rescreening justifies this cost. For purposes of this analysis, we have assumed that all abnormal cases identified by manual rescreening alone would also be identified using PAPNET-assisted rescreening. Our study demonstrates that PAPNET-assisted rescreening identifies a small number of abnormal smears (0.11%) that are not identified on a single manual rescreening. This result is similar to that reported by Ashfaq et al,5 and is substantially lower than that reported by other investigators.615 The differences are readily explained by the study design. Most other studies have been conducted on samples with a relatively high percentage of abnormal specimens, such as known false negatives,6,7,13,16 cases originally diagnosed as ASCUS,12 slides that had been enriched in "abnormal" diagnoses,8 or cases that had not been previously screened.9 Our study was conducted on a sample expected to have a particularly low fraction of false negatives—slides that had already undergone a negative rescreen. In contrast to all the above studies, therefore, ours strictly addressed the question of whether PAPNET-assisted rescreening identified abnormal smears that had not been diagnosed as abnormal on the basis of a primary screen plus a manual rescreen.

While the value of detecting high-grade, or even low-grade squamous intraepithelial lesions is not questioned, the value of ASCUS and AGUS diagnoses in guiding patient treatment has not been established. For this reason, it is difficult to conclude on the basis of our results that there is any clinical value to PAPNET-assisted rescreening. Given the low number of additional abnormal cases that we identified using the PAPNET system, cost-effectiveness analysis is difficult. For the following analysis, we assume that the PAPNET-assisted rescreen might have been expected to identify 2 additional cases of LSIL (which it did not, but which might be expected on the basis of 6 ASCUS diagnoses and other reports in the literature).

We assume in this cost analysis that the total cost associated with manual rescreening is the labor cost for the cytotechnologist (Crescreen). The cost (CPAPNET) for PAPNET-assisted rescreening is shown in the 2 equations below:

PAPNETsystemreviewrescreen

where Csystem consists of the fees charged by Neuromedical Systems, Inc, for digitization and analysis of the slides, Creview is the cost for cytotechnologist review of the PAPNET images, and C*rescreen is the cost for rescreening those slides that are identified as potentially abnormal during the cytotechnologist review of PAPNET images.

rescreenrescreenrescreen

where prescreen is the probability that a given slide will be selected for manual rescreening based on review of the PAPNET images. (Based on the data presented above, we have assumed prescreen = 0.295 in our analyses.)

A cost analysis for our laboratory has identified the cytotechnologist cost for either initial screening or manual rescreening (Crescreen) to be approximately $3 per slide screened, a figure also used by Kaminsky et al17 in their analysis of rescreening strategies. We use this cost for completely rescreening a slide, whether as the sole rescreening strategy or if the slide was identified as potentially abnormal on the basis of PAPNET images. C*rescreen is thus $0.88. Based on the time analysis above, cytotechnologist cost for initial PAPNET examination (ie, review of stored images, or Creview) is one third this cost ($1). The PAPNET system charges (Csystem) were $7.50 per slide, for a total cost of PAPNET-assisted rescreening (CPAPNET) of $9.38 per slide. The marginal cost of PAPNET-assisted rescreening over that of 100% manual rescreening is $6.38 per slide. Based on our data above, the use of PAPNET will yield 1 more ASCUS or AGUS diagnosis than will manual rescreening for every 913 cases rescreened, at a total additional cost of $5825. The use of PAPNET might be expected to yield 1 more LSIL diagnosis than will manual rescreening for every 2739 cases rescreened, at an additional cost of $17475. If the costs quoted in advertisements for the PAPNET system ($40 per case) are used in this analysis, the marginal cost of PAPNET over 100% rescreening is 913×$37, or $33781 per case of ASCUS or AGUS identified, and 2739×$37, or $101343 per expected case of LSIL identified. These marginal costs are somewhat higher than those estimated by Hutchinson18 for PAPNET-assisted rescreening, apparently reflecting the fact that PAPNET-assisted rescreening was less effective in our study than assumed by Hutchinson.

Although Hutchinson's analysis has been criticized for relying on an inappropriate previously published data set,19 our study nevertheless substantiates the general conclusions regarding the cost-effectiveness of PAPNET-associated rescreening.

We may compare the costs of this rescreening strategy with the costs of the routine rescreening program used in our laboratory. Our program consists of 100% rescreening of patients identified as "high risk"20 because of a history of previously positive Pap smears or because the smears were obtained in colposcopy clinic, and 10% "random rescreening." Cases that were identified as abnormal on initial rescreening were subjected to consensus review of 5 cytotechnologists and pathologists. During 1996, rescreening of approximately 4970 cases identified 3 LSIL diagnoses, 3 AGUS diagnoses, and 11 ASCUS diagnoses, at a cost per LSIL diagnosis of $4970, and a cost per ASCUS or AGUS diagnosis of $1065. These costs are also higher than those estimated by Hutchinson,18 reflecting our identification by rescreening of a smaller fraction of "missed" cases than assumed in Hutchinson's analysis.

Reduction of cervical cancer mortality can be achieved by a number of different approaches.1 An attractive alternative to rescreening is to implement more effective screening programs, ie, performing Pap smears on women who do not currently get them. As noted above, our laboratory identifies approximately 2 cases of SIL and 2 to 3 cases of ASCUS for every 100 Pap smears examined. Based on an office visit cost of $13521 and our laboratory's charge of $10 per Pap smear, SIL can be identified for $7250 per case by screening populations similar to ours. Our laboratory's population consists of women with a relatively high degree of access to care. Waugh et al22 have shown that a population-based cervical screening program aimed at women who have not previously had a Pap smear cost approximately £5780 (US$9826 [approx]) per life saved. Schwarz et al23 have demonstrated that patients who have had negative Pap smears reported within the previous 4 years were more likely to have negative Pap smear results than were women who had never been screened. We thus infer that strategies aimed at increasing the number of women screened, particularly among those who have not had a Pap smear, are likely to be much more cost-effective than computer-assisted rescreening at reducing cervical cancer morbidity and mortality.

The relative utility of rescreening, in turn, will be strongly influenced by the prevalence of disease in the screened population, by the sensitivity of initial screening, and by the cost associated with rescreening.24 Clearly, large reductions in the cost of PAPNET-assisted rescreening would contribute substantially to its cost-effectiveness. In addition, cost-effectiveness will be higher for laboratories that screen populations having a higher prevalence of cervical disease and for laboratories that have a higher false-negative rate on initial screening.

Since absence of diagnostic cells in the smear is a more frequent cause of missed cases than is screening error,25 and because additional screening errors may occur because of sample thickness, air-drying artifacts, and other controllable factors, educational efforts that teach more effective sampling of the cervix may also be more cost-effective than any rescreening strategy, whether manual or automated.

In summary, PAPNET-assisted rescreening identifies a few more cases of ASCUS than does manual rescreening, but at a relatively high cost that must be carefully considered given its expected efficacy in reducing cervical cancer mortality.

Wied GL, Bartels PH, Bibbo M.  et al.  Computer-assisted quality assurance.  Acta Cytol.1996;40:1-3.
Wilbur DC, Bonfiglio TA, Rutkowski MA.  et al.  Sensitivity of the AutoPap 300 QC System for cervical cytologic abnormalities: biopsy data confirmation.  Acta Cytol.1996;40:127-132.
Patten Jr SF, Lee JS, Nelson AC. NeoPath, Inc. NeoPath AutoPap 300 Automatic Pap Screener System.  Acta Cytol.1996;40:45-52.
Mango LJ. The FDA review process: obtaining premarket approval for the PAPNET testing system.  Acta Cytol.1996;40:138-140.
Ashfaq R, Liang Y, Saboorian MH. Evaluation of PAPNET system for rescreening of negative cervical smears.  Diagn Cytopathol.1995;13:31-36.
Boon ME, Kok LP, Nygaard-Nielsen M, Holm K, Holund B. Neural network processing of cervical smears can lead to a decrease in diagnostic variability and an increase in screening efficacy: a study of 63 false-negative smears.  Mod Pathol.1994;7:957-961.
Boon ME, Kok LP. Neural network processing can provide means to catch errors that slip through human screening of Pap smears.  Diagn Cytopathol.1993;9:411-416.
Husain OA, Butler EB, Nayagam M, Mango L, Alonzo A. An analysis of the variation of human interpretation: Papnet a mini-challenge.  Anal Cell Pathol.1994;6:157-163.
Kok MR, Boon ME. Consequences of neural network technology for cervical screening: increase in diagnostic consistency and positive scores.  Cancer.1996;78:112-117.
Koss LG, Lin E, Schreiber K, Elgert P, Mango L. Evaluation of the PAPNET cytologic screening system for quality control of cervical smears.  Am J Clin Pathol.1994;101:220-229.
Ouwerkerk-Noordam E, Boon ME, Beck S. Computer-assisted primary screening of cervical smears using the PAPNET method.  Cytopathology.1994;5:211-218.
Ryan MR, Stastny JF, Remmers R, Pedigo MA, Cahill LA, Frable WJ. PAPNET-directed rescreening of cervicovaginal smears.  Am J Clin Pathol.1996;105:711-718.
Sherman ME, Mango LJ, Kelly D.  et al.  PAPNET analysis of reportedly negative smears preceding the diagnosis of a high-grade squamous intraepithelial lesion or carcinoma.  Mod Pathol.1994;7:578-581.
Slagel DD, Zaleski S, Cohen MB. Efficacy of automated cervical cytology screening.  Diagn Cytopathol.1995;13:26-30.
Vuong PN, Vacher-Lavenu MC, Marsan C, Baviera E. Computer-assisted rescreening of cervicovaginal smears stained by the Papanicolaou method.  Arch Anat Cytol Pathol.1995;43:147-153.
Rosenthal DL, Acosta D, Peters RK. Computer-assisted rescreening of clinically important false-negative cervical smears using the PAPNET testing system.  Acta Cytol.1996;40:120-126.
Kaminsky FC, Burke RJ, Haberle KR, Mullins DL. An economic model for comparing alternative policies for cervical cytologic smear screening.  Acta Cytol.1995;39:232-238.
Hutchinson ML. Assessing the costs and benefits of alternative rescreening strategies.  Acta Cytol.1996;40:4-8.
Boon ME. Assessing costs and benefits of PAPNET.  Acta Cytol.1996;40:1109-1111.
Council on Scientific Affairs.  Quality assurance in cervical cytology.  JAMA.1989;262:1672-1679.
Crane M. Third parties keep on chopping fees.  Med Econ.1996;73:176-177, 181-182, 185-186.
Waugh N, Smith I, Robertson A, Reid GS, Halkerston R, Grant A. Costs and benefits of cervical screening, III.  Cytopathology.1996;7:249-255.
Schwarz PJ, Fasal E, Simmons ME. 12-County program: screening of 34318 women for cervical cancer in California, 1975-78.  Public Health Rep.1981;96:547-554.
Neuhauser D, Lewicki AM. What do we gain from the sixth stool guaiac?  N Engl J Med.1975;293:226-228.
Vooijs GP. Opinion poll on quality assurance and quality control.  Acta CytolConducted by the Committee on Continuing Education and Quality Assurance of the International Academy of Cytology. 1996;40:14-24.

Figures

Tables

Table Graphic Jump LocationAge Distribution of Women Whose Smears Were Included in Study (n=5478)

References

Wied GL, Bartels PH, Bibbo M.  et al.  Computer-assisted quality assurance.  Acta Cytol.1996;40:1-3.
Wilbur DC, Bonfiglio TA, Rutkowski MA.  et al.  Sensitivity of the AutoPap 300 QC System for cervical cytologic abnormalities: biopsy data confirmation.  Acta Cytol.1996;40:127-132.
Patten Jr SF, Lee JS, Nelson AC. NeoPath, Inc. NeoPath AutoPap 300 Automatic Pap Screener System.  Acta Cytol.1996;40:45-52.
Mango LJ. The FDA review process: obtaining premarket approval for the PAPNET testing system.  Acta Cytol.1996;40:138-140.
Ashfaq R, Liang Y, Saboorian MH. Evaluation of PAPNET system for rescreening of negative cervical smears.  Diagn Cytopathol.1995;13:31-36.
Boon ME, Kok LP, Nygaard-Nielsen M, Holm K, Holund B. Neural network processing of cervical smears can lead to a decrease in diagnostic variability and an increase in screening efficacy: a study of 63 false-negative smears.  Mod Pathol.1994;7:957-961.
Boon ME, Kok LP. Neural network processing can provide means to catch errors that slip through human screening of Pap smears.  Diagn Cytopathol.1993;9:411-416.
Husain OA, Butler EB, Nayagam M, Mango L, Alonzo A. An analysis of the variation of human interpretation: Papnet a mini-challenge.  Anal Cell Pathol.1994;6:157-163.
Kok MR, Boon ME. Consequences of neural network technology for cervical screening: increase in diagnostic consistency and positive scores.  Cancer.1996;78:112-117.
Koss LG, Lin E, Schreiber K, Elgert P, Mango L. Evaluation of the PAPNET cytologic screening system for quality control of cervical smears.  Am J Clin Pathol.1994;101:220-229.
Ouwerkerk-Noordam E, Boon ME, Beck S. Computer-assisted primary screening of cervical smears using the PAPNET method.  Cytopathology.1994;5:211-218.
Ryan MR, Stastny JF, Remmers R, Pedigo MA, Cahill LA, Frable WJ. PAPNET-directed rescreening of cervicovaginal smears.  Am J Clin Pathol.1996;105:711-718.
Sherman ME, Mango LJ, Kelly D.  et al.  PAPNET analysis of reportedly negative smears preceding the diagnosis of a high-grade squamous intraepithelial lesion or carcinoma.  Mod Pathol.1994;7:578-581.
Slagel DD, Zaleski S, Cohen MB. Efficacy of automated cervical cytology screening.  Diagn Cytopathol.1995;13:26-30.
Vuong PN, Vacher-Lavenu MC, Marsan C, Baviera E. Computer-assisted rescreening of cervicovaginal smears stained by the Papanicolaou method.  Arch Anat Cytol Pathol.1995;43:147-153.
Rosenthal DL, Acosta D, Peters RK. Computer-assisted rescreening of clinically important false-negative cervical smears using the PAPNET testing system.  Acta Cytol.1996;40:120-126.
Kaminsky FC, Burke RJ, Haberle KR, Mullins DL. An economic model for comparing alternative policies for cervical cytologic smear screening.  Acta Cytol.1995;39:232-238.
Hutchinson ML. Assessing the costs and benefits of alternative rescreening strategies.  Acta Cytol.1996;40:4-8.
Boon ME. Assessing costs and benefits of PAPNET.  Acta Cytol.1996;40:1109-1111.
Council on Scientific Affairs.  Quality assurance in cervical cytology.  JAMA.1989;262:1672-1679.
Crane M. Third parties keep on chopping fees.  Med Econ.1996;73:176-177, 181-182, 185-186.
Waugh N, Smith I, Robertson A, Reid GS, Halkerston R, Grant A. Costs and benefits of cervical screening, III.  Cytopathology.1996;7:249-255.
Schwarz PJ, Fasal E, Simmons ME. 12-County program: screening of 34318 women for cervical cancer in California, 1975-78.  Public Health Rep.1981;96:547-554.
Neuhauser D, Lewicki AM. What do we gain from the sixth stool guaiac?  N Engl J Med.1975;293:226-228.
Vooijs GP. Opinion poll on quality assurance and quality control.  Acta CytolConducted by the Committee on Continuing Education and Quality Assurance of the International Academy of Cytology. 1996;40:14-24.
CME
Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Please click the checkbox indicating that you have read the full article in order to submit your answers.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 56

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Collections
PubMed Articles