Documenting and Comparing Medical Students' Clinical Experiences FREE

Medical students are exposed to an array of clinical experiences in hospital and ambulatory settings during their clinical clerkships. Monitoring these experiences is essential to ensure that students acquire an appropriate mix of clinical experiences. Attempts made over the last 25 years¹ to document the clinical experiences of students have used such recording devices as casebooks,² logbooks,^3- 8 optical scan forms,^9- 10 handheld or palmtop computers,^9,11 and pocket-sized encounter cards.¹² These studies have been limited to small samples in isolated clerkships over brief time periods^{3,8- 9,13- 24} and they have largely ignored the severity of illness.

Dramatic changes in the financing and delivery of health care during the past decade have altered the clinical environments in which medical education occurs.^25- 28 As medical education becomes increasingly decentralized, clinical education has partially shifted from a tertiary inpatient setting to community-based and ambulatory sites.^3,29- 31 In this kind of educational environment, it is still essential that medical students encounter a variety of disease entities and are given the opportunity to perform basic diagnostic and therapeutic maneuvers.

Effective curriculum management requires a valid and reliable system to document the range and type of students' clinical experiences. Only by monitoring students' opportunities for clinical encounters with a diverse mix of patients can informed decisions be made regarding the appropriateness of a teaching network, training sites, and the balance between inpatient and ambulatory activities. As medical schools review their learning objectives to better define the competencies needed by future physicians,³² it will become even more important to document the clinical educational opportunities offered to medical students.

We investigated the applications of a new system to document key aspects of the clinical experiences of third-year medical students in 3 clerkships (family medicine, pediatrics, and internal medicine) at Jefferson Medical College, Philadelphia, Pa. We provide evidence of validity and reliability, as well as representative examples of the information produced by the system.

Jefferson's network of affiliated clinical sites includes urban, suburban, and rural locations throughout Pennsylvania, Delaware, and New Jersey, providing training in family medicine (6 sites), pediatrics (5 sites), and internal medicine (8 sites). Family medicine is a 6-week, predominantly ambulatory rotation; pediatrics is a 6-week rotation including a mix of inpatient and ambulatory experiences; and internal medicine is a 12-week, predominantly inpatient rotation.

This project is a collaborative effort involving Jefferson Medical College's Center for Research in Medical Education and Health Care, the Office of the Dean, and clerkship directors. The data collection system was developed first for the family medicine clerkship and then adapted for use in pediatrics and internal medicine. Data collection began in 1993 for family medicine, in 1994 for pediatrics, and in 1997 for internal medicine. The key issues addressed by the Clinical Encounter Project include the location of encounters, case and severity mix of patients, procedures and activities performed by students, and students' workloads.

At the start of each clerkship students are provided with packets of 5 × 8-inch computer-read patient encounter cards and a pocket-sized instruction booklet. While the general structure of the cards and booklets is standard, each department has made modifications to reflect its mix of patients. In addition to instructions for completing the cards, the booklet contains a diagnosis list using a small subset of codes from International Classification of Diseases, Ninth Revision, Clinical Modification.³³ These codes are used by students to translate the diagnoses that they write down on the card into numerical codes for purposes of data analyses. Also in the booklet are examples of the logic and application of Disease Staging for classifying illness severity.^34- 35 Disease Staging is a clinically based classification system with more than 400 disease categories that includes disease etiology, organ system involvement, and severity of complications. Severity for specific medical problems is defined in relation to the risk of organ failure or death. The classification separates each disease into 3 stages based on the severity of its pathophysiologic manifestations. Stage 1 defines a disease with no complications (eg, appendicitis without complications); stage 2 is a disease with local complications (eg, pneumococcal pneumonia with empyema); stage 3 is a disease with multiple site involvement or systemic complications (eg, asthma with respiratory failure).

Students were instructed to complete a card each time they participated in the care of a patient from whom they obtained a history or performed a physical examination. For in-hospital locations (internal medicine and part of pediatrics), multiple encounters with the same patient were recorded on the same card. Students were instructed to encode each patient's age and sex, location of encounter, level of involvement and supervision, and diagnostic procedures that were performed or observed. Principal diagnosis with severity of illness was recorded, with space provided for up to 4 secondary diagnoses. Cards were completed at the end of patient care sessions as charting was completed, or from the students' patient care notes. They were returned and scanned weekly.

Summary reports are prepared twice a year for each department. These reports are reviewed by the clerkship directors and reviewed annually at departmental affiliations meetings. These meetings, as well as the usefulness of the data to the faculty themselves, help to ensure the integrity of the system. Individual reports, with peer group comparisons, are available to the students.

The total cohort included 647 third-year students (98% of the total classes) at Jefferson Medical College who had completed clerkships in either family medicine, pediatrics, or internal medicine in academic years 1997 to 1998, 1998 to 1999, and 1999 to 2000 at Thomas Jefferson University Hospital and 16 affiliated clinical sites. The remaining 2% included those who either failed to properly complete or turn in their cards.

We included in our analyses those students who completed at least 30 cards in family medicine, 26 in pediatrics, or 18 in internal medicine. These criteria were set to represent 75% of the completed cards expected to be received from each student based on clerkship directors' judgment after 1 to 2 years of experience with the project (the threshold was lower for internal medicine due to multiple encounters with the same patient using a single card). While 647 students met the inclusion criteria in at least 1 clerkship, the sample size for each clerkship ranged from 582 (90%) in family medicine, to 531 (82%) in internal medicine, to 469 (72%) in pediatrics.

Frequencies and percentages of diagnostic categories and disease severity were compared using the χ² test for the significance of the association between clerkships and severity of diseases for each diagnostic category. The z test for proportions was used for pairwise comparisons of proportions for each stage of the diagnostic categories. Statistical analyses were performed using SAS version 6.12 (SAS Institute, Cary, NC).

Students returned 86 011 cards (mean, 133 cards per student): 48 367 in family medicine, 22 604 cards representing 48 799 encounters in pediatrics, and 15 040 cards representing 90 240 encounters in internal medicine, or a total of 187 406 encounters. The mean number of encounters per student was 83 in family medicine, 104 in pediatrics, and 170 in internal medicine. The mean number of encounters for internal medicine is larger because the clerkship spans 12 weeks, while the others span only 6 weeks. Family medicine and pediatrics are primarily ambulatory clerkships, having, respectively, 86% and 66% of patient encounters in office environments. Students in internal medicine reported having more than two thirds of their encounters in a hospital and 30% in an emergency department.

Geriatric patients comprised 50% of encounters in internal medicine and 22% in family medicine, while approximately 20% of the family medicine encounters were with either children or adolescents. The proportions of females encountered were 60%, 47%, and 45% in family medicine, pediatrics, and internal medicine, respectively.

The percentages of students who reported encountering at least 1 patient in each clerkship having 1 of the 10 most frequently encountered principal and secondary diagnoses are displayed in Figure 1. Differences among the clerkships primarily reflected the locations of encounters and the ages of patients. The most frequently encountered principal diagnoses in family medicine included essential hypertension, diabetes mellitus, back strain, depression, and sinusitis. Asthma, otitis media, diarrhea/gastroenteritis, and upper respiratory tract infection were the most frequently encountered diagnoses in pediatrics. Pneumonia and cardiac diagnoses (eg, heart failure, arrhythmias, acute myocardial infarction) were the most frequently encountered principal diagnoses in the predominantly inpatient internal medicine clerkship.

Less than half of the students saw any patients with the common medical problems that have high prevalence rates as reported by the Centers for Disease Control and Prevention.³⁶ In pediatrics, for instance, less than half of the students saw a patient with a primary diagnosis of conjunctivitis (data not shown); in internal medicine fewer than half encountered syncope as a primary diagnosis. Peptic ulcer disease is another common condition that was encountered infrequently. Only 11% of students encountered any patients with this condition during family medicine clerkships; 6% encountered it during internal medicine.

Students were instructed to record multiple diagnoses or medical problems for each patient, when appropriate. Figure 1 also displays data for these secondary diagnoses. For example, 42% of students reported encountering a patient with a primary diagnosis of diabetes mellitus during their internal medicine clerkship, and 88% reported seeing a patient with diabetes mellitus as either the primary or secondary diagnosis.

Data on severity of disease are reported in Table 1 for family medicine and internal medicine rotations. Although the rate of encountering those disease categories was similar in the 2 clerkships, severity of the same diseases that students encountered varied. We report data for family medicine and internal medicine because of the compatibility of diseases.

For example, of 400 students in family medicine who had seen patients with diabetes mellitus, 281 (70%) encountered patients in stage 1 of this disease, while in internal medicine only 50 of 193 (26%) saw such patients. However, the converse was true for stage 3 of diabetes mellitus; 58% of students in the internal medicine clerkship, compared with 35% in family medicine, reported encountering a patient in this stage of the disease. Associations between severity of disease encountered in different clerkships were statistically significant by χ² test. With the exception of stage 2 essential hypertension, all other differences in percentages across clerkships were statistically significant by z test for proportions. Data reported in Table 1 indicate that during the family medicine clerkship, students were more likely to encounter early stages of disease and less likely to encounter advanced stages. The opposite was found for internal medicine.

Each department instructed students to track diagnostic and therapeutic tasks, including components of the history and physical examination, special activities such as performance of a Mini-Mental Status Examination, procedures that the students might be expected to perform in a given clerkship (eg, venipuncture), as well as procedures they might observe (eg, sigmoidoscopy). The number and percentage of students who performed the most frequently reported procedures in each clerkship, as well as the mean and median number performed per student, are shown in Table 2. For example, 84% (489/582) of students in family medicine clerkships and 82% (385/469) in pediatrics clerkships performed breast examinations, while 37% (198/531) in internal medicine did so. In family medicine, 81% (472/582) reported doing a pelvic examination. None performed this procedure in pediatrics, and only 13% (67/531) performed the procedure in internal medicine. Health promotion/counseling activities were performed by 91% (532/582) of the students in family medicine, 93% (436/469) in pediatrics, and 77% (410/531) in internal medicine clerkships.

On average, each student performed 9 breast examinations (median, 6), and offered health promotion or counseling 38 times (median, 28) during the 3 clerkships. Some procedures were observed by students more often than performed. For example, while 40% of students reported performing an electrocardiogram in either family medicine or internal medicine, 80% reported observing at least 1 electrocardiographic procedure (data not shown).

To summarize experiences across clerkships, we analyzed the data for 358 students who met our inclusion criteria in all 3 clerkships (Table 2). For example, 13% of students in family medicine, 22% in pediatrics, and 38% in internal medicine reported administering a purified protein derivative (PPD) skin test for tuberulosis (Table 2). However, when we examined across all 3 clerkships, 55% of students reported administering a PPD.

To assess the validity of students' diagnoses, we selected 3 days randomly and examined 112 encounter cards completed by 15 students in the family medicine clerkship. Three of the authors (D.Z.L., C.R., F.W.M.) reviewed charts set aside from office sessions. The reviewers compared the attending faculty's principal diagnosis from the chart to the diagnosis on the previously submitted cards. Concordance was 77% between the students' recorded principal diagnoses and the faculty's confirmation of diagnosis. The concordance rate increased to 97% when either principal or secondary diagnoses were considered. These concordance rates are much higher than those reported in a psychiatry clerkship (range, 33%-49%).⁸ Our concordance rates are comparable with those reported in a primary care clerkship (88% and 87%).²⁴ Findings reported in Figure 1, regarding the most prevalent diseases in each clerkship, provide further support for the validity of the system (eg, concordance between expecting and actually encountering a large number of patients with hypertension in family medicine, otitis media in pediatrics, and pneumonia in internal medicine clerkships). The expected variation in disease severity (Table 1) reflects the settings in which encounters were recorded (outpatient, therefore less disease severity in family medicine; inpatient, more disease severity in internal medicine), providing further validation evidence.

To support the reliability of this documentation system we compared the pattern of diagnostic categories for each clerkship in different years. Similar patterns of diagnostic encounters were found for each clerkship across the 3 academic years (1997-1998, 1998-1999, 1999-2000), indicating that the pattern of patient encounters remained stable over time, supporting the reproducibility of the results. For example, in pediatrics, 8 of the 10 most frequently reported diagnoses/problems were on the list in all 3 years.

Previous studies of student encounters with patients have focused on a single discipline such as family medicine,^3,9,13- 17 internal medicine,^18- 20 or psychiatry,⁸ or on the ambulatory setting.^21- 22,24 Exceptions are a comparison of the encounters of 40 students in family medicine to those in other clerkships,²³ and a study of 17 918 patient encounters during family medicine and internal medicine rotations at 2 medical schools for 1 year.³⁷ Our study reports on a system that has been in place for 8 years and that assesses 3 major clerkships in terms of patient disease severity.

Although the present study focused on only 3 clerkships, we believe that this methodology could be applied to any clerkship that involves encounters with patients. In our educational network this system has been initiated in the third-year clerkships in surgery and obstetrics/gynecology and will soon begin in psychiatry. Objective data on types of clinical experiences could be used to identify curriculum deficiencies, to design supplementary experiences, and to assess the results of interventions designed to enhance various aspects of clinical encounters.

Focused information about the clinical experiences of each medical student can be used in the planning, implementation, and evaluation of educational experiences as well as in counseling individual students. Trend data are essential in the ongoing evaluation of the suitability and effectiveness of each educational site. Data collected by this documentation system can also identify the types of patient problems that students do not encounter and procedures they do not perform. To address clerkship deficiencies and monitor a student's progress, the student could be asked to evaluate a patient with critical problems and perform some of these procedures in the same or other clerkships.

For instance, in an earlier study we noted that in a family medicine clerkship, female students were not performing male genital examinations as frequently as were their male peers.³⁸ By monitoring clinical experiences we were able to correct this deficiency and improve female students' participation. Similarly, in pediatrics, few students reported doing such office procedures as screening patients' hearing and vision. A checklist is now in place at all training sites to remind students and faculty to include these procedures in students' clinical experiences.

Experiences within both inpatient and outpatient educational settings are essential to expose each student to a diversity of common medical problems and to a spectrum of illness severity within specified diagnostic categories. Interviewing a patient with diabetes as an outpatient to address glycemic control and tertiary prevention is a very different educational experience from that of caring for a patient admitted to the hospital for management of diabetic ketoacidosis. It also differs from the educational experience of caring for a patient with a myocardial infarction who also has diabetes. Recording multiple problems highlights the educational importance of each diagnosis.

In summary, this surveillance system provides insight into disease frequency and the diversity of case mix and has the unique feature of taking into account the severity of medical problems that students encounter. Monitoring students' educational opportunities in different clerkships and at different educational sites is necessary if educators are to understand and optimize those clinical experiences.