Association Between Licensure Examination Scores and Practice in Primary Care FREE

The medical profession ensures the basic competence of physicians by requiring them to pass licensing and certifying examinations.¹ Although it is generally assumed that these examinations predict how physicians will practice in the future,² the data in support of this assumption are mostly indirect. Physicians who have more training in a discipline are more knowledgeable^3- 4 and achieve higher scores in their respective discipline on recertification examinations.⁵ More knowledgeable physicians are more likely to adhere to evidence-based guidelines in the delivery of care^6- 7 and achieve better patient outcomes.⁶ Certification status, which represents pass/fail status on certification examinations, is an important predictor of quality of care.^8- 9

It is unknown, however, whether scores achieved by physicians with the same training and specialty are predictive of future performance. A prior study⁸ found that scores on an internal medicine certification examination predicted colleagues' ratings of the quality of care delivered by internists 5 to 8 years later. However, little is known about the relationship between examination scores and more objective measures of quality of care. Assessing this relationship is relevant because important gaps exist between optimal and actual practice in the delivery of preventive care,¹⁰ in the management of acute and chronic disease,^7,11- 12 and in the quality of drug treatment.¹³ All of these are deficiencies that could potentially be predicted by licensing examinations.

We previously reported that physicians who achieved higher scores on the Québec family medicine certification examination were more likely to refer women for mammography screening, to prescribe more disease-specific medication and fewer symptomatic and contraindicated drugs, and to refer more of their patients for consultation.¹⁴ However, assessment of these outcomes was limited to the first 18 months of practice. We used this opportunity to determine if the association between family medicine certification examination scores and practice performance persisted with increasing practice experience.¹⁴ We could not find any studies that explored the relationship between earlier licensing examinations taken at the end of medical school and future practice performance. We investigated whether such scores predict clinical behaviors 4 to 7 years later.

All Canadian provinces provide a universal health insurance program that covers the costs of medical care for provincial residents. In Québec, 14 500 physicians provide services to 7.4 million residents of the province, for whom 92% of services and 93% of physicians are paid by the Québec health insurance agency (Régie de l'assurance maladie du Québec; RAMQ) on a fee-for-service basis.¹⁵

A cohort of all family physicians who passed the Québec family medicine certification examination between 1990 and 1993, and entered fee-for-service practice in Québec, was followed up for the first 4 to 7 years of practice. Annual measures of each physician's practice performance were used to test associations between examination scores and practice performance. Salaried physicians were excluded because there was no accurate way to identify all their patients, as were physicians who subsequently trained in another specialty. Potentially eligible physicians were identified by the Québec College of Physicians, and physicians' license numbers were provided to the RAMQ to retrieve data on practice activity. Annual measures of each physician's clinical behaviors were used to test associations between examination scores and practice performance. Data on all patients seen by each physician, for each year of practice, were retrieved from the RAMQ to assess annual practice activity and case-mix differences between physician practices. For each cohort, the follow-up period was between July 1 of the certification examination and December 31, 1996, providing practice assessment for 4 years (1993 cohort) to 7 years (1990 cohort). The study protocol was approved by the institutional review board at McGill University, the Provincial Access to Information Office, and the RAMQ legal counsel.

The Québec Licensing Examination (QLEX) comprises the College of Family Physicians of Canada Certification Examination (CFPCex) and the Québec Objective Structured Clinical Examination (OSCE).¹⁶ Between 1990 and 1993, the CFPCex assessed diagnosis, management, and prevention with multiple-choice questions and short-answer management problems, and assessed communication skills with a simulated patient. The OSCE measured clinical problem-solving skills by direct observation of performance in 25 standardized patient encounters, rated by physician examiners.¹⁶ To pass the examination, a score of 60% was required in diagnosis, management, and communication; 50% in prevention; and an OSCE score greater than 2 SDs below the mean. The overall reliability of examination scores varied from a low of 0.27 for prevention to a high of 0.72 for the OSCE.¹⁴ Scores were standardized to adjust for differences in the difficulty of examinations using the reference group approach.¹⁴ The College of Physicians provided the license number, medical school, and encrypted examination scores to the RAMQ, who then retrieved data on practice activity and linked this with score files through encrypted physician and patient identifiers.

The Medical Council of Canada Qualifying Examination (MCCQE) is used to test an individual's competence to enter postgraduate training. It is generally taken during the final year of medical school. A passing score on the MCCQE is required for unrestricted licensure in all Canadian provinces except Québec, although most Québec graduates take the examination. The 1988-1991 examinations consisted of 450 multiple-choice questions, which tested knowledge in medicine, surgery, obstetrics/gynecology, psychiatry, pediatrics, and preventive medicine, and also included 28 to 35 patient management problems that tested competence in clinical problem solving.¹⁷ To enable comparisons between the MCCQE and the Québec certification examination, the 1258 multiple-choice questions used between 1988 and 1991 were reclassified by the test committees as diagnosis, management, or other knowledge. Test committees also identified management items that tested drug-specific knowledge. Standardized ability scores were created for each subscore using the same reference group approach as the certification examination.¹⁸ The overall score reliability was 0.92, and subscore reliabilities were 0.71 for prevention, 0.63 for diagnosis, 0.73 for management, and 0.48 for drug knowledge. The Medical Council of Canada linked score data to the Québec cohort file by name, sex, and birth date.

Data Sources and Retrieval. Four previously validated¹⁹ health administrative databases, linked by encrypted beneficiary identifiers, were used to assess practice performance.^20- 21 The registrant database provided patients' age, sex, postal code, and date of death. The medical services database provided type, location, diagnosis, treating and referring physician, and date of all services delivered on a fee-for-service basis. The prescription database provided drug, dose, duration, prescribing physician, and date of each prescription dispensed for patients aged 65 years or older. The hospitalization database provided records of all hospital discharges including discharge diagnoses and admission and discharge dates. The 1991 census data was linked by 6-digit postal code to the registrant database to measure mean family income and educational achievement in the residential area of each patient.²² For each physician, the medical services claims files were used to identify all patients seen by the physician from the licensure date to December 31, 1996. For each patient, the RAMQ retrieved demographic data, as well as all medical services, prescriptions, and hospitalizations provided during the follow-up period. We also obtained data for the year prior to the first contact with the study physician to provide an independent assessment of patient characteristics that could not have been influenced by the study physician.¹⁴

Indicators of Practice Performance. We assessed 6 performance indicators in each follow-up year, selected on the basis of unexplained practice variation, and/or their association with the outcomes or costs of care.¹⁴ First, a mammography screening rate was used to assess preventive care, because physicians who screen for breast cancer are also more likely to perform other preventive services.²³ Second, continuity of care was selected because of its importance in prevention and chronic disease management,^24- 27 and to test the hypothesis that communication and management skills are predictive of better continuity.^28- 29 Third, the differences between disease-specific and symptom-relief prescribing rate and contraindicated prescribing rate were used as indicators of the quality of acute and chronic disease management because variation in disease-specific relative to symptom-relief prescribing rate³⁰ has been linked to diagnostic competence.¹⁴ Fourth, contraindicated prescribing, which accounts for 20% of drug-related adverse events, may be caused by deficiencies in physician knowledge.³¹ Finally, consultation rate was used as an indicator of resource use because referral determines access to higher cost specialty care,³² and primary care physicians with higher self-reported competence appear more likely to refer patients for specialty consultation.^33- 34

Annual mammography screening rate was defined as the proportion of eligible women in the primary care practice who were referred for a bilateral mammogram by the physician. Eligible women were between age 50 and 69 years, had no prior diagnosis of breast cancer, breast disease, or diagnostic mammogram, were due for screening, and were not receiving primary care from a gynecologist or obstetrician.

Annual continuity of care was defined as the mean proportion of visits that were made to or referred by the study physician by all patients in the primary care practice population. Using the medical services claims, we determined the proportion of all visits in the year for each patient that were made either to the study physician or to other physicians based on study physician referral. Each patient's proportion was weighted by the square root of the total number of visits made by the patient in the year. An overall mean annual continuity of care index for each physician's practice was calculated as the weighted mean of individual patient's proportions.

Mammography screening and continuity of care were assessed only in the primary care practice population, which consisted of patients seen in an office or clinic in the respective year, for whom the study physician had provided an annual physical or major (3 system) assessment. Consultation was assessed in all ambulatory patients for whom the study physician had billed at least 1 outpatient, office practice, or emergency department visit. Prescribing rates were assessed in ambulatory patients aged 65 years or older for whom complete information was available on all prescriptions dispensed.

Annual difference between disease-specific and symptom-relief prescribing rates was determined by examining medications prescribed by the study physician to all elderly patients. Disease-specific medication was defined as drugs that would rarely be prescribed without an investigation-confirmed disease (eg, anticoagulants, anticonvulsants, antidepressants, antihypertensives, medications for cardiovascular disease, asthma, and Parkinson disease, corticosteroids, diuretics, antiglaucoma medication, hypoglycemic medication, and thyroid medication). Symptom-relief medication was defined as drugs that relieve symptoms, but have little impact on the disease process (eg, nonsteroidal anti-inflammatory medications, benzodiazepines, low-dose narcotic analgesics) using the McGavock classification.³⁰

Annual contraindicated prescribing rate was the proportion of ambulatory elderly patients for whom the study physician prescribed a relatively contraindicated medication. These were defined by an updated expert review^35- 36 as 30 drugs that should be avoided in elderly patients because of possible toxic effects.

Annual consultation rate was the proportion of all ambulatory patients in the respective calendar year referred, at least once, to a specialist by the respective primary care physician, based on medical services claims. To be reimbursed for a consultation visit, the consultant must record a valid license number for the referring physician.

Time in Practice. Physicians' preceding practice experience was represented as a time-dependent covariate, based on a count of the number of previous months that the physician had at least 1 fee-for-service billing in the medical services claims file.

Relevant data were retrieved for each eligible patient for the 12-month period preceding the first contact with the study physician. Individual characteristics of all patients seen in a given year by the physician were then aggregated to adjust for between-physician differences in practice population case-mix. Each practice population was characterized by age and sex distribution, mean family income, educational achievement, geographic access to health care,¹⁴ propensity for the use of health care services,³⁷ comorbidity (Charlson index),³⁸ and hospitalization rates in the previous year. These annual case-mix measures were modeled as time-dependent covariates. For contraindicated prescribing, no adjustments were made for case-mix because the prescription of relatively contraindicated medication is rarely justified by patient characteristics.¹⁴

Relationships between examination scores and practice performance were tested using multiple linear and Poisson regression for repeated measures using generalized estimating equations.³⁹ Physicians were the unit of analysis. Outcome variables were the 5 annual measures of practice performance. An autoregressive first-order correlation structure of residuals was used to characterize the interdependence between annual performance measures for each physician. To reduce imprecision in outcome measurement, physicians were excluded in years in which they had fewer than 5 patients, and the logarithm of the number of patients in the practice population in a given year was used as a weight in the analysis. Linearity assumptions were evaluated by testing the statistical significance of the quadratic component. Poisson regression was used to assess rates of contraindicated prescribing because such events were rare. The SEs were empirically estimated to account for overdispersion. Annual measures of case-mix were included in all analyses, as were indicators of the medical school to conservatively assess the impact of variation in examination scores within each medical school. To determine whether the association between examination scores and practice outcomes diminished over time, the interactions between examination score and practice experience were tested. For significant relationships between examination score and outcomes that persisted over time, the cumulative difference in the number of outcomes per 1000 patients followed up over the first 5 years in practice by high-scoring (2 SDs above the mean) rather than low-scoring (2 SDs below the mean) physicians was estimated. Regression coefficients for the score-outcome relationship were used to determine the expected annual difference in rates corresponding to a 4-SD difference in examination scores, and then the result was multiplied by 5 to estimate the cumulative impact over 5 years. P<.05 was used as the level of statistical significance. We used SAS statistical software to perform our analyses (Version 8.0, SAS Institute Inc, Cary, NC).

Between 1990 and 1993, a total of 944 family physicians passed the QLEX, 920 (97.5%) started practice in Québec, 912 (96.7%) entered a fee-for-service practice, of whom 58.1% were female (Table 1). Overall, 85.8% of physicians took the MCCQE during the final year of medical school. Québec medical school graduates had modestly lower MCCQE scores in comparison with other Canadian graduates (mean [SD], −0.04 [0.84] vs 0.09 [0.92]; P<.001). Graduates who entered family medicine or general practice training in Canada had slightly lower standardized scores than those entering specialty programs (mean [SD], 0.02 [0.95] vs 0.13 [0.84]; P<.001). Québec family physicians who did not take the MCCQE had lower certification examination scores than those who did (mean [SD], −0.76 [1.18] vs −0.03 [1.02]; P<.001). Mean scores achieved on both the QLEX and MCCQE were equivalent or slightly lower than in the reference group of first-time takers, with a typical range of 6 to 7 SDs (Table 1). The Pearson correlation between the MCCQE and QLEX was 0.55, and ranged from 0.26 (clinical problem solving) to 0.49 (management).

Physicians practiced in a mean of 2.6 (year 1-2) to 3.2 (year 5-7) different types of settings (Table 2). During the first 4 to 7 years of practice, an increasing proportion of physicians moved their primary practice base from rural to urban populations, and established private office practice. After the first 2 years, practice size, number of days worked per year, and visits per day continued to increase, but at a slower rate.

The 912 physicians billed for 3.4 million different patients (45.9% of the Québec population), of whom 1.4 million were in their primary care practice populations, 385 321 were elderly, and 119 866 were women eligible for mammography screening.

The relationship between the QLEX and mammography screening was sustained over the first 4 to 7 years in practice (Table 3). The significant interaction between certification examination score and practice experience indicated that the strength of the relationship increased over time. For every SD increase in QLEX score, the mammography screening rate increased by 13.3 women per 1000 in years 1 to 2 of practice, 21.7 by years 3 to 4, and 19.2 by years 5 to 7. The persistence of this relationship means that, during the first 5 years of practice, high-scoring physicians would be expected to order 347 more mammograms per 1000 women than low-scoring physicians. In contrast, there was no relationship between QLEX scores and continuity of care. Overall, however, family physicians provided or coordinated only 28.3% of all visits made by patients in their primary care practice population.

Consultation rate showed a persistent, but modest, association with QLEX score over the first 7 years of practice (Table 3). Each SD increase in score was associated with an additional 2.92 referrals for specialty consultation per 1000 patients seen per year, resulting, over the first 5 years, in 58 more referrals per 1000 patients by high-scoring than low-scoring physicians.

Diagnosis and management subscores of the QLEX were the only significant predictors of prescribing outcomes. Higher scores were associated with higher rates of disease-specific relative to symptom-relief prescribing, and a lower risk of contraindicated prescribing, although the latter did not achieve statistical significance (Table 3). The strength and significance of the associations between examination scores and practice outcomes increased when medical school was excluded from the model, because there were systematic differences in scores for graduates from different medical schools.

Scores on the MCCQE, taken at the end of medical school, had a similar pattern of relationships as the certification examination (Table 4). For example, the increase in mammography screening rate per SD increase in score was 16.8 per 1000 for the MCCQE score (Table 4) and 17.4 per 1000 for the overall QLEX score (Table 3). The diagnosis subscore was the strongest predictor of differences in the rates of disease-specific and symptom-relief prescribing in both the MCCQE and the QLEX. The drug knowledge subscore was the only significant predictor of contraindicated prescribing—reducing the risk of contraindicated prescribing by 12% per SD increase in score. During the first 5 years, a high-scoring physician would be expected to write 85 fewer contraindicated prescriptions per 1000 elderly patients than a low-scoring physician.

We examined the relationship between licensure and certification examination scores and practice performance in a sample of newly certified family physicians. Linked databases from a universal health care system allowed us to follow-up all licensed physicians, and all their patients in their first 4 to 7 years of practice. Thus, we minimized selection biases that could confound the assessment of relationships. We were also able to assess the population impact of potential licensure regulations. The most important findings of this study were that the relationships between certification examination scores were sustained through the first 4 to 7 years of practice, and that examinations taken in the final year of medical school were also significant predictors of practice performance. However, the use of administrative databases had limitations. The databases restricted the type of performance indicators that could be measured and we were limited in our ability to adjust for confounding by case-mix, but as case-mix was not associated with examination scores, it is unlikely that this introduced substantial bias in the results. These results have several implications for licensing bodies.

Such outcome data could be incorporated into the establishment of examination passing standards so that standard-setting participants can weigh their usual judgments of test content and pass rates against the consequences for patients.^40- 42 For example, an increase in the passing criterion of only 1 SD in drug knowledge would have resulted in failing 16 additional physicians on the MCCQE licensing examination over 4 years. It would also have reduced the expected risk of contraindicated prescriptions for elderly patients seen by these physicians by approximately 42% (from 4.7% for these low-scoring physicians to 3.3% for an average physician). A reduction in risk of this magnitude is equivalent to or greater than that reported for the most effective form of continuing medical education for physicians with high rates of inappropriate prescribing.⁴³ To minimize the risk of adverse outcomes for the population, passing standards may need to be established for individual components of the examination, such as drug knowledge, diagnosis, and management.

Consultation rates increased linearly with examination score. Physicians tend to report higher referral rates in clinical areas in which they felt more competent.^33- 34 More competent physicians may be more aware of their limitations. However, most examinations do not test whether an individual knows when to refer. This may be an important area for test development, as the risk of outcomes related to medical errors has been shown to increase when physicians practice beyond their areas of competence.⁴⁴

The finding that examinations taken at the end of medical school were also predictive of future practice may be useful for residency program directors, who could use this information to target learning opportunities to areas of deficiency. Medical school educators may also be able to identify persons more likely to experience difficulty in practice earlier in training, as scores on medical school examinations are strongly correlated with scores on licensing examinations.⁴⁵

Future research should investigate methods of establishing outcomes-based passing scores, and develop more refined measures of quality of care in disease-specific populations. Longitudinal follow-up studies should be conducted in multijurisdictional cohorts of medical graduates to explore the aspects of admissions, undergraduate, and postgraduate training that may influence practice.