0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Original Contribution |

Association Between Licensure Examination Scores and Practice in Primary Care FREE

Robyn Tamblyn, PhD; Michal Abrahamowicz, PhD; W. Dale Dauphinee, MD; James A. Hanley, PhD; John Norcini, PhD; Nadyne Girard, MSc; Paul Grand'Maison, MD; Carlos Brailovsky, MD, PhD
[+] Author Affiliations

Author Affiliations: Departments of Medicine (Dr Tamblyn) and Epidemiology and Biostatistics (Drs Tamblyn, Abrahamowicz, and Hanley, and Ms Girard), McGill University, Montreal, Québec; Medical Council of Canada, Ottawa, Ontario (Dr Dauphinee); Foundation for Advancement of International Medical Education and Research, Philadelphia, Pa (Dr Norcini); Department of Family Medicine, University of Sherbrooke, Sherbrooke, Québec (Dr Grand'Maison); and Centre d'évaluation des sciences de la santé, University of Laval, Ste-Foy, Québec (Dr Brailovsky).


JAMA. 2002;288(23):3019-3026. doi:10.1001/jama.288.23.3019.
Text Size: A A A
Published online

Context Standards for licensure are designed to provide assurance to the public of a physician's competence to practice. However, there has been little assessment of the relationship between examination scores and subsequent practice performance.

Objective To determine if there is a sustained relationship between certification examination scores and practice performance and if licensing examinations taken at the end of medical school are predictive of future practice in primary care.

Design, Setting, and Participants A total of 912 family physicians, who passed the Québec family medicine certification examination (QLEX) between 1990 and 1993 and entered practice. Linked databases were used to assess physicians' practice performance for 3.4 million patients in the universal health care system in Québec, Canada. Patients were seen during the follow-up period for the first 4 years (1993 cohort of physicians) to 7 years (1990 cohort of physicians) of practice from July 1 of the certification examination to December 31, 1996.

Main Outcome Measures Mammography screening rate, continuity of care index, disease-specific and symptom-relief prescribing rate, contraindicated prescribing rate, and consultation rate.

Results Physicians achieving higher scores on both examinations had higher rates (rate increase per SD increase in score per 1000 persons per year) of mammography screening (β for QLEX, 16.8 [95% confidence interval {CI}, 8.7-24.9]; β for Medical Council of Canada Qualifying Examination [MCCQE], 17.4 [95% CI, 10.6-24.1]) and consultation (β for QLEX, 4.9 [95% CI, 2.1-7.8]; β for MCCQE, 2.9 [95% CI, 0.4-5.4]). Higher subscores in diagnosis were predictive of higher rates in the difference between disease-specific and symptom-relief prescribing (β for QLEX, 3.9 [95% CI, 0.9-7.0]; β for MCCQE, 3.8 [95% CI, 0.3-7.3]). Higher scores of drug knowledge were predictive of a lower rate (relative risk per SD increase in score) of contraindicated prescribing for MCCQE (relative risk, 0.88; 95% CI, 0.77-1.00). Relationships between examination scores and practice performance were sustained through the first 4 to 7 years in practice.

Conclusion Scores achieved on certification examinations and licensure examinations taken at the end of medical school show a sustained relationship, over 4 to 7 years, with indices of preventive care and acute and chronic disease management in primary care practice.

The medical profession ensures the basic competence of physicians by requiring them to pass licensing and certifying examinations.1 Although it is generally assumed that these examinations predict how physicians will practice in the future,2 the data in support of this assumption are mostly indirect. Physicians who have more training in a discipline are more knowledgeable3,4 and achieve higher scores in their respective discipline on recertification examinations.5 More knowledgeable physicians are more likely to adhere to evidence-based guidelines in the delivery of care6,7 and achieve better patient outcomes.6 Certification status, which represents pass/fail status on certification examinations, is an important predictor of quality of care.8,9

It is unknown, however, whether scores achieved by physicians with the same training and specialty are predictive of future performance. A prior study8 found that scores on an internal medicine certification examination predicted colleagues' ratings of the quality of care delivered by internists 5 to 8 years later. However, little is known about the relationship between examination scores and more objective measures of quality of care. Assessing this relationship is relevant because important gaps exist between optimal and actual practice in the delivery of preventive care,10 in the management of acute and chronic disease,7,11,12 and in the quality of drug treatment.13 All of these are deficiencies that could potentially be predicted by licensing examinations.

We previously reported that physicians who achieved higher scores on the Québec family medicine certification examination were more likely to refer women for mammography screening, to prescribe more disease-specific medication and fewer symptomatic and contraindicated drugs, and to refer more of their patients for consultation.14 However, assessment of these outcomes was limited to the first 18 months of practice. We used this opportunity to determine if the association between family medicine certification examination scores and practice performance persisted with increasing practice experience.14 We could not find any studies that explored the relationship between earlier licensing examinations taken at the end of medical school and future practice performance. We investigated whether such scores predict clinical behaviors 4 to 7 years later.

Context

All Canadian provinces provide a universal health insurance program that covers the costs of medical care for provincial residents. In Québec, 14 500 physicians provide services to 7.4 million residents of the province, for whom 92% of services and 93% of physicians are paid by the Québec health insurance agency (Régie de l'assurance maladie du Québec; RAMQ) on a fee-for-service basis.15

Design and Study Population

A cohort of all family physicians who passed the Québec family medicine certification examination between 1990 and 1993, and entered fee-for-service practice in Québec, was followed up for the first 4 to 7 years of practice. Annual measures of each physician's practice performance were used to test associations between examination scores and practice performance. Salaried physicians were excluded because there was no accurate way to identify all their patients, as were physicians who subsequently trained in another specialty. Potentially eligible physicians were identified by the Québec College of Physicians, and physicians' license numbers were provided to the RAMQ to retrieve data on practice activity. Annual measures of each physician's clinical behaviors were used to test associations between examination scores and practice performance. Data on all patients seen by each physician, for each year of practice, were retrieved from the RAMQ to assess annual practice activity and case-mix differences between physician practices. For each cohort, the follow-up period was between July 1 of the certification examination and December 31, 1996, providing practice assessment for 4 years (1993 cohort) to 7 years (1990 cohort). The study protocol was approved by the institutional review board at McGill University, the Provincial Access to Information Office, and the RAMQ legal counsel.

Family Medicine Certification Examination

The Québec Licensing Examination (QLEX) comprises the College of Family Physicians of Canada Certification Examination (CFPCex) and the Québec Objective Structured Clinical Examination (OSCE).16 Between 1990 and 1993, the CFPCex assessed diagnosis, management, and prevention with multiple-choice questions and short-answer management problems, and assessed communication skills with a simulated patient. The OSCE measured clinical problem-solving skills by direct observation of performance in 25 standardized patient encounters, rated by physician examiners.16 To pass the examination, a score of 60% was required in diagnosis, management, and communication; 50% in prevention; and an OSCE score greater than 2 SDs below the mean. The overall reliability of examination scores varied from a low of 0.27 for prevention to a high of 0.72 for the OSCE.14 Scores were standardized to adjust for differences in the difficulty of examinations using the reference group approach.14 The College of Physicians provided the license number, medical school, and encrypted examination scores to the RAMQ, who then retrieved data on practice activity and linked this with score files through encrypted physician and patient identifiers.

Medical Council of Canada Qualifying Examination

The Medical Council of Canada Qualifying Examination (MCCQE) is used to test an individual's competence to enter postgraduate training. It is generally taken during the final year of medical school. A passing score on the MCCQE is required for unrestricted licensure in all Canadian provinces except Québec, although most Québec graduates take the examination. The 1988-1991 examinations consisted of 450 multiple-choice questions, which tested knowledge in medicine, surgery, obstetrics/gynecology, psychiatry, pediatrics, and preventive medicine, and also included 28 to 35 patient management problems that tested competence in clinical problem solving.17 To enable comparisons between the MCCQE and the Québec certification examination, the 1258 multiple-choice questions used between 1988 and 1991 were reclassified by the test committees as diagnosis, management, or other knowledge. Test committees also identified management items that tested drug-specific knowledge. Standardized ability scores were created for each subscore using the same reference group approach as the certification examination.18 The overall score reliability was 0.92, and subscore reliabilities were 0.71 for prevention, 0.63 for diagnosis, 0.73 for management, and 0.48 for drug knowledge. The Medical Council of Canada linked score data to the Québec cohort file by name, sex, and birth date.

Practice Assessment

Data Sources and Retrieval. Four previously validated19 health administrative databases, linked by encrypted beneficiary identifiers, were used to assess practice performance.20,21 The registrant database provided patients' age, sex, postal code, and date of death. The medical services database provided type, location, diagnosis, treating and referring physician, and date of all services delivered on a fee-for-service basis. The prescription database provided drug, dose, duration, prescribing physician, and date of each prescription dispensed for patients aged 65 years or older. The hospitalization database provided records of all hospital discharges including discharge diagnoses and admission and discharge dates. The 1991 census data was linked by 6-digit postal code to the registrant database to measure mean family income and educational achievement in the residential area of each patient.22 For each physician, the medical services claims files were used to identify all patients seen by the physician from the licensure date to December 31, 1996. For each patient, the RAMQ retrieved demographic data, as well as all medical services, prescriptions, and hospitalizations provided during the follow-up period. We also obtained data for the year prior to the first contact with the study physician to provide an independent assessment of patient characteristics that could not have been influenced by the study physician.14

Indicators of Practice Performance. We assessed 6 performance indicators in each follow-up year, selected on the basis of unexplained practice variation, and/or their association with the outcomes or costs of care.14 First, a mammography screening rate was used to assess preventive care, because physicians who screen for breast cancer are also more likely to perform other preventive services.23 Second, continuity of care was selected because of its importance in prevention and chronic disease management,2427 and to test the hypothesis that communication and management skills are predictive of better continuity.28,29 Third, the differences between disease-specific and symptom-relief prescribing rate and contraindicated prescribing rate were used as indicators of the quality of acute and chronic disease management because variation in disease-specific relative to symptom-relief prescribing rate30 has been linked to diagnostic competence.14 Fourth, contraindicated prescribing, which accounts for 20% of drug-related adverse events, may be caused by deficiencies in physician knowledge.31 Finally, consultation rate was used as an indicator of resource use because referral determines access to higher cost specialty care,32 and primary care physicians with higher self-reported competence appear more likely to refer patients for specialty consultation.33,34

Annual mammography screening rate was defined as the proportion of eligible women in the primary care practice who were referred for a bilateral mammogram by the physician. Eligible women were between age 50 and 69 years, had no prior diagnosis of breast cancer, breast disease, or diagnostic mammogram, were due for screening, and were not receiving primary care from a gynecologist or obstetrician.

Annual continuity of care was defined as the mean proportion of visits that were made to or referred by the study physician by all patients in the primary care practice population. Using the medical services claims, we determined the proportion of all visits in the year for each patient that were made either to the study physician or to other physicians based on study physician referral. Each patient's proportion was weighted by the square root of the total number of visits made by the patient in the year. An overall mean annual continuity of care index for each physician's practice was calculated as the weighted mean of individual patient's proportions.

Mammography screening and continuity of care were assessed only in the primary care practice population, which consisted of patients seen in an office or clinic in the respective year, for whom the study physician had provided an annual physical or major (3 system) assessment. Consultation was assessed in all ambulatory patients for whom the study physician had billed at least 1 outpatient, office practice, or emergency department visit. Prescribing rates were assessed in ambulatory patients aged 65 years or older for whom complete information was available on all prescriptions dispensed.

Annual difference between disease-specific and symptom-relief prescribing rates was determined by examining medications prescribed by the study physician to all elderly patients. Disease-specific medication was defined as drugs that would rarely be prescribed without an investigation-confirmed disease (eg, anticoagulants, anticonvulsants, antidepressants, antihypertensives, medications for cardiovascular disease, asthma, and Parkinson disease, corticosteroids, diuretics, antiglaucoma medication, hypoglycemic medication, and thyroid medication). Symptom-relief medication was defined as drugs that relieve symptoms, but have little impact on the disease process (eg, nonsteroidal anti-inflammatory medications, benzodiazepines, low-dose narcotic analgesics) using the McGavock classification.30

Annual contraindicated prescribing rate was the proportion of ambulatory elderly patients for whom the study physician prescribed a relatively contraindicated medication. These were defined by an updated expert review35,36 as 30 drugs that should be avoided in elderly patients because of possible toxic effects.

Annual consultation rate was the proportion of all ambulatory patients in the respective calendar year referred, at least once, to a specialist by the respective primary care physician, based on medical services claims. To be reimbursed for a consultation visit, the consultant must record a valid license number for the referring physician.

Time in Practice. Physicians' preceding practice experience was represented as a time-dependent covariate, based on a count of the number of previous months that the physician had at least 1 fee-for-service billing in the medical services claims file.

Case-Mix Assessment

Relevant data were retrieved for each eligible patient for the 12-month period preceding the first contact with the study physician. Individual characteristics of all patients seen in a given year by the physician were then aggregated to adjust for between-physician differences in practice population case-mix. Each practice population was characterized by age and sex distribution, mean family income, educational achievement, geographic access to health care,14 propensity for the use of health care services,37 comorbidity (Charlson index),38 and hospitalization rates in the previous year. These annual case-mix measures were modeled as time-dependent covariates. For contraindicated prescribing, no adjustments were made for case-mix because the prescription of relatively contraindicated medication is rarely justified by patient characteristics.14

Statistical Analysis

Relationships between examination scores and practice performance were tested using multiple linear and Poisson regression for repeated measures using generalized estimating equations.39 Physicians were the unit of analysis. Outcome variables were the 5 annual measures of practice performance. An autoregressive first-order correlation structure of residuals was used to characterize the interdependence between annual performance measures for each physician. To reduce imprecision in outcome measurement, physicians were excluded in years in which they had fewer than 5 patients, and the logarithm of the number of patients in the practice population in a given year was used as a weight in the analysis. Linearity assumptions were evaluated by testing the statistical significance of the quadratic component. Poisson regression was used to assess rates of contraindicated prescribing because such events were rare. The SEs were empirically estimated to account for overdispersion. Annual measures of case-mix were included in all analyses, as were indicators of the medical school to conservatively assess the impact of variation in examination scores within each medical school. To determine whether the association between examination scores and practice outcomes diminished over time, the interactions between examination score and practice experience were tested. For significant relationships between examination score and outcomes that persisted over time, the cumulative difference in the number of outcomes per 1000 patients followed up over the first 5 years in practice by high-scoring (2 SDs above the mean) rather than low-scoring (2 SDs below the mean) physicians was estimated. Regression coefficients for the score-outcome relationship were used to determine the expected annual difference in rates corresponding to a 4-SD difference in examination scores, and then the result was multiplied by 5 to estimate the cumulative impact over 5 years. P<.05 was used as the level of statistical significance. We used SAS statistical software to perform our analyses (Version 8.0, SAS Institute Inc, Cary, NC).

Certification Examination

Between 1990 and 1993, a total of 944 family physicians passed the QLEX, 920 (97.5%) started practice in Québec, 912 (96.7%) entered a fee-for-service practice, of whom 58.1% were female (Table 1). Overall, 85.8% of physicians took the MCCQE during the final year of medical school. Québec medical school graduates had modestly lower MCCQE scores in comparison with other Canadian graduates (mean [SD], −0.04 [0.84] vs 0.09 [0.92]; P<.001). Graduates who entered family medicine or general practice training in Canada had slightly lower standardized scores than those entering specialty programs (mean [SD], 0.02 [0.95] vs 0.13 [0.84]; P<.001). Québec family physicians who did not take the MCCQE had lower certification examination scores than those who did (mean [SD], −0.76 [1.18] vs −0.03 [1.02]; P<.001). Mean scores achieved on both the QLEX and MCCQE were equivalent or slightly lower than in the reference group of first-time takers, with a typical range of 6 to 7 SDs (Table 1). The Pearson correlation between the MCCQE and QLEX was 0.55, and ranged from 0.26 (clinical problem solving) to 0.49 (management).

Table Graphic Jump LocationTable 1. Physician Characteristics and Examination Scores for 912 Québec Family Physicians*

Physicians practiced in a mean of 2.6 (year 1-2) to 3.2 (year 5-7) different types of settings (Table 2). During the first 4 to 7 years of practice, an increasing proportion of physicians moved their primary practice base from rural to urban populations, and established private office practice. After the first 2 years, practice size, number of days worked per year, and visits per day continued to increase, but at a slower rate.

Table Graphic Jump LocationTable 2. Practice Setting and Workload Characteristics for Family Physicians in the First 4 to 7 Years of Practice

The 912 physicians billed for 3.4 million different patients (45.9% of the Québec population), of whom 1.4 million were in their primary care practice populations, 385 321 were elderly, and 119 866 were women eligible for mammography screening.

The relationship between the QLEX and mammography screening was sustained over the first 4 to 7 years in practice (Table 3). The significant interaction between certification examination score and practice experience indicated that the strength of the relationship increased over time. For every SD increase in QLEX score, the mammography screening rate increased by 13.3 women per 1000 in years 1 to 2 of practice, 21.7 by years 3 to 4, and 19.2 by years 5 to 7. The persistence of this relationship means that, during the first 5 years of practice, high-scoring physicians would be expected to order 347 more mammograms per 1000 women than low-scoring physicians. In contrast, there was no relationship between QLEX scores and continuity of care. Overall, however, family physicians provided or coordinated only 28.3% of all visits made by patients in their primary care practice population.

Table Graphic Jump LocationTable 3. Association Between Family Medicine Certification Examination Scores and Practice Performance in the First 4 to 7 Years of Practice

Consultation rate showed a persistent, but modest, association with QLEX score over the first 7 years of practice (Table 3). Each SD increase in score was associated with an additional 2.92 referrals for specialty consultation per 1000 patients seen per year, resulting, over the first 5 years, in 58 more referrals per 1000 patients by high-scoring than low-scoring physicians.

Diagnosis and management subscores of the QLEX were the only significant predictors of prescribing outcomes. Higher scores were associated with higher rates of disease-specific relative to symptom-relief prescribing, and a lower risk of contraindicated prescribing, although the latter did not achieve statistical significance (Table 3). The strength and significance of the associations between examination scores and practice outcomes increased when medical school was excluded from the model, because there were systematic differences in scores for graduates from different medical schools.

Qualifying Examination

Scores on the MCCQE, taken at the end of medical school, had a similar pattern of relationships as the certification examination (Table 4). For example, the increase in mammography screening rate per SD increase in score was 16.8 per 1000 for the MCCQE score (Table 4) and 17.4 per 1000 for the overall QLEX score (Table 3). The diagnosis subscore was the strongest predictor of differences in the rates of disease-specific and symptom-relief prescribing in both the MCCQE and the QLEX. The drug knowledge subscore was the only significant predictor of contraindicated prescribing—reducing the risk of contraindicated prescribing by 12% per SD increase in score. During the first 5 years, a high-scoring physician would be expected to write 85 fewer contraindicated prescriptions per 1000 elderly patients than a low-scoring physician.

Table Graphic Jump LocationTable 4. Association Between Medical Council of Canada Licensing Examination Scores and Practice Performance in the First 4 to 7 Years of Practice

We examined the relationship between licensure and certification examination scores and practice performance in a sample of newly certified family physicians. Linked databases from a universal health care system allowed us to follow-up all licensed physicians, and all their patients in their first 4 to 7 years of practice. Thus, we minimized selection biases that could confound the assessment of relationships. We were also able to assess the population impact of potential licensure regulations. The most important findings of this study were that the relationships between certification examination scores were sustained through the first 4 to 7 years of practice, and that examinations taken in the final year of medical school were also significant predictors of practice performance. However, the use of administrative databases had limitations. The databases restricted the type of performance indicators that could be measured and we were limited in our ability to adjust for confounding by case-mix, but as case-mix was not associated with examination scores, it is unlikely that this introduced substantial bias in the results. These results have several implications for licensing bodies.

Such outcome data could be incorporated into the establishment of examination passing standards so that standard-setting participants can weigh their usual judgments of test content and pass rates against the consequences for patients.4042 For example, an increase in the passing criterion of only 1 SD in drug knowledge would have resulted in failing 16 additional physicians on the MCCQE licensing examination over 4 years. It would also have reduced the expected risk of contraindicated prescriptions for elderly patients seen by these physicians by approximately 42% (from 4.7% for these low-scoring physicians to 3.3% for an average physician). A reduction in risk of this magnitude is equivalent to or greater than that reported for the most effective form of continuing medical education for physicians with high rates of inappropriate prescribing.43 To minimize the risk of adverse outcomes for the population, passing standards may need to be established for individual components of the examination, such as drug knowledge, diagnosis, and management.

Consultation rates increased linearly with examination score. Physicians tend to report higher referral rates in clinical areas in which they felt more competent.33,34 More competent physicians may be more aware of their limitations. However, most examinations do not test whether an individual knows when to refer. This may be an important area for test development, as the risk of outcomes related to medical errors has been shown to increase when physicians practice beyond their areas of competence.44

The finding that examinations taken at the end of medical school were also predictive of future practice may be useful for residency program directors, who could use this information to target learning opportunities to areas of deficiency. Medical school educators may also be able to identify persons more likely to experience difficulty in practice earlier in training, as scores on medical school examinations are strongly correlated with scores on licensing examinations.45

Future research should investigate methods of establishing outcomes-based passing scores, and develop more refined measures of quality of care in disease-specific populations. Longitudinal follow-up studies should be conducted in multijurisdictional cohorts of medical graduates to explore the aspects of admissions, undergraduate, and postgraduate training that may influence practice.

Irvine D. The performance of doctors, I: professionalism and self-regulation in a changing world.  BMJ.1997;314:1540-1542.
Tamblyn RM. Is the public being protected? prevention of suboptimal medical practice through training programs and credentialing examinations.  Eval Health Prof.1994;17:198-221.
Ayanian JZ, Hauptman PJ, Guadagnoli E.  et al.  Knowledge and practices of generalist and specialist physicians regarding drug therapy for acute myocardial infarction.  N Engl J Med.1994;331:1136-1142.
Edep ME, Shah NB, Tateo IM, Massie BM. Differences between primary care physicians and cardiologists in management of congestive heart failure.  J Am Coll Cardiol.1997;30:518-526.
Norcini J, Lipner RS, Benson Jr JA, Webster GD. An analysis of the knowledge base of practicing internists as measured by the 1980 recertification examination.  Ann Intern Med.1985;102:385-389.
Jollis JG, DeLong ER, Peterson ED.  et al.  Outcome of acute myocardial infarction according to the specialty of the admitting physician.  N Engl J Med.1996;335:1880-1887.
Weiner JP, Parente ST, Garnick DW, Fowles J, Lawthers AG, Palmer HR. Variation in office-based quality: a claims-based profile of care provided to Medicare patients with diabetes.  JAMA.1995;273:1503-1508.
Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD. Predictive validity of certification by the American Board of Internal Medicine.  Ann Intern Med.1989;110:719-726.
McAuley RG, Paul WM, Morrison GH, Beckett RF, Goldsmith CH. Five-year results of the peer assessment program of the College of Physicians and Surgeons of Ontario.  CMAJ.1990;143:1193-1199.
Murata PJ, Li J. Relationship between Pap smear performance and physician ordering a mammogram.  J Fam Pract.1992;35:644-648.
Nicolucci A, Cavaliere D, Scorpiglione N.  et al.  A comprehensive assessment of the avoidability of long-term complications of diabetes.  Diabetes Care.1996;19:927-932.
Legorreta AP, Christian-Herman J, O'Connor RD.  et al.  Compliance with national asthma management guidelines and specialty care.  Arch Intern Med.1998;158:457-464.
Wilcox SM, Himmelstein DU, Woolhandler S. Inappropriate drug prescribing for the community-dwelling elderly.  JAMA.1994;272:292-296.
Tamblyn R, Abrahamowicz M, Brailovsky C.  et al.  Association between licensing examination scores and resource use and quality of care in primary care practice.  JAMA.1998;280:989-996.
 Statistiques Annuelles.  Québec, Québec: Regie de l'assurance-maladie du Québec; 1995.
Grand'Maison P, Lescop J, Rainsberry P, Brailovsky CA. Large-scale use of an objective, structured clinical examination for licensing family physicians.  CMAJ.1992;146:1735-1740.
 Information Pamphlet on the Qualifying Examination.  Ottawa, Ontario: Medical Council of Canada; 1991.
Livingston SA, Zieky M. Passing Scores. Princeton, NJ: Educational Testing Service; 1989.
Tamblyn RM, Lavoie G, Petrella L, Monette J. Use of prescription claims databases in pharmacoepidemiological research.  J Clin Epidemiol.1995;48:999-1009.
Levy AR, Tamblyn RFD, McLeod P, Hanley J. Coding accuracy of hospital discharge data for elderly survivors of myocardial infarction.  Can J Cardiol.1999;15:1277-1282.
Wilchesky M, Tamblyn RM. Predictive values of diagnoses in medical services claims.  J Clin Pharmacol.2001;220(suppl).
Wilkins K. Use of postal codes and addresses in the analysis of health data.  Health Rep.1993;5:157-177.
Salive ME, Guralnik JM, Brock D. Preventive services for breast and cervical cancer in US office-based practices.  Prev Med.1996;25:561-568.
Bindman AB, Grumbach K, Osmond D.  et al.  Preventable hospitalizations and access to health care.  JAMA.1995;274:305-311.
Shea S, Misra D, Ehrlich MH, Field L, Francis CK. Predisposing factors for severe, uncontrolled hypertension in an inner-city minority population.  N Engl J Med.1992;327:776-781.
Wasson JH, Sauvigne AE, Mogielnicki RP.  et al.  Continuity of outpatient medical care in elderly men: a randomized trial.  JAMA.1984;252:2413-2417.
Lambrew JM, DeFriese GH, Carey TS, Ricketts TC, Biddle AK. The effects of having a regular doctor on access to primary care.  Med Care.1996;34:138-154.
Hulscher ME, van Drenth BB, Mokkink HG.  et al.  Barriers to preventive care in general practice.  Br J Gen Pract.1997;47:711-714.
Safran DG, Montgomery JE, Chang H, Murphy J, Rogers H. Switching doctors: predictors of voluntary disenrollment for primary care physician's practice.  J Fam Pract.2001;50:130-136.
McGavock H, Wilson-Davis K, Niblock RWF. Unsuspected patterns of drug utilization revealed by interrogation of a regional general practitioner prescribing database.  Pharmacoepidemiol Drug Saf.1992;1:73-80.
Leape LL, Bates DW, Cullen DJ.  et al.  Systems analysis of adverse drug events.  JAMA.1995;274:35-43.
Franks P, Clancy CM. Gatekeeping revisited: protecting patients from overtreatment.  N Engl J Med.1992;327:424-429.
Reynolds GA, Chitnis JG, Roland MO. General practitioner outpatient referrals: do good doctors refer more patients to hospital?  BMJ.1991;302:1250-1252.
Calman NS, Hyman RB, Licht W. Variability in consultation rates and practitioner level of diagnostic certainty.  J Fam Pract.1992;35:31-38.
Beers MH, Ouslander JG, Rollingher I, Reuben DB, Brooks J, Beck JC. Explicit criteria for determining inappropriate medication use in nursing home residents.  Arch Intern Med.1991;151:1825-1832.
McLeod PJ, Huang AR, Tamblyn RM, Gayton DC. Defining inappropriate practices in prescribing for elderly people.  CMAJ.1997;156:385-391.
Starfield B, Weiner J, Mumford L, Steinwachs D. Ambulatory care groups: a categorization of diagnoses for research and management.  Health Serv Res.1991;26:53-74.
Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM adminstrative databases.  J Clin Epidemiol.1992;45:613-619.
Zeger SL, Liang KY. Longitudinal data analysis for discrete and continous outcomes.  Biometrics.1986;42:121-130.
 Public trust and accountability for clinical performance: lessons from the national press reportage of the Bristol hearing.  J Eval Clin Pract.1999;5:335-342.
Southgate L, Hays RB, Norcini J.  et al.  Setting performance standards for medical practice: a theoretical framework.  Med Educ.2001;35:474-481.
Norcini Jr JJ. Standards and reliability in evaluation.  Acad Med.1999;74:1088-1090.
Schaffner W, Ray WA, Federspiel CF, Miller WO. Improving antibiotic prescribing in office practice.  JAMA.1983;250:1728-1732.
Leape LL, Brennan TA, Laird N.  et al.  Nature of adverse events in hospitalized patients.  N Engl J Med.1991;324:377-384.
Veloski JJ, Hojat M, Gonnella JS. The validity of Part III of the National Board Examination.  Proc Annu Conf Res Med Educ.1987;26:54-59.

Figures

Tables

Table Graphic Jump LocationTable 1. Physician Characteristics and Examination Scores for 912 Québec Family Physicians*
Table Graphic Jump LocationTable 2. Practice Setting and Workload Characteristics for Family Physicians in the First 4 to 7 Years of Practice
Table Graphic Jump LocationTable 3. Association Between Family Medicine Certification Examination Scores and Practice Performance in the First 4 to 7 Years of Practice
Table Graphic Jump LocationTable 4. Association Between Medical Council of Canada Licensing Examination Scores and Practice Performance in the First 4 to 7 Years of Practice

References

Irvine D. The performance of doctors, I: professionalism and self-regulation in a changing world.  BMJ.1997;314:1540-1542.
Tamblyn RM. Is the public being protected? prevention of suboptimal medical practice through training programs and credentialing examinations.  Eval Health Prof.1994;17:198-221.
Ayanian JZ, Hauptman PJ, Guadagnoli E.  et al.  Knowledge and practices of generalist and specialist physicians regarding drug therapy for acute myocardial infarction.  N Engl J Med.1994;331:1136-1142.
Edep ME, Shah NB, Tateo IM, Massie BM. Differences between primary care physicians and cardiologists in management of congestive heart failure.  J Am Coll Cardiol.1997;30:518-526.
Norcini J, Lipner RS, Benson Jr JA, Webster GD. An analysis of the knowledge base of practicing internists as measured by the 1980 recertification examination.  Ann Intern Med.1985;102:385-389.
Jollis JG, DeLong ER, Peterson ED.  et al.  Outcome of acute myocardial infarction according to the specialty of the admitting physician.  N Engl J Med.1996;335:1880-1887.
Weiner JP, Parente ST, Garnick DW, Fowles J, Lawthers AG, Palmer HR. Variation in office-based quality: a claims-based profile of care provided to Medicare patients with diabetes.  JAMA.1995;273:1503-1508.
Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD. Predictive validity of certification by the American Board of Internal Medicine.  Ann Intern Med.1989;110:719-726.
McAuley RG, Paul WM, Morrison GH, Beckett RF, Goldsmith CH. Five-year results of the peer assessment program of the College of Physicians and Surgeons of Ontario.  CMAJ.1990;143:1193-1199.
Murata PJ, Li J. Relationship between Pap smear performance and physician ordering a mammogram.  J Fam Pract.1992;35:644-648.
Nicolucci A, Cavaliere D, Scorpiglione N.  et al.  A comprehensive assessment of the avoidability of long-term complications of diabetes.  Diabetes Care.1996;19:927-932.
Legorreta AP, Christian-Herman J, O'Connor RD.  et al.  Compliance with national asthma management guidelines and specialty care.  Arch Intern Med.1998;158:457-464.
Wilcox SM, Himmelstein DU, Woolhandler S. Inappropriate drug prescribing for the community-dwelling elderly.  JAMA.1994;272:292-296.
Tamblyn R, Abrahamowicz M, Brailovsky C.  et al.  Association between licensing examination scores and resource use and quality of care in primary care practice.  JAMA.1998;280:989-996.
 Statistiques Annuelles.  Québec, Québec: Regie de l'assurance-maladie du Québec; 1995.
Grand'Maison P, Lescop J, Rainsberry P, Brailovsky CA. Large-scale use of an objective, structured clinical examination for licensing family physicians.  CMAJ.1992;146:1735-1740.
 Information Pamphlet on the Qualifying Examination.  Ottawa, Ontario: Medical Council of Canada; 1991.
Livingston SA, Zieky M. Passing Scores. Princeton, NJ: Educational Testing Service; 1989.
Tamblyn RM, Lavoie G, Petrella L, Monette J. Use of prescription claims databases in pharmacoepidemiological research.  J Clin Epidemiol.1995;48:999-1009.
Levy AR, Tamblyn RFD, McLeod P, Hanley J. Coding accuracy of hospital discharge data for elderly survivors of myocardial infarction.  Can J Cardiol.1999;15:1277-1282.
Wilchesky M, Tamblyn RM. Predictive values of diagnoses in medical services claims.  J Clin Pharmacol.2001;220(suppl).
Wilkins K. Use of postal codes and addresses in the analysis of health data.  Health Rep.1993;5:157-177.
Salive ME, Guralnik JM, Brock D. Preventive services for breast and cervical cancer in US office-based practices.  Prev Med.1996;25:561-568.
Bindman AB, Grumbach K, Osmond D.  et al.  Preventable hospitalizations and access to health care.  JAMA.1995;274:305-311.
Shea S, Misra D, Ehrlich MH, Field L, Francis CK. Predisposing factors for severe, uncontrolled hypertension in an inner-city minority population.  N Engl J Med.1992;327:776-781.
Wasson JH, Sauvigne AE, Mogielnicki RP.  et al.  Continuity of outpatient medical care in elderly men: a randomized trial.  JAMA.1984;252:2413-2417.
Lambrew JM, DeFriese GH, Carey TS, Ricketts TC, Biddle AK. The effects of having a regular doctor on access to primary care.  Med Care.1996;34:138-154.
Hulscher ME, van Drenth BB, Mokkink HG.  et al.  Barriers to preventive care in general practice.  Br J Gen Pract.1997;47:711-714.
Safran DG, Montgomery JE, Chang H, Murphy J, Rogers H. Switching doctors: predictors of voluntary disenrollment for primary care physician's practice.  J Fam Pract.2001;50:130-136.
McGavock H, Wilson-Davis K, Niblock RWF. Unsuspected patterns of drug utilization revealed by interrogation of a regional general practitioner prescribing database.  Pharmacoepidemiol Drug Saf.1992;1:73-80.
Leape LL, Bates DW, Cullen DJ.  et al.  Systems analysis of adverse drug events.  JAMA.1995;274:35-43.
Franks P, Clancy CM. Gatekeeping revisited: protecting patients from overtreatment.  N Engl J Med.1992;327:424-429.
Reynolds GA, Chitnis JG, Roland MO. General practitioner outpatient referrals: do good doctors refer more patients to hospital?  BMJ.1991;302:1250-1252.
Calman NS, Hyman RB, Licht W. Variability in consultation rates and practitioner level of diagnostic certainty.  J Fam Pract.1992;35:31-38.
Beers MH, Ouslander JG, Rollingher I, Reuben DB, Brooks J, Beck JC. Explicit criteria for determining inappropriate medication use in nursing home residents.  Arch Intern Med.1991;151:1825-1832.
McLeod PJ, Huang AR, Tamblyn RM, Gayton DC. Defining inappropriate practices in prescribing for elderly people.  CMAJ.1997;156:385-391.
Starfield B, Weiner J, Mumford L, Steinwachs D. Ambulatory care groups: a categorization of diagnoses for research and management.  Health Serv Res.1991;26:53-74.
Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM adminstrative databases.  J Clin Epidemiol.1992;45:613-619.
Zeger SL, Liang KY. Longitudinal data analysis for discrete and continous outcomes.  Biometrics.1986;42:121-130.
 Public trust and accountability for clinical performance: lessons from the national press reportage of the Bristol hearing.  J Eval Clin Pract.1999;5:335-342.
Southgate L, Hays RB, Norcini J.  et al.  Setting performance standards for medical practice: a theoretical framework.  Med Educ.2001;35:474-481.
Norcini Jr JJ. Standards and reliability in evaluation.  Acad Med.1999;74:1088-1090.
Schaffner W, Ray WA, Federspiel CF, Miller WO. Improving antibiotic prescribing in office practice.  JAMA.1983;250:1728-1732.
Leape LL, Brennan TA, Laird N.  et al.  Nature of adverse events in hospitalized patients.  N Engl J Med.1991;324:377-384.
Veloski JJ, Hojat M, Gonnella JS. The validity of Part III of the National Board Examination.  Proc Annu Conf Res Med Educ.1987;26:54-59.
CME
Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 93

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Collections
JAMAevidence.com

Care at the Close of Life EDUCATION GUIDES
Complexities in Prognostication in Advanced Cancer