0
Original Contribution |

Association Between Licensing Examination Scores and Resource Use and Quality of Care in Primary Care Practice FREE

Robyn Tamblyn, PhD; Michael Abrahamowicz, PhD; Carlos Brailovsky, MD, MA; Paul Grand'Maison, MD, MSc; Joelle Lescop, MD, MPH; John Norcini, PhD; Nadyne Girard, MSc; Jeannie Haggerty, MSc
[+] Author Affiliations

From the Departments of Medicine (Dr Tamblyn) and Epidemiology and Biostatistics (Drs Tamblyn and Abrahamowicz and Mss Girard and Haggerty), McGill University, Montreal, QuÉbec; the Centre d'Évaluation des Sciences de la SantÉ, University of Laval, QuÉbec City, QuÉbec (Dr Brailovsky); the Department of Family Medicine, University of Sherbrooke, Sherbrooke, QuÉbec (Dr Grand'Maison); the QuÉbec College of Physicians, Montreal, QuÉbec (Dr Lescop); and the American Board of Internal Medicine, Philadelphia, Pa (Dr Norcini).


JAMA. 1998;280(11):989-996. doi:10.1001/jama.280.11.989.
Text Size: A A A
Published online

Context.— Clinical competence is a determinant of the quality of care delivered, and may be associated with use of health care resources by primary care physicians. Clinical competence is assumed to be assessed by licensing examinations, yet there is a paucity of information on whether scores achieved predict subsequent practice.

Objective.— To determine if licensing examination scores were associated with selected aspects of quality of care and resource use in initial primary care practice.

Design.— Prospective cohort study of recently licensed family physicians, followed up for the first 18 months of practice.

Setting.— The Québec health care system.

Participants.— A total of 614 family physicians who passed the licensing examination between 1991 and 1993 and entered fee-for-service practice in Québec.

Main Outcome Measures.— All patients seen by physicians were identified by the universal health insurance board and all health services provided to these patients were retrieved for the 18 months prior to (baseline) and after (follow-up) the physicians' entry into practice. Medical service and prescription claims files were used to measure rates of resource use (specialty consultation, symptom-relief prescribing compared with disease-specific prescribing) and quality of care (inappropriate prescribing, mammography screening). Baseline data were used to adjust for differences in practice population.

Results.— Study physicians saw a total of 1116389 patients, of whom 113535 (10.2%) were elderly and 83391 (7.5%) were women aged 50 to 69 years. Physicians with higher licensing examination scores referred more of their patients for consultation (3.8/1000 patients per SD increase in score; 95% confidence interval [CI], 1.2-7.0; P=.005), prescribed to elderly patients fewer inappropriate medications (−2.7/1000 patients per SD increase in score; 95% CI, −4.8 to −0.7; P =.009) and more disease-specific medications relative to symptom-relief medications (3.9/1000 patients per SD increase in score; 95% CI, 0.3 to 7.4; P=.03), and referred more women aged 50 to 69 years (6.6/1000 patients per SD increase in score; 95% CI, 1.2-11.9; P=.02) for mammography screening. If patients of physicians with the lowest scores had experienced the same rates of consultation, prescribing, and screening as patients of physicians with the highest scores, an additional 3027 patients would have been referred, 179 fewer elderly patients would have been prescribed symptom-relief medication, 912 more elderly patients would have been prescribed disease-specific medication, 189 fewer patients would have received inappropriate medication, and 121 more women would have received mammography screening.

Conclusions.— Licensing examination scores are significant predictors of consultation, prescribing, and mammography screening rates in initial primary care practice.

PRIMARY CARE physicians are the foundation of westernized health care. They manage episodic and chronic illness, play a central role in instituting interventions for primary and secondary disease prevention, prescribe and monitor the majority of drug treatment, and function as the entry point to costly health care resources.17 The quality of care delivered by primary care physicians and their use of resources can have a considerable impact on the cost-effectiveness of health care delivery.3 Preventive care,811 prescribing,1220 and chronic disease management2128 have been identified as areas in which quality of care could be improved.

Continuing medical education has been identified as a solution to this problem because low self-reported rates of preventive care are associated with inadequate knowledge of preventive guidelines,29,30 and improvements in prescribing and chronic disease management have been reported with disease-specific education in diagnosis and management.31,32 Yet mandatory continuing medical education requirements have not been associated with improvements in practice,33,34 and this observation has likely contributed to a call for mandatory assessment of the competence of practicing physicians through relicensure and recertification examinations.35,36

The question that has not been adequately addressed is whether tests that purport to measure clinical competence (licensing and certifying examinations) actually predict performance in subsequent practice. Ramsey et al,37 who were the first investigators to address this question, found that scores on the American Board of Internal Medicine certification examination predicted colleagues' ratings of the quality of care delivered by internists 5 to 8 years after certification. Although these results are promising, we do not know whether they are generalizable to family physicians and general practitioners, physicians who provide the majority of primary care, or if colleagues' ratings actually predict what is done in practice. The possibility that licensing examination scores may predict performance in practice needs to be examined because interventions to improve practice through licensure and certification provide the ultimate opportunity for an ounce of prevention to be worth a pound of cure.

Clinical competence has not only been considered to be relevant to the quality of care delivered, but it has also been associated with variation in the use of resources by primary care physicians. Two studies, one British38 and one American,39 have found that primary care physicians with higher self-reported competence were more likely to refer their patients for specialty consultation. This is relevant because referral is the main mechanism by which patients gain access to hospital resources and expensive surgical and investigative procedures,3 and unexplained differences in referral rates exist among primary care physicians.4043 Considerable variation also exists among primary care physicians in prescription rates,12,4449 and it has been suspected that these differences are related to clinical competence.50,51 In particular, higher rates of prescription of medications often used for the relief of undifferentiated symptoms49,5260 in the elderly (eg, benzodiazepines, nonsteroidal anti-inflammatory drugs [NSAIDs], analgesics) are seen among physicians who have low rates of prescribing for disease-specific treatments (eg, hypoglycemics, cardiac medications).46,47 It has been postulated that this pattern is related to an underdiagnosis and undertreatment of disease because of poorer skills in diagnosis and management. There has been no direct test of this association.

Several methodological problems have limited research on the relationship between licensing and certification examination scores and practice. First, physicians with higher scores are more likely to participate in follow-up studies.61 Second, mandatory examinations have traditionally been completed at the end of medical school, with only selected groups of physicians going on to take voluntary certification examinations. Thus, to include all physicians, evaluation would have to take place at a point in training when competence in diagnosis and management would not be fully developed. Finally, follow-up studies of physicians into practice are expensive because assessment of 50 to 200 patients may be needed to obtain stable estimates of performance, and the alternate of self-reported activity is not a valid assessment of what physicians actually do in practice.62,63

An opportunity to overcome 4 major methodological difficulties existed in Québec, a Canadian province with a population of 7.2 million people, 4 medical schools, and approximately 17000 physicians. First, in this province, mandatory assessment of a physician's competence is carried out shortly before an individual enters practice. To be eligible for licensure, physicians must complete a family medicine or other specialty residency and then pass a combination licensing and specialty certification examination. Second, physicians are tested in specific areas of competence and must meet minimum standards in each area to be eligible for licensure. As a result, the subset of competencies that would be expected to have the strongest link with specific practice activities could be assessed. In addition, Québec was the first jurisdiction to include an objective structured clinical examination (OSCE) in its licensure examination process.64 This provided the opportunity to evaluate the utility of this new form of testing of clinical skills. Third, all medical services provided to provincial residents are covered by a universal health insurance plan with 1 government agency responsible for payment of all physician services. This provided the opportunity to obtain a complete census of the practice activity of fee-for-service physicians. Fourth, the availability of high security encryption technology and third-party linkage provided methods of protecting the confidentiality of individual physicians so that all physicians could be included in a study. Although differences in licensing examination requirements and medical service data acquisition exist in Québec, undergraduate education and family medicine training are similar to other accredited North American programs.

In the current study, we used this opportunity to evaluate several aspects of primary care in which variation in care had been identified and in which practice activity could be measured using information retained in provincial health care databases. In relationship to quality of care, we tested the hypotheses that better clinical skills and prevention knowledge would be related to higher rates of mammography screening and that higher scores in diagnosis and management would be related to lower rates of potentially inappropriate prescribing. We included the assessment of clinical skills as a potential predictor of mammography screening because of the reported association between physical examination and preventive care.65 In relationship to resource use, we tested the hypotheses that higher overall competence would be related to higher consultation rates and that higher scores in diagnosis and management would be related to lower rates of symptom-relief prescribing relative to disease-specific prescribing.

Design and Study Population

A prospective follow-up study of a cohort of newly licensed family physicians was conducted. Physicians were eligible for inclusion in the study if they had passed the family medicine licensing examination in 1991, 1992, or 1993; applied for a license to practice in Québec; and entered fee-for-service practice. Physicians paid by salary were excluded because there was no accurate way to identify all patients who had been seen by these physicians even though all prescriptions and referrals for consultation and mammography could be ascertained. Potentially eligible physicians were identified by the Québec College of Physicians, Montreal, Québec, and the license numbers of these physicians were provided to the provincial health insurance agency (Régie de l'Assurance-Maladie du Québec [RAMQ], Québec City, Québec). Practice activity was assessed in the 18-month period after the earliest possible entry into practice. For each cohort, the follow-up period was between July 1 of the examination year and December 31 of the following year.

Licensing Examination Measures of Clinical Competence

Candidates applying for licensure in Québec must pass the College of Family Physicians of Canada Certification Examination (CFPCEx) and the Québec OSCE.64,66 The CFPCEx assesses 4 areas of competence: diagnosis, management, prevention, and communication.66 Diagnosis, management, and prevention are measured by multiple-choice questions and short-answer management problems and communication is measured by a simulated patient office oral examination. To pass the CFPCEx, a candidate must obtain a score of at least 60% each in diagnosis, management, and communication and a score of at least 50% in prevention. The OSCE primarily measures clinical skills by direct, structured observation of a candidate's performance in 24 to 26 clinical problems.64 Clinical problems are presented by standardized patients and performance is assessed by physician examiners. Scores for each case are averaged to produce an overall OSCE score. Candidates fail the OSCE if their score is more than 2 SDs below the mean. Candidates who pass some but not all components of the 2 examinations are reviewed by the licensing examination board and decisions are made on a case-by-case basis.

The subscores from the CFPCEx and the OSCE and an overall average score (the mean of all scores) were retrieved for eligible physicians. There were 6 administrations of the licensing examination between 1991 and 1993. To adjust for potential differences in the difficulty of the examination, scores were standardized using the reference-group method.67 A large homogeneous group of examinees (first-time takers from North American medical schools) is assumed to have similar ability over time and this group was used as the reference standard. Reliability of scores was estimated in each year and summarized across years using a weighted intraclass correlation coefficient.68 Standardized examination scores for each physician and their license number were sent to the RAMQ. Personal identifiers on the score file and health care databases were encrypted to protect confidentiality but permit database linkage.

Practice Assessment

Data Source. Practice activity and practice population characteristics were assessed using information from 4 provincial health care databases and the national 1992 census data. The physician claims file provides, for all medical services delivered on a fee-for-service basis (95% of services69 ), information on the treating physician; referring physician; recipient; type, location, and date of service; and the diagnosis for the visit. The prescription claims file provides information on all prescription drugs dispensed to the elderly, including the recipient, prescribing physician, drug, quantity dispensed, and dispensing date. These reimbursement files are systematically audited and have been independently validated.70,71 The registrant database contains the registration number, age, sex, residential address, and health district of each provincial resident. The master physician file contains information on the license number, specialty, year of graduation, medical school, age, and sex of Québec physicians. The national 1991 census data72 were linked to the registrant database to provide information on the average family income and mean educational achievement in the health district of each registrant.

Data Retrieval. A 3-step process was used to obtain practice data for each physician. First, the RAMQ identified all registrants who had been seen on a fee-for-service basis by each study physician in the 18 months following their earliest possible entry date into practice. Second, all services and prescriptions provided to these patients by the study physician as well as other physicians in the province were retrieved from the 18-month follow-up window established for each physician. Third, all services and prescriptions provided to these patients were retrieved from the 18-month period before the follow-up window. These baseline data were used to account for differences in the practice population characteristics that existed prior to the first contact with the study physicians and, thus, could not be attributed to their practice patterns.

Practice Activity

Practice activity for each physician was measured in 3 practice populations: all outpatients, elderly patients, and women aged 50 to 69 years. Patients were considered to be in a physician's practice population if they received 1 or more services from the study physician in an outpatient setting (emergency department, outpatient clinic, or office practice). For women to be considered eligible for mammography screening, we further restricted the practice population to women seen in an outpatient setting where the institution of preventive care would normally be expected. The setting in which care was delivered was determined by the location recorded on the billing claim, and age restrictions, when relevant, were determined by the age recorded for an individual in the registrant database. Each practice population denominator was composed of the number of different patients seen by a physician during the follow-up period. Patients who were seen by multiple study physicians were represented in more than 1 physician's practice population. Four practice indicators were generated for each physician: consultation rate, symptom-relief prescribing rate compared with disease-specific prescribing rate, inappropriate prescribing rate, and mammography screening rate in women aged 50 to 69 years.

Health Care Resource Use

Consultation rate was defined as the proportion of outpatients in a physician's practice who were referred at least once for a specialty consultation by the study physician. A patient was designated as having been referred to the specialist when the study physician was identified by the specialist as the referring physician on the claim for a consultation.

Symptom-relief prescribing rate relative to disease-specific prescribing rate was defined as the difference between the proportion of elderly patients for whom the study physician prescribed disease-specific medication and the proportion of elderly patients who were prescribed symptomatic medication by the study physician. Symptomatic medication, defined as drugs that relieve symptoms but have little impact on the disease process,47 included NSAIDs, benzodiazepines, and low-dose narcotic analgesics. Disease-specific medication, defined as drugs that would likely be used exclusively to treat an investigation-confirmed disease state,47 included anticoagulants, anticonvulsants, antidepressants, antihypertensives, anti–Parkinson disease medication, asthma preparations, corticosteroids, diuretics, antiglaucoma medication, cardiac medication, hypoglycemic medication, and thyroid and antithyroid medication. (The list of medications included in symptom-relief and disease-specific prescribing rates is available on request from the authors.) The list of disease-specific and symptomatic medications was created using the classification established by McGavock et al,47 excluding drugs that could be obtained as over-the-counter products. The prescription of drugs started by other physicians and refilled by the study physician was distinguished from the prescription of drugs started by the study physician (no evidence of a previous prescription for the drug in the 6 months preceding the study physician's prescription) for both disease-specific and symptom-relief medication.

Quality of Care

Inappropriate prescribing rate was defined as the proportion of patients prescribed a relatively contraindicated psychotropic drug, cardiovascular drug, or NSAID. Inappropriate drugs were identified from a published international expert consensus73 and were defined as medications that should be avoided in the elderly because of the risk of toxic effects (long-acting benzodiazepines, meprobamate, barbiturates, amitriptyline, indomethacin, phenylbutazone, propoxyphene, pentazocine, reserpine, methyldopa, and propranolol). Prescriptions of inappropriate drugs that were started by other physicians and refilled by the study physician were distinguished from drug prescriptions that were started by the study physician without evidence of a previous prescription in the past 6 months.

Mammography screening rate in women aged 50 to 69 years was defined as the proportion of eligible women aged 50 to 69 years who received bilateral mammography at the request of the study physician. Eligible women were those who had no diagnosis of breast cancer or breast disease or a diagnostic mammogram before their first contact with the study physician. To assess whether omissions in care were due to the provision of mammography screening by other physicians, we also determined the overall rate of screening for each study physician's practice, irrespective of the referring physician.

Practice Population Characteristics and Access to Health Care

For each practice subpopulation (all outpatients, elderly patients, and women aged 50-69 years), information was first retrieved for each patient and then aggregated to produce summary measures for the practice (available from the authors on request). Variables used to summarize the practice population included age, sex, average income, education, geographic access to health care, comorbidity, prior health care use, and outcome-specific utilization of health services. Geographic access was treated as a potential confounder in comparisons of specialty consultation (distance to an urban center) and mammography screening rates (distance to a mammography screening facility) and practice population characteristics and outcome-specific utilization behavior as confounders for consultation, mammography, and symptom-relief prescribing rates. For inappropriate prescribing, no confounders were considered relevant because the decision to prescribe a relatively contraindicated medication is rarely justified by patient characteristics.

Statistical Analysis

Multiple linear regression was used to test hypothesized associations between examination subscores and practice outcomes when adjusted for potential confounders. To account for differences in the precision of the estimated outcome rates for each physician, observations were weighted by the square root of the number of eligible patients in each physician's practice. The residuals were examined to assess whether a linear model was appropriate. All practice population characteristics were included in the regression models for consultation rate, symptom-relief prescribing rate compared with disease-specific prescribing rate, and mammography screening rate to control for potential confounding due to differences in the composition of the practice population. Because physicians entered practice at different points in the 18-month period, we controlled for varying amounts of follow-up by including months of active practice as a continuous covariate in the regression models. Each examination score was modeled separately to avoid problems of multicollinearity in the estimation of the regression coefficients. The population impact of variation in examination scores was determined based on the differences in the rate of each outcome among patients seen by physicians at the upper (≥2 SDs above the mean) and lower (≥2 SDs below the mean) end of the score distribution. We calculated the difference in the number of outcomes that would be expected among patients seen by physicians with lowest scores if they experienced the same outcome rate as patients of physicians with the highest scores.

Study Population

A total of 810 family medicine residents took the licensing examination in 1991, 1992, or 1993, and 614 were eligible for inclusion. Among the ineligible residents, 21 failed, 46 did not apply for Québec license, 17 had not started practice within the first 18 months, and 112 entered salaried practice. Of the 614 physicians included in the analysis, 56.0% were female, 85.8% were graduates from Québec medical schools, and 6.5% were foreign medical graduates. Most (90%) had graduated between 1989 and 1991; 36.5% took the examination in 1991, 36.6% in 1992, and 26.9% in 1993.

Examination Scores

Average standardized examination scores for the 614 physicians were slightly below the mean of 0 for the reference group (Table 1). This is because the reference group had, on average, higher scores than the entire group of physicians who passed the examination, and they were relatively homogeneous in their performance. The percentage of study physicians who were 3 or more SDs below the reference group mean varied from 0.8% (OSCE) to 3.1% (prevention). For prevention, the average score for the reference group varied from 74% to 80% in different administrations (SD, 7.5%-7.6%). Thus, physicians who obtained the passing score of 50% for prevention would have a standardized score of –2.8 to –3.9 depending on the year in which they took the examination. Score reliability varied from 0.27 to 0.72. Correlations among examination scores varied from a low of r=0.36 for prevention and OSCE scores to a high of r=0.63 for diagnosis and management scores.

Table Graphic Jump LocationTable 1.—Standardized Examination (z) Scores for the 614 Physicians Eligible for Inclusion in the Study Population*
Practice Activity

During the 18-month follow-up period, the 614 study physicians saw a total of 1116389 individuals, 16% of the total population in the province. Seventeen percent of individuals were seen by more than 1 study physician. Over 74 million claims for medical services and prescriptions were retrieved for these patients, 31574888 from the baseline period and 42850337 from the follow-up period. Physicians practiced in an average of 3.6 settings (median, 4; range, 1-7) in the first 18 months and the distribution by geographic location was 71.8% in urban or periurban areas, 25.4% in rural and remote settings, and 2.8% in mixed locations. The average outpatient practice population size was 2250 patients per study physician, of which 251 were eligible elderly patients and 161 were eligible women aged 50 to 69 years. The average (SD) length of follow-up was 15.8 (3.2) months.

During the follow-up period, the average consultation rate in the practices of study physicians was 71 per 1000 patients (SD, 51). On average, 108 elderly patients per 1000 (SD, 109; range, 0-563) were prescribed disease-specific medication and 126 elderly patients per 1000 (SD, 103) were prescribed symptom-relief medication, the most common drug being a benzodiazepine (54/1000). The average difference in prescribing rates was negative (mean [SD], −18 [68]/1000; range, −167 to 188), indicating that physicians prescribed symptom-relief drugs to proportionately more elderly patients than they prescribed disease-specific medications. However, the range indicates that differences in disease-specific and symptom-relief prescribing rates varied considerably among physicians. Seventy-one percent of study physicians prescribed an inappropriate drug to at least 1 of their elderly patients. The average rate of inappropriate prescribing across practices was 20 per 1000 elderly patients (SD, 31; range, 0-310). Approximately half (9/1000) of all patients who were prescribed an inappropriate medication were started on the drug therapy by the study physician, the most common drug being a long-acting benzodiazepine. The mammography screening rate was 51 per 1000 women aged 50 to 69 years (SD, 81; range, 0-750) when restricted to mammograms requested by the study physician and 218 per 1000 when all screening mammograms ordered for study physicians' patients were considered.

Examination Scores and Practice Activity

Higher consultation rates were associated with higher overall examination scores (Table 2). For every SD increase in a physician's overall competence score, 3.8 additional patients per 1000 would be referred for consultation.

Table Graphic Jump LocationTable 2.—Association Between Licensing Examination Scores and Practice Activity After Adjustment for Practice Population*

Physicians with higher diagnostic and management scores were more likely to have higher rates of disease-specific prescribing relative to symptom-relief prescribing and were less likely to prescribe symptom-relief medication or inappropriate medication. With every SD increase in diagnostic score, 3.7 more elderly patients per 1000 would be prescribed disease-specific medication relative to symptom-relief medication, 6.5 fewer elderly patients per 1000 would be prescribed symptom-relief medication and 2.7 fewer elderly patients per 1000 would be prescribed potentially inappropriate medication. Higher management scores were not significantly associated with symptom-relief prescribing but were associated with higher relative rates of disease-specific to symptom-relief prescribing and lower rates of potentially inappropriate prescribing. The regression coefficients were similar when the analysis was limited to treatment that was started by the physician. To determine if the exclusion of salaried physicians biased our estimates, we evaluated the rate of inappropriate prescribing among elderly patients prescribed a drug by the study physician. Diagnostic score was the most significant predictor in models that included and excluded salaried physicians and the estimated regression coefficient was the same in both models (β, −5/1000; P =.01).

Higher rates of mammography screening were associated with higher scores in prevention and clinical assessment. For every SD increase in these scores, we estimated that an additional 5.4 to 6.6 women per 1000 would be screened for breast cancer. The inclusion of mammograms ordered by other physicians tended to lower the estimated regression coefficients for each of the examination scores.

Population Impact

Although a relatively small number of physicians obtained subscores that were 2 or more SDs below (7.2%-9.9%) or above (4.7%-6.5%) the mean, these physicians saw a considerable number of patients in the initial practice period. If patients of physicians with the lowest scores experienced the same rates of consultation, prescribing, and screening as patients of physicians with the highest scores, an additional 3027 of the 125383 outpatients seen by the lowest-scoring physicians would have been referred. Of the 9718 elderly patients seen, 179 fewer would have been prescribed symptom-relief medication, 912 more would have been prescribed disease-specific medication, and 189 fewer would have received inappropriate medication. For the 5335 women aged 50 to 69 years who were seen, 121 more would have received mammography screening.

In this study, we found that clinical competence, assessed by a combined licensing and certifying examination at the end of training, predicted consultation, prescribing, and mammography screening in initial primary care practice. The strengths of this study were that we were able to study all physicians who entered fee-for-service practice, assess their clinical competence at the end of training with state-of-the-art methods of assessment, and capture information on all patients seen by these physicians in the initial practice period. Both the strengths and main limitations of the study relate to the use of secondary databases for outcome assessment. On the positive side, we were able to avoid biases related to self-selection, self-report, and poor documentation. On the negative side, we could not measure practice organization or many outcomes that would be of relevance in assessing the adequacy of primary care delivery such as episodic and chronic disease management and patient satisfaction. The relationship of examination scores to these outcomes remains unknown. As clinical information is limited, we were unable to determine the appropriateness of resource use or whether extenuating circumstances explained rates of mammography referral or inappropriate prescribing. Limited precision in classifying outcomes would have likely led to an underestimation of the strength of the relationship between examination scores and practice outcomes. Overestimation of the effect would have been possible only if physicians with lower scores were more likely to see patients who did not require referral; were more likely to have indications for benzodiazepines, NSAIDs, or relatively contraindicated therapy; and were less likely to comply with physician referral for mammography.

Implications for Training and Licensure

This study supports the validity of licensing examination as measures of clinical competence. This is good news for licensing and certifying bodies because these examinations have been used to judge whether an individual has sufficient knowledge, skills, and judgment to deliver safe and effective medical care, and yet there has been a paucity of information about whether these assumptions are correct.74 Furthermore, this study supports the contention that specific competencies are directly related to the aspects of practice in which these abilities are assumed to be required. For example, knowledge of prevention was an important predictor of mammography referral rate. These findings legitimize the approach taken in this examination to require demonstration of a minimum level of proficiency in each of the major aspects of clinical competence: diagnosis, management, prevention, and clinical skills. It also raises questions about the more common practice of establishing 1 overall passing score since this may mask important weaknesses in specific domains.

The observation that 71% of newly licensed physicians prescribed potentially inappropriate medication to their elderly patients, and for 1 physician that 31% of his or her elderly patients were prescribed potentially inappropriate medication, raises the question of whether the passing standard was high enough. Methods of establishing the passing scores for examinations have not had empirical information that could be used to assess the risk of specific deficiencies in knowledge or ability on the quality of care that would be delivered to the population or consequences for health outcome. Either a specified proportion of examinees at the lower tail of the score distribution fail, regardless of their ability (norm-referenced approaches), or a minimum absolute standard is established that would be expected to fail about half of theoretically "borderline" candidates (criterion-referenced approaches). Practice outcome information such as that produced in this study could be incorporated into criterion-referenced methods of standard setting, and this may be an important area for future research.

From the perspective of training programs, there are 2 relevant issues. First, from the literature we know that the quality of a training program has been shown to be positively associated with the scores achieved on certification examinations,75 and furthermore, that certification examination scores, at least in internal medicine, are associated with quality of practice as rated by colleagues.37 This means that improvements in training programs may have an appreciable effect on the quality of care delivered by graduates. Second, scores on licensing examinations tend to be highly correlated with scores on the same types of tests taken during medical school.7680 If licensing examination scores predict some aspects of future practice, then scores achieved during medical school may also be predictive. This supposition should be tested because undergraduate and postgraduate training programs have the opportunity to provide remedial programs for deficiencies identified at a much earlier point in training.

Implications for Population Health and Health Care System Policy

The focus in medical education and licensing examinations has usually been on the individual physician, not on the impact that a physician will have on his or her patient population. This study quantified the number of patients who are influenced by new physicians. Indeed, one sixth of the provincial population, more than a million patients, received services from these physicians in the first 1.5 years of practice. Even when risks are small, such as the case in this study in which only modest associations were evident between examination scores and practice outcomes, the effects of potentially suboptimal care delivery by lower-scoring physicians may be appreciable if a substantial proportion of the population is affected. If this effect persists over time, the population-attributable risk of receiving treatment from physicians with lower levels of competence may be considerable. This study could not answer this question. We do not know if the behavior exhibited in the early period of practice had an impact on health outcome or if these behaviors persist over time. Nevertheless, the associations we did observe raise some questions about current policy and practice.

First, we substantiated the observations of earlier studies40,41; physicians who have higher competency scores are more likely to refer their patients. What does this mean? We could assume that high-scoring physicians are overly fastidious in early practice. Alternately, at a time when health policy has encouraged a reduction in use of specialty consultation for cost containment, we could and should worry that physicians who have lower scores may be less likely to recognize the significance of a patient's problem or their own limitations. If this were the case, the apparent cost savings associated with low referral rates may mask poorer quality of care and even greater costs, both human and administrative, of avoidable morbidity.81

Prescribing activity of physicians has also come under the spotlight in health policy reforms and drug review assessments.1320,8284 Annual expenditures for prescription drugs have increased substantially,8587 and drug-related illness is now claimed to be the sixth leading cause of mortality in the United States.88 The obvious question for both consumers and payers is whether prescription drugs are being used wisely. This study showed that drugs that may be overused or relatively contraindicated in the elderly are more likely to be prescribed by physicians with lower scores in diagnosis and management. We have no way of knowing whether these prescriptions were justified, but our results supported the conviction of McGavock et al46,47 that high rates of symptom-relief prescribing compared with disease-specific prescribing may be more common in physicians with poorer skills in diagnosis and management. A considerable investment has been made in interventions to improve the drug-prescribing habits of practicing physicians.31,56,89 This study provides new insights into this problem, namely, that training and licensure may provide a new avenue for preventive intervention that ultimately may be far less costly than remedial interventions for practicing physicians.90

Physician recommendation has been identified as the most important determinant of the likelihood that a woman will receive a mammogram.9194 Optimal levels of screening have not been achieved,6,810,95 and physician knowledge and attitudes have been identified as significant determinants of their intention to institute screening practices.29,30 To our knowledge, our study provided the first direct assessment of the association between competence in clinical assessment and prevention and mammography screening behavior. Our findings are important because physicians who are more likely to screen for breast cancer are also more likely to carry out other preventive services.6,96 Changes in training programs and licensing examinations may provide a means of optimizing preventive care, and this should be the subject of future research.

Conclusions

Licensing examination scores are significantly associated with some aspects of initial practice behavior. Changes in the passing standard and training may provide an appropriate and effective method of optimizing practice. To determine if this is true, future studies should broaden the scope of practice activities assessed to include continuity of care, other aspects of preventive care and counseling, and the quality of episodic and chronic disease management. The association between examination scores and patient outcomes such as patient satisfaction, quality of life, and avoidable morbidity and mortality would also be important to assess. Future research should also determine if the association between examination scores and practice activity persist over time and whether examinations taken earlier in training are also predictive of future practice activity.

Canadian Medical Association Working Group on Primary Care.  Strengthening the Foundation: The Role of the Physician in Primary Health Care in Canada . Ottawa, Ontario: Canadian Medical Association; 1994:1.
Gervas J, Perez-Fernandez M, Starfield BH. Primary care, financing and gatekeeping in western Europe.  Fam Pract.1994;11:307-317.
Franks P, Clancy CM. Gatekeeping revisited: protecting patients from overtreatment.  N Engl J Med.1992;327:424-429.
Wright RA. Community-oriented primary care: the cornerstone of health care reform.  JAMA.1993;269:2544-2547.
Bindman AB, Grumbach K, Osmond D, Vranizan K, Stewart AL. Primary care and receipt of preventive services.  J Gen Intern Med.1996;11:269-276.
Salive ME, Guralnik JM, Brock D. Preventive services for breast and cervical cancer in US office-based practices.  Prev Med.1996;25:561-568.
Williams CL, Bollella M, Wynder E. Preventive cardiology in primary care.  Atherosclerosis.1994;108(suppl):S117-S126.
Osborn EH, Bird JA, McPhee SJ, Rodnick JE, Fordham D. Cancer screening by primary care physicians: can we explain the differences?  J Fam Pract.1991;32:465-471.
Gann P, Melville SK, Luckmann R. Characteristics of primary care office systems as predictors of mammography utilization.  Ann Intern Med.1993;118:893-898.
Selinger HA, Goldfarb NI, Perkel RL, Lepidus B, Carlson L. Physician compliance with mammography guidelines: a retrospective chart review.  Fam Med.1989;21:56-58.
Mustin HD, Holt VL, Connel FA. Adequacy of well-child care and immunizations in US infants born in 1988.  JAMA.1994;272:1111.
Davidson W, Malloy W, Bédard M. Physician characteristics and prescribing for elderly people in New Brunswick: relation to patient outcomes.  CMAJ.1995;152:1227-1234.
Beers MH, Ouslander JG, Fingold SF.  et al.  Inappropriate medication prescribing in skilled-nursing facilities.  Ann Intern Med.1992;117:684-689.
Wilcox SM, Himmelstein DU, Woolhandler S. Inappropriate drug prescribing for the community-dwelling elderly.  JAMA.1994;272:292-296.
Lesar TS, Bricestand LL, Delcoure K, Parmalee JC, Masta-Garnic V, Pohl H. Medication prescribing errors in a teaching hospital.  JAMA.1990;263:2329-2334.
Beers MH, Storrie M, Lee G. Potential adverse drug interactions in the emergency room.  Ann Intern Med.1990;112:61-64.
Tamblyn RM, McLeod P, Abrahamowicz M.  et al.  Questionable prescribing for elderly patients in Quebec.  CMAJ.1994;150:1801-1809.
Lindley CM, Tully MP, Paramsothy V, Tallis RC. Inappropriate medication is a major cause of adverse drug reactions in elderly patients.  Age Ageing.1992;21:294-300.
Johnson JA, Bootman JL. Drug-related morbidity and mortality: a cost-of-illness model.  Arch Intern Med.1995;155:1949-1956.
Bates DW, Leape LL, Petrycki S. Incidence and preventability of adverse drug events in hospitalized adults.  J Gen Intern Med.1993;8:289-294.
Lawler F, Viviani N. Patient and physician perspectives regarding treatment of diabetes: compliance with practice guidelines.  J Fam Pract.1997;44:369-373.
Wareham NJ, Harrison BD, Jenkins PF, Nicholls J, Stableforth DE. A district confidential enquiry into deaths due to asthma.  Thorax.1993;48:1117-1120.
Nicolucci A, Cavalieri D, Scorpiglioni N.  et al. for the SID-AMD Italian Study Group for the Implementation of the St. Vincent Declaration.  A comprehensive assessment of the avoidability of long-term complications of diabetes: a case-control study.  Diabetes Care.1996;19:927-933.
Ashton CM, Kuykendall DH, Johnson ML, Wray NP, Wu L. The association between the qualtiy of inpatient care and early readmission.  Ann Intern Med.1995;122:415-421.
Legorreta AP, Christian-Herman J, O'Connor RD, Hassan MM, Evans R, Leung KM. Compliance with national asthma management guidelines and specialty care: a health maintenance organization experience.  Arch Intern Med.1998;158:457-464.
Partridge MR, Latouche D, Trako E, Thurston JG.for the UK National Asthma Task Force.  A national census of those attending UK accident and emergency departments with asthma.  J Accid Emerg Med.1997;14:16-20.
Friday GA, Khine H, Lin MS, Caliguiri LA. Profile of children requiring emergency treatment for asthma.  Ann Allergy Asthma Immunol.1997;78:221-224.
Homer CJ, Szilagyi P, Rodewald L.  et al.  Does quality of care affect rates of hospitalization for childhood asthma?  Pediatrics.1996;98:18-23.
Cockburn J, Irwig L, Turnbull D, Simpson JM, Mock P, Tattersall M. Encouraging attendance at screening mammography: knowledge, attitudes and intentions of general practitioners.  Med J Aust.1989;151:391-396.
Battista RN, Williams JI, MacFarlane LA. Determinants of preventive practices in fee-for-service primary care.  Am J Prev Med.1990;6:6-11.
Soumerai SB, McLaughlin TJ, Avorn J. Improving drug prescribing in primary care: a critical analysis of the experimental literature.  Milbank Q.1989;67:268-317.
Haynes RB, Davis D, McKibbon A, Tugwell P. A critical appraisal of the efficacy of continuing medical education.  JAMA.1984;251:61-64.
Davis DA, Thomson MA, Oxman AD, Haynes B. Changing physician performance: a systematic review to the effect of continuing medical education strategies.  JAMA.1995;274:700-705.
Tamblyn RM, Battista RN. Changing clinical practice: which interventions work?  J Continuing Educ Health Professions.1993;13:273-288.
Gabel RA. Quality assurance/peer review for recredentialing/relicensure in New York State.  Int Anesthesiol Clin.1992;30:93-101.
Bashook PG, Parboosingh J. Recertification and the maintenance of competence.  BMJ.1998;316:545-548.
Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD. Predictive validity of certification by the American Board of Internal Medicine.  Ann Intern Med.1989;110:719-726.
Reynolds GA, Chitnis JG, Roland MO. General practitioner outpatient referrals: do good doctors refer more patients to hospital?  BMJ.1991;302:1250-1252.
Calman NS, Hyman RB, Licht W. Variability in consultation rates and practitioner level of diagnostic certainty.  J Fam Pract.1992;35:31-38.
Wilkin D, Smith AG. Variation in general practitioners' referral rates to consultants.  J R Coll Gen Pract.1987;37:350-353.
Madeley RJ, Evans JR, Miur B. The use of routine referral data in the development of clinical audit and management in North Lincolnshire.  J Public Health Med.1990;12:22-27.
Knottnerus JA, Joosten J, Daams J. Comparing the quality of referrals of general practitioners with high and average referral rates: an independent panel review.  Br J Gen Pract.1990;40:178-181.
Noone A, Goldacre M, Coulter A, Seagroatt V. Do referral rates vary widely between practices and does supply of services affect demand? a study in Milton Keynes and the Oxford region.  J R Coll Gen Pract.1989;39:404-407.
Davidson W, Malloy W, Somers G, Bédard M. Relationships between physician practice characteristics and prescribing behaviour for the elderly in New Brunswick.  Can Med Assoc J.1994;150:917.
Molstad S, Hovelius B, Kroon L, Melander A. Presciption of antibiotics to out-patients in hospital clinics, community health centres and private practice.  Eur J Clin Pharmacol.1990;39:9-12.
McGavock H. Some patterns of prescribing by urban general practitioners.  Br Med J (Clin Res Ed).1988;296:900-902.
McGavock H, Wilson-Davis K, Niblock RWF. Unsuspected patterns of drug utilization revealed by interrogation of a regional general practitioner prescribing database.  Pharmacoepidemiol Drug Safety.1992;1:73-80.
Bellamy N, Brooks PM, Emmerson BT, Gilbert JR, Campell J, McCredie M. A survey of current prescribing practices of anti-inflammatory and urate-lowering drugs in gouty arthritis in New South Wales and Queensland.  Med J Aust.1989;151:531-537.
Tamblyn R, Berkson L, Dauphinee WD.  et al.  Unnecessary prescribing of NSAIDs and the management of NSAID-related gastrophy in medical practice.  Ann Intern Med.1997;127:429-438.
Ferry ME, Lamy PP, Becker LA. Physicians' knowledge of prescribing for the elderly.  J Am Geriatr Soc.1985;33:616-625.
Bradley CP. Uncomfortable prescribing decisions: a critical incident study.  BMJ.1992;304:294-296.
Hohmann AA. Gender bias in psychotropic drug prescribing in primary care.  Med Care.1989;27:478-490.
Gabe J. Towards a sociology of tranquillizer prescribing.  Br J Addict.1990;85:41-48.
Holm M. Prescription of benzodiazepines in general practice in the County of Arhus, Denmark.  Dan Med Bull.1988;35:495-499.
Van der Waals FW, Mohrs J, Foets M. Sex differences among recipients of benzodiazepines in Dutch general practice.  BMJ.1993;307:363-366.
Iliffe S. Medication review for older people in general practice.  J R Soc Med.1994;87(suppl 23):11-13.
Lexchin J. Why are we still poisoning the elderly so often?  Can Fam Physician.1993;39:2298-2300.
Holt WS, Mazzuca SA. Prescribing behaviors of family physicians in the treatment of osteoarthritis.  Fam Med.1992;24:524-527.
Committee on Safety of Medicines.  Non-steroidal anti-inflammatory drugs and serious gastrointestinal adverse reactions, I.  BMJ.1986;292:614.
Roth SH. Nonsteroidal anti-inflammatory drugs: gastropathy, deaths, and medical practice.  Ann Intern Med.1988;109:353-354.
Verhulst SJ, Distlehorst LH. Examination of nonresponse bias in a major residency follow-up study.  Acad Med.1993;68(suppl):S61-S63.
Leaf DA, Neighbor WE, Schaad D, Scott CS. A comparison of self-report and chart audit in studying resident physician assessment of cardiac risk factors.  J Gen Intern Med.1995;10:194-198.
Montano DE, Phillips WR. Cancer screening by primary care physicians: a comparison of rates obtained from physician self-report, patient survey, and chart audit.  Am J Public Health.1995;85:795-800.
Grand'Maison P, Lescop J, Rainsberry P, Brailovsky CA. Large-scale use of an objective, structured clinical examination for licensing family physicians.  CMAJ.1992;146:1735-1740.
Battista RN, Williams JI, MacFarlane LA. Determinants of primary medical practice in adult cancer prevention.  Med Care.1986;24:216-224.
Handfield-Jones R, Hollingworth GR. CFPC's certification examination: an overview.  Can Fam Physician.1990;36:2069-2074.
Livingston SA, Zieky M. Passing Scores . Princeton, NJ: Educational Testing Service; 1989.
Sirotnik KA. Introduction to matrix sampling for the practitioner. In: Popham WJ, ed. Evaluation in Education . Berkeley, Calif: McCurtchau Publishing Corp; 1974:453-529.
Régie de l'Assurance-Maladie du Québec.  Statistiques Annuelles 1992 . Québec City, Québec: Québec Gouvernement; 1994:1.
Tamblyn RM, Abrahamowicz M, Smith MC.  et al.  Final Report: Validation of Database Measures of Injury in the Elderly, Part A: Validation of Information in Physician Claims for the Measurement of Fractures, Subluxation, and Laceration. Part B: Validation of Information in the Hospitalization Database for the Measurement of Accident (E) Codes for Falls and Other Injuries and for Hip Fracture . Ottawa, Ontario: National Health Research and Development Program; 1996. Project Reference No. 6605-4441-503.
Tamblyn RM, Lavoie G, Petrella L, Monette J. The use of prescription claims databases in pharmacoepidemiological research: the accuracy and comprehensiveness of the prescription claims database in Québec.  J Clin Epidemiol.1995;48:999-1009.
Willkins R. Use of postal codes and addresses in the analysis of health data.  Health Rep.1993;5:157-177.
Beers MH, Ouslander JG, Rollingher I, Reuben DB, Brooks J, Beck JC. Explicit criteria for determining inappropriate medication use in nursing home residents.  Arch Intern Med.1991;151:1825-1832.
Tamblyn RM. Is the public being protected? prevention of suboptimal medical practice through training programs and credentialing examinations.  Eval Health Professions.1994;17:198-221.
Norcini JJ, Grosso LJ, Shea JA, Webster GD. The relationship between features of residency training and ABIM certifying examination performance.  J Gen Intern Med.1987;2:330-336.
Case SM, Swanson DB, Ripkey DR. Practically Cradle to Grave: A Look at the Relationship Between Undergraduate GPAs, MCATs, NBME Part I and II Scores, and Orthopaedic Surgery Board Scores . Presented at: American Educational Research Association; April 1995; San Francisco, Calif.
Elam CL, Johnson MM. Using Preadmission and medical school performances to predict scores on the USMLE Step 2 examination.  Acad Med.1994;69:852.
Swanson DB, Ripkey DR, Case SM.and the 1994-1995 Validity Study Group for USMLE Step 1/2 Pass/Fail Standards.  USMLE relationship between achievement in basic science coursework and performance on 1994 USMLE Step 1 test administrations.  Acad Med.1996;71(suppl):S28-S30.
Case WG, Ripkey DR, Swanson DB.and the 1994-95 Validity Study Group for USMLE Step 1 and 2 Pass/Fail Standards.  The relationship between clinical science performance in 20 medical schools and performance on Step 2 of the USMLE Licensing Examination.  Acad Med.1996;71(suppl):S31-S33.
Elam CL, Johnson MM. NBME Part 1 versus USMLE Step 1: predicting scores based on preadmission and medical school performances.  Acad Med.1994;69:155.
Dillon EC, Sergott RC, Savino PJ, Bosley TM. Diagnostic management by gatekeepers is not cost effective for neuro-ophthalmology.  Ophthalmology.1994;101:1627-1630.
Ferguson RI, Maling TJB. The Nelson general practice prescribing project, part I: a pilot audit of the regional prescribing profile.  N Z Med J.1990;103:558-560.
Lipton HL, Bird JA. Drug utilization review in ambulatory settings: state of the science and directions for outcomes research.  Med Care.1993;31:1069-1082.
Lowy FH, Williams JI. Prescription drug policy: a Canadian perspective.  Health Aff (Millwood).Spring 1991:215-216.
Anderson GM, Kerluke KJ, Pulcins IR, Hertzman C, Barer ML. Trends and determinants of prescription drug expenditures in the elderly: data from the British Columbia Pharmacare Program.  Inquiry.1993;30:199-207.
Santell JP. Projecting future drug expenditures.  Am J Hosp Pharm.1994;51:177-187.
The National Pharmaceutical Strategy Office.  National Pharmaceutical Strategy Discussion Document . Ottawa, Ontario: Drugs Directorate, Health Protection Branch; 1993.
Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies.  JAMA.1998;279:1200-1205.
Lipton HL, Byrns PJ, Soumerai SB, Chrischilles EA. Pharmacists as agents of change for rational drug therapy.  Int J Technol Assess Health Care.1995;11:485-508.
Soumerai SB, Avorn J. Economic and policy analysis of university-based drug "detailing.".  Med Care.1986;24:313-331.
Fox SA, Murata PJ, Stein JA. The impact of physician compliance on screening mammography for older women.  Arch Intern Med.1991;151:50-56.
Kruse J, Phillips DM. Factors influencing women's decision to undergo mammography.  Obstet Gynecol.1987;70:744-748.
Coll PP, O'Connor PJ, Crabtree BF, Besdine RW. Effects of age, education, and physician advice on utilization of screening mammography.  J Am Geriatr Soc.1989;37:957-962.
O'Connor AM, Perrault DJ. Importance of physician's role highlighted in survey of women's breast screening practices.  Can J Public Health.1995;86:42-45.
Yaphe J, Sangster J, Guttman D. Chart audit of mammography in a family practice.  Fam Physician.1992;38:2314-2317.
Maheux B, Pineault R, Lambert J, Beland F, Berthiaume B. Factors influencing physicians' preventive practices.  Am J Prev Med.1989;5:201-206.

Figures

Tables

Table Graphic Jump LocationTable 1.—Standardized Examination (z) Scores for the 614 Physicians Eligible for Inclusion in the Study Population*
Table Graphic Jump LocationTable 2.—Association Between Licensing Examination Scores and Practice Activity After Adjustment for Practice Population*

References

Canadian Medical Association Working Group on Primary Care.  Strengthening the Foundation: The Role of the Physician in Primary Health Care in Canada . Ottawa, Ontario: Canadian Medical Association; 1994:1.
Gervas J, Perez-Fernandez M, Starfield BH. Primary care, financing and gatekeeping in western Europe.  Fam Pract.1994;11:307-317.
Franks P, Clancy CM. Gatekeeping revisited: protecting patients from overtreatment.  N Engl J Med.1992;327:424-429.
Wright RA. Community-oriented primary care: the cornerstone of health care reform.  JAMA.1993;269:2544-2547.
Bindman AB, Grumbach K, Osmond D, Vranizan K, Stewart AL. Primary care and receipt of preventive services.  J Gen Intern Med.1996;11:269-276.
Salive ME, Guralnik JM, Brock D. Preventive services for breast and cervical cancer in US office-based practices.  Prev Med.1996;25:561-568.
Williams CL, Bollella M, Wynder E. Preventive cardiology in primary care.  Atherosclerosis.1994;108(suppl):S117-S126.
Osborn EH, Bird JA, McPhee SJ, Rodnick JE, Fordham D. Cancer screening by primary care physicians: can we explain the differences?  J Fam Pract.1991;32:465-471.
Gann P, Melville SK, Luckmann R. Characteristics of primary care office systems as predictors of mammography utilization.  Ann Intern Med.1993;118:893-898.
Selinger HA, Goldfarb NI, Perkel RL, Lepidus B, Carlson L. Physician compliance with mammography guidelines: a retrospective chart review.  Fam Med.1989;21:56-58.
Mustin HD, Holt VL, Connel FA. Adequacy of well-child care and immunizations in US infants born in 1988.  JAMA.1994;272:1111.
Davidson W, Malloy W, Bédard M. Physician characteristics and prescribing for elderly people in New Brunswick: relation to patient outcomes.  CMAJ.1995;152:1227-1234.
Beers MH, Ouslander JG, Fingold SF.  et al.  Inappropriate medication prescribing in skilled-nursing facilities.  Ann Intern Med.1992;117:684-689.
Wilcox SM, Himmelstein DU, Woolhandler S. Inappropriate drug prescribing for the community-dwelling elderly.  JAMA.1994;272:292-296.
Lesar TS, Bricestand LL, Delcoure K, Parmalee JC, Masta-Garnic V, Pohl H. Medication prescribing errors in a teaching hospital.  JAMA.1990;263:2329-2334.
Beers MH, Storrie M, Lee G. Potential adverse drug interactions in the emergency room.  Ann Intern Med.1990;112:61-64.
Tamblyn RM, McLeod P, Abrahamowicz M.  et al.  Questionable prescribing for elderly patients in Quebec.  CMAJ.1994;150:1801-1809.
Lindley CM, Tully MP, Paramsothy V, Tallis RC. Inappropriate medication is a major cause of adverse drug reactions in elderly patients.  Age Ageing.1992;21:294-300.
Johnson JA, Bootman JL. Drug-related morbidity and mortality: a cost-of-illness model.  Arch Intern Med.1995;155:1949-1956.
Bates DW, Leape LL, Petrycki S. Incidence and preventability of adverse drug events in hospitalized adults.  J Gen Intern Med.1993;8:289-294.
Lawler F, Viviani N. Patient and physician perspectives regarding treatment of diabetes: compliance with practice guidelines.  J Fam Pract.1997;44:369-373.
Wareham NJ, Harrison BD, Jenkins PF, Nicholls J, Stableforth DE. A district confidential enquiry into deaths due to asthma.  Thorax.1993;48:1117-1120.
Nicolucci A, Cavalieri D, Scorpiglioni N.  et al. for the SID-AMD Italian Study Group for the Implementation of the St. Vincent Declaration.  A comprehensive assessment of the avoidability of long-term complications of diabetes: a case-control study.  Diabetes Care.1996;19:927-933.
Ashton CM, Kuykendall DH, Johnson ML, Wray NP, Wu L. The association between the qualtiy of inpatient care and early readmission.  Ann Intern Med.1995;122:415-421.
Legorreta AP, Christian-Herman J, O'Connor RD, Hassan MM, Evans R, Leung KM. Compliance with national asthma management guidelines and specialty care: a health maintenance organization experience.  Arch Intern Med.1998;158:457-464.
Partridge MR, Latouche D, Trako E, Thurston JG.for the UK National Asthma Task Force.  A national census of those attending UK accident and emergency departments with asthma.  J Accid Emerg Med.1997;14:16-20.
Friday GA, Khine H, Lin MS, Caliguiri LA. Profile of children requiring emergency treatment for asthma.  Ann Allergy Asthma Immunol.1997;78:221-224.
Homer CJ, Szilagyi P, Rodewald L.  et al.  Does quality of care affect rates of hospitalization for childhood asthma?  Pediatrics.1996;98:18-23.
Cockburn J, Irwig L, Turnbull D, Simpson JM, Mock P, Tattersall M. Encouraging attendance at screening mammography: knowledge, attitudes and intentions of general practitioners.  Med J Aust.1989;151:391-396.
Battista RN, Williams JI, MacFarlane LA. Determinants of preventive practices in fee-for-service primary care.  Am J Prev Med.1990;6:6-11.
Soumerai SB, McLaughlin TJ, Avorn J. Improving drug prescribing in primary care: a critical analysis of the experimental literature.  Milbank Q.1989;67:268-317.
Haynes RB, Davis D, McKibbon A, Tugwell P. A critical appraisal of the efficacy of continuing medical education.  JAMA.1984;251:61-64.
Davis DA, Thomson MA, Oxman AD, Haynes B. Changing physician performance: a systematic review to the effect of continuing medical education strategies.  JAMA.1995;274:700-705.
Tamblyn RM, Battista RN. Changing clinical practice: which interventions work?  J Continuing Educ Health Professions.1993;13:273-288.
Gabel RA. Quality assurance/peer review for recredentialing/relicensure in New York State.  Int Anesthesiol Clin.1992;30:93-101.
Bashook PG, Parboosingh J. Recertification and the maintenance of competence.  BMJ.1998;316:545-548.
Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD. Predictive validity of certification by the American Board of Internal Medicine.  Ann Intern Med.1989;110:719-726.
Reynolds GA, Chitnis JG, Roland MO. General practitioner outpatient referrals: do good doctors refer more patients to hospital?  BMJ.1991;302:1250-1252.
Calman NS, Hyman RB, Licht W. Variability in consultation rates and practitioner level of diagnostic certainty.  J Fam Pract.1992;35:31-38.
Wilkin D, Smith AG. Variation in general practitioners' referral rates to consultants.  J R Coll Gen Pract.1987;37:350-353.
Madeley RJ, Evans JR, Miur B. The use of routine referral data in the development of clinical audit and management in North Lincolnshire.  J Public Health Med.1990;12:22-27.
Knottnerus JA, Joosten J, Daams J. Comparing the quality of referrals of general practitioners with high and average referral rates: an independent panel review.  Br J Gen Pract.1990;40:178-181.
Noone A, Goldacre M, Coulter A, Seagroatt V. Do referral rates vary widely between practices and does supply of services affect demand? a study in Milton Keynes and the Oxford region.  J R Coll Gen Pract.1989;39:404-407.
Davidson W, Malloy W, Somers G, Bédard M. Relationships between physician practice characteristics and prescribing behaviour for the elderly in New Brunswick.  Can Med Assoc J.1994;150:917.
Molstad S, Hovelius B, Kroon L, Melander A. Presciption of antibiotics to out-patients in hospital clinics, community health centres and private practice.  Eur J Clin Pharmacol.1990;39:9-12.
McGavock H. Some patterns of prescribing by urban general practitioners.  Br Med J (Clin Res Ed).1988;296:900-902.
McGavock H, Wilson-Davis K, Niblock RWF. Unsuspected patterns of drug utilization revealed by interrogation of a regional general practitioner prescribing database.  Pharmacoepidemiol Drug Safety.1992;1:73-80.
Bellamy N, Brooks PM, Emmerson BT, Gilbert JR, Campell J, McCredie M. A survey of current prescribing practices of anti-inflammatory and urate-lowering drugs in gouty arthritis in New South Wales and Queensland.  Med J Aust.1989;151:531-537.
Tamblyn R, Berkson L, Dauphinee WD.  et al.  Unnecessary prescribing of NSAIDs and the management of NSAID-related gastrophy in medical practice.  Ann Intern Med.1997;127:429-438.
Ferry ME, Lamy PP, Becker LA. Physicians' knowledge of prescribing for the elderly.  J Am Geriatr Soc.1985;33:616-625.
Bradley CP. Uncomfortable prescribing decisions: a critical incident study.  BMJ.1992;304:294-296.
Hohmann AA. Gender bias in psychotropic drug prescribing in primary care.  Med Care.1989;27:478-490.
Gabe J. Towards a sociology of tranquillizer prescribing.  Br J Addict.1990;85:41-48.
Holm M. Prescription of benzodiazepines in general practice in the County of Arhus, Denmark.  Dan Med Bull.1988;35:495-499.
Van der Waals FW, Mohrs J, Foets M. Sex differences among recipients of benzodiazepines in Dutch general practice.  BMJ.1993;307:363-366.
Iliffe S. Medication review for older people in general practice.  J R Soc Med.1994;87(suppl 23):11-13.
Lexchin J. Why are we still poisoning the elderly so often?  Can Fam Physician.1993;39:2298-2300.
Holt WS, Mazzuca SA. Prescribing behaviors of family physicians in the treatment of osteoarthritis.  Fam Med.1992;24:524-527.
Committee on Safety of Medicines.  Non-steroidal anti-inflammatory drugs and serious gastrointestinal adverse reactions, I.  BMJ.1986;292:614.
Roth SH. Nonsteroidal anti-inflammatory drugs: gastropathy, deaths, and medical practice.  Ann Intern Med.1988;109:353-354.
Verhulst SJ, Distlehorst LH. Examination of nonresponse bias in a major residency follow-up study.  Acad Med.1993;68(suppl):S61-S63.
Leaf DA, Neighbor WE, Schaad D, Scott CS. A comparison of self-report and chart audit in studying resident physician assessment of cardiac risk factors.  J Gen Intern Med.1995;10:194-198.
Montano DE, Phillips WR. Cancer screening by primary care physicians: a comparison of rates obtained from physician self-report, patient survey, and chart audit.  Am J Public Health.1995;85:795-800.
Grand'Maison P, Lescop J, Rainsberry P, Brailovsky CA. Large-scale use of an objective, structured clinical examination for licensing family physicians.  CMAJ.1992;146:1735-1740.
Battista RN, Williams JI, MacFarlane LA. Determinants of primary medical practice in adult cancer prevention.  Med Care.1986;24:216-224.
Handfield-Jones R, Hollingworth GR. CFPC's certification examination: an overview.  Can Fam Physician.1990;36:2069-2074.
Livingston SA, Zieky M. Passing Scores . Princeton, NJ: Educational Testing Service; 1989.
Sirotnik KA. Introduction to matrix sampling for the practitioner. In: Popham WJ, ed. Evaluation in Education . Berkeley, Calif: McCurtchau Publishing Corp; 1974:453-529.
Régie de l'Assurance-Maladie du Québec.  Statistiques Annuelles 1992 . Québec City, Québec: Québec Gouvernement; 1994:1.
Tamblyn RM, Abrahamowicz M, Smith MC.  et al.  Final Report: Validation of Database Measures of Injury in the Elderly, Part A: Validation of Information in Physician Claims for the Measurement of Fractures, Subluxation, and Laceration. Part B: Validation of Information in the Hospitalization Database for the Measurement of Accident (E) Codes for Falls and Other Injuries and for Hip Fracture . Ottawa, Ontario: National Health Research and Development Program; 1996. Project Reference No. 6605-4441-503.
Tamblyn RM, Lavoie G, Petrella L, Monette J. The use of prescription claims databases in pharmacoepidemiological research: the accuracy and comprehensiveness of the prescription claims database in Québec.  J Clin Epidemiol.1995;48:999-1009.
Willkins R. Use of postal codes and addresses in the analysis of health data.  Health Rep.1993;5:157-177.
Beers MH, Ouslander JG, Rollingher I, Reuben DB, Brooks J, Beck JC. Explicit criteria for determining inappropriate medication use in nursing home residents.  Arch Intern Med.1991;151:1825-1832.
Tamblyn RM. Is the public being protected? prevention of suboptimal medical practice through training programs and credentialing examinations.  Eval Health Professions.1994;17:198-221.
Norcini JJ, Grosso LJ, Shea JA, Webster GD. The relationship between features of residency training and ABIM certifying examination performance.  J Gen Intern Med.1987;2:330-336.
Case SM, Swanson DB, Ripkey DR. Practically Cradle to Grave: A Look at the Relationship Between Undergraduate GPAs, MCATs, NBME Part I and II Scores, and Orthopaedic Surgery Board Scores . Presented at: American Educational Research Association; April 1995; San Francisco, Calif.
Elam CL, Johnson MM. Using Preadmission and medical school performances to predict scores on the USMLE Step 2 examination.  Acad Med.1994;69:852.
Swanson DB, Ripkey DR, Case SM.and the 1994-1995 Validity Study Group for USMLE Step 1/2 Pass/Fail Standards.  USMLE relationship between achievement in basic science coursework and performance on 1994 USMLE Step 1 test administrations.  Acad Med.1996;71(suppl):S28-S30.
Case WG, Ripkey DR, Swanson DB.and the 1994-95 Validity Study Group for USMLE Step 1 and 2 Pass/Fail Standards.  The relationship between clinical science performance in 20 medical schools and performance on Step 2 of the USMLE Licensing Examination.  Acad Med.1996;71(suppl):S31-S33.
Elam CL, Johnson MM. NBME Part 1 versus USMLE Step 1: predicting scores based on preadmission and medical school performances.  Acad Med.1994;69:155.
Dillon EC, Sergott RC, Savino PJ, Bosley TM. Diagnostic management by gatekeepers is not cost effective for neuro-ophthalmology.  Ophthalmology.1994;101:1627-1630.
Ferguson RI, Maling TJB. The Nelson general practice prescribing project, part I: a pilot audit of the regional prescribing profile.  N Z Med J.1990;103:558-560.
Lipton HL, Bird JA. Drug utilization review in ambulatory settings: state of the science and directions for outcomes research.  Med Care.1993;31:1069-1082.
Lowy FH, Williams JI. Prescription drug policy: a Canadian perspective.  Health Aff (Millwood).Spring 1991:215-216.
Anderson GM, Kerluke KJ, Pulcins IR, Hertzman C, Barer ML. Trends and determinants of prescription drug expenditures in the elderly: data from the British Columbia Pharmacare Program.  Inquiry.1993;30:199-207.
Santell JP. Projecting future drug expenditures.  Am J Hosp Pharm.1994;51:177-187.
The National Pharmaceutical Strategy Office.  National Pharmaceutical Strategy Discussion Document . Ottawa, Ontario: Drugs Directorate, Health Protection Branch; 1993.
Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies.  JAMA.1998;279:1200-1205.
Lipton HL, Byrns PJ, Soumerai SB, Chrischilles EA. Pharmacists as agents of change for rational drug therapy.  Int J Technol Assess Health Care.1995;11:485-508.
Soumerai SB, Avorn J. Economic and policy analysis of university-based drug "detailing.".  Med Care.1986;24:313-331.
Fox SA, Murata PJ, Stein JA. The impact of physician compliance on screening mammography for older women.  Arch Intern Med.1991;151:50-56.
Kruse J, Phillips DM. Factors influencing women's decision to undergo mammography.  Obstet Gynecol.1987;70:744-748.
Coll PP, O'Connor PJ, Crabtree BF, Besdine RW. Effects of age, education, and physician advice on utilization of screening mammography.  J Am Geriatr Soc.1989;37:957-962.
O'Connor AM, Perrault DJ. Importance of physician's role highlighted in survey of women's breast screening practices.  Can J Public Health.1995;86:42-45.
Yaphe J, Sangster J, Guttman D. Chart audit of mammography in a family practice.  Fam Physician.1992;38:2314-2317.
Maheux B, Pineault R, Lambert J, Beland F, Berthiaume B. Factors influencing physicians' preventive practices.  Am J Prev Med.1989;5:201-206.
CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 66

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Topics
PubMed Articles
JAMAevidence.com

Users' Guides to the Medical Literature
Chapter e23. How to Use an Article About Quality Improvement