Assessing Quality of Care: Title and subTitle BreakKnowledge Matters

Significant quality-of-care gaps are well documented in the United States.^{1 - 2} These reports have focused mostly on underuse of performance measures of important processes of care, and some outcomes of care.¹ Others have argued that the cause of underuse of these evidence-based processes of care is usually not deficient physician knowledge about whether to perform the examination or order the test, but rather poorly designed, dysfunctional microsystems of care unable to deliver effective, efficient, and reliable care.^{3 - 4}

Consequently, much of the recent work in quality improvement has focused on changing microsystems of care “to deliver the right care for the right patient at the right time, all the time.”⁵ What is often overlooked in quality improvement, but equally important, is that effective microsystems must have highly competent clinicians, who possess sufficient knowledge and clinical skills to make and execute evidence-based decisions, exercise informed clinical judgment, and deal effectively with uncertainty.⁶ Clinical judgment and the ability to deal with uncertainty are especially critical with respect to misuse and overuse of processes of care. Misuse and overuse of processes of care (eg, overprescribing antibiotics and unnecessary imaging and procedures) put patients at greater risk for unnecessary complications.^{7 - 8}

Physician knowledge and clinical judgment also are central to making correct diagnoses.⁶ The majority of current performance measures assume a correct diagnosis, but more than that current measures cover only a fraction of the myriad health problems seen by physicians on a daily basis and likely will never address unusual or less common but no less important or serious conditions. Furthermore, many symptoms and signs that prompt patients to see physicians are often not well-defined and a diagnosis often remains uncertain after the initial visit.⁹ Clinical judgment is crucial in determining when further intervention is necessary or when watchful waiting may be the best approach. Even when an accurate diagnosis is made, prudent clinical judgment is necessary to determine appropriate care, including the correct diagnostic tests, critical to the efficiency and effectiveness aspects of quality.

Our objectives in this Commentary are to discuss the relationship between medical knowledge and quality and how the secure examination component of specialty board certification—with its primary focus on assessing physician knowledge, diagnostic acumen, and clinical judgment—is an important complement to current performance measures. Recognizing this importance, in 2006 the American Board of Internal Medicine instituted a new requirement for all physicians with time-limited certificates to evaluate their performance in practice to address physician competence in practice-based learning and improvement and systems-based practice. We hope this discussion will stimulate dialogue about the need for more comprehensive physician performance measurement in the era of public reporting.

The last 20 years has witnessed a rapid expansion of the understanding of how physicians integrate medical knowledge and clinical skills in the clinical judgment process. The crux of this process is the creation of a problem representation, which refers to what the physician thinks is going on with the patient (eg, diagnosis or state of his or her medical condition) based on the synthesis and integration of 2 key elements: information collected from the patient through an accurate, complete medical history and focused physical examination, and the physician's working medical knowledge.⁶ Physicians must then evaluate their synthesis before moving to the last step in this complex process, which is the action or management step (ie, the processes of care such as ordering a test for diabetes or prescribing an antibiotic for an infection). A physician must accurately and skillfully perform substantial information processing and clinical judgment before he or she can execute appropriate management, testing, or therapy for the patient. Prior knowledge is crucial and physicians cannot rely solely on looking everything up but must start with some basic level of knowledge and understanding. Without this, a physician will not recognize gaps in his or her knowledge.

Surprisingly, the quality of diagnosis and clinical reasoning has received little attention in the quality literature. However, researchers working on clinical reasoning have found that diagnostic errors are prevalent and consequential among physicians and may not simply resolve with more practice experience.^{10 - 11} For example, a recent study found that adverse events in a hospital setting were a function not only of system-related issues but also problems with physician clinical judgment skills. The majority of these judgment errors were categorized as faulty synthesis, such as not considering alternative clinical diagnoses after the initial diagnosis is made or misjudging the importance of a patient's clinical findings.¹²

Research suggests that, on average, clinical skills tend to decline over time; a meta-analysis by Choudry and Fletcher¹¹ illustrates that practice does not make perfect and supports the argument that physicians must engage in continuous professional development, including board certification, to retain competency. A study by Eva¹³ highlighted the problem of aging knowledge among experienced physicians; older physicians may tend to rely too heavily on nonanalytic thinking (eg, pattern recognition) over time, leading to premature closure about a diagnosis. Nonanalytic thinking is likely to be acceptable and efficient as long as the medical knowledge for a specific medical condition does not change. However, when new knowledge emerges that should change the approach to patient care, the physician must incorporate this new information into clinical reasoning. Evidence suggests, however, that this does not happen effectively over time. In a study of a recertification examination, Day et al¹⁴ found that physicians did much better on test questions of stable, unchanged medical knowledge than on new knowledge developed since their medical training.

Specialty certification board examinations provide a cognitive simulator that can assess the ability of physicians to integrate multiple pieces of clinical information required for effective clinical judgment. Decades of research work in test development and psychometrics¹⁵ has led to current high-stakes cognitive examinations with high reliability and reproducibility (reliability coefficients consistently above 0.9), as well as face, concurrent and construct validity. Secure examinations of medical knowledge and clinical judgment can provide an effective means to assess whether physicians have incorporated new knowledge over time. This is one reason an examination is a required part of maintenance of certification programs.

There is evidence to support the link between board certification examinations and quality.^{16 - 23} For instance, Norcini et al²¹ found that mortality was lower for patients with acute myocardial infarction cared for by certified physicians. In another study, Prystowsky¹⁹ reported that certification in surgery was a significant predictor of lower mortality and complication rates for colorectal surgery. Pham et al²⁰ found an association between the rate at which preventive care services were delivered for Medicare patients and the certification status in internal medicine or family medicine. More research is needed to better understand how an examination of knowledge links to quality of care, but it should be reassuring that evidence exists that medical knowledge, one of the foundational competencies for clinical practice, matters.

Understandably, taking a high-stakes examination is rarely cited by physicians as one of their favorite activities—it is anxiety-provoking and requires a monetary cost and time to prepare. Yet the majority of physicians who have taken the American Board of Internal Medicine's maintenance of certification examination reported on the survey they completed at the end of the examination that the content was fair and relevant (Louis Grosso, written communication). Perhaps more importantly, the public expects, in return for the privilege of self-regulation, that physicians undergo a rigorous, periodic examination of knowledge.²⁴ The willingness to demonstrate competence in knowledge through a secure examination can be viewed as an important act of professionalism, consistent with the charter on medical professionalism's commitment to maintain professional competence.²⁵ In return, certification boards are obligated to ensure their examinations are a relevant and meaningful measure of cognitive competence.

Understanding the relationship between cognitive skills and quality over a physician's practice career is just beginning to be explored and elucidated.^{11 ,13}

The research efforts of the medical specialty boards and others should continue to build the evidence base on the importance of medical knowledge as an essential component of the quality calculus. Failure to do so will leave large gaps in the comprehensive performance measurement of patient care; knowledge matters to patients too. New research should focus on the development of rigorous and comprehensive assessment measures for key aspects of clinical judgment such as dealing with uncertainty, the relationship between knowledge and misuse and overuse measures, and skill in answering clinical questions at the point of care. Physicians and the public should ask how assessment of knowledge can facilitate and accelerate quality improvement.