The GAAP in Quality Measurement and Reporting

Catalyzed by evidence of poor-quality care and remarkable variations in processes and outcomes, the interest in quality measurement has increased exponentially. Manifestations of this interest include widespread promulgation of quality measures, an increase in public reporting of these measures, and early experiments in paying for quality.^{1 - 2} Now that quality of care is being measured rather than assumed, there seems little doubt that better quality scores will lead to major competitive advantages for clinicians and organizations.

Although many quality measures are used internally by health care organizations to improve quality of care, an increasing number of measures are being reported publicly. Yet the measurement of quality in health care is neither standardized nor consistently accurate and reliable. Because any organization or company can advertise the quality of its products, it is important to hold health care quality measures to a higher standard than claims about, for example, household products. Without such assurances, publicly reported quality measures may misinform and possibly erode the trust that underlies the medical profession. If patients cannot trust what is reported, they may lose confidence in the quality of care that is provided.

Invalid publicly reported quality-of-care measures pose significant risks to patients, clinicians, and potentially to payers. Patients might choose care according to misinformation and make poor decisions.^{3 - 4} Health care organizations may become overconfident about the quality of care provided and reduce or eliminate improvement efforts and introduce preventable harm. Payers may mistakenly provide financial rewards, channel patients to low-quality clinicians, or make inaccurate inferences about the value (quality per cost) they purchase.^{5 - 7}

In support of this movement to make health care quality and value⁵ transparent, the young science of measuring quality is maturing rapidly, although in a relatively frenetic manner. These efforts are promoted by a deeper understanding of 2 issues that markedly influence the validity of quality measures: do the measures truly represent the quality of care provided, and are the results tainted by systematic and random error?⁸ This first issue has experienced considerable flux after recent statements that certain widely promoted measures are problematic.⁹ For example, the measure of door-to-antibiotics time for patients with pneumonia (publicly reported by the Centers for Medicare & Medicaid Services [CMS]) has been critiqued as being poorly supported by the literature and promoting significant amounts of antibiotic overuse.¹⁰ Even better-established measures, such as those targeting acute myocardial infarction management, have demonstrated marginal relationships to ultimate outcomes.¹¹ Still others, such as smoking cessation counseling, could be important if measured correctly; however, this measure is assessed by whether a box has been checked—likely a poor surrogate for the true quality and impact of counseling.¹²

To truly reflect the quality of care, measures must not only be supported by evidence but also capture the phenomenon of interest (the intervention that is associated with improved outcomes) in the context of actual practice.¹³ Efforts to make measures less burdensome often pose problems for the latter requirement. For example, rather than measure whether surgical patients receive prophylaxis for deep venous thrombosis daily, whether the patient has an order for prophylaxis any time in the postoperative period is measured; how this correlates with clinical outcomes is uncertain.

As the stakes have increased, an infrastructure has developed to ensure that some quality measures are scientifically sound. For example, some measures approved by the National Quality Forum or developed by the CMS or The Joint Commission undergo an extensive vetting process to help improve their validity. When measures prove invalid (because of poor or changing science), there are mechanisms for reexamination and potential refinement. Some mechanisms are formal (planned rereviews), whereas others involve inevitable pushback from organizations and researchers who believe the measures are inappropriate. This transparent environment of systems to create and reassess measures helps ensure that quality measures are relatively valid.

Even if measures are supported by evidence, whether the results are tainted by error must be considered. Does the measure have standard definitions, inclusion and exclusion criteria, and methods of surveillance? Is staff trained to use these measures? Has a data quality-control plan that includes reporting and minimizing missing data been implemented? Is risk adjustment needed or the amount of random error estimated? Although measures publicly reported by federal agencies (eg, CMS measures) have mechanisms to minimize error (penalties of censure, legal action, or no reimbursement if submitted data are found to be fraudulent), no such mechanisms exist for other measures that hospitals voluntarily report publicly.

Many organizations are now reporting quality data on Web sites and in brochures, independent of national reporting initiatives, but there is no assurance of their accuracy. As quality measurement expands, new measures of uncertain or poor validity are being promulgated, with little assurance about their validity. These measures may lead patients to change physicians or hospitals or payers to change reimbursement or preferred providers.^{3 ,14} All of this raises another key and largely unexplored question: how valid are the data that health care organizations independently report about their own quality?

Examples of problematic reports on health care organization Web sites are easy to find. For example, one hospital Web site reported that it saved 242 lives during 18 months (4 lives/1000 discharges). But the sample size, methods of risk adjustment, and a measure of precision (eg, confidence intervals) for their mortality estimates were not reported. Another hospital Web site indicated that 90% of patients with pneumonia were screened and given pneumococcal vaccination, whereas the CMS's Hospital Compare Web site¹⁵ on the same day reported that only 64% of patients at that hospital were vaccinated. Hospital Compare has a significant time delay, so it is possible that this hospital's reported screening and vaccination rate was accurate. However, the hospital Web site did not post dates of data collection, the sample size, or the confidence interval, making it impossible to confirm the veracity of the data or account for differences in the reports.

Another hospital reported the ratio of central line–associated bloodstream infections and ventilator-associated pneumonia compared with “CDC national averages.” Yet the hospital did not describe the Centers for Disease Control and Prevention benchmark rates used, which vary by intensive care unit type, or the number of patients or period included. Not surprisingly, this hospital performed well, with both a ventilator-associated pneumonia ratio and a central line–associated bloodstream infection ratio of 0.00. For how long have these rates been zero? Does this mean that patients will never get these infections at this hospital? Without more detail, it is difficult to know how to interpret these data, including whether this performance is really laudable.¹⁶

The theme emerging from these and other examples is that health care organizations are rapidly embracing quality measurement and reporting. Invalid measures used internally by health care organizations for quality improvement may misinform staff and senior leaders yet pose few risks to the public.⁹ On the other hand, invalid measures that are publicly reported for nationally mandated efforts or for hospitals' broader quality-improvement efforts may misinform the public and erode trust. Of great concern are the rapidly growing measures that hospitals voluntarily develop and publicly report. Although this enthusiasm is good and long overdue, there is little assurance that these publicly reported measures are accurate, including whether there are unintentional biases or outright falsehoods. Even though our search of hospital Web sites was limited and informal, we could not find an example of a hospital that reported poor performance on voluntarily reported performance measures.

Health care should ensure that the resources spent on quality measurement are used efficiently and effectively to improve patient outcomes and reduce costs. As the burgeoning area of quality measurement and reporting is examined, how other industries report high-stakes data for public consumption should be considered. The reporting of clinical quality measures should be contrasted with the reporting of financial performance by industry and drug treatment information by pharmaceutical companies.

Investors generally have confidence that the figures in financial reports, including those of hospitals, are correct. The Securities and Exchange Commission designated and authorized the Financial Accounting Standards Board, an independent body, to establish and improve standards for financial accounting and reporting. “Such standards are essential to the efficient functioning of the economy because investors, creditors, auditors, and others rely on credible, transparent, and comparable financial information.”¹⁷ These standards force organizations to comply with generally accepted accounting principles (GAAP) in reporting data.

Pharmaceutical companies must adhere to the Code of Federal Regulations when making statements to consumers about drug products. Promotional claims by companies must be supported by evidence or clinical experience and, consistent with product labeling, must include a balanced discussion of risks and benefits and must not be misleading or otherwise false. The oversight of these communications comes predominantly from the US Food and Drug Administration's Division of Drug Marketing, Advertising, and Communications. Failure of companies to follow the guidelines can result in enforcement action by the agency.¹⁸

Public reporting of quality measures should have the same reporting standards as financial and pharmaceutical data entities. Their systems have explicit rules to reduce bias in reports, well-trained and qualified professionals who oversee the reports, mechanisms for auditing the validity of the reports, and methods to hold organizations accountable for the reports. Except for a relatively small number of national measures, health care organizations lack guidance on how to develop and report scientifically sound measures. The stakes are high and increasing. Invalid quality-of-care measures pose significant risks to patients, payers, clinicians, and, perhaps most important, public trust in the medical profession.

To improve the reporting of quality measures, we recommend a partnership between public and private stakeholders to start a dialogue and move to what the Securities and Exchange Commission did in 1934—ensure the accuracy of financial data. This new panel could do the following: first, define standards for measuring and reporting quality of care. These standards, informed by clinical research methods, could describe how to reduce and make transparent selection bias, measurement bias, analytic bias, confounding, and random error.⁸

Second, the panel could describe the optimal training and certification needed to measure and report quality of care. It is difficult to imagine a chief financial officer without advanced training in financial management. Yet many chief medical officers, chief quality officers, chief safety officers, and directors of quality lack formal training in quality measurement, such as epidemiology. Training in quality improvement without a rigorous training program evaluation is likely insufficient. Once the required competencies are defined, training and certification programs (eg, master's or doctoral degree) in epidemiology or health services research from a school of public health could emerge.

Third, the panel might design an auditing system to ensure that quality reports are accurate and potential biases transparent.

Fourth, the panel could develop a system to ensure that health care organizations are held accountable for the quality of care they report. These auditing and accountability efforts should be informed by what does and does not work when financial and pharmaceutical data are reported. Although some of these rules certainly are burdensome, costly, and likely do little to protect consumers, they appear to work and provide the public with credible financial and pharmaceutical information. Health care would be wise to learn from these efforts. An important unanswered question for this much-needed system is who will finance it. Yet the costs may help the health care industry improve the efficiency and effectiveness of quality measurement.

The quality movement has matured. Improving its value and making measurement transparent will likely be the cornerstone of efforts to improve quality in health care. Improvements in measurement could help provide a basis for the evaluation of health policy changes.¹⁹ Yet today's infrastructure for accurately measuring and publicly reporting quality is inadequate. The time has come to create a stronger environment to help protect patients, clinicians, and payers from misinformation regarding quality of care and to help ensure public trust in the health care profession.

Health care can learn from the reporting standards for financial data and pharmaceutical information and create standards for reporting, certification for those who report, methods for auditing performance data, and strong systems of accountability. Without significant improvements in the reporting of health care quality, the promise of value-based health care purchasing will remain unrealized.