Transparency Standards for Diabetes Performance Measures

Regardless of the direction US health care reform takes, performance measurement, public reporting, and accountability will play major roles. Presently, a number of entities have staked out business models for developing, collecting, and disseminating operational data about performance at the hospital, health plan, and clinician levels. Critical to such a system are valid measures that permit fair comparisons. Indeed, generalization of randomized controlled trials to larger populations for performance measurement is a complex process that invariably involves judgment of the health benefit of the proposed measure as well as technical issues. Such factors include importance, scientific evidence, reproducibility, validity, precision, specification (including inclusion and exclusion criteria for the measure denominator), and feasibility in practice. The validity of methods measurement development, and adoption must be transparent. Transparency has been defined as “a process by which information about existing conditions, decisions and actions is made accessible, visible and understandable.”¹ One concern is that in the rush to value-based purchasing, the views into the process by which measures are vetted are opaque, rather than transparent.

While there is an emerging literature about industry influence on the development of professional society guidelines that may inform performance measurement,² the industry influence on organizations that develop, endorse, and report on performance measurement is less well appreciated.³ Because of its high profile, the initial adoption and subsequent revision of glycemic control measures for diabetes provides a unique window into both the internal and external factors that can shape measure development.

Hemoglobin A_1c control is an important intermediate outcome for measuring quality of care provided to the more than 20 million US individuals with diabetes because it is the major determinant of microvascular complications. Moreover, considerable debate has occurred over a decade concerning what hemoglobin A_1c levels are appropriate for public reporting when quality assessments are compared across multiple health care systems as opposed to internal use for quality improvement. The Diabetes Quality Improvement Project, a federal–private sector coalition, was formed in 1997 to develop a diabetes measure set. However, the factors that led to decision making, especially for the hemoglobin A_1c level, were described in an article published nearly 4 years later.⁴ Specifically, it was noted that hemoglobin A_1c threshold values lower than the accountability measure (then 9.5%) were made quality-improvement measures because of the inability to adjust outcomes for case mix, a sine qua non for fairly comparing health plans and clinicians.⁴

The technical specifications for the measures were developed by the National Committee for Quality Assurance (NCQA), which branded the set as the NCQA Comprehensive Diabetes Care Measure Set. Subsequent revision of the measure set became the responsibility of the National Diabetes Quality Improvement Alliance, which was a coalition of 13 private and public national organizations dedicated to developing and maintaining a national performance measurement set for diabetes from 2001 through 2006. During this time frame, a number of quality-improvement and public service campaigns to improve glycemic control were established.

In 2002, the Aim-Believe-Achieve: Diabetes A_1C Initiative was funded by Aventis Pharmaceuticals, developed by the marketing firm Burson-Marsteller, and launched on World Diabetes Day.⁵ This national campaign involved more than 2 dozen medical organizations and national political leaders and encouraged patients with diabetes to have their hemoglobin A_1c levels regularly tested and maintained under 7%. Bridges to Excellence is a not-for-profit, coalition-based organization created in 2003 to encourage significant increases in the quality of care by recognizing and rewarding clinicians who demonstrate they deliver safe, timely, effective, and patient-centered care.

In April 2004, Bridges to Excellence rolled out a diabetes module for physicians. This module, cosponsored by the American Diabetes Association and the NCQA, awarded recognition for those physicians who voluntarily participated and achieved at least a threshold score on a weighted scale based on a number of measures. Among these measures was a hemoglobin A_1c level of less than 7% in more than 50% of a physician's patients with diabetes. Quality vignettes on the Bridges to Excellence diabetes module were copublished by industry and the NCQA.⁶ Also in 2004, Burson-Marsteller, created “A_1C <7% by 2007,” which was launched on World Diabetes Day 2004 with public support from many political groups, including the Congressional Diabetes Caucus. The “A_1C <7% by 2007” program was a 3-year plan of action to increase the number of US individuals who achieve target blood glucose control by 2007. A key component was the formation of a steering committee of nationally recognized experts to develop and support a related program entitled “A Blueprint for Change.” According to information about this program, it “serves to align the medical community, public health advocates, managed care decision-makers, educators, and executives from private corporations.”⁵

Until 2006, the National Diabetes Quality Improvement Alliance, the NCQA, and the National Quality Forum maintained the performance measure for accountability for hemoglobin A_1c at greater than 9%. In 2006, the NCQA adopted a hemoglobin A_1c level of less than 7% measure for all persons aged 18 to 75 years without exclusion criteria in addition to the existing poor glycemic control measure. Collecting data for this measure would begin in 2007 and would be publicly reported in 2008.⁷ In contrast, the National Quality Forum's Ambulatory Care Performance Measures Project, in late 2006, endorsed a level of hemoglobin A_1c of greater than 9% for public reporting.⁸

It is not clear why these different organizations came to different decisions regarding the level of hemoglobin A_1c of less than 7% measure. Coincidentally, 2 weeks after the NCQA adoption of the measure, the Agency for Healthcare Research and Quality sponsored a conference of more than 40 scientists, federal government officials, and representatives of measurement organizations. However, the proceedings of this conference were not posted on the Internet for nearly 17 months.⁹ One possibility is that despite the potential availability of the same information from the National Diabetes Quality Improvement Alliance, not all decision makers were provided this information. For example, a member of the technical advisory committee of the National Diabetes Quality Improvement Alliance stated that “almost everyone [on the NCQA Committee on Performance Measurement] who voted for the 2 new HEDIS [Healthcare Employer Data and Information Set] measures did not know that the Technical Expert Panel (TEP) of the Diabetes Alliance unanimously rejected the proposed A1c <7% and BP <130/80 measures.” However, there were concerns about the balance of benefits and harms as well as technical specifications of the measure.¹⁰ Nonetheless, during the next year, there was a plethora of reports promoting hemoglobin A_1c level of less than 7% as a standard of quality of care.

However, everything changed in 2008. In February 2008, the National Institute of Diabetes and Digestive and Kidney Diseases announced the termination of the Action to Control Cardiovascular Risk in Diabetes (ACCORD) study's tight glycemic control treatment group because of increased mortality.¹¹ This study raised concerns about aggressive target values for performance measures. As a result of the ACCORD study, and other trials showing no significant cardiovascular benefit of tight glycemic control, the NCQA indicated that the level of hemoglobin A_1c of less than 7% measure would be reevaluated and not be reported for 2008 because “new evidence demands a new examination of this measure.”¹²

In the absence of transparent processes, the influence of nonscientific factors on organizational decision making cannot be completely ascertained. However, publicly available information suggests that an industry-driven campaign was successful in coalescing numerous organizations, key professional opinion leaders (many with significant formal leadership roles in professional societies, guideline development, and national educational programs), business leaders, and politicians around a national goal of hemoglobin A_1c level of less than 7% for glycemic control.⁵ Yet, although the National Quality Forum and the NCQA reached different conclusions in 2006, neither organization provided a public report of how they reached their decision. There appears to be no detailed rationale and empirical data regarding the ability of the hemoglobin A_1c measure to distinguish plans or clinicians on the basis of quality, especially given the absence of case-mix adjustment. There was no discussion of potential harms of the measure (the most obvious are harms due to hypoglycemia from insulin, especially in the elderly and those with comorbid illness). Although a major scientific conference on the subject of performance measures for glycemic control held immediately after the NCQA endorsement revealed significant disagreements, the results were not widely publicized.

Furthermore, differences of opinion among national guidelines were not addressed. For example, a review of existing North American and United Kingdom diabetes guidelines rated the 2 US professional societies—the American Diabetes Association and the American Association of Clinical Endocrinologists—as having significantly less evidence-based development processes than other professional society and government guidelines.¹³ Although the results of the ACCORD trial and other studies could have been quite different, resulting in support of the level of hemoglobin A_1c of less than 7% measure, the initial decision to use a level of hemoglobin A_1c of less than 7% was made 2 years before the data were available. It appears that a bandwagon effect, openly funded by industry, may have contributed to the adoption of a measure despite significant and documented scientific disagreement of opinion. How could this have been stopped earlier?

One approach is a process of open source–like measure development that could build on existing successful examples of transparency in guideline development.¹⁴ This process would minimize the asymmetrical information and private influences that seem to dominate the current quasi-governmental model of health care.^{14 - 15} During the development process, there would be full vetting of each proposed measure for each domain considered essential. There would then be an open public comment period in which all member organizations, as well as representatives from all government organizations with an interest, would post their comments. A separate section could be reserved for professional societies, advocacy groups, and for academics through their universities. Each organization would be responsible for obtaining notarized disclosures of conflict of interests that would be posted. This would include all industry support, as well as participation in public service announcement campaigns.

There are potential downsides, depending on how the procedures are operationalized. These include the time and effort required to deal with all of the comments and the difficulties associated with permitting individuals, as opposed to organizations, to participate. However, approaches could be developed to address these difficulties (eg, use of a comment template that required citation of evidence rather than opinion).

During this time, the Agency for Healthcare Research and Quality could contract with an existing evidence-based center to assess the accuracy of the evidence and metrics of the measure, and render a decision regarding the benefits and harms of the measure. An instrument could be adopted, such as Appraisal of Guidelines, Research and Evaluation in Europe (AGREE) that would rate the scientific rigor of the process.¹³ The meeting at which the measure would be discussed should be open, and proceedings published. The votes of each member would be recorded. Each member would have filed a complete disclosure for himself or herself personally, and for the organization that he or she represents. This proposal would not add additional time or significant costs to the organizations. Presumably, minutes would be kept of all proceedings, and the incremental cost of posting on Web sites would be minimal. The cost of a structured performance measurement review by the AHRQ could be low because primary data collection and validation are not required. Considering the potential effect on millions of patients and potentially billions of dollars in antiglycemic medications alone, the upfront investment in ensuring evidence-based, transparently developed performance measures would be worthwhile to protect the public health and restore public and professional confidence.