Implications of the Principle of Question Propagation for Comparative-Effectiveness and “Data Mining” Research

Recent legislation incorporated comparative-effectiveness research (CER) as a scientific mechanism to help improve health care.¹ The law expresses particular interest in discovering which treatments work “in a real world setting” and encourages conduct of observational studies using data mining techniques of standardized electronic records.^{1 - 2} Ideally, CER will identify effective interventions in the subgroup of patients, since traditional randomized trials typically provide efficacy data for an “average” patient only.^{1 - 2} It is likely that the amount of observational research will increase significantly, especially studies involving data mining of large administrative databases and electronic medical records. However, epistemological arguments suggest that data mining efforts cannot provide definitive answers to the questions asked by the CER program. Rather, CER should be considered hypothesis-generating research aiming to inform future prospective studies that will invariably require new (and better) data collection.

Science is an open-ended system. For every explanation science generates, a new ensuing explanation is required, because “there can be no explanation which is not in need of a further explanation.”³ This characteristic of scientific activity as open-ended instead of a finite, closed-end system was described by Rescher³ to be the consequence of Kant's Principle of Question Propagation as “ answering our factual (scientific) questions paves the way to further yet unanswered questions.” The essence of scientific inquiry is characterized by the cycle of answers generating new questions. The driving force behind this cycle of questions-answers-questions (Q-A-Q) is the attempt to improve scientific explanatory power, descriptive precision, and predictive accuracy.³ Consequently, science is characterized by escalating theoretical complexity accompanied by increasingly sophisticated scientific explanations. Scientific knowledge is purported knowledge—what counts as scientific knowledge is tentative and remains revisable as science progresses.^{3 - 4}

Ever-Increasing Use of Technology to Arrive at Definitive Answers in Clinical Medicine

The capabilities of physicians to understand ailments, diagnose diseases, improve prognosis, and treat patients have dramatically improved during the past century, and especially during the past 25 years. These improvements have been achieved via systematic scientific activity, which has provided new answers to old questions, asked new questions, and replaced old knowledge with new.³ It is estimated that the amount of medical information is doubled every decade and that only approximately 22% to 50% of clinical research conclusions remain recognizably correct after 50 years.⁵

Despite the improved understandings of the mechanisms of diseases, medical knowledge will never be complete. Although new diagnostic tests, prognostic tools, and new treatments continue to be developed, and new clinical and research programs continue to discover new disease (eg, National Institutes of Health Program for Undiagnosed Diseases), biomedical scientific efforts will remain “inherently incompletable, with the ever-receding horizon, separating where we are from where we would ideally like to be.”³ In the ever-increasing desire to practice personalized medicine and to tailor the use of treatments or diagnostic devices to individual patients, clinicians and scientists seem to have no limit on the number of questions they can ask. Therefore, the number of possible Q-A-Q cycles is so large that the cycles should be seen as a permanent feature of research and practice of medicine. For example, it is estimated that in general practice alone, there are 48 000 key clinical questions that ought to be answered⁶ ; this is likely an underestimate.

Epistemological Feasibility of Conducting Health Research Using “Data Mining” Approach

The epistemological view that medicine is an open, incomplete system means that theory driven and hypothesis testing should continue to dominate scientific practice in medicine.⁷ Testing one or few key hypotheses at a time also represents the best way to make discoveries rapidly.⁸

To increase the rate of scientific discoveries, however, CER is shifting from the hypothesis-testing approach to the “data mining” scientific approach. In data mining, existing data are analyzed from different perspectives that may reveal new patterns or correlations in relational databases. This approach, which requires access to large data sets and massive investment in information technology (IT), has often been justified by the potential for new discoveries. Data mining has been strongly advocated for health services research.

The application of IT and analysis of the existing data sets have been promoted as one of the important aspects of CER (ie, the program that represents a scientific foundation of the ongoing health care reform in the United States).² The premise is that by connecting different data sets (eg, patient outcome data, electronic medical records, administrative databases, genomic or proteomic data sets) using various data mining tools, new discoveries will emerge that had not been anticipated when databases were originally designed. An implied assumption is that the answer provided from such analyses will require no new data collection.

By definition, data mining presupposes the use of data that were originally designed and collected for different purposes. This is akin to the epistemological stand describing clinical medicine as a closed system within which data that had already been collected contain a desired answer. Given that medicine is an incomplete, open-ended system, the authors of the original database used for retrospective data mining could not possibly have anticipated or realized what new advances in medical science will be introduced and what kind of new discoveries will be made.³ This creates a paradox, which is particularly evident when searching for treatment effects in subgroups—one of the purported goals of the IT CER initiative. As new research generates new evidence of the importance for tailoring treatments to a given subpopulation of patients, the existing databases will need to be updated, in turn undermining the original purpose to discover new relationships via existing records. An example of the moving target nature of the open-endedness of scientific inquiry is shown in the Box. Consequently, the data mining approach can never result in credible discoveries that will obviate the need for new data collection. The best data mining research can hope to accomplish is to provide hypothesis-generating results, which will then need to be subjected to further scrutiny using the hypothesis-testing paradigm.

As the nation embarks on a multibillion-dollar investment to develop new registries, data warehouses, and other standardized collections of data in an electronic format, it is important to heed some long-known principles in the philosophy of science.