Clinical Protocols and Trainee Knowledge About Mechanical Ventilation FREE

A necessary tension exists between the educational needs of trainees and the safety and quality of patient care.¹ In general, this tension is accepted because of the importance of experiential learning to maintain a physician workforce. One organizational factor designed to improve care that may contribute to this tension is the clinical protocol. The barriers between evidence and practice are complex,² and protocols may be one means to overcome them. Protocols provide management algorithms that can minimize practice variation^3- 4 and can often be implemented by nonphysicians.⁵ However, protocols may distance physicians and physicians-in-training from direct patient care. Development of clinical competency requires that trainees have an active role in decision making,⁶ and protocols may attenuate their experiences. Conversely, clinical protocols may be one of many means to facilitate learning by providing a summary of best practices that can serve as a tool for knowledge dissemination.⁷

The relationship between clinical protocols and education is particularly salient in intensive care. In recent years, protocol-directed care has permeated the practice of mechanical ventilation management. For example, in trial settings, protocols for ventilation liberation,⁸ lung-protective strategies for acute lung injury,⁹ and sedation management¹⁰ demonstrated improved clinical outcomes. In response, professional societies have endorsed the use of clinical protocols for mechanical ventilation management.^11- 12 However, it is unknown if unintended consequences of protocols exist, such as the potential creation of a physician workforce unable to think beyond an algorithm,¹³ and whether these consequences may outweigh the benefits to patients. Therefore, the goal of our study was to examine the relationship between clinical protocols and trainee knowledge, using mechanical ventilation management as a model system.

We performed a retrospective cohort equivalence study to test the association of the availability of mechanical ventilation protocols in a fellow's training institution with knowledge about mechanical ventilation management among examinees of the American Board of Internal Medicine (ABIM) Critical Care Medicine Certification Examination administered in 2008 and 2009. Examinees' performance on the mechanical ventilation questions from the examination was linked to the results of a previously performed national survey of the availability of protocols for mechanical ventilation management.¹⁴ That study administered an electronic questionnaire to the medical directors of adult medical intensive care units (ICUs) in hospitals with accredited pulmonary and critical care fellowship training programs in the United States. Quiz Ref IDThe questionnaire inquired about the availability of clinical protocols in 3 domains of mechanical ventilation: (1) liberation from mechanical ventilation, (2) management of sedation for patients who were mechanically ventilated, and (3) lung-protective ventilation for patients with acute lung injury.

The study was exempted from review by the institutional review board of the University of Pennsylvania, Philadelphia, because all identifying data were removed from the study databases.

Study participants were first-time examinees of the ABIM Critical Care Medicine Certification Examination in 2008 and 2009. We excluded examinees from training programs that did not respond to the national survey; all others were included in the final analysis. Quiz Ref IDThe examination measures medical knowledge, diagnostic acumen, and clinical judgment in critical care medicine. Questions are posed as mini-case simulations (patient vignettes) and require fellows to use clinical judgment to reach an appropriate decision about a patient. Individuals are eligible to take the examination if they have completed training in a program accredited by the Accreditation Council of Graduate Medical Education either in combined pulmonary and critical care or medical critical care and have passed the ABIM Internal Medicine (IM) Certification Examination previously. The examination is composed of 200 scoreable questions and is completed in 1 day.

Examinee demographic data were collected as part of registration for the examination. Characteristics of the hospitals, ICUs, and training programs were reported as part of the survey.

For the primary analysis, the exposure was defined as a binary variable representing intensity of protocol availability at the primary training ICU of a pulmonary and critical care or medical critical care training program. The original survey inquired about availability of 3 different mechanical ventilation protocols, as detailed above. High intensity was defined as the availability of 2 or more protocols and low intensity was defined as the availability of 1 or 0 protocol.

The outcome variable was a measure of performance on all mechanical ventilation questions included in each form of the examination. There were 21 questions on the 2008 form and 24 questions on the 2009 form (approximately 66% of the mechanical ventilation questions were the same across the 2 forms). These questions were identified by 1 of the study investigators (E.S.H.) through review of all individual questions on each examination.

The performance measure was a score estimated using the measurement model of item response theory (IRT), a powerful and well-accepted model in the testing industry.¹⁵ All ABIM examinations are scored using this model, which uses maximum likelihood estimation to estimate examinees' scores, and considers the entire pattern of responses as well as the characteristics of each question to derive the score. The ABIM collects evidence on validity showing that examination scores have correlation with real-world practice.^16- 17 The ABIM applied the IRT model to the selected questions for our study to estimate a mechanical ventilation score. The score was developed a priori, blinded to any information about protocols or other scoring data. In this application, the score reflects the amount of medical knowledge and clinical judgment an examinee exhibits in the area of mechanical ventilation management. Scores are scaled to have a mean (SD) of 500 (100) in the entire population of examinees and, because IRT is used, are comparable across the 2008 and 2009 examination forms. Based on the study's context and Cohen guidelines for interpreting effect sizes,¹⁸ one-fourth of an SD, or approximately 25 points on the standardized score scale, is considered to be a meaningful difference. This outcome measure was chosen a priori, with the recognition that performance on examination questions may not completely correlate with clinical performance, but appears to be the best available measure of knowledge and judgment, and prior work has shown a positive association between a measure of knowledge and clinical quality.¹⁶

Potential confounders to be included in the analyses were identified a priori by consensus of the study investigators and included birth country, residency training country, and overall first-attempt score on the ABIM IM Certification Examination taken previously. The latter variable was used as a measure of overall medical knowledge, because examinees who trained in a certain type of program (either with or without protocols) may have greater medical knowledge independent of the protocols, potentially confounding the results.

The unit of analysis was the examinee. Data were summarized using standard descriptive statistics. Mechanical ventilation scores of examinees who were included and excluded from the study were compared (based on participation of their training programs in the survey) using a t test. Characteristics of high-intensity and low-intensity programs were also compared by using χ² and t tests, as appropriate. A bivariable analysis of the association of protocol intensity of examinees' training programs with examinees' scores was conducted using a t test. Multivariable analysis was conducted to determine if protocol intensity was independently associated with the mechanical ventilation score, controlling for the factors listed above. For this, a population-averaged generalized estimating equation was used, which was clustered on training program to account for the nonindependence of within-program observations.¹⁹ All potential confounders listed above were included as covariates in the model. In addition, examination year was included as a covariate to account for fixed effects of differences between the examinations administered in each year.

In addition to the primary multivariable analysis, separate analyses were performed for each individual protocol to explore if any 1 protocol in particular has a stronger or weaker association with examination performance. To evaluate the robustness of the findings and to test the assumptions, multiple additional sensitivity analyses were conducted. In these analyses, the multivariable generalized estimating equation analyses were repeated with the following adjustments to the assumptions: (1) the exposure (protocol intensity) was defined as a linear ordinal variable, with 1 category for each number of protocols present; (2) exposure was redefined to include only those programs with all 3 protocols; (3) exposure was redefined to include those programs with at least 1 protocol; (4) exposure was redefined to include those programs with at least 2 protocols with no physician involvement as determined by the training ICU's design and reported in the survey, because we hypothesized that removal of physician trainees from clinical decision making may adversely affect education in the presence of protocols; and (5) overall trainee knowledge was controlled for using each examinee's overall competency rating as reported to the ABIM, instead of his or her score on the ABIM IM Certification Examination. This rating is subjectively assigned by each examinee's residency training program director and ranges between 1 and 9, with a higher rating indicating a higher level of overall competence.

All testing was 2-sided, with a significance threshold of α≤.05. We designed the studies of association of protocol intensity with examination performance to test an equivalence hypothesis. We considered the 2 exposure groups to be nonequivalent if the entire 95% CI was outside of the equivalence margins,²⁰ defined as ±25 points, which is the threshold for a significant difference in knowledge based on IRT.

Because the sample size was fixed, we did not conduct an a priori power calculation. Instead, we predetermined the significant difference in the mechanical ventilation score at 25. Based on our set sample size, the observed distribution of the score, a significance level of α = .05, and varying the proportion of exposure from 30% to 50%, this sample size achieves between 80% and 86% power when the true difference in means is 0 and the equivalence margins are ±25 points.

Power calculations were performed using PASS 11 statistical software (NCSS, Kaysville, Utah). All other analyses were performed using Stata version 10 (StataCorp LP, College Station, Texas).

Of 129 pulmonary critical care training programs, 90 (70%) responded. Of these, 77 (86%) had protocols for ventilation liberation, 66 (73%) had protocols for sedation management, and 54 (60%) had protocols for lung-protective ventilation. No significant differences were observed in hospital or program characteristics among responding and nonresponding programs in terms of hospital size, financial structure, or medical school affiliation.¹⁴

Of the 90 programs responding to the survey, 88 (98%) graduated trainees who completed the ABIM Critical Care Medicine Certification Examination in 2008, 2009, or both. Hospital and program characteristics are shown in Table 1. Of these 88 programs, 27 (31%) had 0 protocols, 19 (22%) had 1 protocol, 24 (27%) had 2 protocols, and 18 (20%) had 3 protocols for at least 3 years. Forty-two programs (48%) were classified as high intensity (≥2 mechanical ventilation protocols for at least 3 years) and 46 programs (52%) were classified as low intensity.

The initial study population included 778 examinees, 553 (71%) of whom trained at programs that responded to the survey. Examinees from responding and nonresponding programs did not differ significantly in their scores (responding programs: mean, 497; 95% CI, 489-505; vs nonresponding programs: mean, 485; 95% CI, 473-499; P = .14). Characteristics of examinees from responding programs, who were thus included in the study, are shown in Table 2.

Within the study population, 503 of 553 examinees passed the examination, defined as a score of 379 or higher, for an overall pass rate of 91%. The Figure shows the distribution of mechanical ventilation scores of the study population (mean [SD], 497 [93]; range, 200-703). A total of 304 examinees (55%) trained in high-intensity protocol programs and 249 (45%) in low-intensity protocol programs. In the bivariable analysis, the scores on mechanical ventilation questions were equivalent between the 2 groups (mean, 497; 95% CI, 485-509, for the low-intensity group vs 497; 95% CI, 486-507, for the high-intensity group; mean difference, 0; 95% CI, −16 to 16).

In the multivariable analysis, which controlled for examination year, ABIM IM Certification Examination score, birth country, and residency training country as potential confounders, mechanical ventilation scores were still equivalent among graduates of high-intensity and low-intensity protocol programs (β coefficient, −5.36; 95% CI, −20.7 to 10.0; in which the coefficient reflects the point difference in the score between intensity groups and a negative value indicates that the high-intensity exposure is associated with a lower score). Furthermore, the CI was entirely included within the equivalence margins (predefined as ±25 points). The coefficients for all the model variables are shown in Table 3. The only covariates significantly associated with the mechanical ventilation score in the multivariable analysis were examination year (ie, the mean score was higher in 2009), ABIM IM Certification Examination score, and birth and training country (individuals who were both born and trained internationally had lower scores). In separate multivariable analyses, no individual protocol was associated with the mechanical ventilation score.

The results of multivariable analyses were similar in multiple sensitivity analyses with 1 exception. Scores were equivalent in both exposure groups when the following adjustments were made to model assumptions: (1) exposure was redefined as only those programs with all 3 protocols, (2) exposure was redefined as those programs with at least 1 protocol, (3) exposure was redefined as only those programs with at least 2 available protocols that do not require any physician involvement, and (4) the program director rating of overall clinical competence was used instead of the ABIM IM Certification Examination score (eTable). In the sensitivity analysis in which exposure was defined as a categorical variable (for 0, 1, 2, or 3 available protocols), the CIs for the point estimate of association for 1 and 2 protocols (compared with the reference group of 0 protocols) were not entirely within the equivalence margins of ±25 points (95% CI, −30.73 to 19.31; and 95% CI, −30.47 to 8.89; respectively). Therefore, we could not conclude that scores were equivalent among groups; however, this analysis was relatively underpowered due to the categorization of the exposure variable, as evidenced by the wide CIs.

Quiz Ref IDIn a population of junior critical care physicians, having trained in a high-intensity protocol environment with respect to mechanical ventilation practice was not associated with any significant differences in performance on certification examination questions in the topic of mechanical ventilation management. One other study has investigated the relationship between clinical protocols and trainee knowledge. Stoller et al²¹ examined the association between the availability of respiratory therapy protocols for asthma management and resident knowledge about respiratory care. The authors demonstrated no difference by exposure to protocols, but the interpretation of the results was limited because protocol exposure correlated completely with training hospital. As a result, the independent association of the protocol could not be teased out from other educational factors. Our study builds on this work to provide more robust empirical evidence consistent with clinical protocols not worsening trainee knowledge.

As the use of clinical protocols increases in prevalence in response to concerns about patient safety and practice variation,⁴ understanding the potential relationship of protocols with education in clinical settings is essential. There is a reasonable concern about the potential unintended consequences of protocols on education, because some protocols will by design remove junior clinicians from direct clinical decision making experience necessary for building their knowledge, decision making skills, and confidence.¹³ Our study suggests that this phenomenon may not occur.

Several possible explanations exist for the lack of association of mechanical ventilation protocols with performance of examinees on mechanical ventilation certification examination questions. First, there may be no association to be found, in that protocols might not interact in any way with education at all. Second, this population of examinees is highly motivated to learn about mechanical ventilation. Critical care physicians have an interest in mechanical ventilation management and, regardless of their learning experience in actual clinical settings, may find ways to learn what they believe is important. Another related explanation is critical care fellows may learn from many sources, including nonphysician practitioners in the ICUs (such as respiratory therapists). The protocols, grounded in evidence-based practice, may facilitate such cross-disciplinary learning and may also provide a clear, concise framework for teaching about the medical evidence in a certain clinical situation and through standardization and repetition of best practices.⁶

Third, the examination questions may be limited in their ability to measure actual skills in mechanical ventilation management. Quiz Ref IDExamination questions are the best available measure of knowledge, because they are rigorously developed and tested and are standardized across many examinees. However, examination questions cannot measure all aspects of decision making in real clinical settings and may not discern subtle differences in overall competence in mechanical ventilation management.

Quiz Ref IDFourth, protocols may not in fact remove the trainee from decision making. Although protocols may provide an algorithm, the typical conduct of clinical care permits continued active involvement of clinicians and therefore may not reduce the direct experience so important to a trainee's education. Fifth, protocols may actually have both positive and negative relationships with education that counteract. Were this true, there would still be no overall association with education.

Mechanical ventilation management is only one clinical topic in which protocol use and education may be related. Another example is management of severe sepsis. A complex algorithm of early, goal-directed therapy resulted in improved patient outcomes²²; a clinical protocol guiding decision making for such patients could either enable passivity that might impede experiential learning or provide a framework for learning about the pathophysiology of sepsis. Other similar examples include management of diabetic ketoacidosis and titration of outpatient anticoagulation. We chose to study mechanical ventilation management as a model for this question for several reasons. First, there is compelling clinical evidence in support of using ventilation clinical protocols for patient care.^{8- 10,23- 25} Second, the high risk of mortality associated with mechanical ventilation²⁶ makes it likely that the quality of training in this area is most likely to have a significant effect on patients. Third, because of heterogeneity in critical care education,²⁷ we anticipated a high degree of variability in performance on certification examination questions, a necessary characteristic of an outcome variable for an observational study. Therefore, we believe that if clinical protocols do indeed interact with medical education, we would most likely expose the relationship in this clinical area.

We acknowledge important limitations to our study. First, the availability of a clinical protocol does not equal its active use. We chose not to inquire about the actual or perceived use of the protocols, because we believe that reports would likely be inaccurate and not interpretable due to recall bias,²⁸ and measurement of actual protocol use would likely require direct observation and would be infeasible in a sample of programs of this size. In the previous study of the association between clinical protocols and trainee knowledge, limited to 2 centers, investigators knew of the actual use of the protocols under study. Their finding of no association between protocols and trainee knowledge about respiratory care complements our findings.²¹

Second, there is likely to be significant variability in the protocol designs and how much they may actually interfere with the decision making of critical care fellows. Our survey was limited in how much information we could obtain about the content of the protocols; however, we attempted to explore the variation by inquiring about the involvement of the physicians in ordering and implementing the protocols and accounting for these differences among programs in sensitivity analyses.

Third, we believed that overall trainee competence was an important potential confounder in this study and that it was essential to address this issue in the analysis, but we had limited tools to do so. In the primary analysis, we used examinees' scores on previously taken ABIM IM Certification Examinations, because there is evidence to support a correlation between examination scores and clinical competence.¹⁶ In the sensitivity analysis, we additionally used residency program director ratings of clinical competence to adjust for potential confounding and found similar results.

Fourth, performance on examination questions is only one component of clinical competence; therefore, we interpret our findings with caution to avoid overstating the relationship, or lack thereof, between protocol exposure and clinical performance. Board certification examination questions are the most rigorously developed and tested measures of knowledge that are currently available and have reasonable correlation with other measures of clinical performance¹⁶; however, they may not be sufficiently sensitive to educational harm of protocols. Further work is required to explore other domains of clinical competence.

In conclusion, as medical education evolves, with work-hours limitations, increasing compartmentalization of care, reductions in procedural requirements, and emphasis on patient safety in the educational setting, it is important to understand all the effects of the systems that are implemented to respond to these changes and principles. Our study provided empirical evidence that clinical protocols, designed to standardize and optimize patient care, may not hinder high-quality medical education in mechanical ventilation management. Further exploration in other content areas and with other measures of competence should investigate the balance between patient safety and the needs of medical trainees to ensure present and future high-quality care.