Critical Thinking in Graduate Medical Education: Title and subTitle BreakA Role for Concept Mapping Assessment? FREE

During the past few decades medical curricula have increasingly reflected the belief that students should understand concepts, as opposed to rote memorization of facts or procedural algorithms. Many medical schools have adopted alternative teaching and learning strategies that challenge students to become self-directed learners who continuously expand the limits of their knowledge and seek evidence for clinical decision making.¹ The root of these ideas lies in the constructivist theory of learning, which states that meaningful learning occurs when students assimilate new knowledge into a preexisting conceptual framework.^{2 - 9} This evolving conceptual framework supports expertise in critical thinking and clinical problem solving.^{10 - 12} Despite its critical importance in the education of future physicians, tools to evaluate the quality of students' conceptual frameworks are limited because of the difficulty in designing and administering objective measures.

In this article, we describe an application of concept mapping assessment (CMA) that provides a "window" into learners' organization of knowledge at various points in training. Concept mapping is a technique developed by Joseph Novak in the 1970s in which students create a 2-dimensional diagram outlining their understanding of the relationships between and among important concepts within a given subject area.^{13 - 15} Unlike most conventional examinations, CMA may provide unique insights into how an individual organizes his or her knowledge or comes to think about a problem. While concept mapping has been used frequently as a learning tool, especially in high school and college science education, it has only rarely been used as an assessment method, primarily because there is limited information regarding its validity and reliability.^{12 ,16 - 18}

Therefore, we undertook this preliminary study to assess whether CMA can measure predicted differences and expected changes in the conceptual framework of resident physicians. In addition, we sought to determine whether concept maps could be scored in a reliable way.

All pediatric residents (33 total) at our institution were eligible to participate in this study, which was approved by our institutional human subjects committee. Written informed consent was obtained. A total of 21 residents enrolled: 9 first-year residents, 8 second-year residents, and 4 third-year residents. Each participant underwent concept map training followed by drawing a preinstruction concept map about the topic seizures. They then participated in a 3-session seizure education course followed by drawing a postinstruction map about seizures.

All study subjects participated in a standardized concept map training exercise based on previously described methods.¹⁴ Within 2 to 3 weeks of completing map training, participants were given 30 minutes to independently, and without study aids, draw their concept of seizures on a blank sheet of paper (preinstruction map). Residents then participated in a 3-session (1 hour per session) seizure education course consisting of case-based instruction. One week after completing the course, residents drew a postinstruction concept map about seizures under the same conditions as before.

We used a modification of the "hierarchical technique" of concept mapping in which a concept map consists of 5 components: concepts, concept links, hierarchies, cross-links, and examples (see Figure 1 for an example concept map of water).^{14 ,19}

A concept is defined as a perceived regularity in events or objects designated by a label and, when depicted on a map, enclosed with a circle. Creating a concept map involves connecting related concepts using arrow lines with a statement or proposition written above the line describing how the concepts are related.

A concept linked to another concept by an arrow line with a propositional statement, called a concept link, represents the foundational unit of a concept map.^{3 ,14 ,20 - 21} In Figure 1, the concept water connected to living things by the proposition needed by represents a concept link. The expression of the hierarchy between concepts is indicated by the direction of the arrow in the concept link with the most general concepts arranged at the top and more specific concepts arranged hierarchically below. Three levels of hierarchy are represented in Figure 1. The linking of a series of related concept links in a hierarchical fashion represents a domain of knowledge. Cross-links are the method by which the map author indicates how different domains of knowledge are related to each other and are a critical measure of map complexity. In a manner similar to propositions, cross-links are labeled with a phrase that describes the relationship of the domains. In the example map, motion connected to states by the proposition determines defines a cross-link.

Examples are labeled and connected to the related concept in the most subordinate position on the map. In Figure 1, Lake Tahoe is a specific example of a lake. Theoretically, as a learner's conceptual framework changes, he or she links new propositions sequentially (concept links), general to specific (hierarchy), and across domains (cross-links), thus demonstrating integration of knowledge.²²

Maps were scored independently by 3 different raters, blinded to the identity of the map author, and using a modification of the structural scoring method previously described.^{7 ,14} Each rater was a board-certified pediatrician who underwent a 30-minute concept map scoring training exercise immediately prior to grading the maps. Maps were scored in 4 categories: concept links, hierarchy, cross-links, and examples. The value of each scoring category represents the depth of thinking required to form the category.^{7 ,14} Each valid concept link was given 2 points, and each level of hierarchy was given 5 points. A cross-link was given 10 points, while an example counted for 1 point. Invalid links or concepts were given 0 points. A total score and subscore for each category (concept links, cross-links, hierarchy, and examples) were generated for each map. Standard total scores and standard subscores consisting of the sum of individual rater total scores and subscores, respectively, were assigned to each map.

Statistical analysis was done using STATA, version 6.0 (College Station, Tex). The correlation of each rater's total scores and subscores for each map was tested using Spearman rank correlation. All other data analyses were done using standard total scores and standard subscores. We compared preinstruction map to postinstruction map scores using the Wilcoxon signed rank test. To assess whether there was a difference in map scores based on level of training or experience, we combined the more experienced second- and third-year residents (n = 12) into 1 group and compared their scores with those of the first-year (n = 9) residents using the 2-sample Wilcoxon rank sum test. To assess the relationship of concept map scores to standard written examinations, we compared the most recent American Board of Pediatrics In-training Examination (ITE) scores of second- and third-year residents to first-year residents using the 2-sample Wilcoxon rank sum test. First- and second-year residents took the examination within 1 month of completing this study, while third-year residents took the examination 10 months prior to completing the study. We then tested the relationship of concept map scores to ITE scores using Spearman rank correlation.

All 21 participants completed all phases of the study including concept map training, the seizure education course, and both preinstruction and postinstruction concept maps.

Figure 2 shows a complete low-scoring concept map from a first-year resident, while Figure 3 shows a high-scoring map from a resident in the second- and third-year group. Note the complexity, with numerous cross-links in the map from the second- and third-year group compared with the linear simplicity of the first-year resident's map. Table 1 summarizes the mean standard total and mean standard subscores of study participants by level of training. The second- and third-year residents had significantly higher total (P = .04) and cross-link scores on the preinstruction map (P = .01) compared with first-year residents. There was no significant difference between concept link, hierarchy, and example scores based on level of training. The postinstruction maps demonstrated that while the total and subscores for second- and third-year residents remained higher, these differences did not reach significance. Of particular note is that the significant difference in total score and cross-link score seen on the preinstruction maps was no longer apparent after instruction (P = .16 and .15, respectively).

Total scores increased significantly after the educational intervention (P = .03). In addition, cross-link (P = .02) and concept link (P = .01) subscores also increased significantly. Subscores for hierarchy and examples both decreased slightly, but this change did not reach significance. Preinstruction and postinstruction maps from 1 subject are shown in Figure 4. The resident added a new conceptual domain for prevention and education as well as new cross-links to increased intracranial pressure. The vague concept cerebral irritation included as a focal point in the preinstruction map was omitted in the postinstruction map. Table 2 summarizes the mean standard score and subscores of all subjects before (preinstruction map) and after (postinstruction map) the seizure education course. Total scores increased significantly after the educational intervention (P = .03). In addition, cross-link (P = .02) and concept link (P = .01) subscores also increased significantly. Subscores for hierarchy and examples both decreased slightly, but this change did not reach significance.

The mean (SD) ITE score of second- and third-year residents was 358 (135) and of first-year residents, 243 (94). The ITE scores of second- and third-year residents were significantly higher than those of first-year residents (P = .05). There was weak to no correlation between CMA scores (total and subscores) and ITE scores (r = 0.10-0.54). The strongest correlation was between ITE score and the example score on the preinstruction (r = 0.54; P = .01) and postinstruction map total score (r = 0.53; P = .01). The correlation between ITE scores and postinstruction map total scores, preinstruction map cross-link scores, and postinstruction map cross-link scores was r = 0.29, 0.34, and 0.20. respectively.

The interrater correlation was weak to moderate (r = 0.51-0.69) for the preinstruction map and moderate to strong for the postinstruction map (r = 0.74-0.88). The interrater correlation for subscores demonstrated an almost identical range to that seen in the analysis of total scores.

We found that concept map scores of residents increased significantly following completion of an educational program after which conceptual framework would be expected to change. This change is perhaps best illustrated by qualitative comparison of the preinstruction and postinstruction maps (eg, Figure 4), which demonstrates the incorporation of additional concepts into the postintervention map, further differentiation of existing concepts, and a change in cross-linking between concepts. Quantitatively, these changes are reflected in a statistically significant increase in total map score and concept link and cross-link subscores. Despite the small sample size, we also found that CMA scores of residents with more training and expertise were significantly higher than those with less training. This difference was the result of greater map complexity, as reflected in the significantly higher cross-link scores of second- and third-year residents. Thus, more experienced residents appeared to organize their knowledge differently, expressing links between concepts that were not observed by less experienced residents. It is important to note that this difference narrowed, as one might anticipate, after all residents participated in the same educational intervention. While not conclusive, these findings provide important preliminary evidence supporting our hypothesis that CMA is a valid measure of conceptual framework and the expected change in that framework with new learning.

We also found that while ITE scores of second- and third-year residents were significantly higher than those of first-year residents, these scores did not correlate well with CMA scores. Thus, residents who scored highly on CMA were not the same residents who scored highly on the ITE examination. This finding is consistent with other reports, which note that CMA scores of college students do not correlate with conventional measures of learning, such as final course grades or scores on standardized tests.¹⁶ While there may be other explanations, the absence of a positive correlation suggests that CMA measures a different knowledge characteristic than do multiple choice examinations.

The issue of scoring reliability is critically important to any future use of CMA. We found that interrater reliability was moderate to strong and, notably, was greater on postinstruction maps. This level of scoring reliability is similar to that reported by others and suggests that gains in scoring experience lead to increased consistency across raters.²³ Therefore, rater training would be important to any future applications.

The vast majority of previous research regarding concept mapping has involved high school and college science education. In reviewing this literature, there is clear evidence that concept mapping can be an effective learning tool in a variety of classroom settings—including the education of health care professionals.^{16 - 18 ,24 - 30} However, evidence supporting the validity of concept mapping as an assessment tool is much more limited, probably because there is no criterion standard measure of conceptual framework to which CMA can be compared. Traditional measures of competence, such as board scores or even newer competency-based assessments (eg, Objective Structured Clinical Examinations), may not represent appropriate comparisons since they shed little light on how a learner thinks about a problem.¹² Thus, other investigators have used the same strategy to build a case for the validity of CMA. For example, in a variety of secondary school and undergraduate settings, CMA has consistently been found to measure conceptual change in situations in which such change would be expected and to detect differences between students with more training in a particular field compared with those with less.^{14 ,16 - 18 ,22 ,31 - 32} In medical education, we identified only a single related study in which Pathfinder networks, a substantially different form of concept mapping, was used to assess student understanding of pulmonary physiology.³³ In that study, the investigators found that, after instruction, student Pathfinder networks became more similar to the networks generated by faculty experts. These findings appear consistent with our study, and while not conclusive, provide additional evidence suggesting the predictive validity of CMA.

We have provided encouraging preliminary evidence supporting the validity of CMA as a way to measure conceptual change and differences among resident physicians. Our findings also indicate that rater training is critical to the use of CMA. Future research must focus on further validating this method and addressing issues of reliability before CMA can be applied on a larger scale. Nevertheless, CMA has the potential to evaluate how students or residents organize and use knowledge in a way that traditional objective tests cannot. It also could provide a tool for educators to identify unique distortions in students' understanding of content and to identify errors of omission. Finally, CMA might provide insight into why some residents score well on objective written examinations but have difficulty applying this knowledge to clinical situations.