ABSTRACT
Research on the comprehension of text and pictures has blossomed in recent years. Most of this research investigates the interaction between verbal information and the information contained in pictures in order to understand the conditions under which pictures are more likely to facilitate the comprehension of a given text. Various theories try to explain this text/picture interaction, such as the Dual Coding Theory (Clark & Paivio, 1991; Paivio, 1986), the Conjoint Processing Theory (Mayer, 1997), and the Integrative Model of Text and Picture Comprehension (Schnotz, 2001, 2002). In these theories the emphasis is usually placed on understanding the cognitive mechanisms that underlie the interplay between verbal and pictorial representation. Mayer and his colleagues (Mayer, 1997, 2003; Mayer & Moreno, 2002,
2003), for instance, argue that students learn better when both words and pictures are present in book-based or computer-based environments, than when only words are presented to them. They also claim that there are certain methods or principles that optimise the effectiveness of such multimedia environments, such as the coherence effect – in which students learn more deeply when extraneous material is excluded rather than included – the spatial contiguity effect – in which students learn more deeply when printed words are placed near rather than far from corresponding pictures – and the personalisation effect – in which students learn more deeply when words are presented in conversational rather than formal style. Schnotz and his colleagues (Schnotz, 2002; Schnotz & Bannert, 2003;
Schnotz & Kürschner, 2008), on the other hand, have looked at the interaction between internal and external representations, the effects of multiple representations, and the relationship between external representations and cognitive
processes including the amount of cognitive load. They argue that the inclusion of different pictures in the same text can result in the construction of different mental representations by the reader and in different patterns of performance in subsequently presented tasks. They also explain some of the conditions under which the presentation of external representations may not always be beneficial for the acquisition of new knowledge. Finally, Lowe and Schnotz (2007) and Ainsworth (1999; Ainsworth &
VanLabeke, 2004) have investigated dynamic representations and animations, and have found that it is not always beneficial to have multiple representations, as students, and particularly young students, may find it difficult to translate from one representation to another. The explanatory framework used to interpret these research findings is usually
that of cognitive load theory. In other words, external representations are said to facilitate performance because they reduce the cognitive load of the task or because they make specific cognitive processes easier, faster, and more accessible. The above-mentioned studies have provided a great deal of information about
the ways in which readers comprehend pictures in texts and have generated very useful recommendations for the improvement of curriculum materials and their instructional uses. There are two areas, however, where we believe that more research is needed in order better to understand the interaction between verbal and pictorial information in text comprehension. The first area concerns the distinction between external representations that are perceptually based depictions1
and those that represent conceptual models. The existing research has not explicitly addressed students’ difficulties in understanding the meaning of these two types of external representation. As will be explained later in detail, conceptual models usually have an analogical relationship to the perceptual situation that they represent and they may differ from that situation substantially in surface similarity, compared with perceptually based depictions. The second area where more research is needed has to do with the effects
of prior knowledge. While many researchers are concerned about how pictorial representations interact with verbal information, there is little discussion of how external representations may interact with domain-specific prior knowledge. The interaction between prior knowledge and the comprehension of pictorial information is particularly relevant in the case of conceptual models, because conceptual models depend more on domain-specific knowledge in order to be understood, compared with perceptually based depictions. In this chapter we attempt to show that a distinction can be made between
perceptually based external representations and those that can be considered conceptual models. Moreover, we argue that conceptual models are more difficult to understand than perceptually based representations, because they demand (a) specific scientific and mathematical domain knowledge, and (b) substantial epistemological sophistication. For this reason, we claim, it is important to distinguish between these two kinds of external representations and to be careful about how each one is used in curricula and instruction.
