ABSTRACT

This chapter explores the research challenges and opportunities presented by narratives that exploit more than one semiotic channel to evoke a storyworld. To outline directions for inquiry into multimodal storytelling, I discuss two case studies: on the one hand, word-image combinations in comics and graphic novels; on the other hand, utterance-gesture combinations in videotaped occasions of face-to-face narration. With the word-image combinations, I consider strategies for analyzing a key aspect of narrative structure-namely, character or role-in print texts that deploy a visual as well as a verbal information track. Then, turning to gesture use in face-toface storytelling, I draw on gesture research, theories of deixis, and recent work on space and place to investigate how utterances and gestures interact in real-time narration. My chapter thus focuses on two quite different case studies, with the fi rst case study emphasizing issues of story interpretation and the second case study foregrounding issues of story production; in this way, I seek to suggest the diversity of the corpora, disciplinary frameworks, and methods of analysis relevant for research on narrative and multimodality. What is more, to anticipate remarks that I expand upon in my concluding section, the approach developed here suggests the need to bring into closer dialogue two strands of postclassical narratology that have for the most part been pursued separately up to now. The two strands at issue are transmedial narratology, or the study of narrative across media, and cognitive narratology, or the study of mind-relevant aspects of storytelling practices, wherever-and by whatever means-those practices occur.1