The term ‘multimodality’ dates from the 1920s. It was a technical term in the then relatively new ﬁeld of the psychology of perception, denoting the eﬀect diﬀerent sensory perceptions have on each other. An example of this is the so-called McGurk eﬀect: if people are shown a video of someone articulating a particular syllable, e.g. /ga/, while hearing another syllable, e.g. /ba/, they perceive neither /ga/, nor /ba/, but /da/ (Stork 1997: 239). In other words, perception is multimodal. It integrates information received by diﬀerent senses. More recently, linguists and discourse analysts have taken up the term, broadening it to
denote the integrated use of diﬀerent communicative resources, such as language, image, sound and music in multimodal texts and communicative events. As soon as they had begun to study texts and communicative events rather than isolated sentences, they realized what they should have known all along: that communication is multimodal; that spoken language cannot be adequately understood without taking non-verbal communication into account; and that many forms of contemporary written language cannot be adequately understood unless we look, not just at language, but also at images, layout, typography and colour. In the past twenty or so years this led to the development of multimodality as a ﬁeld of study investigating the common as well as the distinct properties of the diﬀerent modes in the multimodal mix and the way they integrate in multimodal texts and communicative events. It is not diﬃcult to see why such a ﬁeld of study should have developed. From the 1920s
onwards, public communication had become increasingly multimodal. Film had changed acting, enlarging subtle aspects of non-verbal communication, and so inﬂuencing how people talk and move and smile the world over. Later, television had made non-verbal communication a decisive factor in politics, most famously in the televised debate between Nixon and Kennedy. Writing, too, had become multimodal, as illustrations and layout elements such as boxes and sidebars broke up and reshaped the pages of books and magazines. Like scholars in other ﬁelds of study, linguists took notice. In the course of the twentieth century, several schools of linguistics engaged with communicative modes other than language. The ﬁrst was the Prague School, which, in the 1930s and 1940s, extended linguistics into the visual arts and the non-verbal aspects of theatre, and which included studies of folklore, and collaborations
with avant-garde artists (see, for example, Matejka and Titunik 1976). Paris School structuralist semiotics of the 1960s also used concepts and methods from linguistics to understand communicative modes other than language. Largely inspired by the work of Roland Barthes, it mostly focused on analyses of popular culture and the mass media, rather than on folklore or avant-garde art (e.g. Barthes 1967, 1977, 1983). In roughly the same period, American linguists began to take an interest in the multimodal analysis of spoken language and non-verbal communication. Birdwhistell (e.g. 1973) developed an intricate set of tools for analyzing body motion, and Pittenger et al. (1960) published a highly detailed and groundbreaking multimodal analysis of the ﬁrst ﬁve minutes of a psychiatric interview. In the late 1960s, conversation analysis replaced the 16mm ﬁlm sound camera with the cassette recorder as the research tool of choice, which diminished attention to non-verbal communication, although some scholars in this tradition have, more recently, re-introduced it (Ochs 1979; Goodwin 2001). A fourth school emerged in the 1990s. Inspired by the linguistics of M. A. K. Halliday (1978, 1985), it was this school which adopted and broadened the term ‘multimodality’, and introduced it into applied linguistics, and especially into the study of language and literacy in education. More recently a further approach, mediated discourse analysis, inspired by the work of Ron and Suzie Scollon (2003, 2004), returned to American micro-analysis of social interaction, but in a new way, linking it to the wider social and political context, and adding a new emphasis on technological mediation (see, for example, Jones 2009). By now, multimodality has its own bi-annual conference and a range of edited books
(e.g. O’Halloran 2004; Ventola et al. 2004; Levine and Scollon 2004; Norris and Jones 2005; Unsworth 2008; Jewitt 2009), and it is regularly included in handbooks and encyclopedias of linguistics, discourse analysis, visual communication and so on. Although it encompasses a number of distinct theoretical and methodological approaches, it has nevertheless remained a united ﬁeld of study, with productive dialogue and mutual inﬂuence between the diﬀerent ‘schools’ (cf. Jewitt 2009, for a more extensive overview of the diﬀerent approaches in the ﬁeld).