ABSTRACT
Multimodality invites a re-conceptualisation of what is meant by the term text. In the past, the word text has largely been associated with language written on paper, but increasingly in the 21st century, texts are multimodal, and writing can be on screen. Digital technology has increased the range of text-making tools available to children, but it has also broadened the child’s use of different semiotic systems for meaning making. Writers and creators often have to be able to work with several modes of communication as they create a text. Each mode has its own constituent signs or semiotic system, and readers and viewers must be able to ‘read’ the interconnections across and between modes. In this sense, ‘reading’ involves not just the process of deriving meaning from the written word alone but comprehending meanings derived from ‘reading’ the interplay of several semiotic systems operating simultaneously. These semiotic systems may complement one another in a number of ways. Asha (2022) identifies three ways illustrations in picture books either reflect (concurrence), embellish (enhancement), or extend and even diverge from the written word (contrast). The picture books of Anthony Browne and Shaun Tan, amongst others, include illustrations that are highly symbolic, making the texts so multi-layered they invite the reader to return to them time and time again to find new meanings in linguistic and visual details that may have been missed in previous readings.
When we watch a film, we are not just seeing a sequence of static images move quickly before our eyes to give the appearance of movement; the images are more often than not accompanied by sound effects and music, which create a particular mood. Colour and lighting, along with the expressions and gestures of the actors and their positioning in relation to each other, all signal meanings that inform us of the emotional and mental states of individual characters, their behaviour and the nature of their relationships. Imagine a scene in which a woman stands at the front door of her house waving to her husband as he drives off for work. As his car disappears down the street, the music is slow and slightly melancholy, and her smile fades to sadness. We might presume she is sad because she will be alone all day, but the refrain in the accompanying lyric signals another reason: ‘When you call me baby, I know I am not the only one’. Music videos along with filmic advertisements have evolved into a narrative art form that requires the viewer to ‘read’ several modes simultaneously. The description of the film to Sam Smith’s ‘I’m Not the Only One’ has a clear beginning, specific characters whose different relationships become the narrative complication and a conclusion. The lyrics and visual images and the woman’s changing gestures and expressions evoke emotional responses which might not be captured by listening to the lyrics alone (https://www.youtube.com/watch?v=nCkpzqqog4k) (accessed 24 April 2023). It is the interplay between and complementary nature of different modes working together that make multimodality a rich, multi-layered, polysemic means of communication. Multimodal texts simultaneously appeal to multiple senses.
