ABSTRACT
With the availability of new technologies, computer users are increasingly able to integrate multiple communicative modes (language, images, music) for expression in multiple media. Multimodality and multimediality pose distinct analytic challenges for sociocultural researchers who investigate language-based practices. In the case of video gaming and streaming video, moving images appear in activity sequences that may involve little or no linguistic mediation. In such contexts, users’ perceptual habits and cognitive operations orient to spatial signifi cation, a communicative context constituted by syntactical units that are imagistic rather than linguistic. As Tchertov (2002) suggests, “Spatial semiosis allows syntactic structures to be built in an essentially other way than the successive ordered chains of discrete signs, known for linguistics” (p. 443). In particular, the visual processing of spatial signs in real and virtual contexts involves distinct features not associated with the sequencing of linguistic signs. In the interpretation of spatially situated signs, the eyes move in sudden, rapid motions across the visual fi eld in jumps known as “saccades” (Jacobs, 1995, p. 260). Each saccade “is followed by a fi xation, a period of relative stability during which an object can be viewed. Even during a fi xation, an eye does not remain completely still, but engages in several types of small motion. It drifts slowly, then is corrected by a tiny saccade-like jump (a microsaccade), which corrects the eff ect of the drift” (p. 260). Print texts require linear processing with few of the jumps, acceleration, and deceleration characterizing saccadic shifts in spatial signifi cation, a process that is notably nonlinear and reversible across a visual fi eld.
