ABSTRACT
In making cross-modality matches with respect to perceived intensity—for example, in matching the loudness of sound and the brightness of light—people recognize quantitative similarity in the face of qualitative difference. In fact, people recognize several kinds of auditory-visual similarity: between pitch and brightness (low pitch = dim; high pitch = bright) and pitch and size (low pitch = large; high pitch = small), as well as loudness and brightness (soft = dim; loud = bright): Such similarity relations can be characterized through a multidimensional spatial representation that is also multimodal, with different modalities “sharing” dimensions such as pitch, loudness, and brightness. Evidence that very young children, like adults, perceive similarity between pitch and brightness and between loudness and brightness, but not between pitch and size, accords with the view that pitch-brightness and loudness-brightness relations are intrinsic to perception, whereas the pitch-size relation may derive from experience. Auditory-visual similarities express themselves widely—in language as well as perception, and in functional interactions as well as structural relations: Similarities appear in synesthetic perception, in cross-modal matches, in judgments of cross-modal metaphors, and in response-time interactions of perceptual and linguistic stimuli. Even when they are initially perceptual, intermodal similarities infiltrate language, so that semantic codes may even come to dominate perceptual ones.
