DAC Home   Form, Substance, Correspondence:
Intersensory Composition in
Digital Media

Paul Hertz, Northwestern University
paul-hertz@northwestern.edu
<back to
program
 
Wagner's "total art work" (Gesamtkunstwerk) and the correspondence of sound, color and scent proposed in Charles Baudelaire's poem "Correspondances" influenced a whole generation of artists in their time. Their influence is still felt today, when digital media offer us new possibilities for creating multisensory art. With the computer we can compose and perform multisensory events with a degree of precision beyond the threshold of human perception. This capacity is not just of interest to artists: immersive interfaces and multisensory representations of data have become an important tool for scientific research and the exploration of large data sets. In their separate ways, both scientists and artists attempt to make sense of the world. Arguably, their methods if not their styles converge in the propagation of digitally-mediated multisensory experiences. This paper contends that a specific area of these experiences, which we shall call digital intermedia, can prove particularly fruitful. "Intermedia" is a term coined by the Fluxus artist and theorist Dick Higgins. In intermedia, the compositional process works across the boundaries between media or even fuses media. While it is sometimes called "synesthetic art," intermedia does not seek to imitate the physiological phenomenon of synesthesia. It extends the creation of form across sensory modalities without necessarily promoting a tight coupling of multisensory events. At the same time, intermedia implies something more specific than multisensory opera. It is not so much a "total art work" as a hybrid art work.

Now that digital media enable us to generate multisensory experiences with all sorts of structural couplings between modalities, questions old and new arise. Can a synthetic experience of sensory coupling stimulate the heightened mental states sometimes associated with neurological synesthesia? Can cross-sensory structures actually be perceived, or are they just a compositional device? Whether perceived as forms or not, will cross-sensory composition prove to be memorable in the same ways that paintings or music are?

We would like to suggest that the answer to these questions is a qualified "yes." Intermedia can be considered a form of immersion. Not unlike the 3D immersion of virtual reality, it involves a threshold experience, a point at which the senses become accommodated to artifice. While virtual reality involves problems of latency for which we have laboratory measurements, immersion in VR also has elements in common with immersion in a musical work, or with the willing suspension of disbelief in reading a novel. All of them represent moments when our cognitive system adjusts to and becomes absorbed in a particular form of representation. The scientist who repeatedly explores a data set with visual, sonic and possibly haptic mappings is not unlike the person who listens repeatedly to a symphony, hearing new things each time. Only the emotional responses are distinct. Whether we are looking into the inner processes of the physical world or the inner processes of the imagination, digitally-mediated multisensory experiences will open us to new insights.

DAC2001 Home