Title: Deixis, pointing and the experience of time: music cognition between discrete particulars and synoptic overview
Authors: Reybrouck, Mark # ×
Issue Date: 2001
Host Document: VII International symposium on systematic and comparative musicology. III International conference on cognitive musicology August 16-19 pages:151-155
Conference: VII International symposium on systematic and comparative musicology. III International conference on cognitive musicology location:Yväskylä date:16-19 August 2001
Abstract: This paper is programmatic in its claims. It tries to bring together the ‘objectivist’ and ‘non-objectivist’ approach to music cognition. Music, in fact, can be considered as a sonorous articulation through time. This is a ‘realistic’ claim, as there is an objective acoustic signal which the listener can process. Listening, however, is highly idiosyncratic and leans upon subjective and intentional aspects as well. What matters here is the mental involvement of the listener as the music proceeds. The listener can select and focus at will and has many possibilities for making sense out of the perceptual flux. It is the listener who is doing the cognizing, and this is a ‘non-objectivist’ or cognitive claim. Both conceptions are not totally opposed but can complement each other, and it is a principal aim of this paper to argue for such hybrid approach. The technological means that are available at this moment can be very helpful here. They have reached a state of sophistication which makes it possible to give an objective description of the sound, but they make it possible to manipulate the sound as well. This means a quasi-total control of the input side of perceptual processing, which is a basic requirement for all experimental research. The same holds true for the output side of perceptual processing, as is obvious for manifest reactions to the sound, but the internal neurophysiological processes of ideomotor processing (ERP’s) and motor imagery can be studied experimentally as well. The major challenge is the modeling of the processing proper of the acoustical input, as there is no linear relation between perceptual input and behavioral output. All higher psychic functions, in fact, are mediated functions, with the mind intervening actively in the processing of the stimuli. Cognitive processes, further, can be studied directly by neurophysiological research, which allows us to penetrate in the black box of consciousness. It is possible, however, to study cognitive processes indirectly by studying overt or manifest reactions to the sound, and the act of pointing is a possible starting point.

In this paper we investigate the problem of music cognition and the experience of time. Two major questions must be considered here: the transition from the acoustic signal to perceptual processing, and the transition from discrete particulars to relational continuity. In order to do so, we lean upon two conceptual domains: the domain of deixis
with its related notion of pointing and the ecological approach to perception.

The notion of pointing provides the necessary conceptual tools for describing the listener’s mental involvement and interaction with the sound, both at a manifest or internalized level. Pointing thus holds a position between mental pointing ( a kind of auditive tracking) and the gestural approach to music, with the coupling of action and perception.

The ecological approach to perception stresses the role of event perception and their ecological validity. Somewhat related to this approach is the construction of music knowledge as a kind of adaptation to the sonic world. Two things are important here: the elements the listener selects out of the perceptual flux and the way these elements are connected in a relational network. Music, in fact, is a temporal art, and the experience of time is not reducible to the concatenation of isolated elements. Hence the importance of a conceptual construction of time. What the listener experiences is not absolute time, but time that is filled with acoustic events and their interrelationships.

A central question in this is the representation of the music, both in time and outside of time. This representational aspect involves both local and global features of the music, allowing the listener to navigate through the sound, both at a manifest or virtual level. What is meant is a kind of sonic browsing and interaction with the sound with the listener being actively engaged in the construction of a mental map of the unfolding through time. Two options are possible here: the listener keeps step with the unfolding through time or he/she transcends himself/herself from the inexorable character of the arrow of time. The latter option is possible only if the listener has knowledge of the piece as a whole, stressing the difference between first listening and repeated listening. As such, the representational aspect of music is linked directly with knowledge construction, and these epistemological claims are central in the experience of time.

1. The temporality of music

Time has been considered as the defining character of music. There is, however, a tension between real and outer time - time as objectified - and perceived or experienced time. Music is not to be considered as a mere concatenation of temporal slices, but calls upon a relational network of acoustic events that provides structure and coherence. But structure involves simultaneity of interrelated events, and music is dependent upon consumption of time. The structural approach therefore considers time as a modality, stressing the fundamental achronous character of the music. Memory and imagination can be helpful here, in providing virtual instantianations of not actually sounding events. Time, then, is only the framework for auditory events. It is events, not time, that are in flux.

The first task in making sense of music is to extract auditory events out of the perceptual flux. This is an ecological claim that stresses the notion of ‘extraction’ and ‘information pickup’ and that takes for granted that the information is already structured in the music. Two things are important here: the sonorous flux and the extraction of events. What is needed therefore is a veridical description of the sounding acoustical signal and explanatory and heuristic conceptual tools for delimitating the events and their relationships. We see two options here: the empiricist claim that takes the sonorous articulation as a starting point, and the ecological claim that offers a description of the sonic environment rather than giving a merely acoustical description.

As to the empiricist claim, much work has already be done. The major objective here is to objectify the temporal unfolding through time, using the timeline for the representation of time ordering, and providing means for portraying the continuous acoustic signal. Several techniques are available at the moment (wavelet, sonogram, spectrogram, spectral images, sonic design) and many of them can be described in phenophysical and morphodynamical terms. Especially the domain of spectromorphology is likely to become an interesting contribution to this domain. There are, however, some shortcomings that apply to the lack of representation of pitch relationships, phrasings, overall dynamic levels, and particular processing techniques. What is needed, in fact, is a kind of time-varying imagery or time-based media representation that allows us to capture and to visualize sound on the basis of feature extraction and temporal segmentation. But the visual medium is not the only one. Much can be expected from the multimedia approach and the multimodal representation of sound, including interactional aspects of dealing with the sound (zooming, panning, temporal access control as start, stop, fast forward, reverse ...).

2. Event perception and the ecological approach

There is no translation algorithm from the acoustic signal to perceptual significance. At a lower sensational level, however, there are psychophysical and psychological constraints that function as gate-keepers for making sense of the acoustic flux, but it is the listener, who finally selects and focuses with acts of deliberate attention. The ecological approach, however, is a promising area of research, as it replaces the acoustic description of the sound by a description of the sonic environment to be perceived. What matters here are not primarily the acoustic properties, but ecological acoustic events. These are to be considered as higher-order variables with time-varying complex acoustic properties that conserve their relational structure during transformation. Events, further, are auditory patterns with a clearly defined beginning and ending and a gross temporal patterning. Essential in this event description is the interplay between invariance and change.

Events, further, can be picked up at different levels of grain-size. What is meant, is an aspect of discretization that makes it possible for the listener to label something as something, and that allows identification through discrimination. This quantal aspect of perception involves a kind of distantiation and polarization that calls forth cognitive economy, as it is much simpler to handle discrete labels than to be involved in processual aspects of predication. Auditive events, however, also have a continuous aspect as they are time-consuming. As such there are continuity requirements of the analog and continuous representation of the sound. What is needed, therefore, is an ongoing dynamic time-based representation that meets both the requirement of continuity and discrete labeling. And it is our conviction that the act of pointing and the gestural approach to dealing with music can provide the means for fulfilling these requirements.

3. Introducing the observer

Pointing to something is a kind of mediating tool between mental involvement and the external things or events that are pointed at. As such the act of pointing is a heuristic tool for semanticity: it provides both an external and internal reconstruction of the sonorous articulation and its focal points. Two things are important: the things which are pointed at and the act of pointing. They bring together the objective or allocentric and subjective or egocentric aspects of dealing with music.

The starting point is the setting up of an external frame of reference. What is needed primarily for pointing to the music as a temporal art is a time line and a temporal scale, which can function as a temporal grid for locating events in time. Hence the role of temporal indexing and time tags. Sounding events thus can be located against an external frame of reference but they can be related to other events as well. This is, in fact, an objectivist or realist approach to dealing with music.

The listener, however, also uses an internal frame of reference. The act of pointing, in fact, uses indexical an deictic devices, which take the listener as the origo for attributing dynamic-vectorial qualities to the sounding environment. What matters here is the temporal and spatial anchoring of the subject as origo, and the role of the actual listening situation. It is an interactional approach that fits well the claims of embodied and enactive or experiential cognition, which state that our knowledge is the result of having a body with perceptual and motor capacities. Music cognition, in this approach, is not the recovery of an objective reality or the projection of an inner world, but an active construction as the result of interaction with the sound. What is meant, is a kind of adaptive behavior and adaptive perceptual categorization that allows the listener to delimitate, to discriminate and to associate acoustic objects or events. The listener, then, can be considered as an adaptive device, which can alter its receptor, conceptual and motor tools. In doing this he acquires a kind of epistemic autonomy and semantic closure.

The concept of pointing is an interesting conceptual tool. It provides operational means for describing the listener’s interaction with the sound, both as a primitive marking system and a mechanism of selection. The act of pointing, further, can be manifest or internalized, and provides an external or internal reconstruction of the sonorous articulation and its focal points. Pointing, finally, is not restricted to isolated temporal slices, but can involve a continuous gesture as well. The whole problem of pointing, then, is the relation between the unfolding of time, the sonorous articulation and the mental involvement of the listener as the music proceeds. And the whole domain of deixis can be very helpful here.

4. Deixis and deictic

The concept of deixis is an old one that did not yet receive much attention in music research. It was introduced by Bühler who first coined the term ‘deictic field’ (Zeigfeld der Sprache). Such a deictic field has a zero point or origo, which can be personal (I), spatial (here) or temporal (now). In this sense it is possible to speak of a socio-spatio-temporal anchoring of a communication act. Deixis thus offers the spatiotemporal and social coordinates of the typical context of uttering and stresses the interactional situation. As such it can provide interesting theoretical tools for an interactional approach to music cognition.

The notion of deixis is linked with the concept of indexicality and refers basically to the notion of pointing. Three options are possible: demonstratio ad oculos, anaphora and deixis ad phantasma. Or in more operational terms: ostentive pointing, pointing back to something that has already occurred and the imaginative act of pointing mentally to something. All of them are applicable to the process of dealing with music, especially if one considers the possibilities of new technologies for sonic browsing and navigating through the sound. Digital signal processing in fact makes it very easy to have access to tracks and segments of sound files. There are the technological means of instantaneous representation of the sound with the possibility of capturing and freezing and subsequent inspection of ephemeral sounds. And the usage of a slide bar in musical representation offers the possibility for moving the origo and changing the viewpoint of the listener at will. Navigating through the sound, however, involves a kind of representation of the complete piece. First listening therefore calls upon the imagination rather than upon the music as objectified.

5. From discrete particulars to relational continuity: elements, relations and operations

Music cognition is not reducible to the concatenation of discrete slices of time. Music as experienced involves sequential and simultaneous processing of the sound, leaning heavily on memory and imagination, as the musical structure is not present at a glance. Grasping the structure of music needs consummation of time, but it is possible to transcend the articulation through time and to represent it as an achronous relational network. Two modalities are involved here: the existential mode of actuality and the virtual mode of potentiality. All actual sounding events are real and objectified, all possible future events are possible but not real. The whole process of listening, therefore cannot be objectified as the listener has at his/her disposal a whole universe of potential sounding events. This calls forth the background of the listener and could be implemented in an expert system that must be constructed for each individual listener. Rather than doing this, we argue for an operational description of the process of dealing with music, focusing primarily on the relational aspect of structure building. We make a distinction here between the structure that is intrinsic to the music and the structuring by the listener, leaning heavily on the pioneering work of Piaget in his analysis of logico-mathematical operations of the human mind. Cognitive or mental operations essentially deal with operations of grouping, ordering and bringing in correspondence. For doing this the human observer compares and relates, both at a manifest and at a virtual level of representation.

6. The problem of representation

Comparing and relating are mental operations. Mental operations, however, can be facilitated by using of visible signs or lasting marks. Hence the need of graphical notation and visible forms of representation. The same holds true for music, which is a continuous non-notational sounding system. The whole problem of transcription and representation is the translation of a continuous acoustic signal to a kind of discrete and symbolic representation. Or to put it in other terms: the transition from a deictic to a symbolic field. A deictic field is restricted to the actual sounding situation, whereas a symbolic field transcends the time-bound articulation and the non-commutative character of the unfolding. It makes possible an access to the flow of discourse as a whole and wandering through the sound, using mechanisms of catadeixis (pointing backwards) and anadeixis (pointing forward). It allows the transition from a motoric-continuous to a cognitive-discontinuous involvement with the sound. What is needed therefore is a kind of lasting representation that allows the listener to navigate through the music. We further argue for a representation that does justice to both the continuous and discrete representation of sound. Much is to be expected here from the general techniques of route-description and musical cartography, and the usage of charts, diagrams and graphs, as well as from the more general techniques of sonification and auditory display or time-varying imagery in particular. Representing music, then, is to be considered as a kind of motion graphics that visualizes music’s flow through time and allows the listener to point at it and to perform relations and operations on it.
Description: Musicologie (OE) Academische lerarenopleiding Letteren.
Publication status: published
KU Leuven publication type: IC
Appears in Collections:Musicology, Leuven
× corresponding author
# (joint) last author

Files in This Item:

There are no files associated with this item.

Request a copy


All items in Lirias are protected by copyright, with all rights reserved.