Mediate

• Allows audio-input as one of the sensors of MEDIATE.

• Provides generation and output of Sound and Music and multi-channel audio-projection of sound-sources.

• Gives adaptive composition and interactivity reacting on the behaviour of the user.

• Translates input from other sensors to relevant Sonic output being either sonic structures or musical structures.

• The user is in control of the generated output and is invited to discover and experiment with the adaptive musical behaviour of the system.

• Enables the user to express and explore in terms of sound and music using all the sensors present in the MEDIATE environment.

back to top

Overall Schematic of the Sonic Interface

back to top

• 8-channel audio-input is provided.
  (microphones pick up sounds made by the user - the volume of this sound may vary widely.)

• Each input can be limited or compressed.

• Each input can be supplied with automatic gain control.

back to top

• Model 1
  
  • Pitch, amplitude, harmonic profile
  • Phrase recognition (start, stop)

• Model 2
  
  • Formant-analysis
  • A, e, i recognition

• Model 3
  
  • Vocoder-analysis (set of band-filters)

• Model 4
  
  • Complex analysis of spectral properties
    (prototype on basis of FFT is ready)
  • Data-reduction of the results

back to top

Communication of Analysis Results and Control Data

• Communication of data to and from other components of MEDIATE is anticipated to use Ethernet and TCP/IP. TCP/IP connections have been tested; modules have been implemented.

• The sonic interface can be controlled by commands; implementation of command-handling is in progress.

• Any other network and/or network protocol may be used instead of the ones mentioned.

• The final implementation will be part of the integration phase.

back to top

Mapping of Parameters of Other Sensors to Sonic and/or Musical Structures

• Several mapping strategies are implemented.

• The final implementation is part of the total Sound Design, which is part of the integration phase.

• The Sound Design will be discussed and tested with the PAS user group.

• Mappings of analysed video-images to sound structures have been implemented using soft-VNS (a video-processing package within MAX/MSP).

back to top

Interaction Models Acting On the Mapped Parameters

• Prototypes:
  
  • Several Interaction Models have been implemented for MIDI->MIDI and audio->MIDI.
  • Interaction Models acting on analysed video have been implemented and have been used in dance-performances.
  • Several interactive installations have been developed using video to sound interaction.

back to top

Generation of Functional Sonic and/or Muscial Structures

• Several synthesis-techniques have been implemented to generate Sonic Structures from mapped parameters.

back to top

Examples of Potential Mappings

example video 1 

The first is an extract from a performance by Marijke de Moel (27 May 1977 - 27 October 2002) - this is a tribute to her as a remarkable human being and her innovative and artistically creative use of space as an orchestrated virtual instrument generated by personal vocal production which is placed, manipulated and overlaid with further real time vocal output.

example video 2 

The second example is an extract from a dance performance by Isabel Ariel and Benjamin Petitjean with musical composition and interactive design by Marcel Wierckx. Here the dancers' movements manipulate the musical output in real time and are also captured and interact with the real time visually generated backdrop on both wall and floor.

back to top

Audio Output and Localisation of Sound Sources

• 8-channel audio-output is implemented.

• 8-channel localisation of Sound Sources is implemented using modified volume-panning techniques with interpolation during a time-interval in which sound-sources move from the initial position to a new position. This transition-interval can be varied from 0 msec to several seconds.

• Audio-engines have been implemented that can play stereo-sound files (like a sampler) localised as described before.

• Dependent on processor-speed 8 to 12 (stereo)-voice polyphony is available.

• A MIDI-sampler will be used for extra polyphony. The sampler output can also be localised.

back to top

Final Design Implementation and Integration

• In the integration-phase the final versions of several components will be developed and adapted to the overall design.
  
  
• The Sound Design of the system, the final design of Adaptive Behaviour, Interaction Models etc. will be an important part of the integration phase.

• Some of the topics are:
  
  • Data-communication and command-syntaxes
  • Data-reduction and mappings
  • Sound Design
  • Interaction Models

back to top

Interaction Models (Some Examples)

• Imitation
• Mirror
• Shadow
• Variation
• Reversal
• Echo
• Dialogue

back to top

Examples at Barcelona Meeting

• Through detection of an individual's position in relation to a quadraphonic arrangement of speakers a series of sounds could either follow or entice the movement of the individual. The potential delight of this technique was particularly evident when the song of a small bird featured as the attractor.

• Using higher resolution detection an individual's hand location in relation to a horizontal surface created a means to 'play' with sound elements and their sound characteristics e.g. frequency and duration.

• Using the 'interaction models' above these engagements will eventually orchestrate the final environment response.

back to top