Scenarios ("use cases")

The CO-ME-DI-A project began by defining together some scenarios (“use cases”) that the partners would like to explore collectively. This is a first draft of the scenarios which will evolve, be refined etc. as they are implemented.

Tester for Multiple Network Ports. Testing basic network connection between all partners using audio, video, motion capture data and messaging. This scenario serves as a way of guaranteeing that basic network infrastructure (bandwidth, open ports etc..) is in place for subsequent projects. Connection starts with one-to-one then multiple nodes are added. Tests to be conducted by one node managing all remote computers. The aim is to map out the network conditions across all partners.

A lecturer, at one end of the network, is making a presentation of a particular technology (software or hardware). The audience, at the other end of the network, is made up of people who will follow the lecture. The audience includes a person, a host, who will field questions from the audience to the lecturer, if necessary. Aside from the audio and visual components, it is possible that other forms of data can come from the lecturer.

Robotic instruments are placed in different places over the network. Each instrument has a sensory setup in parallel to capture the audio, and/or any other generative source for data. Data is delivered to the other places and interpreted by the possibilities of the robotic instrument either directly or with a dialog reaction to the data. Each place has a different sound and character, so mapping is not be linear, (eg: automatic piano player, drum-android, ...) The instruments should be constructed or adapted in a development process before with the focus on the network performance, either as installation or a concert. Each place provides a video and audio stream for distribution and remote audience. It’s about mapping data in different domains of the instruments. The abstraction of the data leaves room for interpretation, but structures and gestures should be respected.

This scenario addresses the embodiment of remote performers through gesture-modulated visual display. 2 sites with 2 musicians/groups of musicians are linked with audio and motion capture data (MOCAP). MOCAP data is used for animation on one site and audio descriptors/analysis is used for animating abstract objects/forms on the other. Visualisation is intended both or the audience and for performers to interact with one another. The aim is to explore how remote performance can be rendered (visually and acoustically) in another site in a way that is readable by both performers and audiences. Audiences in both sites experience a specific perspective of a shared performance. Avatars (human-based and abstract) are used to address interaction between performers.

A conductor and three (or different number of) choirs. One choir is in the same place (and in eye contact) of the conductor, the others are remote.

Test configuration: conductor and three singers, two in remote places.

Interaction of the conductor with choirs, and direct interaction among choirs.

Standard condition: A video bidirectional connection between conductor and choirs, and an audio connection between choirs. Extension: the choirs can see each other and can interact accordingly. We use the normal video image of the conductor and the analysis multimodal data extracted in real time on the gesture of conductor. Alternate between real video and mapping of specific gestural cues, to explore the effectiveness of communication with choirs, the impact on interpretation of choirs.

“Etude” conditions: to trace and evaluate which are the expressive cues that are most relevant and best (naturally) perceived by performers: we can experiment with individuals one by one, given a restricted but artistically significant score and gestural conducting.

Track the state of the choir (individually or as a whole organism) in terms of relevant gestural cues, explicit signs and conventions (colored signs), and, for example, changes in postural attitude, facial expression and eye contact: we aim at measuring the empathic relation between conductor and each choir. Audience is in each place, but it is not involved actively: there no contact among the different audiences in the three places.

Improvisational piece, exploit latency (in audio and video) artistically.

Communication channels: which cues from director to choirs and vice-versa.

A concert in a source place is transmitted to a remote place where the audience is located. The concert “audio signature” is replicated in the destination. The audience is able to return to the source (performers) a quasi real time feedback in term of visual (e.g., movements) and audio (e.g. clapping, shouting…).

This simple scenario aims at exploring and studying specific problems of networked bidirectional expressive communication:

• How to achieve the awareness in audience that their feedback is perceived and understood by the source: amplify cues?
• How to transmit a high quality “audio signature” (spatialization problem, and room acoustics coherence between source and destination)

An ensemble is distributed over the network. At each node there are one or more musicians, all or part of which participate in the network. Each musician is playing an acoustic or electronic instrument and is in front of a computer whose screen is projected to the audience. Each node hears all the other nodes. Feedback is done via audio or controllers back to central computer and is analyzed at each node as well as injected (in a previously understood form by the musicians) into the central computer and thereby influences the composition. There is a conductor who masters the central computer. Performance is valid both in local and wide area networks. Conductor manages the global form of the work. A composer sets up the basic musical material, the rules of generation and transformation of the feedback. The composer may be the conductor.

Two musicians are situated in two places and communicate via audio, visual, and generated data. Each instrument is analyzed and transformed and generates data (audio and numerical) which is sent to the other musician. Each instrument can generate gestural control information. Visual gestural information is also sent. A composer sets up the basic musical material, the rules of generation and transformation of the data and audio. Each instrumentalist has a local audience. The audience can perceive the overall result. The audience can see the other performer.

A new approach towards controlling sound processing through an ambient spatial interface installed on stage. Distributed collaborative instruments can be developed with MELE creating participative performance spaces which invite to be inhabited rather than to function as spaces of control. Virtual instruments are played by the presence motions of the musicians. The virtual space is mapped over the network.

This scenario explores the network as medium with acoustic implications and potential. It addresses distance, latency, jitter and other network conditions between nodes as an integral part of a characteristic audible space. This scenario exploits the phenomena of synchronization and a-synchronization for the creation of material that is unique on each node.

How to build the awareness of community among two remote audiences.Mutual influence between remote audiences, sound/music (interactive audio scene) used to cause the emergence of a sense of community, shared goals, competition, cooperation, Shared experience, context-driven sense of community, entrainment between audiences. This scenario can be built step-by-step, e.g. starting from the condition of a couple of individuals, with minimal degrees of freedom in interaction.

This scenario explores a virtual environment as a performance situation to which 3 sites with 3 performing groups (music/movement) contribute. MOCAP data used for rendering remote performers and for audio modulated objects/forms for audience visualization. Content from all sites “meet” in a virtual environment which is open to an internet audience.

One Application is set up distributed over the net on different locations. Computer musicians, doing live coding, are playing together on one big computer music instrument, a kind of Hyper-instrument, to play and program this in real time as a concert. This could be done free as improvisation or with rough score to tell what should be done. The generated sound is played in every location, with same spatialisation. There is a communication line between the players in parallel (chat, low scan video, audio-speech).

Based on the idea of "active listening" – a concept developed by the project SAME – where a listener can explore a piece of music, for example, by changing his listening focus (by paying attention only to the string section, for example) or modifying the performance (by changing the tempo, for example), this scenario proposes active listening over the network where each node contains a part of the performance.