Mediate

• The Data flow diagram presents an overview of communication between the sensory component technologies.
  (Please use your browser's back arrow to return you to this page on choosing to view this interactive diagram)

• The following issues wil be described in relation to the software integration strategy:
  • Processes and Messaging.

  • Speed and Prioritisation.

  • Latency and 'Frame' Rates.

back to top

Processes and Messaging

• Distributed Processing

  Each of the modules are processor intensive, especially video, audio and signature. Furthermore, the native software environment for each is different. Some are hand-coded. Some are designed using an off-the-shelf runtime engine that must be present in the final system. Some are PC-based, while others are Mac based. Some can also be entirely hardware (eg. smart peripherals).

  In order to integrate the system over a multi-platform environment, it has been necessary to design a compact, fast and robust messaging system. The solution chosen is:

  • Hardware network using CAN (Controller Area Network)
    • CAN is proven fast and robust technology and is well-suited to intefacing with smart peripherals.

  • Software network using a Server-Client model (developed in C++)
    • Server provides message-routing power and communications over the network
    • Client enables ease of integration into each module with a compact set of fixed commands.

  • Messages use a header+payload format, where the header is standardised across the system. The header protocol is designed to facilitate expansion.

  At a functional level (ignoring the physical architecture), there is a messaging hub through which all messages flow. This enables flexibility and scalability of the system, since modules do not have to be individually integrated and are not dependent on the physical location of any another module.

• Parallel Processing

  Whilst the system will be distributed over several computers, there will also be several modules running within each computer. To facilitate simultaneous computation and load balancing, a multi-threading framework has been designed.

back to top

Speed and Prioritisation

• Rapid Response 'Event' Messages
  • Short and high priority.
  • Used where it is more important to know that something significant has happened, than to know the specific details.
  • Sent by media-sensing modules to media-stimulus modules (eg. sudden vocalisation captured by audio and responded to by video).
  • Enables low latency sense-to-stimulus response within existing interaction models.

• Regular 'Control', 'Status', 'Position', etc. Messages
  • Longer messages that occur more regularly.
  • Enables continuity of response and higher-level behaviour.

back to top

Latency and 'Frame' Rates

• Some modules must iterate with a higher frame rate than others.

• Some activities / messages must take effect more rapidly than others.

• Example:
  • It is important that synthesised video is generated for each video frame based on very recent sense data so that the user perceives no visual latency.

  • In contrast, control information from the Decision Maker, requiring the tactile module to begin running a new interaction model, occurs less regularly and can tolerate a relatively high latency without there being a noticeable sluggishness and without affecting the overall system behaviour.

• The orders of magnitude are:
  • 1000s + sensing & stimulus i/o
  • 100s media pre-analysis & synthesis positional information
  • 10s interaction model control (parameter updates)
  • 1s behavioural control

back to top