Partner und Internationale Organisationen
(Englisch)
|
AT, BE, CY, CZ, DK, FI, FR, DE, EL, HU, IT, LT, NL, NO, PT, SK, SI, ES, SE, CH, TR, UK
|
Abstract
(Englisch)
|
In the main phase of this project a concept was developed for generating a multimodal user interface (MMUI) for a target application from an abstract description of the tasks and concepts in this application: The generation process starts with a description of the tasks and concepts of the application in an OOA-Model augmented with additional information for the different modalities (GUI, speech input/output). In a first step this description is transformed into an abstract description of the whole MMUI comprising different screens. Each screen is build from a hierarchy of so-called multimodal objects (MMO). An MMO is a new concept of a multimodal component, we propose, which is is described further down. In a second step the actual MMUI for a specific device is generated from the abstract description of the MMUI. In this step, the MMUI-designer can fine tune the whole MMUI interaction. The result of this step is either a running MMUI, which finally must be connected with the application logic. Another possibility is to generate a standardized XML-description of the actual MMUI. MMOs are a new kind of multimodal components we propose in order to simplify the construction of MMUIs. An MMO is in the simplest case a basic interaction element like a single button or textfield. These MMOs are build according to the model-view-controller pattern. The model of the MMO is responsible for the logic of the MMO. The view of the MMO has different aspects, one for each modality. One aspect of the view is the graphical view well known from corresponding GUI-components. Another aspect of the view of an MMO is the acoustic aspect, which gives an acoustic representation of the MMO and is responsible for the spoken dialogue with the MMO. The controller of the MMO is responsible for the event fusion, i.e. to coordinate the different event coming from the different aspects of the view and compiling them into a semantic event of the MMO. This semantic event is then transmitted to the model of the MMO which normally results in a state change of the model. From these kind of basic MMO larger MMOs can be built by simply adding MMOs to other MMOs which than act as a container. The task of this container MMO is to coordinate the interaction of all MMOs it contains. Finally, the root-MMO is the screen itself which coordinates the whole multimodal dialogue with the user for a specific task of the application and therefore contains all interaction components necessary to perform this specific task. We have developed a simple proof-of-concept demonstrator for a flight information application based on MMOs. This demonstrator shows on the one hand how an MMUI for an application can be composed from a hierarchy of MMOs. On the other hand this demonstrator gave us a proof of comcept for the proposed generation of such MMUIs from an abstract description.
|
