VIDAS: Video assisted with audio coding and representation

• Research unit: EU RFP
• Project number: 95.0689-1

Key words (English)

H.263; MPEG-4; video-telephony; model-based video; speech analysis; video analysis

Alternative project number (English)

EU project number: AC057

Research programs (English)

EU-programme: 4. Frame Research Programme - 1.2 Communications technologies

Short description (English)

See abstract

Partners and International Organizations (English)

DIST (University of Genova), Matra Communication, Modis SPA, Linköping University, Universitat Polytecnica Catalunya, IRISA, EPFL, Universite of Geneva, Philips

Abstract (English)

The basic objective of the project is that of approaching the problem of videophone coding from a joint audio and visual point of view, for both the analysis and the synthesis. The motivating idea is that personal audio-visual communication represents an information source that may be easily modelled, characterised in audio by a human speaker's voice and in video by the same speaker's face.

From the technical point of view the following goals have been reached in this project. The first goal is that of improving the recommendation H.324 through speech-assisted frame interpolation. The quality of the interpolation is to be determined through subjective evaluation of the interpolated sequences by deaf people focusing on lip movement. The subsequent goal is that of implementing a software prototype of a hybrid scheme based on the segmentation of the scene into the modelled component (the speaker's face) and into a non-modelled component (the background). This hybrid scheme has been developed in the MPEG-4 framework such as to provide a standard environment that may be used in a world-wide implementation. The face of the speaker is modelled by a 3-dimensional polygonal mesh. That mesh may either be a pre-defined standard face, or a user-defined face that can be transmitted along with the animation parameters in a compressed form.

Two demonstrators have been provided. First, a software platform with H.324 encoder/decoder has been provided, integrated with a module for speech analysis and articulatory estimation, lips extraction and tracking, audio assisted frame interpolation for increasing the frame frequency. The results have then been submitted for subjective evaluation. Second, a software demonstrator of a hybrid coding scheme compliant with the MPEG-4 standard has been provided, where speech analysis is used for applying suitable deformations onto the mesh representing the face of the speaker.

References in databases (English)

Swiss Database: Euro-DB of the
State Secretariat for Education and Research
Hallwylstrasse 4
CH-3003 Berne, Switzerland
Tel. [phone]
Swiss Project-Number: 95.0689-1