Partner und Internationale Organisationen
(Englisch)
|
Royal Institute of Technology KTH (S), Weizmann Institute of Science ( IL), Institut National de Recherche en Informatique (F), Katholieke Universiteit Leuven (B), University of Oxford (UK), Ecole Polytechnique Fédérale de Lausanne, EPFL (CH)
|
Abstract
(Englisch)
|
Video is a rich source of information. It provides temporal and spatial information about scenes. However, this information is implicitly buried inside the raw video data, and is very inefficiently organized. While the standard sequential frame-based representation of video data is adequate for viewing in a 'movie mode', it fails to support rapid access to information of interest that is required in many emerging applications. VIBES proposes new content-based representations of video data, which explicitly emphasize the geometric, photometric, and dynamic components of information. In particular, VIBES will develop methods for rapid video search, hyper-linking, re-animation and view synthesis, which will enhance the content and enrich the experience of video sequences.
Objectives:
Video provides continuous coverage of scenes over an extended region both in time and in space. That is what makes it more than a plain collection of images. In VIBES, our objective is to make video a first class data type, which can be searched on content, annotated, hyper-linked, and edited much as text can be now. Furthermore, video has many more modes of information than simple text. For example, it contains scene geometry and extended actions over multiple frames.
Our objectives are also to extract and use these 'modes'. With these aims, VIBES proposes new ways of exploring and using video that have the potential of leading to significant breakthroughs in video consumption and new industrial, commercial, and home entertainment applications. The tools we develop will enable cut detection, indexing, synthesis, and classification of non-static and non-rigid scenes.
Work description:
The project contains eight interlinked workpackages investigating two main themes:
1. Rapid browsing and retrieval: A video or a DVD will be automatically augmented with hyperlinks connecting shots containing a particular actor, type of action, or scene. E.g. all scenes inside the casino in 'Casablanca'. Such facilities will change the way in which video is addressed, significantly reducing the tedium and inefficiency of current serial video browsing.
2. 3D scene synthesis and human animation models: 3D scene geometry for virtual reality environments will be automatically generated for particular shots. E.g., the yellow brick road in the 'Wizard of Oz' could be reconstructed, and a viewer could then walk down it using VR together with virtual actors. 3D dynamical models of actors from classic movies will be learned and used to generate new scenes involving the actors -synthetic thespians- or to replace one actor by another. For example, replace the 'hero' in Home Alone or Toy Story by a texture mapped dynamical model of your son or daughter.
Milestones:
YEAR 1: Simple video unit segmentation. Feature based matching between shots. Simple object recognition. 3D scene models of shots. Simple within-shot human tracking.
YEAR 2: General video unit segmentation. Multi-shot matching. 3D scene models from multiple shots. Advanced human modelling and tracking. Initial hyper-linking demonstrator.
YEAR 3: Classification of action and certain object and scene types. Sequence-to-sequence alignment. Merging human models from multiple shots. Final web-based hyper-linking and video synthesis demonstrators.
|
