ELSE: Evaluation in language and speech engineering

• Research unit: EU RFP
• Project number: 98.0011

Key words (English)

Language and speech evaluation; natural language processing; speech processing

Alternative project number (English)

EU project number: LE4-8340

Research programs (English)

EU-programme: 4. Frame Research Programme - 1.1 Information technologies

Short description (English)

See abstract

Further information (English)

Full name of research-institution/enterprise:
EPF Lausanne
Faculté I & C
Laboratoire d'Intelligence Artificielle

Partners and International Organizations (English)

Uni. Odense, Uni. Saarland, Uni. Pisa, Uni. Sheffield, LIMSI-CNRS, CECOJI-CNRS, Xerox Research Center

Abstract (English)

The general objective of the ELSE project is to develop and test a generic infrastructure and a (semi)-automatic task-independent protocol for black-box quantitative evaluation of Language and Speech Engineering in a multilingual set-up. The choice of a the black-box approach offers the advantage of leading to an evaluation scheme easily automated, with reproducible results, and allowing multi-dimensional analysis taking into account various point of views. While being generic in its nature, the project is grounded in reality through its concrete implementation for a specific NLP task: part-of-speech tagging, for which a demonstration evaluation scheme will be implemented.
In addition to its specific sample instantiation, ELSE participates to the standardization efforts initiated by the European Commission and provides a generic framework that developers can further customize to adapt to their own expertise.
All ELSE participants are known for their experience in real-life NLP application development for text and/or speech processing. In addition, most of them have already organized or participated to evaluation programs, either in a national or an international context.
In the framework of the ELSE project, EPFL has contributed to the following work packages
WP1: Evaluation blueprint. Contribution to the definition of the generic evaluation protocol blue print for a task independent semi-automatic quantitative black-box evaluation scheme in a multi-lingual setup.
WP3: Data format and standards. Analysis of various issues (specific mark up, scalable confidentiality) related with data formatting, format standardization, and data distribution in the framework of an evaluation scheme.
WP4, WP5: Lexica and Corpora. Requirements and specifications to impose on lexica and corpora for their use in a quantitative black-box evaluation scheme
WP6, WP7: Software tools and sample implementation for POS tagger evaluation. \ Definition and functional specification of a minimal set of software tools required for the concrete implementation the evaluation chain described
\f0\ in the blueprint. mplementation of a sample evaluation for part-of-speech tagging in 3 European languages (French, English, and German), with real size data and prototype tools.\b\

References in databases (English)

Swiss Database: Euro-DB of the
State Secretariat for Education and Research
Hallwylstrasse 4
CH-3003 Berne, Switzerland
Tel. [phone]
Swiss Project-Number: 98.0011