|
Search | Back Issues | Author Index | Title Index | Contents |
|
D-Lib Magazine
|
|
Carol Peters |
|
|
The results of the seventh campaign of the Cross-Language Evaluation Forum were presented at a two-and-a-half day workshop held in Alicante, Spain, 20-22 September, immediately following the tenth European Conference on Digital Libraries (ECDL 2006). The workshop was attended by over 130 researchers and system developers from academia and industry. The main objectives of the Cross-Language Evaluation Forum (CLEF) are to stimulate the development of mono- and multilingual information retrieval systems for European languages and to contribute to the building of a research community in the multidisciplinary area of multilingual information access (MLIA). These objectives are realised through the organisation of annual evaluation campaigns and workshops. The scope of CLEF has gradually expanded over the years. While in the early years, the main interest was in textual document retrieval, the focus is now diversified to include different kinds of text retrieval across languages and on different kinds of media (i.e., not just plain text but collections containing images and speech as well). In addition, attention is given to issues that regard system usability and user satisfaction with tasks to measure the effectiveness of interactive systems. Evaluation TracksIn CLEF 2006 eight tracks were offered to evaluate the performance of systems for:
Test SuitesMost of the tracks adopt a corpus-based automatic scoring method for the assessment of system performance. The test collections consist of sets of statements representing information needs known as topics (queries) and collections of documents (corpora). System performance is evaluated by judging the documents retrieved in response to a topic with respect to their relevance (relevance assessments) and computing recall and precision measures. The following document collections were used in CLEF 2006:
The ImageCLEF track used collections for both general photographic and medical image retrieval:
Diverse sets of topics or queries were prepared in many languages according to the needs of the various tracks. For example, the Ad Hoc track used document collections in nine different European languages and topics were offered in more than fifteen languages including non-European languages such as Amharic, Hindi, Indonesian, Oromo and Telugu. ParticipationParticipation was up again this year with 90 groups submitting results for one or more of the different tracks: 60 from Europe, 14 from North America, 10 from Asia, 4 from South America and 2 from Australia.
WorkshopThe campaign culminated in the workshop held in Alicante, 20-22 September. The workshop was divided between plenary track overviews and, parallel, poster and breakout sessions. In her opening talk (see Figure 1), Carol Peters, the CLEF Coordinator, stressed the need for more technical transfer activities. She commented that although many advances had been made in the multilingual information access research field, there were still few real-world operational cross-language systems. In her opinion, CLEF should be paying more attention to issues that directly regard the user and the needs of the application communities rather than focusing most attention on system performance only in terms of precision and recall. In fact, one of the most interesting activities this year was the real-time question answering exercise, organised on-site by Fernando Llopis and Elisa Noguera, U. Alicante (Figure 2). Here the aim was to examine the ability of question answering systems to respond within a time constraint. The need for more technical transfer was taken up again in the final session in two talks. Martin Braschler, U. Applied Sciences Winterthur, Switzerland, gave an insightful talk on "What MLIA Applications can learn from Evaluation Campaigns" while Fredric Gey from U.C. Berkeley, USA, summarised some of the main conclusions of the MLIA workshop at SIGIR 2006 in Seattle, where much of the discussion was concentrated on problems involved in building and marketing commercial MLIA systems. There was also an invited talk by Noriko Kando, National Institute of Informatics, Tokyo, Japan, on new evaluation activities at the NTCIR evaluation initiative for Asian languages.
ResultsCLEF has been running since 2000. The main results over the years can be summarised in the following points:
The presentations given at the CLEF Workshops and detailed reports on the experiments of CLEF 2006 and previous years can be found on the CLEF website at <http://www.clef-campaign.org/>. The preliminary agenda for CLEF 2007 will be available from mid-November. CLEF is an activity of the DELOS Network of Excellence for Digital Libraries. The CLEF website is at <http://www.clef-campaign.org>. Copyright © 2006 Carol Peters |
|
|
|
|
Top
| Contents |
|
|
|
D-Lib Magazine Access Terms and Conditions doi:10.1045/october2006-peters
|
