Type

Conference Proceedings

Authors

Gareth J. F. Jones
Walid Magdy
Maria Eskevich

Subjects

Computer Science

Topics
information retrieval mean average precision multimedia systems informally structured speech structured data evaluation metrics speech retrieval speech search

New metrics for meaningful evaluation of informally structured speech retrieval (2012)

Abstract Search effectiveness for tasks where the retrieval units are clearly defined documents is generally evaluated using standard measures such as mean average precision (MAP). However, many practical speech search tasks focus on content within large spoken files lacking defined structure. These data must be segmented into smaller units for search which may only partially overlap with relevant material. We introduce two new metrics for the evaluation of search effectiveness for informally structured speech data: mean average segment precision (MASP) which measures retrieval performance in terms of both content segmentation and ranking with respect to relevance; and mean average segment distance-weighted precision (MASDWP) which takes into account the distance between the start of the relevant segment and the retrieved segment. We demonstrate the eectiveness of these new metrics on a retrieval test collection based on the AMI meeting corpus.
Collections Ireland -> Dublin City University -> DCU Faculties and Centres = Research Initiatives and Centres: Centre for Digital Video Processing (CDVP)
Ireland -> Dublin City University -> Status = Published
Ireland -> Dublin City University -> Subject = Computer Science: Multimedia systems
Ireland -> Dublin City University -> DCU Faculties and Centres = Research Initiatives and Centres
Ireland -> Dublin City University -> Publication Type = Conference or Workshop Item
Ireland -> Dublin City University -> DCU Faculties and Centres = DCU Faculties and Schools: Faculty of Engineering and Computing: School of Computing
Ireland -> Dublin City University -> Subject = Computer Science
Ireland -> Dublin City University -> DCU Faculties and Centres = Research Initiatives and Centres: Centre for Next Generation Localisation (CNGL)
Ireland -> Dublin City University -> DCU Faculties and Centres = DCU Faculties and Schools
Ireland -> Dublin City University -> Subject = Computer Science: Information retrieval
Ireland -> Dublin City University -> DCU Faculties and Centres = DCU Faculties and Schools: Faculty of Engineering and Computing

Full list of authors on original publication

Gareth J. F. Jones, Walid Magdy, Maria Eskevich

Experts in our system

1
Gareth J. F. Jones
Dublin City University
Total Publications: 297
 
2
Maria Eskevich
Dublin City University
Total Publications: 21