Type

Conference Proceedings

Authors

Benoit Huet
Roeland Ordelman
Robin Aly
Gareth J.F. Jones
Maria Eskevich

Subjects

Computer Science

Topics
approaches individual experiences trecvid benchmark benchmarking ad hoc information retrieval research challenges

Pursuing a moving target: iterative use of benchmarking of a task to understand the task (2016)

Abstract Individual tasks carried out within benchmarking initiatives, or campaigns, enable direct comparison of alternative approaches to tackling shared research challenges and ideally promote new research ideas and foster communities of researchers interested in common or related scientific topics. When a task has a clear predefined use case, it might straightforwardly adopt a well established framework and methodology. For example, an ad hoc information retrieval task adopting the standard Cranfield paradigm. On the other hand, in cases of new and emerging tasks which pose more complex challenges in terms of use scenarios or dataset design, the development of a new task is far from a straightforward process. This letter summarises our reflections on our experiences as task organisers of the Search and Hyperlinking task from its origins as a Brave New Task at the MediaEval benchmarking campaign (2011–2014) to its current instantiation as a task at the NIST TRECVid benchmark (since 2015). We highlight the challenges encountered in the development of the task over a number of annual iterations, the solutions found so far, and our process for maintaining a vision for the ongoing advancement of the task’s ambition.
Collections Ireland -> Dublin City University -> Publication Type = Conference or Workshop Item
Ireland -> Dublin City University -> DCU Faculties and Centres = DCU Faculties and Schools: Faculty of Engineering and Computing: School of Computing
Ireland -> Dublin City University -> DCU Faculties and Centres = Research Initiatives and Centres: ADAPT
Ireland -> Dublin City University -> Status = Published
Ireland -> Dublin City University -> Subject = Computer Science: Information retrieval

Full list of authors on original publication

Benoit Huet, Roeland Ordelman, Robin Aly, Gareth J.F. Jones, Maria Eskevich

Experts in our system

1
Benoit Huet
Dublin City University
Total Publications: 3
 
2
Roeland Ordelman
Dublin City University
Total Publications: 12
 
3
Robin Aly
Dublin City University
Total Publications: 17
 
4
Gareth J. F. Jones
Dublin City University
Total Publications: 297
 
5
Maria Eskevich
Dublin City University
Total Publications: 21