Type

Conference Proceedings

Authors

Robert Villa
Alan F. Smeaton
P. Punitha
Alia Amin
Daragh Byrne
Martin Halvey
Raphael Troncy
Peter Wilkins

Subjects

Computer Science

Topics
video retrieval interactive computer systems multimedia systems qualitative data user interfaces dublin city trecvid information retrieval

User variance and its impact on video retrieval benchmarking (2009)

Abstract In this paper, we describe one of the largest multi-site interactive video retrieval experiments conducted in a laboratory setting. Interactive video retrieval performance is difficult to cross-compare as variables exist across users, interfaces and the underlying retrieval engine. Conducted within the framework of TRECVID 2008, we completed a multi-site, multi-interface experiment. Three institutes participated involving 36 users, 12 each from Dublin City University (DCU, Ireland), University of Glasgow (GU, Scotland) and Centrum Wiskunde \& Informatica (CWI, the Netherlands). Three user interfaces were developed which all used the same search service. Using a latin squares arrangement, each user completed 12 topics, leading to 6 TRECVID runs per site, 18 in total. This allowed us to isolate the factors of users and interfaces from retrieval performance. In this paper we present an analysis of both the quantitative and qualitative data generated from this experiment, demonstrating that for interactive video retrieval with ``novice'' users, performance can vary by up to 300\% for the same system using different sets of users, whilst differences in performance of interface variants was in comparison not statistically different. Our results have implications for the manner in which interactive video retrieval experiments using non-expert users are evaluated. The primary focus of this paper is in highlighting that non-expert users generate very large performance fluctuations, which may either mask or create system variability. The discussion of why this happened is not covered by this paper.
Collections Ireland -> Dublin City University -> Publication Type = Conference or Workshop Item
Ireland -> Dublin City University -> Subject = Computer Science: Interactive computer systems
Ireland -> Dublin City University -> Subject = Computer Science
Ireland -> Dublin City University -> Status = Published
Ireland -> Dublin City University -> DCU Faculties and Centres = Research Initiatives and Centres: CLARITY: The Centre for Sensor Web Technologies
Ireland -> Dublin City University -> Subject = Computer Science: Multimedia systems
Ireland -> Dublin City University -> Subject = Computer Science: Information retrieval
Ireland -> Dublin City University -> DCU Faculties and Centres = Research Initiatives and Centres

Full list of authors on original publication

Robert Villa, Alan F. Smeaton, P. Punitha, Alia Amin, Daragh Byrne, Martin Halvey, Raphael Troncy, Peter Wilkins

Experts in our system

1
Alan F. Smeaton
Dublin City University
Total Publications: 492
 
2
Daragh Byrne
Dublin City University
Total Publications: 58