UNED at iCLEF 2008: Analysis of a large log of multilingual image searches in Flickr Victor Peinado, Javier Artiles, Julio Gonzalo and Fernando López-Ostenero.

Slides:

Advertisements

Similar presentations

Multilinguality & Semantic Search Eelco Mossel (University of Hamburg) Review Meeting, January 2008, Zürich.

Advertisements

SINAI-GIR A Multilingual Geographical IR System University of Jaén (Spain) José Manuel Perea Ortega CLEF 2008, 18 September, Aarhus (Denmark) Computer.

Terrapin Trader Transformation by Oliver Stohr - Olga Kuznetsova Tyler Cordrey - Brett Holbert December 9, 2008.

Implementing Website Search in a Major University Dan Jackson Senior Web Architect,

ImageCLEF breakout session Please help us to prepare ImageCLEF2010.

Overview of Collaborative Information Retrieval (CIR) at FIRE 2012 Debasis Ganguly, Johannes Leveling, Gareth Jones School of Computing, CNGL, Dublin City.

The Challenges of Multilingual Search Paul Clough The Information School University of Sheffield ISKO UK conference 8-9 July 2013.

Multilingual Information Access in a Digital Library Vamshi Ambati, Rohini U, Pramod, N Balakrishnan and Raj Reddy International Institute of Information.

East Asian Content in Integrated Systems— HathiTrust and the New Melvyl: Progress and Challenges UCEAB/EAALC Meeting September 15, 2011 Emily Lin UC Merced.

11 September 2002IR/LM workshop, Amherst1 Information retrieval, language and ‘language models’ Stephen Robertson Microsoft Research Cambridge and City.

Modern Information Retrieval

Information Retrieval Concerned with the: Representation of Storage of Organization of, and Access to Information items.

Cross Language IR Philip Resnik Salim Roukos Workshop on Challenges in Information Retrieval and Language Modeling Amherst, Massachusetts, September 11-12,

Measuring the quality of academic library electronic services and resources Jillian R Griffiths Research Associate CERLIM – Centre for Research in Library.

Semantic (Language) Models: Robustness, Structure & Beyond Thomas Hofmann Department of Computer Science Brown University Chief Scientist.

Advance Information Retrieval Topics Hassan Bashiri.

Retrieval Evaluation. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity. With large document.

Personalizing the Digital Library Experience Nicholas J. Belkin, Jacek Gwizdka, Xiangmin Zhang SCILS, Rutgers University

With or without users? Julio Gonzalo UNEDhttp://nlp.uned.es.

 Ad-hoc - This track tests mono- and cross- language text retrieval. Tasks in 2009 will test both CL and IR aspects.

Cross-Language Retrieval INST 734 Module 11 Doug Oard.

The SemEval-2007 Web People Search Evaluation The SemEval-2007 Web People Search Evaluatin Javier Artiles, Julio Gonzalo, Satoshi SekineThe SemEval-2007.

CS344: Introduction to Artificial Intelligence Vishal Vachhani M.Tech, CSE Lecture 34-35: CLIR and Ranking in IR.

The ECHA-term project Multilingual REACH and CLP Terminology Dieter Rummel, Translation Centre for the Bodies of the EU Luxembourg EAFT - Oslo, 11 October.

Assessing higher education learning outcomes globally Professor Hamish Coates

Challenges in Information Retrieval and Language Modeling Michael Shepherd Dalhousie University Halifax, NS Canada.

Personalization in Local Search Personalization of Content Ranking in the Context of Local Search Philip O’Brien, Xiao Luo, Tony Abou-Assaleh, Weizheng.

IAEA International Atomic Energy Agency Agenda item 2.6 INIS Collection Search 36 th Consultative Meeting of INIS Liaison Officers 4-5 October 2012, Vienna,

August 21, 2002Szechenyi National Library Support for Multilingual Information Access Douglas W. Oard College of Information Studies and Institute for.

CONFIDENTIAL Solving the Online Merchandising Challenge… 1 st May 2008 Frank Lombos – VP EMEA, Fredhopper.

COMPUTER-ASSISTED PLAGIARISM DETECTION PRESENTER: CSCI 6530 STUDENT.

JULIO GONZALO, VÍCTOR PEINADO, PAUL CLOUGH & JUSSI KARLGREN CLEF 2009, CORFU iCLEF 2009 overview tags : image_search, multilinguality, interactivity, log_analysis,

1 Cross-Lingual Query Suggestion Using Query Logs of Different Languages SIGIR 07.

CLEF 2004 – Interactive Xling Bookmarking, thesaurus, and cooperation in bilingual Q & A Jussi Karlgren – Preben Hansen –

Thanks to Bill Arms, Marti Hearst Documents. Last time Size of information –Continues to grow IR an old field, goes back to the ‘40s IR iterative process.

Mining the Web to Create Minority Language Corpora Rayid Ghani Accenture Technology Labs - Research Rosie Jones Carnegie Mellon University Dunja Mladenic.

Interactive Probabilistic Search for GikiCLEF Ray R Larson School of Information University of California, Berkeley Ray R Larson School of Information.

IAEA International Atomic Energy Agency Agenda item 2.2 INIS Collection Search 13th INIS/ETDE Joint Technical Committee Meeting October 2011, Vienna.

Intelligent Database Systems Lab 國立雲林科技大學 National Yunlin University of Science and Technology Advisor ： Dr. Hsu Presenter ： Chien Shing Chen Author: Wei-Hao.

Access to Information in Digital Libraries: Users and Digital Divide Gobinda G. Chowdhury Graduate School of Informatics, Department of Computer and Information.

Comparing syntactic semantic patterns and passages in Interactive Cross Language Information Access (iCLEF at the University of Alicante) Borja Navarro,

WIRED Week 3 Syllabus Update (next week) Readings Overview - Quick Review of Last Week’s IR Models (if time) - Evaluating IR Systems - Understanding Queries.

The Structure of Information Retrieval Systems LBSC 708A/CMSC 838L Douglas W. Oard and Philip Resnik Session 1: September 4, 2001.

First Meeting of the AgentLink SIG on Intelligent Information Agents (I2A) Hotel Euroflat, Brussels September 24-25, 1998 Report Innes A. Ferguson, SIG.

United Nations Economic Commission for Europe Statistical Division Data Initiatives: The UNECE Gender Database and Website Victoria Velkoff On behalf of.

Measuring How Good Your Search Engine Is. *. Information System Evaluation l Before 1993 evaluations were done using a few small, well-known corpora of.

Recuperação de Informação Cap. 01: Introdução 21 de Fevereiro de 1999 Berthier Ribeiro-Neto.

Information Retrieval Transfer Cycle Dania Bilal IS 530 Fall 2007.

Comparing Document Segmentation for Passage Retrieval in Question Answering Jorg Tiedemann University of Groningen presented by: Moy’awiah Al-Shannaq

What’s happening in iCLEF? (the iCLEF Flickr Challenge) Julio Gonzalo (UNED), Paul Clough (U. Sheffield), Jussi Karlgren (SICS), Javier Artiles (UNED),

Ranking of Database Query Results Nitesh Maan, Arujn Saraswat, Nishant Kapoor.

The Cross Language Image Retrieval Track: ImageCLEF Breakout session discussion.

1 13/05/07 1/20 LIST – DTSI – Interfaces, Cognitics and Virtual Reality Unit The INFILE project: a crosslingual filtering systems evaluation campaign Romaric.

Acceso a la información mediante exploración de sintagmas Anselmo Peñas, Julio Gonzalo y Felisa Verdejo Dpto. Lenguajes y Sistemas Informáticos UNED III.

The CLEF 2005 interactive track (iCLEF) Julio Gonzalo 1, Paul Clough 2 and Alessandro Vallin Departamento de Lenguajes y Sistemas Informáticos, Universidad.

Analysis of Experiments on Hybridization of different approaches in mono and cross-language information retrieval DAEDALUS – Data, Decisions and Language,

WHAT WOULD GOOGLE DO? Tracy Gilmore, Virginia Tech Lesley Moyo, Virginia Tech Elizabeth German, University of Houston Andrew Nagy, Serials Solutions Data.

From CLEF to TrebleCLEF Promoting Technology Transfer

WP5: Semantic Multimedia

Kyriaki Dimitriadou, Brandeis University

College of Information

User Requirements in the Cultural Heritage Domain

Multilingual Information Access in a Digital Library

News like you’ve never seen before… Alan Vien Bui, Fall 2009

INFORMATION RETRIEVAL TECHNIQUES BY DR. ADNAN ABID

Evaluation of IR Performance

CSE 635 Multimedia Information Retrieval

Web Mining Department of Computer Science and Engg.

Recuperação de Informação

Mobile Language Learning:

Presentation transcript:

UNED at iCLEF 2008: Analysis of a large log of multilingual image searches in Flickr Victor Peinado, Javier Artiles, Julio Gonzalo and Fernando López-Ostenero nlp.uned.es, Madrid CLEF workshop, 19 September 2008

iCLEF 2008 task Log analysis task. Data: logs from an online game (known item retrieval in Flickr) which implies searching in six languages simultaneously. Game interface implements reasonable baseline facilities for MLIA. Registered users with language profiles (native, active, passive, unknown languages)

Simultaneous search in six languages

Boolean search with translations

Relevance feedback

Assisted query translation

User profiles

User rank (Hall of Fame)

Group rank

Hint mechanism

15 global session attributes

84 search behaviour attributes

Harvested logs 109 topics 312 users / 41 teams 5101 complete search sessions Linguistics students, photography fans, IR researchers from industry and academia, monitored groups, other.

About users: procedence 312 users, 41 teams, 43 countries

About users: language skills

About user/image search session pairs The target image was annotated in a language that for the user was…

User’s behaviour and language skills Active Passive: ~ same success, more effort (~ +5%) Unknown: - 12% success, more effort (~ +10%)

You failed… what was the problem?

Overall… what were the challenges? Users learn that cross- linguality is a challenge!

Does learning improve success rate?

Cross-language interactive facilities: are they useful? At best, 1 out of 10 sessions manipulate translations provided by the system. So? Is that good or bad? Low or high?

Cross-language interactive facilities: are they useful? It’s good if compared with relevance feedback facilities!

Cross-Language Interactive facilities: are they perceived as useful?

Use of interactive facilities: Is there a learning effect?

Same with number of queries

Same with navigation beyond first page of results

Which interface facilities did you miss?

How were translations selected?

Conclusions Interactive CL features used more frequently than standard RF features. Occasional use of interactive features may have a big impact on the perceived usefulness of the interface! We finally have quantitatively relevant evidence on how much harder is to look for information in foreign (active/passive/unknown) languages… …And there is much space for improvement over baseline interface features!

Thank you!