1. What’s happening in iCLEF? (the iCLEF Flickr Challenge) Julio Gonzalo (UNED), Paul Clough (U. Sheffield), Jussi Karlgren (SICS), Javier Artiles (UNED), Víctor Peinado (UNED)

2. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news A piece of good news for CLEF! People have started to hyperlink MT versions of web pages!

3. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news The original URL

4. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news The world is in danger

5. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news The world needs Multilingual Information Access research/solutions …(that do care about users)

6. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Retrospective: goals for iCLEF 2006 Find realistic scenarios where multilingual search arises naturally  multilingual web communities growing around (mostly) non-textual items (ebay, Flickr)  Explore Web 2.0 “new text” challenges: folksonomies, blogs, wikis, etc. Reduce the cost of entry for new participants.  Provide a default multilingual search interface to Flickr. Improve the cost-effectiveness of the traditional experiment design.  Let participants work on their own experimental designs.

7. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Flickr

8. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Flickr advantages Naturally multilingual Web 2.0 challenges (folksonomies) Replicability (unlike general web image search facilities)

9. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news iCLEF 2006 Topics  Ad hoc: find as many photographs of (different) european parliaments as possible.  Creative: find five illustrations for this article about saffron in Italy.  Visual: What is the name of the beach where this crab is lying on? Methodology  Participants must propose their own evaluation measures and experiment design.

10. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Experiments UNED focused on user’s behaviour  How users deal with native/passive/unknown languages?  Do they actually use CLIR facilities when available? SICS focused on user’s perceptions  Satisfaction (all tasks)  Completeness (creative,ad-hoc)  Quality (creative) Sheffield also measured search effectiveness  How many facets were retrieved (creative, ad-hoc)  Was the image found? (visual)

11. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Example result: choice of target languages along time.

12. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Achievements Found a natural CLIR task Flickr works for CLIR testing New perspectives for experiment design. Moved from bilingual to multilingual search problems.  Stick to Flickr in 2007

13. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Why participation was low? Kept on recruiting users and forcing them to search whatever we asked for.  Experiment design still too costly.  User populations still too small. Cost of entry still too high  Building a Flickr interface / adapting the provided interface was not trivial.  Experiment design too open for newcomers. Schedule collapses with standard tracks.  A problem for CLEF groups potentially interested.

14. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news How to improve? Kept on recruiting users and forcing them to search whatever we asked for.  Experiment design still too costly.  User populations still too small. Cost of entry still too high  Building a Flickr interface / adapting the provided interface was not trivial.  Experiment design too open for newcomers. Schedule collapses with standard tracks.  A problem for CLEF groups potentially interested.  Move to Flickr/Web users  Make search task a game Provide experiment design  Participants ≈ Searchers  Adjust iCLEF calendar

15. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news iCLEF 2007 (i)Target search task 海 (sea, wave) Find this image Clear goal for the user Clearly defined measures of success (appropriate for a “Hall of Fame” experience) Invokes different searching strategies

16. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news iCLEF 2007 (ii) Evaluation aspects User’s behaviour (Observational/Log Analysis)  Search strategies with native/passive/unknown languages User’s perceptions  Usefulness  Difficulty  Clarity, etc. Search effectiveness  Success rate and its correlation with user profiles, search strategies, etc.

17. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news iCLEF 2007 (iii) CL search interface Single web interface for all experiments. Available for:web interface  Groups willing to recruit users and conduct their own experiments.  Any web user wishing to enter the iCLEF Hall of Fame (by finding more images than anyone else!).  Groups willing to experience CL searching from the user’s perspective: who is the best CLEF team searching cross-language? Full experiments Log analysis

18. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

19. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

20. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

21. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

22. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

23. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

24. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

25. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

26. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

27. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news iCLEF 2007/2008 schedule End of October: Feedback from you! on  the multilingual search interface  experiment design  evaluation measures Mid-December: Guidelines + interface goes public  Groups start their own experiments.  People can start competing in the search challenge.  Organization starts collecting logs from Flickr users Mid-April:  Flickr/Web user Logs released to interested groups  Thematic session in Chorus Conference on Multimedia Search and Access (hosted by Yahoo!). Paper submission: aligned with CLEF 2008

28. Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Start testing the interface now! http://bender.lsi.uned.es:8080/iCLEF2007 Feedback to julio@lsi.uned.es