1. AlltheWeb Torbjørn Kanestrøm January 30 th, 2003

2. Agenda Who is FAST ? What do we do? Libraries; Relevant projects we have done What is AlltheWeb? Under the Hood: Phrasing & Lemmatization Take a tour of AlltheWeb –Simple searches (Web, News, Multimedia, FTP) –Advanced Web Search –Results Page Q & A

3. Who is FAST? San Francisco Tokyo Boston Norway Munich Rome London Paris Fast Search & Transfer (FAST) –Founded 1997 –Public company (Oslo Stock Exchange – June 2001) –One of the fastest growing companies in Europe –Profitable –200 employees –40+ Phds –12 offices world wide

4. What we do… Understand the Intention of a query Understanding content

5. //TECHNOLOGY Common Technology Platform FAST Solutions

6. Enterprise Portals Partners //BACKGROUND FAST Customers & Partners FAST is the creator of the real-time integrated search and filter technology solutions that are behind the scenes at some of the world's best known companies with the world's most demanding search problems

7. A few selected projects we have done - Relevant to every librarian

8. Questia

9. Questia – the online library

10. Nordic Web Archive w The Nordic Web Archive is a cooperation between the Nordic National Libraries (Finland, Sweden, Denmark, Norway, Iceland). w Project started in 2000, datacenter built deep inside a mountain in northern Norway w Collecting and archiving web documents of national interest and importance. w Everything published in the national domains (.NO,.DK,.FI etc.) w Everything written on the web in the respective languages w Everything referring to one of the countries (city, company, person, etc.) w Continuous project designed to scale indefinitely w Available to the research community, not a public site.

11. Elesevier Engineering Information Compendex® is the most comprehensive interdisciplinary engineering database in the world with almost seven million records referencing 5,000 engineering journals and conference materials dating from 1970. The database is updated weekly.

12. Combining scientific classification of the deep web and proprietary publications FASTs core search technology has enabled us to provide the best scientific search results, period - John Regazzi - Managing Director, Elsevier Science Web Server XML //BUSINESS CASES 120M web pages 17M Elsevier Science publications Scientific classification Grouping and identification of related articles Leading science Index Understanding content Scientific navigation Scirus.com – the webs Science search

13. What is AlltheWeb?

14. Showcase for FAST technology –Test new search features with real live audience –Several milion queries per day –40% North America, 30% Europe, and 30% rest of World Integrated interface for searching –2.1+ billion web pages, PDF docs, MS Word docs, & Flash objects –Continuously refreshed news from 5000+ global/local news sources –150 million images and videos –130 million ftp files –2 million mp3 files Targeted at advand searches

15. What makes AlltheWeb different? Versatility –Searching in 49 languages –Six seperate catalogues (Web, News, Pictures, Videos, MP3, FTP) –Fully customizable front-end (only major search site that is XHTML/CSS compl.) Solid Index –2.5 billion web objects (pages, pictures, videos, mp3s, etc.) –One of the fastest refresh cycle (every 7 – 14 days) Advanced search features –Boolean search –Embedded content selectors –Domain & IP filtering –File format and size filtering –Much more...

16. Under the Hood - Phrasing & Lemmatization

17. Under the Hood: Phrasing/Anti-Phrasing Phrasing: Known phrases are matched as a phrase –New York New York –Based on common phrases, names, movie names, geographic names, etc. –Can detect multiple phrases within same query Anti-Phrasing: Remove words irrelevant to the query –Who is… –What is… Combines to create a better query –Who is George Bush George Bush –What is the age of the earth the age of the earth –How do I get to train station in New York get to train station in New York

18. Under the Hood: Lemmatization Lemmatization improves recall –Literal matching only finds a fraction of candidates for a query Ratio between base and full forms –English: 2 –German, French, Spanish: 5 – 10 –Russian, Polish: 40+ Typical Cases: Singular/plural variation, case marking, etc. Stemming vs. Lemmatization –Traditional stemming Term is stemmed according to rules, e.g. walking walk Can easily result in false stemmings, e.g. Bobby Browning Bobby Brown –Lemmatization Rewriting of terms are controlled by language-sensitive dictionaries Very comprehensive dictionaries; about 20 man years

19. Take a Tour

20. AlltheWeb Home Page

21. Simple Search (Web/News) Web- and News Search Picture-, Video- and MP3 Search FTP Search Language detection Your query Match exact phrase Similar to using quotes around your query WebSearch University

22. Simple Search (Rich Media) Web- and News Search Picture-, Video- and MP3 Search FTP Search

23. Simple Search (FTP) Web- and News Search Picture-, Video- and MP3 Search FTP Search Query/ Expression Select between 13 different matching algorithms

24. Advanced Web Search Select Search Type All the words (AND) Any of the words (OR) The exact phrase Boolean expression Language/Charset 49 languages Most used characters sets Term / Phrase Only one phrase/word per filter. Add more filters if necessary. Where To Match Body text Page title URL Hostname Links on the page Embedded Content Exclude or include pages based on embedded content on these pages Specific Date range and Document depth

25. File Type Limits results to PDF, MS Word, and Macromedia Flash files Save Settings Keep knobs in the same position when you return Page Depth/Type Filter based on depth of URL and whether ~ occurs in URL IP-address filter For especially interested. Supports most common IP- address/-range syntaxes Advanced Web Search (cont.) Domain Filters Only include and/or exclude results from a domain Region Filter Limit results to different regions Document Size Specify size of document. Supports exact, less than or more than Presentation How many search results to list per page Offensive Content Whether or not to filter out/reduce results with offensive content

26. The Result Page Search Bar Click tabs to send query to other catalogs News Result Flashed in results from real-time News Search catalog Web Results Search results from Web Pages, PDF & MS Word files and Macromedia Flash files Site Collapsing Show the other hits from this site Query Rewriting Did we rewrite your query? Gives you full control! Paid content Revenue funds new features at AlltheWeb Related Queries Multimedia Results Results from other catalogues

27. www.AllTheWeb.com Has all the advanced search features and functions that you can find on all other major web search engines – combined... And we innovate at a faster pace and invest more in R&D than ever before.

28. AlltheWeb Q&A