1. how does a web search engine work?

3. search  google (started 1998 … now worth $365 billion)  bing  amazon  web, images, news, maps, books, shopping, apps, videos, music – and much more!  sometimes people tell you what they want  type your query into google  sometimes you have to guess and offer stuff to them  amazon gives you shopping ideas

4. our objective search engines help people find things saves them time and effort as a search engine, your job is to make sure they can find things quickly and easily!

5. overview 1.collect the things that you want to search... take a snapshot of all the internet 2.figure out what those things are “about”... words in text documents, speech in videos, notes in sound 3.allow people to find what they want from your collection … decide which things you have are relevant … like finding a needle in a haystack!

6. 3 core parts crawling the web ‘spider’ crawls across all the pages on the internet indexing like a librarian categorising books retrieving letting people find the stuff they want

7. but it gets complicated …  the internet is huge (trillions of webpages)  useless information (old, poorly written, advertising, duplicates)  inappropriate stuff  different languages  spam

8. search engines are expensive  lots of computers to do you work for you!  but you need to tell them what work to do – programming.  they all have to work together  what if some break?  24/7/365  mobile phones, ipads, computers etc  in every corner of the world  lots of fast internet connections

10. servers

11. cooling

12. let’s take a little look at how each part works…

13. 1. crawling  start with a bunch of website you know about  and just follow the links…  Imagine if you kept clicking all the links forever  How long would it take to get back to the page you started on, if you were clicking on a different link each time?  Could you cover all the pages on the internet?  Is it equally likely you will cover all pages? What about more popular pages, for example: bbc.com, facebook.com etc?

15. 2. indexing activity...

16. 3. retrieval activity...

17. challenges  words like ‘the’, ‘a’, ‘and’, ‘what’ – useless!  tiger/tigers, bengali, tyger, big cat – plurals, spelling mistakes, synonyms  what people search for ‘mismatches’ exactly what people write, even though it means the same!  how easy to read, or “helpful” is a web page?  how about the search ‘topic’ is the page, really?