Presentation on theme: "How does a web search engine work?. search google (started 1998 … now worth $365 billion) bing amazon web, images, news, maps, books, shopping,"— Presentation transcript:
how does a web search engine work?
search google (started 1998 … now worth $365 billion) bing amazon web, images, news, maps, books, shopping, apps, videos, music – and much more! sometimes people tell you what they want type your query into google sometimes you have to guess and offer stuff to them amazon gives you shopping ideas
our objective search engines help people find things saves them time and effort as a search engine, your job is to make sure they can find things quickly and easily!
overview 1.collect the things that you want to search... take a snapshot of all the internet 2.figure out what those things are “about”... words in text documents, speech in videos, notes in sound 3.allow people to find what they want from your collection … decide which things you have are relevant … like finding a needle in a haystack!
3 core parts crawling the web ‘spider’ crawls across all the pages on the internet indexing like a librarian categorising books retrieving letting people find the stuff they want
but it gets complicated … the internet is huge (trillions of webpages) useless information (old, poorly written, advertising, duplicates) inappropriate stuff different languages spam
search engines are expensive lots of computers to do you work for you! but you need to tell them what work to do – programming. they all have to work together what if some break? 24/7/365 mobile phones, ipads, computers etc in every corner of the world lots of fast internet connections
let’s take a little look at how each part works…
1. crawling start with a bunch of website you know about and just follow the links… Imagine if you kept clicking all the links forever How long would it take to get back to the page you started on, if you were clicking on a different link each time? Could you cover all the pages on the internet? Is it equally likely you will cover all pages? What about more popular pages, for example: bbc.com, facebook.com etc?
2. indexing activity...
3. retrieval activity...
challenges words like ‘the’, ‘a’, ‘and’, ‘what’ – useless! tiger/tigers, bengali, tyger, big cat – plurals, spelling mistakes, synonyms what people search for ‘mismatches’ exactly what people write, even though it means the same! how easy to read, or “helpful” is a web page? how about the search ‘topic’ is the page, really?