Hypersearching the Web Hira Bashir - June 22, 2010 Soumen Chakarbarti, Byron Dom, S. Ravi Kumar, Prabhakar Raghavan, Sridhar Rajagopalan & Andrew Tomkins

… … … … … … … … … …

… … … … … … … … … … CHAOS!

Enter your text here… Search I’m a Search Engine!

Learn Pashto Language| Search I’m a Search Engine!

Learn Pashto Language Search I’m a Search Engine!

A Abate Abash. A Abate Abash. Webpage 1, Webpage 2, …, Webpage L Webpage 1, Webpage 2, …, Webpage M Webpage 1, Webpage 2, …, Webpage N. Webpage 1, Webpage 2, …, Webpage L Webpage 1, Webpage 2, …, Webpage M Webpage 1, Webpage 2, …, Webpage N. Words WebPages containing the word

A Abate Abash. A Abate Abash. Webpage 1, Webpage 2, …, Webpage L Webpage 1, Webpage 2, …, Webpage M Webpage 1, Webpage 2, …, Webpage N. Webpage 1, Webpage 2, …, Webpage L Webpage 1, Webpage 2, …, Webpage M Webpage 1, Webpage 2, …, Webpage N. Words WebPages containing the word Index

A Abate Abash. A Abate Abash. Webpage 1, Webpage 2, …, Webpage L Webpage 1, Webpage 2, …, Webpage M Webpage 1, Webpage 2, …, Webpage N. Webpage 1, Webpage 2, …, Webpage L Webpage 1, Webpage 2, …, Webpage M Webpage 1, Webpage 2, …, Webpage N. Words WebPages containing the word Challenge: Creating, maintaining, using Index

Ranking Function Heuristics

Ranking Function Heuristics Pashto is the National language of Afghanistan. Pashto is extensively spoken by Pashtuns tribes in Pakistan’s rural cities. Pashto has two major Dialects. The Pashto spoken in Peshawar differs from the Pashto spoken in Afghanistan or Baluchistan. This text would cover the basic vocabulary to familiarize you with this language. If you know Urdu, it would be easier you learn written…………. Pashto is the National language of Afghanistan. Pashto is extensively spoken by Pashtuns tribes in Pakistan’s rural cities. Pashto has two major Dialects. The Pashto spoken in Peshawar differs from the Pashto spoken in Afghanistan. Pashto poetry and pashto songs have been a highlight of pashto language. Pashto has a rich vocabulary. Pashto is much older then………………………………….

Ranking Function Heuristics Learn Pashto Language Welcome to the world’s best resource to learn…. -Pashto Poetry -Pashto Poets -Greetings in Pashto Welcome to the world’s best resource to learn…. Pashto is the National language of Afghanistan. Pashto is extensively spoken by Pashtuns tribes in Pakistan’s rural cities. Pashto has two major Dialects. The Pashto spoken in Peshawar differs from the Pashto spoken in Afghanistan or Baluchistan. This text would cover the basic vocabulary to familiarize you with this language. If you know Urdu, it would be easier you learn written…………. Pashto is the National language of Afghanistan. Pashto is extensively spoken by Pashtuns tribes in Pakistan’s rural cities. Pashto has two major Dialects. The Pashto spoken in Peshawar differs from the Pashto spoken in Afghanistan. Pashto poetry and pashto songs have been a highlight of pashto language. Pashto has a rich vocabulary. Pashto is much older then………………………………….

Ranking Function Heuristics Learn Pashto Language Welcome to the world’s best resource to learn…. -Pashto Poetry -Pashto Poets -Greetings in Pashto Welcome to the world’s best resource to learn…. Pashto is the National language of Afghanistan. Pashto is extensively spoken by Pashtuns tribes in Pakistan’s rural cities. Pashto has two major Dialects. The Pashto spoken in Peshawar differs from the Pashto spoken in Afghanistan or Baluchistan. This text would cover the basic vocabulary to familiarize you with this language. If you know Urdu, it would be easier you learn written…………. Pashto is the National language of Afghanistan. Pashto is extensively spoken by Pashtuns tribes in Pakistan’s rural cities. Pashto has two major Dialects. The Pashto spoken in Peshawar differs from the Pashto spoken in Afghanistan. Pashto poetry and pashto songs have been a highlight of pashto language. Pashto has a rich vocabulary. Pashto is much older then…………………………………. What’s the problem?

Cheap airfare cheap airfare cheap airfare cheap airfare…………….!!!

Ok, I do spamming but it ain’t just me!

Synonymy Polysemy Cheap airfare cheap airfare cheap airfare cheap airfare…………….!!! Ok, I do spamming but it ain’t just me!

Any Solution?

WordNet Project Indexed-based Search Engine By George A. Miller & Colleagues At Princeton University

WordNet Project Indexed-based Search Engine By George A. Miller & Colleagues At Princeton University Semantic Network Group of human linguistics

WordNet Project Indexed-based Search Engine By George A. Miller & Colleagues At Princeton University Car Automobile Semantic Network

WordNet Project Indexed-based Search Engine By George A. Miller & Colleagues At Princeton University Car Automobile Now search Semantic Network

Any Challenges?

Polysemy would aggravate!

And even with Synonymy….

Polysemy would aggravate! FAQs BOTs Browse

Clever Project -at IBM Are we ignoring something useful…

Clever Project -at IBM Yes…

...More than a billion carefully placed hyperlinks! Clever Project -at IBM

Some issues in search queries: - Harvard example - IBM example - Website Design

Some issues in search queries: - Harvard example - IBM example - Website Design Let’s fix this! Query List Query List Override with Predetermined RIGHT answers

Some issues in search queries: - Harvard example - IBM example - Website Design Let’s fix this! Query List Query List Override with Predetermined RIGHT answers An Observation

Clever Project -at IBM Underlying Approach/Idea Location A link to Location B Location B An implicit endorsement of Location B by Location A

Hub Authority Recommendation Hub - My Fav Links -Commercial Links -Personal Inventories Clever Project -at IBM Finding authoritative sites on broad topics with the help of hyperlinks

Clever Project -at IBM What computational method is used to identify hubs and authorities? Page 1 Page 2 Page 3. Candidate Pages Good Hub Good Authority Yes No Yes. No Yes. Initial estimates by guessing Estimate about Hubs Guess about Authorities Used to Improve

Clever Project -at IBM What computational method is used to identify hubs and authorities? Guess about Authorities Used to Improve Hub Estimate about Hubs

Clever Project -at IBM What computational method is used to identify hubs and authorities? Guess about Authorities Used to Improve Hub Estimate about Hubs

Clever Project -at IBM What computational method is used to identify hubs and authorities? Guess about Authorities Used to Improve Hub Estimate about Hubs Where does the best hubs point most heavily at?

Clever Project -at IBM More light on implementation Topic: Acupuncture Initial List (200 pages) By any standard text index, such as Alta Vista

Clever Project -at IBM More light on implementation Topic: Acupuncture Augmented List Initial List (200 pages) By any standard text index, such as Alta Vista Pages that link to and from the pages in the Initial list

Clever Project -at IBM More light on implementation Topic: Acupuncture Augmented List Initial List (200 pages) Initial Authority Score Initial Hub Score Sum of hub scores of other locations pointing to it Sum of authority scores of other locations pointing to it Root Set

Clever Project -at IBM More light on implementation Topic: Acupuncture Augmented List Initial List (200 pages) Initial Authority Score Initial Hub Score Sum of hub scores of other locations pointing to it Sum of authority scores of other locations pointing to it Root Set These scores will be re-adjusted iteratively, until results are fine-tuned & start settling down!

Clever Project -at IBM In visual terms...

Clever Project -at IBM Looking at the mathematics behind... Vector Matrix (Hub score & Authority score) Numerical values defining the hyperlinking structure of the root set Iteratively Result Hub & Authority Vector (Eigen Vector) Equilibrated to a certain number!

Clever Project -at IBM Looking at the mathematics behind... Vector Matrix (Hub score & Authority score) Numerical values defining the hyperlinking structure of the root set Iteratively Result Hub & Authority Vector (Eigen Vector) Equilibrated to a certain number! Observations -If root set’s size is 3000 pages, 5 rounds of calculations will be enough to steady the Hub and Authority scores -Algorithm is independent of initial scores

Clever Project -at IBM Iterative Process Separation of Websites Cluster 1 Cluster 3 A By-product of Clever!

Clever Project -at IBM Abortion Iterative Process Separation of Websites Cluster 1 Cluster 3 Pro-life Pro- choice A By-product of Clever!

Clever Project -at IBM Chaotic cover A Larger Perspective... Based on how pages are linked! Inherent albeit inchoate order

Clever Project -at IBM Paper 2 … Refrences … paper 4 … Garfield Measure Paper 4 … Refrences … paper 1 … Paper 5 … Refrences … paper 4 … Paper 1 … Refrences … paper 4 … Paper 3 … Refrences … paper 2 … Reference: Eugene Garfield, founder of Science Citation Index

Clever Project -at IBM Paper 2 … Refrences … paper 4 … Garfield Measure Paper 4 … Refrences … paper 1 … Paper 5 … Refrences … paper 4 … Paper 1 … Refrences … paper 4 … Paper 3 … Refrences … paper 2 … Reference: Eugene Garfield, founder of Science Citation Index

Clever Project -at IBM Paper 2 … Refrences … paper 4 … Garfield Measure (High) Impact Factor Paper 4 … Refrences … paper 1 … Paper 5 … Refrences … paper 4 … Paper 1 … Refrences … paper 4 … Paper 3 … Refrences … paper 2 … A metric that judges a paper by the number of citation it gets Reference: Eugene Garfield, founder of Science Citation Index

Any Challenges?

Any solution to this…?

Improvement to Garfield Measure Journal 1 Weight = X Journal 2 Weight = Y

Improvement to Garfield Measure Definition of Importance Journal 1 Weight = X Journal 2 Weight = Y But… c

Improvement to Garfield Measure Definition of Importance Journal 1 Weight = X Reference: Gabriel Pinski and Francis Narin (1976), CHI Research Journal 2 Weight = Y But… c An Iterative method for computing a stable set of adjusted scores (they called it influence weights) So there was indeed a very early solution! Authority Hub No distinction

A fundamental difference… Traditional Printed Scientific LiteratureWeb Hub Needed!

Investigating Power of Hyperlinks

Ranking Measure “Influence Weight” Idea (Pinski, Narin) Related to Link based

Investigating Power of Hyperlinks Ranking Measure “Influence Weight” Idea (Pinski, Narin) Related to Link based Heavy visited location Haphazard jumps

Investigating Power of Hyperlinks Ranking Measure “Influence Weight” Idea (Pinski, Narin) Related to Link based Heavy visited location Web 1101 Web 2304 Web 3060 Web page No. of Links to it In practice.. Haphazard jumps

Investigating Power of Hyperlinks Ranking Measure “Influence Weight” Idea (Pinski, Narin) Related to Link based Heavy visited location -Random Traversal -Finding a single kind of universally important page intuitively Web 1101 Web 2304 Web 3060 Web page No. of Links to it In practice.. Haphazard jumps

Difference Clever Different root set for each search Forward & Backward Google Initial ranking retained Faster Forward (link to link) Sociological Phenomenon

Future... Integrating Text & Hyperlinks Overcomes a shortcoming Listing Web resources Knitting communities Next 5 years? Challenges Fundamental changes?

Questions?