LIS618 lecture 6 Thomas Krichel

Structure Google –news –interfaces to non-web sources Usenet ODP relational databases OpenURL file sharing

Google news Is a gathering of top stories from news stories. The entire pages in built by computer. Which stories make it to the top depends on –how prominently the stories appear on news sites –which sites the stories appear on –when the articles were published –how many articles cover the same story Note the side bar with stories of different topic sections.

special syntax for news I source: gives news from a source only –example source:cnn works –examples source:bbc, source:nytimes source:"new your times" dont seem to get anywhere. location: gives a location. Can by a two- letter state or a country –location:ny –location:russia

special syntax for news II allintitle: searches for words in the title of the article (not of the page) –example allintitle: dead injured allintext: searches for words in the text –example: allintext: saarland government allinurl: searches in article URLs –example: allinurl:bbc Wales

Google interfaces to 3 rd party data Google groups are an interface to Usenet news, called Google Groups. Google directory is an interface to the Open Directory Project. In both cases Google is dependent on the quality of these underlying data source.

Usenet news Usenet is a collection of user-submitted notes on various subjects that are posted to servers on a worldwide network. Each subject collection of posted notes is known as a newsgroup. A newsgroup is a discussion about a particular subject consisting of notes written to a networked site and distributed through Usenet. Newsgroups are hierarchical. Hierarchical levels are separated by dots example: comp.text.tex. alt, news, Info, biz, rec, comp, sci, humanities, soc, misc, talk are classic world-wide groups. –alt stands for anarchists, lunatics and terrorists.

Usenet history The idea of network news was born in 1979 when two graduate students, Tom Truscott and Jim Ellis, thought of using UUCP to connect machines for the purpose of information exchange among users. They set up a small network of three machines in North Carolina. UUCP is ``UNIX to UNIX copy'' a protocol that is used to copy files between machines running some flavor of UNIX, without the need for IP protocol. Usenet is older than the Internet

decline of Usenet essentially open to all (peer-to-peer system) used by spammers for –posting –gathering addresses steady decline of quality of contribution steady decline of quantity of contributions

Usenet worth checking out independent reviews of products, often written by experts. Example: interpretation of beethoven sonatas by Wilhelm Kempff. Sorting by date reveals that the newsgroup rec.music.classical.recordings is still active. On a good day, you will find no finer guide to records.

special syntax for Google Groups group: limits posting to a certain group title: limits to titles of postings author: searches for author name or address Mixing syntaxes works well. Example: intitle:kempff group:rec.music.classical.recordings

the open directory project The Open Directory Project is the largest, most comprehensive human-edited directory of the Web. It is constructed and maintained by a vast, global community of volunteer editors. Claim that there is a historic precedence in the Oxford English Dictionary. Formerly known as ``GnuHoo'', then ``NewHoo'', then acquired by NetScape, and called ``dmoz''.

dmoz.org dmoz is maintained by volunteers ``net-citizen''. No special qualifications required, but claimed to be experts. There are about 30,000 volunteers (they claim). Powers the core directory services for the Web's largest and most popular search engines and portals –Netscape Search AOL Search –GoogleLycos –HotBot DirectHit Headquarters run by Netscape

Appearance of ODP If Google finds a relevant category it put A Google response is a list of results. Each result has –title –snippet –URL Some results have optionally a category attached. Following such categories is a winner if your information need is broad.

full-text databases These databases have an emphasis on providing full-text information in a web environment. Their particular strength is the aggregation of material from a range of publishers. This especially concerns scholarly publishing, where the source material are distributed among a large number of sources.

Access Some of the is arranged via the Brooklyn LIU campus. The ones we can play with a –proquest, webauth.exe?rs=pq&lb=b –ebsco host, webauth.exe?rs=eh&lb=b The databases have some full-text, but not a lot.

Proquest go into the database selection, delete everything and then use the research library. we can search for Paul Levine. It appears that –not all articles have full-text –there is no distinction between different Paul Levines Otherwise it appears straightforward to use

aggregators Proquest and ebsco work as aggregators. They put different scholarly journals in one database together, so you dont have do deal with publishers different interfaces. Publishers are reluctant to join and impose moving-wall embargos on full-text release. So you can not access the full-text via them. But your library may have the text somewhere.

the library as aggregator typically, a library buys holdings from a publisher, as well as cross-publisher abstract and indexing data. when users finds a reference in an abstract and wants to access the full text, they are stuck Herbert Van de Sompel has been working on this problem.

special effects (SFX) Herberts idea was to equip the interface with a special effects button. When users press the button, the interface would transmit metadata such as –author name –journal name –title –date to a special database, called a resolver.

resolver The resolver examines the metadata and makes a decision on what to show to the user. –if the journal is subscribed to and the date is recent, it may formulate a query to the publishers database and fetch the record and/or full text there. –if the journal is not held, suggest ILL –etc…

configuring the resolver librarians, who know the local setting, will configure the server so that users are given the appropriate extended services given the local circumstance. Note that what is returned is a set of extended services, not the response to a specific query.

Bison Futé model This refers to further work by Herbert to generalize the idea. –On a web page, you find a link. It has been made by the provider of the web pages. –But this link may not be a appropriate. There maybe better technology that allows you to move in the same direction but with your own link. –In other words we talk about context-sensitive linking.

OpenURL This is now a draft standard with NISO to standardize the special effects request. The OpenURL is a transport architecture for context objects. Context objects unite descriptions of –the reference found –the context in which is was found

implications for information retrieval The implications on the library world are already important. –many library systems software already implement OpenURLs and provide resolvers But impact could be wider and could cover a whole new structure for the web, replacing static links with on-the-fly dynamic ones.

Databases Databases are collection of data with some organization to them. The classic example is the relational database. But not all database need to be relational databases.

Relational databases A relational database is a set of tables. There may be relations between the tables. Each table has a number of record. Each record has a number of fields. When the database is being set up, we fix –the size of each field –relationships between tables

Example: Movie database ID| title | director| date M1| Gone with the wind | F. Ford Coppola| 1963 M2| Room with a view| Coppola, F Ford| 1985 M3| High Noon| Woody Allan| 1974 M4 | Star Wars| Steve Spielberg| 1993 M5| Alien| Allen, Woody | 1987 M6| Blowing in the Wind| Spielberg, Steven| 1962 Single table No relations between tables, of course

Problem with this database I made up all the data. It is just for illustration. Name covered inconsistently. There is no way to find films by Woody Allan without having to go through all spelling variations. Mistakes are difficult to correct. We have to wade through all records, a masochists pleasure.

Better movie database ID| title | director| year M1| Gone with the wind | D1| 1963 M2| Room with a view| D1| 1985 M3| High Noon| D2| 1974 M4 | Star Wars| D3| 1993 M5| Alien| D2 | 1987 M6| Blowing in the Wind| D3| 1962 ID| director name| birth year D1| Ford Coppola, Francis| 1942 D2| Allan, Woody| 1957 D3| Spielberg, Steven| 1942

Relational database We have a one to many relationship between directors and film –Each film has one director –Each director has produced many films Here it becomes possible for the computer, and then the user –To know which films have been directed by Woody Allen –To find which films have been directed by a director born in 1942

Many-to-many relationships Each film has one director, but many actors star in it. Relationship between actors and films is a many to many relationship. Here are a few actors ID| sex| actor name| birth year A1| f| Brigitte Bardot | 1972 A2| m| George Clooney| 1927 A3| f| Marilyn Monroe| 1934

Actor/Movie table actor id| movie id A1| M4 A2| M3 A3| M2 A1| M5 A1| M3 A2| M6 A3| M4 … as many lines as required

SQL Once we have the relational database, we can ask sophisticated questions: –Which director has had the most female actors working for him? –In which years films have been shot that starred actors born between 1926 and 1935? Such questions can be encoded in a language know as structured query language or SQL. All relational database vendors implement a dialect of SQL.

importance of relational databases Relational databases dominate the world of structured information. Examples –employment and payroll in a company –stock management –e-commerce There are quite easy ways to get relational databases to work with web interfaces. Some are freely available. The most common one is the LAMP (Linux Apache MySQL PHP) architecture.

relational databases in libraries A 2004 enquiry on the LITA revealed that many respondents said that they did regret most not having learned more about relational databases in library school. But there are problems with relational databases in libraries –Slow on very large databases (such as catalogs) –Library data has nasty ad-hoc relationships, e.g. Translation of the first edition of a book CD supplement that comes with the print version Difficult to deal with in a system where all relations and field have to be set up at the start, can not be changed easily later.

off-web Internet information retrieval Under this heading, I principally think about activities known as file-sharing. They concern the (mostly illegal) exchange of files between users. Such files many encode –music –films There is a lot of it going on, but we are not sure how much.

Napster Napster was the first prominent file- sharing service. Napster ran a central server. You connected to that server and announced what files you had to share. Every search was conducted on the dataset assembled at the central server. Connections to download files were done between peer machines only.

end of Napster Napster argued since it was only involved in collecting the information about files available, it was legal. Napster never shared any illegal file. The courts thought otherwise. It was shut down. Napster network died without a central machine. To enable true piracy, we need a truly distributed system.

gnutella protocol This protocol underlies much of the current file-sharing activity on the Internet. It enables a peer-to-peer network between machines. To connect to a gnutella network, you need the IP address of one single machine that is already part of the network.

connection to the guntella network Once you establish connection to the first servent, you announce your presence. The first servent will pass on that message to all the servents that it is connected to, and so on. This quickly adds up to a lot of traffic!

time to live Every gnutella message has a time to live TTL. It is decremented every time it passes at a servent. The TTL is usually quite small. It can be arbitrarily reduced by servents. Therefore you only talk to servents that are close to you. But your software will determine which servents to try to contact first. That usually depends on previous query results.

searches When you do a search, it is passed on from servent to servent through the p2p network. Servents have their own rule how to respond to queries. –Most of the time search strings are matched against a file name. –Some may try to match against the directory name. –Some general queries may be rejected. –Some results sets may be truncated.

downloading If you see a file that you like to have, you can try to download it. To implement downloads the servents use http. Thus everyone who is connected to a file sharing network run a web server! However, there usually is a tight limit on how many downloads a server will accept. Modern servents have the ability to download from several servents.

ease to infringe Clearly all the traffic on gnutella, with current technology, can be observed. But the infringement is so massive that it appears difficult to clamp down on. The easy to infringe is technological. RIAA have sued. They reach the tippy top of the iceberg, with the hope to dissuade.

Thank you for your attention!