1. Searching very large bodies of data using a transparent peer-to-peer proxy Mike Taylor and Marc Cromme, Index Data mike@indexdata.com marc@indexdata.dk Albertosaurus sacophagus skull modified from Carr 1999. (Not relevant to the talk, but pretty.)

2. Overview Where we're headed in the next half-hour: The problem Standardised semantically rich search-and-retrieve protocols ANSI/NISO Z39.50 SRU Transparent protocol proxies Fan-out proxies, singly and in combination Peer-to-peer proxies Operation of the peer-to-peer proxy network Using the peer-to-peer proxy in Alvis Conclusions Transparent peer-to-peer proxy Mike Taylor, Index Data

3. The Problem The key advantage of the Internet is distribution – That's why there is so much information out there. The key problem of the Internet is aggregation – That's why it's so darned hard to find anything! How can we get at all that tasty data? Monolithic systems can only get us so far. Even Google – with its huge index – is limited by its inability to probe into the deep web. It is limited to dumb screen-scraping. We propose a solution made up of many autonomous nodes. We will approach this in several steps. Transparent peer-to-peer proxy Mike Taylor, Index Data

4. Step 1: standardised search-and-retrieve protocols Transparent peer-to-peer proxy Mike Taylor, Index Data Z39.50 client Z39.50 Library of Congress Z39.50 server

5. British Library Z39.50 server Library of Congress Z39.50 server Step 1: standardised search-and-retrieve protocols Transparent peer-to-peer proxy Mike Taylor, Index Data Z39.50 client Z39.50

6. Step 1: standardised search-and-retrieve protocols Transparent peer-to-peer proxy Mike Taylor, Index Data Z39.50 client Z39.50 Library of Congress Z39.50 server British Library Z39.50 server Local catalogue Z39.50 server

7. Step 1: standardised search-and-retrieve protocols Transparent peer-to-peer proxy Mike Taylor, Index Data Library of Congress Z39.50 server Metasearching Z39.50 client Z39.50 British Library Z39.50 server Local catalogue Z39.50 server Z39.50 This is possible because of the semantic alignment of the servers.

8. So can Z39.50 save the world? Transparent peer-to-peer proxy Mike Taylor, Index Data

9. No. Transparent peer-to-peer proxy Mike Taylor, Index Data Then the serpent saith unto Adam, Lo, why doth thine information service not use XML? And Adam saith, Verily, Z39.50 worketh just fine. But the serpent, who was subtle of tongue, saith unto him, But XML is more fashionable. And, behold, Adam was deceived, and did fall. – The Book of Standards, ch. 3, v. 4-6.

10. Welcome to the 21 st Century Transparent peer-to-peer proxy Mike Taylor, Index Data Library of Congress Z39.50 server Metasearching Z39.50 client Z39.50 British Library Z39.50 server Local catalogue Z39.50 server Z39.50 Everything must be XML

11. Welcome to the 21 st Century Transparent peer-to-peer proxy Mike Taylor, Index Data Library of Congress Z39.50 server Metasearching Z39.50 client Z39.50 British Library Z39.50 server Local catalogue Z39.50 server Z39.50 Resistance is useless!

12. XML-based search-and-retrieve protocols Transparent peer-to-peer proxy Mike Taylor, Index Data The binary Z39.50 protocol is superseded by SRU. (Search/Retrieve by Url). This is a NISO-registered standard for expressing queries using rich URLs, to obtain XML responses that contain records matching the query. h ttp://sru.miketaylor.org.uk/sru.pl? version=1.1& operation=searchRetrieve& query=dinosaur& startRecord=1& maximumRecords=1& recordSchema=dc

13. An SRU response (single DC record) Transparent peer-to-peer proxy Mike Taylor, Index Data 1.1 29 info:srw/schema/1/dc-v1.1 xml 1 <srw_dc:dc xmlns:srw_dc="info:srw/schema/1/dc-schema" xmlns="http://purl.org/dc/elements/1.1/"> Fossils Lappi, Megan. text New York, NY: Weigl Publishers 2005 en Studying fossils -- Fossil facts -- Gone forever -- A fossil is born -- From bone to stone -- Insects in amber -- Dinosaur footprints http://www.loc.gov/catdir/toc/ecip0415/2004004136.html URN:ISBN:1590362136

14. An SRU response (single DC record) Transparent peer-to-peer proxy Mike Taylor, Index Data 1.1 29 info:srw/schema/1/dc-v1.1 xml 1 <srw_dc:dc xmlns:srw_dc="info:srw/schema/1/dc-schema" xmlns="http://purl.org/dc/elements/1.1/"> Fossils Lappi, Megan. text New York, NY: Weigl Publishers 2005 en Studying fossils -- Fossil facts -- Gone forever -- A fossil is born -- From bone to stone -- Insects in amber -- Dinosaur footprints http://www.loc.gov/catdir/toc/ecip0415/2004004136.html URN:ISBN:1590362136

15. An SRU response (single DC record) Transparent peer-to-peer proxy Mike Taylor, Index Data 1.1 29 info:srw/schema/1/dc-v1.1 xml 1 <srw_dc:dc xmlns:srw_dc="info:srw/schema/1/dc-schema" xmlns="http://purl.org/dc/elements/1.1/"> Fossils Lappi, Megan. text New York, NY: Weigl Publishers 2005 en Studying fossils -- Fossil facts -- Gone forever -- A fossil is born -- From bone to stone -- Insects in amber -- Dinosaur footprints http://www.loc.gov/catdir/toc/ecip0415/2004004136.html URN:ISBN:1590362136

16. So we can go back to doing what we did before Transparent peer-to-peer proxy Mike Taylor, Index Data Library of Congress Z39.50 server Metasearching Z39.50 client Z39.50 British Library Z39.50 server Local catalogue Z39.50 server Z39.50

17. So we can go back to doing what we did before Transparent peer-to-peer proxy Mike Taylor, Index Data Library of Congress Z39.50 server Metasearching Z39.50 client Z39.50 British Library Z39.50 server Local catalogue Z39.50 server Z39.50

18. So we can go back to doing what we did before Transparent peer-to-peer proxy Mike Taylor, Index Data Library of Congress SRU server Metasearching SRU client SRU British Library SRU server Local catalogue SRU server SRU SRU gives us the same semantic alignment as Z39.50.

19. SRU's query language: CQL Transparent peer-to-peer proxy Mike Taylor, Index Data CQL (Common Query Language) is used by SRU. It may also be used in other contexts (including Z39.50). Its syntax is easy to learn, but very expressive. dinosaur title=dinosaur title=(dinosaur or pterosaur) and author=martill dc.title=*saur and dc.author=martill title exact "the complete dinosaur" and date < 2000 name=/phonetic "smith" fish prox/distance<3/unit=sentence frog

20. 2. Transparent protocol proxies Transparent peer-to-peer proxy Mike Taylor, Index Data Just a Squid acts as a proxy for the dumb HTTP protocol, so we can have proxies for semantically rich search-and-retrieve protocols. YAZ Proxy is one such – http://indexdata.com/yazproxy Because the protocol is rich, the proxy can do more than Squid: Performance improvements: Cache and re-use intialised sessions Cache and re-use search results Cache and re-use fetched records Server protection: Query sanitisation (for broken servers... you know who you are) Client throttling, based on request frequency or bandwidth Protocol-level and application-level logging.

21. 2. Transparent protocol proxies Transparent peer-to-peer proxy Mike Taylor, Index Data Library of Congress SRU server SRU proxy SRU client SRU

22. 3. Fan-out proxies – free metasearching for simple clients Transparent peer-to-peer proxy Mike Taylor, Index Data Library of Congress SRU server Metasearching SRU proxy SRU British Library SRU server Local catalogue SRU server SRU SRU client SRU The client knows nothing about what is happening to its innocent requests. All the metasearching intelligence goes here.

23. 3. Cascading fan-out proxies Transparent peer-to-peer proxy Mike Taylor, Index Data Server 1 Proxy 1 Proxy 2Proxy 3 Client Server 5Server 6Server 3Server 2Server 4

24. Hey, go nuts Transparent peer-to-peer proxy Mike Taylor, Index Data Server Proxy Client ServerProxyServerProxyServer Proxy ServerProxy Server etc., etc., etc....

25. Why this doesn't actually work Transparent peer-to-peer proxy Mike Taylor, Index Data Scaling problems! Every proxy must be administrated: Information about searched resources kept up to date Proxies must be kept running – a single failure knocks out a whole subtree Load on servers Every server is visited by every query What happens when a proxy calls another proxy higher up the tree? Loop detection is difficult in protocols such as Z39.50 and SRU.

26. 4. The peer-to-peer proxy Transparent peer-to-peer proxy Mike Taylor, Index Data Another appropriate SRU server SRU Some appropriate SRU server Really cool SRU server SRU SRU client SRU Big cloud of peers, acting as a proxy

27. What's going on here? Transparent peer-to-peer proxy Mike Taylor, Index Data Life is simple at the edges of the cloud: SRU clients connect to peers that act as SRU servers SRU servers respond to requests from peers that act as SRU clients This means that off-the-shelf SRU clients and servers can be used. Web-based SRU clients can be redeployed Servers such as the Library of Congress catalogue are available You can use our free Z39.50/SRU-enabled XML database, Zebra http://indexdata.com/zebra Although the cloud has its own structure, it is opaque to the clients and servers at the edge.

28. What's going on here? Transparent peer-to-peer proxy Mike Taylor, Index Data Life is a little more complex within the cloud: Peers must communicate between themselves using a P2P protocol Peers associated with a server must also make SRU requests Peers associated with a client must also handle SRU requests Each peer may act for a client, a server, both, or neither: Client peers are the entry-points into the P2P cloud Server peers actually get the job of searching done Servent peers behave as both clients and servers Some peers may participate in the network for routing purposes only. Clearly the servent peer is the general case that all the other specialise. This only needs to be written once.

29. Why this rocks Transparent peer-to-peer proxy Mike Taylor, Index Data The separation between the edges and the cloud is very important. SRU clients and servers are easy to write There will be lots of them out there: Clients providing many different user interfaces Servers providing access to many different collections Servent peers are difficult to write But that's OK, because only one such peer need ever be built Many instances make up a single peer-to-peer proxy cloud The cloud can be used by many clients, and can use many servers Client and server writers don't have to think about the hard stuff.

30. 5. Operation of the peer-to-peer proxy network Transparent peer-to-peer proxy Mike Taylor, Index Data What goes on inside the mysterious cloud? There is a dedicated peer-to-peer protocol used to: Introduce a new peer to the network Welcome a new peer with an initial list of neighbours Pass queries in a chain between peers Return search results back along the chain We won't cover the protocol in detail here, but: Many P2P protocols have Ping and Pong messages for introductions In ours, we have different messages: New peers cry Cathy! when they nuzzle up the network Existing peers respond with a cry of Heathcliff!

31. Everybody needs good neighbours Transparent peer-to-peer proxy Mike Taylor, Index Data Key principle: the network contains NO global information. (So there is no single point of failure.) Each peer knows only about a few nearby peers – its neighbours Old neighbours are dropped from the pool if they don't prove useful New neighbours are discovered in search responses: Peer A forwards a query to its neighbour Peer B Peer B can't answer it, so it forwards it to its neighbour, Peer X Peer X responds with useful information Peer A accepts this response, and remembers Peer X for next time. The usefulness of peers may be judged relative to specific subject areas rather than with a single absolute score.

32. Transparent peer-to-peer proxy Mike Taylor, Index Data SRU client SRU Everybody needs good neighbours Peer A

33. Transparent peer-to-peer proxy Mike Taylor, Index Data SRU client SRU Everybody needs good neighbours Peer BPeer A Can you help me with this?

34. Transparent peer-to-peer proxy Mike Taylor, Index Data SRU client SRU Everybody needs good neighbours Peer BPeer XPeer A Can you help me with this? No, but I'll ask my friend.

35. Transparent peer-to-peer proxy Mike Taylor, Index Data SRU Dinosaur Data SRU server SRU client SRU Everybody needs good neighbours Peer BPeer XPeer A Can you help me with this? No, but I'll ask my friend.

36. Transparent peer-to-peer proxy Mike Taylor, Index Data SRU Dinosaur Data SRU server SRU client SRU Everybody needs good neighbours Peer BPeer XPeer A Can you help me with this? No, but I'll ask my friend. Here you go, pal.

37. Transparent peer-to-peer proxy Mike Taylor, Index Data SRU Dinosaur Data SRU server SRU client SRU Everybody needs good neighbours Peer BPeer XPeer A Can you help me with this? No, but I'll ask my friend. Here you go, pal. (I'll remember that.)

38. Transparent peer-to-peer proxy Mike Taylor, Index Data Why this rocks, part II The peer-to-peer network has the following desirable properties: joinable – easy for new peers to join the network adaptive – the system evolves to improve through time autonomous – each peer can have its own strategies and policies robust – can cope seamlessly with holes appearing in the network efficient – generates minimal network traffic tunable – has parameters that we can play with ecologically diverse – different peers may be tuned differently Most of these properties are related to the key issue: NO GLOBAL KNOWLEDGE!

39. Transparent peer-to-peer proxy Mike Taylor, Index Data Life history of a query Queries can't be allowed to wander around the network forever. Each query begins with a certain lifespan. Each peer that accepts a query decrements its lifespan by one. The remaining lifespan travels with the query to other peers. If a query is passed to multiple peers, the lifespan is divided between them: It might be divided equally between two relevant neighbours It might be split between many neighbours It might be allocated to a single promising neighbour When a query's lifespan is expired, the peer may not propagate it further.

40. Transparent peer-to-peer proxy Mike Taylor, Index Data Ecological diversity Different peers will have different strategies for propagating queries. Some will tend to fan out in a broad but shallow pattern Some will produce tend to pass almost all lifespan to a single neighbour. Some will behave differently depending on the query We hope that diversity of peer strategies will help make the network robust. A query may carry with it a hint about how it likes its lifetime to be spent. Tracer bullet queries have a long, thin trajectory, then fan out. This is a useful way to periodically probe remote parts of the network, in order to discover new and relevant neighbours.

41. Transparent peer-to-peer proxy Mike Taylor, Index Data 6. Using the peer-to-peer proxy in Alvis Alvis is an ongoing European collaborative project to build what the proposal document calls a Superpeer semantic search engine. Named after the dwarf Alvis (all-wise) from Norse mythology, who answered Thor's questions all night... (And then turned to stone when the sun rose.) The Alvis superpeers are what we just call peers in this presentation. This is because Alvis also has another whole layer of peers. These implement a distributed hash table (DHT) of individual keys to find a suitable entry-point to the superpeer network. Testing will show us how much this optimisation buys us.

42. Transparent peer-to-peer proxy Mike Taylor, Index Data 7. Conclusions Standardised search-and-retrieve protocols facilitate interoperability. A well-defined protocol can be proxied. Proxies may transparently perform many different services. They may perform metasearching (fan-out proxy). Metasearching proxies may be cascaded. In practice, such cascades are hard to maintain, and scale poorly. Instead, an entire cloud of peers may function as a proxy. Queries are routed through the cloud to reach the most appropriate servers. Neither clients nor servers need know anything at all about the proxy. Tunable parameters allow us to tweak performance. The European project Alvis is built on such a peer-to-peer proxy. We want to see this kind of network running in the wild with many nodes.

43. Thanks for listening! Mike Taylor and Marc Cromme, Index Data mike@indexdata.com marc@indexdata.dk Albertosaurus sacophagus skull modified from Carr 1999. (We should all take the time to look at more dinosaurs.)