1. Logically Centralized, Physically Distributed Mark Stuart Day Cisco Systems

2. Standard disclaimer No matter what I say in this talk, I’m not making any Lotus product commitments. Cisco

3. Outline What people want What people can have An ancient example: –Replicated mail repository A recent example: –Content distribution network Conclusions

4. What people want Single name/location for single logical service Service never goes down Service grows/shrinks smoothly

5. What people can have Single name/location for single logical service Service never goes down Service grows/shrinks smoothly Occasional weird errors that violate user expectations

6. Some ancient history MIT-LCS-TR-376, Date: May 1987 REPLICATION AND RECONFIGURATION IN A DISTRIBUTED MAIL REPOSITORY Author(s): Day, M.S. Pages: 110 Price: $18.00AD Number: A186967 Keywords: data replication, software reconfiguration, availability, reliability, scalable systems, distributed programs, electronic mail repositories, programming languages

7. Mail system architecture (think of Grapevine) Mailbox 1 Mailbox 2 Mailbox 3 Mailbox 4 Client Directory

8. Highly available email Mailbox 1 Mailbox 2 Mailbox 1 Mailbox 2 Mailbox 1 Mailbox 2 Client Directory

9. How did it work? Systems success –Nice capability for quorum adjustment –New directory algorithm for deletions –Cool dynamic reconfiguration User failure –“What do you mean I can’t delete that message?” –“Where’s that message gone?”

10. A recent example: Content distribution networks Akamai, Digital Island, Mirror Image, Adero, … $Millions in revenue $Billions in market capitalization Might be worth knowing something about

11. The bad old days (without content distribution) Client Origin Server GET some/piece/o/content

12. New and improved (with content distribution) Client Origin Server Delivery Node Delivery Node Delivery Node Request Router Request Router Content Router Content Router GET

13. Virtues Client unchanged Origin server mostly unchanged –Content URLs may be modified Add delivery nodes transparently Move content around transparently

14. Caveats Lots of detail missing –Request routing: HTTP redirection, DNS interception, IP hijacking –Content routing: application-level multicast, IP multicast Both request routing and content routing are nontrivial problems

15. Weird user-visible errors Routed to failed box –Content fails to appear –Depending on routing/caching, maybe no content from that domain ever appears again for that client

16. Making weird errors into not-so-weird errors Deploy “next-click failover” –Delivery nodes clustered into “supernodes” with switch –Supernode monitors failures –IP addresses of failed nodes remapped onto live nodes Result is similar to common Web behavior –“What the hey?” [click] “Oh, OK.”

17. Conclusion People want something that’s logically centralized, physically distributed But they don’t want the weird errors that come with distribution A great thing about the Web: –People are already used to some weird errors