1. LOCKSS: Lots of Copies Keeps Stuff Safe UNIVERSITY of WISCONSIN-MADISON Computer Sciences Department CS 739 Distributed Systems Andrea C. Arpaci-Dusseau Preserving Peer Replicas By Rate-Limited Sampled Voting, Maniatis, Roussopoulos, Giuli, Rosenthal, Baker, Muliadi (Stanford) -- SOSP’03

2. Motivation Librarians: Responsibility to preserve important materials Traditional approach: Acquire lots of copies Distribute around world Lend or copy to provide access Academic publishing is moving to Web LOCKSS: Real system used by many libraries (1999) How to apply techniques to digital preservation? Strength: Real problem that people care about, real solution being used

3. Design Goals and Assumptions Must be cheap to build and maintain No RAID systems Need not operate quickly Want to prevent change, not expedite it Must function properly for decades No centralized control Handle failures Handle malicious attackers Handle catastrophic random failures How is this different from other P2P systems?

4. Design Principles Cheap storage is unreliable No long-term secrets Can’t hold private keys for arbitrary time periods Use inertia Rate limit the amount of activity and change Avoid third-party reputation Malicious users can lie about good users Attackers can “cash in” history of good behavior Reduce predictability Make difficult for attackers to predict behavior of victims Make intrusion detection intrinsic Part of the system itself Assume strong adversary May want to change, suppress, or steal content

5. LOCKSS Overview Libraries run persistent web caches Collect by crawling journal web-sites Distribute by acting as limited proxy cache Preserve by cooperating with others to detect and repair damage Peers vote on large archival units (AU’s) AU == year’s run of a journal Each peer holds different AU’s If AU damaged, call increasingly specific partial polls

6. Opinion Poll Protocol Terminology: Loyal, malign, healthy, damaged peers Goal: High probability loyal peers are healthy (despite attacks by malign peers and failures) Low probability even powerful adversary can damage significant proportion of loyal peers without detection Overview Poll initiator calls opinion poll on AU >> rate of random damage Invites small subset of known peers (poll participant or voter) Voter computes and returns digest of AU Vote results for poll initiator: –Landslide win: Votes overwhelmingly agree with own version –Landslide loss: Repair AU by fetching copy of AU from peer –Inconclusive poll: Raise alarm for human attention Who can benefit from the poll? What if voter disagrees?

7. Peer Lists per AU Lists for every AU Friends list: Peers have outside relationships with friends Reference list: Peers encountered recently –Bootstrap: Init with friends list –Inner circle: Those invited to influence poll results –Outer circle: Nominated by inner circle

8. Poll Initiation Poll initiation: (about every 3 months per AU) Choose N random peers from ref list: Inner circle Send Poll [Poll ID, Diffie-Hellman Public Key] Wait for responses.. Voter from inner circle: Decide if want to participate Why might a peer not participate? Pick new DH public key, compute symmetric session key How does Diffie-Hellman work? –A chooses secret a, sends g^a mod p –B chooses secret b, sends g^b mod p –Each computes secret (g^b mod p)^a mod p = (g^a mod p)^b mod p Why encrypt messages?? Send back encrypted YES or NO to participate –Send PollChallenge [Poll ID, DH public key, {challenge, YES}]

9. Poll Effort Initiator: Produce computational effort for voter Why proof of computation by initiator needed? Use memory-bound functions (MBF) with poll id and challenge as input –Why are MBF good? Send back PollProof [Poll Id, poll effort proof] –Even send this to voters who responded NO. Why? Voter: Verifies result Less computation needed to verify result than compute Nominate outer circle peers (more later) –Randomly selected from reference list Send Vote messages for AU –Also send proof of computational effort in rounds –Why proof of computation by voter needed? Why in rounds?

10. Vote Tabulation Initiatator: Tabulates valid votes from inner circle Three cases: Landslide loss: Agreeing votes <= D –Repair AU Landslide win: Agreeing votes > V-D –Opinion poll concludes successfully; reschedule poll Inconclusive: Raise alarm Repair Initiator picks disagreeing voter and requests repair When is voter willing to supply content? Retabulate results with new content

11. Outer Circle What is the purpose of the outer circle? Initiator: Picks same number from every nominator Repeat same steps of protocol with outer circle –Why? Differences? Update reference list What is a malign peer trying to do? Who is removed? Insert: Valid/agreeing outer circle peers and random friends –Why?

12. Adversary Attacks Assume powerful adversary Total information awareness Perfect work balancing Perfect digital preservation Local eavesdropping Local spoofing Stealth Unconstrained identities Exploitation of Common peer vulnerabilities Complete parameter knowledge

13. Adversary Attacks Stealth modification Convince loyal peer has damaged AU Replace protected content with bad version Focus of paper Nuisance Raise alarms Attrition Make loyal peers waste computational resources so can’t repair damage Theft Acquire published content from peers without fee How does LOCKSS prevent? Free-loading Obtain services without supplying to others

14. Stealth Modification Attack 1)Lurk phase Increase foothold: malign peers in reference list (inner circle) –Wait until invited into circle –Act loyal –Nominate more malign peers 2) Attack phase When see poll is vulnerable (I.e., overwhelming majority of inner circle is malign), vote bad Why is attacking successfully hard? Rate limiting: Must wait for vulnerable polls to occur Damaged loyal peers call and vote in polls using bad copy –Can be repaired or raise alarms (doesn’t act differently when don’t have majority) Must expend effort calling polls too –Loyal peer only requests repair if voted in malign peer’s polls

15. Simulations Environment 1000 peers Clusters of 30 peers; 80% for friends, 20% random Call polls every 3 months on average N (size of inner circle): 20, Q: 10 How many false alarms with no adversaries? 20 years, random damage at every peer: 5-10 years

16. Simulation: Lurking Time How long must lurk for desired foothold ratio? 10% malign; how many years for 40% ratio? 50%? 30% malign; how many years for 50% ratio? 70%?

17. Simulation: Alarm Time How long before attack detected (I.e., inconclusive poll alarm raised)?

18. Simulation: Damage to AU How many bad replicas? How many years? When is irrecoverable damage caused?

19. Simulation: Worst-case How long should adversary lurk before attack?

20. Simulation: Benefit of Churn What churn rates are best?

21. Conclusions Interesting motivation Real problem and deployed solution Opinion Poll Protocol has many attractive properties Uses problem domain to guide protocol Inertia: Adversaries can’t influence poll timing Friend list: Use outside relationships to influence trust Attacking is very costly Must lurk long period to increase foothold in inner circle Must continually pay through proofs of computation (MBF) Immediately removed from lists if disagree Easy to set off alarms If voting results are inconclusive, human notified