1. Carnegie Mellon Approved for Public Release, Distribution Unlimited Increasing Intrusion Tolerance Via Scalable Redundancy Michael Reiter reiter@cmu.edu Anastasia Ailamaki Greg Ganger Priya Narasimhan Chuck Cranor

2. Carnegie Mellon Approved for Public Release, Distribution Unlimited The Problem Space Distributed services manage redundant state across servers to tolerate faults We consider tolerance to Byzantine faults, as might result from an intrusion into a server or client  A faulty server or client may behave arbitrarily We also make no timing assumptions in this work  An “asynchronous” system

3. Carnegie Mellon Approved for Public Release, Distribution Unlimited Our Goals To design, implement and evaluate new protocols for implementing intrusion-tolerant services that scale better  Here, “scale” refers to efficiency as number of servers and number of failures tolerated grows Targeting three types of services  Read-write data objects  Custom “flat” object types for particular applications, notably directories for implementing an intrusion-tolerant file system  Arbitrary objects that support object nesting

4. Carnegie Mellon Approved for Public Release, Distribution Unlimited Expected Impact Significant efficiency and scalability benefits over today’s approaches to intrusion tolerance For example, for data services, we anticipate  At-least twofold latency improvement even at small configurations (e.g., tolerating 3-5 Byzantine server failures) over current best  And improvements will grow as system scales up  A twofold improvement in throughput, again growing with system size Without such improvements, intrusion tolerance will remain relegated to small deployments in narrow application areas

5. Carnegie Mellon Approved for Public Release, Distribution Unlimited Outline Concepts Challenges Techniques Systems Technology transfer

6. Carnegie Mellon Approved for Public Release, Distribution Unlimited Concepts: Distributed Services Service, or object, abstractionImplementation pushpopsort invocation response

7. Carnegie Mellon Approved for Public Release, Distribution Unlimited Concepts: Linearizability [Herlihy & Wing 1991] A strong and accepted semantics for shared objects  mimics semantics of a centralized object implementation  each method appears to be executed at a distinct point between its invocation and response time c1c1 c2c2 Object invocations Apparent execution

8. Carnegie Mellon Approved for Public Release, Distribution Unlimited inv Concepts: State Machine Replication Offers no load dispersion, and degrades as system scales Servers inv

9. Carnegie Mellon Approved for Public Release, Distribution Unlimited Concepts: Wait-Freedom [Herlihy 1990] A liveness property for object invocations Informally, an implementation is wait-free if any client’s operation is guaranteed to complete  Assuming a limit on the number of faulty servers [Jayanti et al.]  But not assuming a limit on the number of faulty clients Intuitively, wait-freedom precludes synchronization mechanisms that must be “unlocked” by a client Only read-write objects can be implemented in a wait-free way  Virtually any other object cannot (in an asynchronous system)

10. Carnegie Mellon Approved for Public Release, Distribution Unlimited Challenges: Concurrency Concurrent updates can violate linearizability Data 45123 Servers 45123

11. Carnegie Mellon Approved for Public Release, Distribution Unlimited Challenges: Server Failures Can attempt to mislead clients  Typically addressed by “voting” Servers ???? 31245 4’

12. Carnegie Mellon Approved for Public Release, Distribution Unlimited 54 Challenges: Client Failures Byzantine client failures can also mislead clients  Typically addressed by submitting a request via an agreement protocol Servers Data? 1234’?2’

13. Carnegie Mellon Approved for Public Release, Distribution Unlimited Challenges: Object Nesting Distributed objects have stubs and replicas Servers

14. Carnegie Mellon Approved for Public Release, Distribution Unlimited Challenges: Object Nesting

15. Carnegie Mellon Approved for Public Release, Distribution Unlimited Techniques: Versioning D 0 determined complete, returned Time ØØØØØ D0D0 D0D0 D0D0 D1D1 T0T0 T1T1 D0D0 D1D1 Ø D0D0 T1T1 Client read operation after T 1 12345 Ø D0D0 D 1 latest candidateD 1 incompleteD 0 latest candidate 3 writes required

16. Carnegie Mellon Approved for Public Release, Distribution Unlimited Techniques: Repair Time ØØØØØ D0D0 D0D0 D0D0 D1D1 T0T0 T1T1 T2T2 D0D0 D1D1 D2D2 T2T2 Client read operation after T 2 D2D2 12345 D2D2 D2D2 D2D2 Unreachable D 2 unclassifiableRepair D 2 D2D2 D2D2 D2D2 D2D2 Return D 2 D 2 latest candidate

17. Carnegie Mellon Approved for Public Release, Distribution Unlimited Techniques: Quorum Systems A quorum system is a data redundancy technique that supports load dispersion among servers Only a subset of servers are accessed in each operation Ex: Grid with n =49, b =3

18. Carnegie Mellon Approved for Public Release, Distribution Unlimited Techniques: Cross Checksums [Gong 1989] A mechanism for defending against Byzantine servers that attempts to alter data in their possession  Each data fragment is appended with a hash of all data fragments  When retrieved, hashes are used as “votes” to determine correct data fragments Data-item Data-fragments Hashes Cross checksum

19. Carnegie Mellon Approved for Public Release, Distribution Unlimited Techniques: Validating Timestamps A technique for defending against Byzantine clients that attempt to write different data values at the same timestamp  Cross-checksum of write value recorded in its timestamp  Read results are used to regenerate all data fragments and compare them to the timestamp Hashes Cross checksum All data-fragments Data-item Hash in timestamp Timestamp Read results

20. Carnegie Mellon Approved for Public Release, Distribution Unlimited Techniques: Replicated Invocation  b stub replicas cannot invoke > b stub replicas can

21. Carnegie Mellon Approved for Public Release, Distribution Unlimited Our Research To summarize, we will explore the use of these techniques for implementing  Read-write block storage (linearizable, wait-free)  Specialized metadata objects (e.g., directories) necessary to construct a fully functional file system (linearizable)  A general framework for arbitrary deterministic objects (linearizable) Not all techniques will be appropriate for all cases  “Flat” objects as found in file systems will generally not utilize replicated clients  Nested objects may not benefit from versioning (TBD)

22. Carnegie Mellon Approved for Public Release, Distribution Unlimited Systems: PASIS PASIS is a survivable storage system developed in a DARPA IPTO project  Funding ended December 2003 Examined the use of encoding schemes for efficiently distributing data storage while protecting confidentiality/integrity Did not address concurrency control  Clients would have to handle explicitly, e.g., using locking Explored use of versioning for other purposes: recovery from user mistakes, system failures, penetrations  Showed viability of comprehensive versioning

23. Carnegie Mellon Approved for Public Release, Distribution Unlimited Systems: Fleet Fleet is a Java-based distributed object architecture developed in previous projects in DARPA ATO  Funding ended June 2004 Focused on the use of quorum systems for efficient object replication Fleet does not support nested objects and nested method invocations Nor does it support potentially faulty clients

24. Carnegie Mellon Approved for Public Release, Distribution Unlimited Technology Transition Two primary channels are the industry consortia of two research centers at Carnegie Mellon: CyLab and the Parallel Data Lab CyLab  A center focused on trustworthy and measurable computing  Founded in 2003 through the merger of the Center for Computer and Communications Security and the Sustainable Computing Consortium  Corporate affiliate program includes over fifty companies, including defense suppliers, tech companies and IT-based critical infrastructures Parallel Data Lab  A ten-year-old center focused on storage infrastructures  Corporate affiliates include most major storage vendors Both have a track record of technology transfer

25. Carnegie Mellon Approved for Public Release, Distribution Unlimited Questions?