1. The Cost of Inconsistency in Chord Shelley Zhuang, Ion Stoica, Randy Katz OASIS/i3 Retreat, January 2005

2. Introduction DHTs support a hashtable-like lookup interface Each node maintains a routing table about a few other (typically O(log n)) nodes A node communicates with other nodes to perform a lookup The routing state may become inconsistent as nodes continuously join and leave What is the impact of routing state inconsistencies on lookup performance?

3. Outline Introduction Cost of Inconsistency Performance Evaluation Conclusion

4. Routing State Inconsistencies InconsistencyGenerated byCorrected by False negative: a node falsely thinks a neighbor is up LeavesFailure detection algorithms False positive: a node falsely removes a neighbor that is up Failure detection algorithms Failure recovery algorithms Join hole: a new node joins but has not been fully incorporated into the routing state JoinsJoin algorithms Leave recovery: a node leaves and disrupts the routing state LeavesFailure recovery algorithms

5. Possible Outcomes of a Lookup Lookup Timeout Reply Correct Incorrect Loss Loop Premature Timeout

6. Chord Protocol Each node has an identifier Nodes are ordered on a circular identifier space Key k is assigned to the first node whose identifier is equal to or follows k on the circular identifier space Each node maintains a predecessor, successors, and fingers Correctness of lookup depends only on predecessor and successor pointers

7. Cost of Inconsistency - Loss Network loss False negative: a node forwards a lookup to a neighbor that has failed A B C

8. Cost of Inconsistency - Loop A forwards lookup to B, B forwards it on because B's pred is between key and B A B key

9. Cost of Inconsistency - Loop A forwards lookup to B, B forwards it on because B's pred is between key and B A B A's succ is wrong, B's pred is correct or wrong key  A's correct succ should be some node between A and key

10. Cost of Inconsistency - Loop A forwards lookup to B, B forwards it on because B's pred is between key and B A B A's succ is wrong, B's pred is correct or wrong key  A's correct succ should be some node between A and key  A's correct succ should be some node between key and B  The above two cases can be caused by FP, join, leave

11. Cost of Inconsistency - Loop A forwards lookup to B, B forwards it on because B's pred is between key and B A B A's succ is wrong, B's pred is correct or wrong key A's succ is correct, B's pred is wrong  A's correct succ should be some node between A and key  A's correct succ should be some node between key and B  The above two cases can be caused by FP, join, leave  B's correct pred should be A  The above case can be caused by FN

12. Cost of Inconsistency - Incorrect A forwards lookup to B, B accepts it A's succ must be wrong because there is some node between key and B B's pred must be wrong because B only accepts a lookup if its pred is before key, but there is a node between key and B A B Y key

13. Cost of Inconsistency - Incorrect A forwards lookup to B, B accepts it A's succ must be wrong because there is some node between key and B B's pred must be wrong because B only accepts a lookup if its pred is before key, but there is a node between key and B A B A's succ must be wrong, B's pred must be wrong key Y  B’s correct pred should be some node between key and B (i.e. Y)  the above case can be caused by FP, join, leave

14. Cost of Inconsistency - Incorrect A forwards lookup to B, B accepts it A's succ must be wrong because there is some node between key and B B's pred must be wrong because B only accepts a lookup if its pred is before key, but there is a node between key and B A B A's succ must be wrong, B's pred must be wrong key Y  A's correct succ should be some node between A and key  B’s correct pred should be some node between key and B (i.e. Y)  the above case can be caused by FP, join, leave

15. Cost of Inconsistency - Incorrect A forwards lookup to B, B accepts it A's succ must be wrong because there is some node between key and B B's pred must be wrong because B only accepts a lookup if its pred is before key, but there is a node between key and B A B A's succ must be wrong, B's pred must be wrong key Y  A's correct succ should be some node between A and key  B’s correct pred should be some node between key and B (i.e. Y)  the above case can be caused by FP, join, leave  A's correct succ should be some node between key and B  the above cases can be caused by FP, join, leave

16. Outline Introduction Cost of Inconsistency Performance Evaluation Conclusion

17. Lookup Timeouts vs. Network Loss Rate

18. Lookup Timeouts vs. Churn Rate

19. Incorrect Lookups vs. Churn Rate

20. Outline Introduction Cost of Inconsistency Performance Evaluation Conclusion

21. Routing invariant has a first order impact on the relative cost of different types of inconsistencies Cost of false negatives is higher than false positives  more important to ensure timely failure detection than low probability of false positive Cost of join holes is higher than false positives and leave recoveries due to the routing invariant of Chord

22. Ongoing / Future Work Lookup losses  Perhop retry  End to end retry What is the best waypoint to try? Incorrect lookups  Predecessor list Cost of inconsistency in other DHTs  Correctness in Chord depends only on predecessor and successor pointers  Many other protocols do not have similar entries in the routing state that is sufficient for correctness