1. Non-blocking Data Structures for High- Performance Computing Håkan Sundell, PhD

2. 5 August 2005EPCC 2005 2 Outline Shared Memory Synchronization Methods Memory Management Shared Data Structures Dictionary Performance Conclusions

3. 5 August 2005EPCC 2005 3 Shared Memory CPU Cache Cache bus Memory... - Uniform Memory Access (UMA) - Non-Uniform Memory Access (NUMA)

4. 5 August 2005EPCC 2005 4 Synchronization Shared data structures needs synchronization! Accesses and updates must be coordinated to establish consistency. P1 P2 P3

5. 5 August 2005EPCC 2005 5 Hardware Synchronization Primitives Consensus 1 Atomic Read/Write Consensus 2 Atomic Test-And-Set (TAS), Fetch-And-Add (FAA), Swap Consensus Infinite Atomic Compare-And-Swap (CAS) Atomic Load-Linked/Store-Conditionally Read Write Read M=f(M,…)

6. 5 August 2005EPCC 2005 6 Mutual Exclusion Access to shared data will be atomic because of lock Reduced Parallelism by definition Blocking, Danger of priority inversion and deadlocks. Solutions exists, but with high overhead, especially for multi-processor systems P1 P2 P3

7. 5 August 2005EPCC 2005 7 Non-blocking Synchronization Perform operation/changes using atomic primitives Lock-Free Synchronization Optimistic approach Retries until succeeding Guarantees progress of at least one operation Wait-Free Synchronization Always finishes in a finite number of its own steps Coordination with all participants

8. 5 August 2005EPCC 2005 8 Memory Management Dynamic data structures need dynamic memory management Concurrent D.S. need concurrent M.M.!

9. 5 August 2005EPCC 2005 9 Concurrent Memory Management Concurrent Memory Allocation i.e. malloc/free functionality Concurrent Garbage Collection Questions (among many): When to re-use memory? How to de-reference pointers safely? P2P1P3

10. 5 August 2005EPCC 2005 10 Lock-Free Memory Management Memory Allocation Valois 1995: fixed block-size, fixed purpose Michael 2004: Gidenstam et al. 2004, any size, any purpose Garbage Collection Valois 1995, Detlefs et al. 2001: reference counting Michael 2002, Herlihy et al. 2002: hazard pointers Gidenstam, Papatriantafilou, Sundell and Tsigas 2005: hazard pointer + reference counting

11. 5 August 2005EPCC 2005 11 Lock-Free Reference Counting De-referencing links 1. Read the link contents, i.e. a pointer. 2. Increment (FAA) the reference count on the corresponding object. What if the link is changed between step 1 and 2? Solution by Detlefs et al: Use DCAS on step 2 that operates on two arbitrary memory words. Retries if link is changed after step 2. Solution by Valois et al: The reference count field is present indefinitely. Decrement reference count and retries if link is changed after step 2.

12. 5 August 2005EPCC 2005 12 Lock-Free Hazard Pointers (Michael 2002) De-referencing links 1. Read the link contents, i.e. a pointer. 2. Set a hazard pointer to the read pointer value. 3. Read the link contents again; if not same as in step 1 then restart from step 1. Deletion After deleted from data structure, put node on a local list. When the local list reaches a certain size; scan all hazard pointers globally, reclaim memory of all nodes which address does not match the scan.

13. 5 August 2005EPCC 2005 13 Lock-Free Memory Allocation Solution (lock-free), IBM freelists: Create a linked-list of the free nodes, allocate/reclaim using CAS Needs some mechanism to avoid the ABA problem. HeadMem 1Mem 2Mem n … Used 1 Reclaim Allocate

14. 5 August 2005EPCC 2005 14 Shared Data Structure: Dictionaries (Sets) Fundamental data structure Works on a set of pairs Three basic operations: Insert(k,v): Adds a new item v=FindKey(k): Finds the item v=DeleteKey(k): Finds and removes the item

15. 5 August 2005EPCC 2005 15 Randomized Algorithm: Skip Lists William Pugh: ”Skip Lists: A Probabilistic Alternative to Balanced Trees”, 1990 Layers of ordered lists with different densities, achieves a tree-like behavior Time complexity: O(log 2 N) – probabilistic! 1234567 HeadTail 50% 25% …

16. 5 August 2005EPCC 2005 16 New Lock-Free Concurrent Skip List Define node state to depend on the insertion status at lowest level as well as a deletion flag Insert from lowest level going upwards Set deletion flag. Delete from highest level going downwards 1234567 DDDDDDD 1 2 3 p 1 2 3 p D

17. 5 August 2005EPCC 2005 17 Overlapping operations on shared data Example: Insert operation - which of 2 or 3 gets inserted? Solution: Compare-And-Swap atomic primitive: CAS(p:pointer to word, old:word, new:word):boolean atomic do if *p = old then *p := new; return true; else return false; 1 2 3 4 Insert 3 Insert 2

18. 5 August 2005EPCC 2005 18 Concurrent Insert vs. Delete operations Problem: - both nodes are deleted! Solution (Harris et al): Use bit 0 of pointer to mark deletion status 1 3 42 Delete Insert a) b) 1 3 42* a) b) c)

19. 5 August 2005EPCC 2005 19 Helping Scheme Threads need to traverse safely Need to remove marked-to-be-deleted nodes while traversing – Help! Finds previous node, finish deletion and continues traversing from previous node 1 42* 1 42* or ? ? 1 42*

20. 5 August 2005EPCC 2005 20 Lock-Free Skip List - Techniques Summary The Skip List is treated as layers of ordered lists Uses CAS atomic primitive Lock-Free memory management IBM Freelists Reference counting (Valois+Michael&Scott) Helping scheme Back-Off strategy All together proved to be linearizable

21. 5 August 2005EPCC 2005 21 Lock-Free Skip List publications First publications in literature: H. Sundell and P. Tsigas, ”Fast and Lock- Free Concurrent Priority Queues for Multi- thread Systems”, IPDPS 2003 H. Sundell and P. Tsigas, ”Scalable and Lock-Free Concurrent Dictionaries”, SAC 2004 Later publications: M. Fomitchev and E. Ruppert, “Lock-free linked lists and skip lists”, PODC 2004 K. Fraser, “Practical lock-freedom”, PhD thesis, 2004

22. 5 August 2005EPCC 2005 22 New Lock-Free Skip List ! The thread that fulfils the deletion of a node removes the next pointer when finished. Allows other threads to traverse through even marked next pointers. If not possible to traverse forward, go back to the remembered position on previous (upper) levels. Helps deletions-in-progress only when absolutely necessary. Works with a modified version of Michael’s Hazard Pointer memory management!

23. 5 August 2005EPCC 2005 23 Correctness Linearizability (Herlihy 1991) In order for an implementation to be linearizable, for every concurrent execution, there should exist an equal sequential execution that respects the partial order of the operations in the concurrent execution

24. 5 August 2005EPCC 2005 24 Correctness Define precise sequential semantics Define abstract state and its interpretation Show that state is atomically updated Define linearizability points Show that operations take effect atomically at these points with respect to sequential semantics Creates a total order using the linearizability points that respects the partial order The algorithm is linearizable

25. 5 August 2005EPCC 2005 25 Memory Consistency and Out-Of-Order execution Models on actual multiprocessor architectures: Relaxed Memory Order etc. Must insert special machine instructions (memory barriers) to enforce stronger memory consistency models! t W(x,1) TiTi TjTj TkTk W(y,0) R(y)=0W(x,0) R(x)=1 W(y,1) R(y)=1 R(x)=1 R(x)=0 R(y)=1

26. 5 August 2005EPCC 2005 26 Experiments Experiment with 1-32 threads performed on Sun Fire 15K with 48 cpu’s. Each thread performs 20000 operations, whereof the first total 50-10000 operations are Insert’s, remaining are equally randomly distributed over Insert, FindKey and DeleteKey’s. Fixed Skiplist maximum level of 10. Compare with implementations of other skip list-based dictionaries and a singly linked list by Michael, using same scenarios. Averaged execution time of 10 experiments.

27. 5 August 2005EPCC 2005 27

28. 5 August 2005EPCC 2005 28

29. 5 August 2005EPCC 2005 29

30. 5 August 2005EPCC 2005 30

31. 5 August 2005EPCC 2005 31

32. 5 August 2005EPCC 2005 32

33. 5 August 2005EPCC 2005 33

34. 5 August 2005EPCC 2005 34

35. 5 August 2005EPCC 2005 35 Multi-Word Compare-And- Swap Operations: bool CASN(int *p1, int o1, int n1, …); int Read(int *p); Standard algoritmic approach: 1. Try to acquire a lock on all positions of interest. 2. If already taken, help corresponding operation 3. If all taken and all match change status of operation 4. Remove locks and possibly write new values My approach: Wait-free memory management (IPDPS 2005) Lock stealing and lock hand-over Allow un-sorted pointers

36. 5 August 2005EPCC 2005 36

37. 5 August 2005EPCC 2005 37

38. 5 August 2005EPCC 2005 38

39. 5 August 2005EPCC 2005 39

40. 5 August 2005EPCC 2005 40

41. 5 August 2005EPCC 2005 41

42. 5 August 2005EPCC 2005 42

43. 5 August 2005EPCC 2005 43

44. 5 August 2005EPCC 2005 44

45. 5 August 2005EPCC 2005 45

46. 5 August 2005EPCC 2005 46

47. 5 August 2005EPCC 2005 47

48. 5 August 2005EPCC 2005 48 Lock-Free Deque Practical algorithms in literature: Michael 2003, ”CAS-based lock-free algorithm for shared deques”, Euro-Par 2003 Sundell and Tsigas, ”Lock-Free and Practical Doubly Linked List-Based Deques using Single-Word Compare-And-Swap”, OPODIS 2004 Approach Apply new memory management on lock- free deque

49. 5 August 2005EPCC 2005 49

50. 5 August 2005EPCC 2005 50

51. 5 August 2005EPCC 2005 51 Conclusions Work performed at EPCC Improved algorithm of lock-free skip list Improved Michael’s hazard pointer algorithm Experiments comparing with other recent dictionary algorithms New implementation of CASN. Experiments comparing with other recent CASN algorithms. Experiments comparing a lock-free deque algorithm using different memory management techniques. Future work Implement new lock-free/ wait-free dynamic data structures. More experiments.

52. 5 August 2005EPCC 2005 52 Questions? Contact Information: Address: Håkan Sundell Computing Science Chalmers University of Technology Email: phs@cs.chalmers.se Web: http://www.cs.chalmers.se/~phs