1. Parallel External Directed Model Checking with Linear I/O Shahid Jabbar Stefan Edelkamp Computer Science Department University of Dortmund, Dortmund, Germany

2. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking2 Model Checking  Given A model of a system. A specification property  Model Checking Problem: Does the system satisfy the property ?  Method: An exhaustive exploration of the state space to search for a state that does not satisfy the property.  Problem: How to cope with large state spaces that do not fit into the main memory?

3. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking3 Directed Model Checking (Edelkamp, Leue, Lluch-Lafuente, 2004)  A guided search in the state space.  Usually by some heuristic estimate.  Only promising states are explored.  Under-certain conditions proved to be complete.  Shorter error trails Better for human comprehension  Problem: The inevitable demands of the model.. Space, space and space.

4. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking4 Possible Solution  Use Virtual Memory. Assume a bigger address space divided into pages. Saved on the hard disk but are moved back to the main memory whenever they are “called” – Page Faults. Pages are mapped to physical locations within the main memory and the desired content is returned from the main memory location.

5. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking5 Problem with the Virtual Memory 0x000…000 0xFFF…FFF Virtual Address Space Memory Page

6. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking6 External Memory Model (Aggarwal and Vitter) Input of size N and N >> M M B Disk If the input size is very large, running time depends on the I/Os rather than on the number of instructions. Scan(N) = O(N / B) Sort(N) = O(N/B log M/B N/B)

7. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking7 External BFS (Munagala & Ranade) A t t+1 t+2 BCDBCD XYZAXXYZAX XYZAXYZA XYZXYZ Duplicates’ Removal I: Remove Duplicates by sorting the nodes according to the indices and doing an scan and compaction phase. II: Subtract layers t and t+1 from t+2.

8. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking8 A* Algorithm a.k.a Goal-directed Dijkstra  A heuristic estimate is used to guide the search. E.g. Straight line distance from the current node to the goal in case of a graph with a geometric layout.  Reweighing: w’(u,v) = w(u,v) – h(u) + h(v)  Problems: A* needs to store all the states during exploration. A* generates large amount of duplicates that can be removed using an internal hash table – only if it can fit in the main memory. A* do not exhibit any locality of expansion. For large state spaces, standard virtual memory management can result in excessive page faults.

9. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking9 Bringing Locality …  Implicit, unweighted, undirected graphs  Consistent heuristic estimates. => ∆h ={-1,0,1} g 0 1 2 3 4 5 0123456 h A Bucket !!

10. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking10 External A* [Edelkamp, Jabbar, and Schroedl, 2004]  Buckets represent temporal locality – cache efficient order of expansion.  If we store the states in the same bucket together we can exploit the spatial locality.  Munagala and Ranade’s BFS and Korf’s delayed duplicate detection for implicit graphs. External A*

11. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking11 External Search For Model Checking [Jabbar and Edelkamp VMCAI – 05] + Uses Harddisk to store the state space divided in the form of Buckets. + Implemented on top of SPIN model checker. + Promising: Largest exploration so far took ~20 GB – much larger than even the address limits of most computers. + Pause and Resume support – Can add more harddisks. Problems: - Slow duplicate detection phase - Internal Processing Time >> External I/O time

12. Solution Distribute the internal working on multiple processors.

13. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking13 Distributed Directed Model Checking Observations:  Since each state in a Bucket is independent of the other – they can be expanded in a parallel fashion.  Duplicates removal can be distributed on different processors.  Bulk (Streamed) transfers between processors are much better than single transfers.

14. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking14 Distributed Queue P0 P1 P2 TOP Beware of the Mutual Exclusion Problem!!!

15. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking15 Delayed Duplicate Detection  Each state can appear several times in a bucket.  A bucket has to be searched completely for the duplicates. P0P1P2P3 GOAL Problem: Concurrent Writes !!!! Sorted buffers Single Files

16. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking16 Multiple Processors - Multiple Disks variant Sorted buffers w.r.t the hash val Sorted Files P1 P2 P3P4 Divide w.r.t the hash ranges Sorted buffers from every processor Sorted File h 0 ….. h k-1 h k ….. h l-1

17. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking17 I/O Complexity External memory algorithms are evaluated on the number of I/Os. Expansion: Linear in I/O => O( Scan ( V ))  Delayed Duplicate Detection: Phase I: Given that enough file pointers are provided by the operating system  Ω( scan ( E ))  Else Ω( sort ( E )) Phase 2: Subtracting previous levels: k.O( Scan ( E )) where k is bounded by the size of the largest cycle in the combined automata.

18. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking18 Comparison with other approaches  Delayed transfer.  Bulk transfer is much better than individual transfers over a network.  External Memory provides the space for large state spaces.

19. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking19 Deadlock Detection in CORBA-GIOP 3-2 (Space Consumption = 2.1 Gigabytes) Time taken on Proc. 1 Time taken on Proc. 2 Time taken on Proc. 3 Speed-up 1 Processor 25m 59s 18m 20s 2 Processors 17m 30s17m 29s1.48 9m 49s9m 44s1.89 3 Processors 15m 55s16m 6s15m 58s1.64 7m 32s7m 28s7m 22s2.44 Multiple Processors Machine Real-time CPU-time

20. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking20 Deadlock Detection in CORBA-GIOP 4-1 (Space Consumption = 5.2 Gigabytes) Time taken on Proc. 1 Time taken on Proc. 2 Time taken on Proc. 3 Speed-up 1 Processor 73m 10s 52m 50s 2 Processors 41m 42s41m 38s1.75 25m 56s25m 49s2.04 3 Processors 37m 24s34m 27s37m 20s2.12 18m 8s18m 11s18m 20s2.91 Multiple Processors Machine Real-time CPU-time

21. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking21 Deadlock Detection in CORBA-GIOP 4-2 (Space Consumption = 20 Gigabytes) Time taken on Proc. 1 Time taken on Proc. 2 Time taken on Proc. 3 Speed-up 1 Processor 269m 9s 186m 12s 2 Processors 165m 25s 1.62 91m 10s90m 32s2.04 3 Processors 151m 6s151m 3s151m 5s1.78 63m 12s63m 35s63m 59s2.93 Multiple Processors Machine Real-time CPU-time

22. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking22 Deadlock Detection in Optical Telegraph (Space Consumption = 4.3 Gigabytes) Time on Proc. 1 Time on Proc. 2 Time on Proc. 3 Speed-up 1 Processor 55m 53s 43m 26s 2 Processors 31m 43s31m 36s1.76 22m 46s22m 58s1.89 3 Processors 23m 32s23m 17s23m 10s2.41 15m 20s14m 24s14m 25s3.01 Multiple Processors Machine Real-time CPU-time

23. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking23 Deadlock Detection in Optical Telegraph (Space Consumption = 4.3 Gigabytes) Time taken on Proc. 1 Time taken on Proc. 2 Speed-up 1 Processor 76m 33s 26m 37s 2 Processors 54m 20s54m 6s1.41 14m 11s14m 12s1.87 Workstations connected via NFS Real-time CPU-time

24. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking24 Deadlock Detection in CORBA-GIOP 4-1 (Space Consumption = 5.2 Gigabytes) Time taken on Proc. 1 Time taken on Proc. 2 Speed-up 1 Processor 100m 27s 31m 6s 2 Processors 76m 38s76m 39s1.3 15m 52s15m 31s1.96 Workstations connected via NFS Real-time CPU-time

25. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking25 Summary  State space explosion problem can be circumvented by Directed External Model Checking  Time turns out to be a bottle-neck. not for the External I/O but for Expansion  Internal work is divided on multiple processors.  Delayed transfer of state sets  low network cost.  Implemented on top of IO-HSF-SPIN – SPIN model checker with external heuristic search.  Significant speed-up.

26. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking26 External Directed LTL Model Checking – under review  Schuppan and Biere approach => liveness as reachability.  Liveness requires searching for an acceptance cycle A path to a previously seen state that also visits an accepting state.  Save a tuple of states.  Two new heuristics to accelerate the search.

27. Shahid Jabbar (Dortmund)External Parallel Directed Model Checking27 External Directed LTL Model Checking – under review Arrives at the final state 0 1 2 3 4 Arrives again at the same final state Same states in both parts Current state Already seen final state