1. How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp 2, Mariëlle Stoelinga 2, Ivan Zapreev 1,2 1 MOVES Group, RWTH Aachen University 2 FMT Group, University of Twente, Enschede 3 ICIS, Radboud University, Nijmegen

2. Probabilistic Model Checker Probabilistic Model Checking Probabilistic SystemProbabilistic Requirement  Probabilistic ModelProbabilistic Formula YesNo Probability ETMCC MRMC PRISM (sparse) PRISM (hybrid) YMER VESTA

3. Why This Work? Used more often –applications: distributed systems, security, biology, quantum computing... Powerful tools Problem: Which tool to choose?

4. Probabilistic Model Checker Probabilistic Model Checkers Probabilistic SystemProbabilistic Requirement Probabilistic ModelProbabilistic Formula YesNo Probability Choices made Four examples Overall evaluation

5. Tools ETMC C numerical CTMCJava MRMCDTMC + CTMCC PRISM hybrid DTMC + CTMC MTBDD for transition matrix C(++ ) and Java PRISM sparse DTMC + CTMC sparse transition matrix VESTA statist CTMC, no SJava YMERCTMC, only U ≤t C(++ )

6. Experiment Relevance Repeatable Verifiable Significant Encapsulated

7. Selected Benchmarks Synchronous Leader Electiondiscrete time Randomized Dining Philosophersdiscrete time Birth–Death Processdiscrete time Tandem Queuing Networkcontinuous time Cyclic Server Pollingcontinuous time

8. Synchronous Leader Election nodes in a ring elect a leader –each node selects random number as id –passes it around the ring (synchronously) –if  unique id, node with maximum unique id is leader [Itai & Rodeh, 1990]

9. Synchronous Leader Election 1 3 2 1 5 5 4 2

10. 5 4 1 2 2 3 5 14 1 2 2 3 5 1 51 2 2 3 5 1 5 4 1 3 2 1 5 5 4 2

11. Randomized Dining Philosophers Dining Philosophers –pick up chopsticks in random order Deadlocks resolved –if there is no second chopstick, give up eating [Pnueli & Zuck, 1986]

12. Birth–Death Process Models a waiting queue Standard model in performance evaluation Limit queue size to get finite model

13. Tandem Queueing Network Two queues after each other [Hermanns, Meyer-Kayser & Siegle, 1999] checkin counter two-phase security check exponential

14. Cyclic Polling System server cycles over n stations and serves each one in turn –e.g. teacher walks through class, each pupil may ask a question [Ibe & Trivedi, 1990]

15. Modelling PRISM model.tra format model PRISM YMER model ETMCCMRMCYMERVESTA model adapt syntax informal description

16. Experiment 1 Qualitative Properties unbounded reachability with prob 1 Cyclic Polling System: busy 1  P ≥1 (true U poll 1 ) If station 1 is busy, the server will poll it eventually

18. PRISM: MTBDD Size Multi-Terminal BDD = data structure for transition matrix size heavily depends on model large MTBDD  slow Model# states# MTBDD nodes Synchronous leader election 500.0001.000.000. Cyclic polling7.000.000< 3.000.

19. CPS versus SLE runtime 7.077.888 states 2.745 MTBDD nodes 458.847 states 1.131.806 MTBDD nodes

20. VESTA: simulation problem actual probability close to bound P ≥p (...) estimate is almost always in [p– ,p+  ] some irregularity stops the simulation 0.95  Prob(yes  actual Prob≥p)  Prob(actual Prob≥p  yes)

21. P ≥1 (... U...) Timing Overview

22. Analysis ETMCCslow, out of memory MRMCfastest tool for small models PRISMonly symbolic  sparse=hybrid depends on MTBDD size VESTAexcessive number of samples, slow YMERnot implemented

23. depends heavily on MTBDD size depends heavily on MTBDD size depends heavily on MTBDD size Result Overview: Timing ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ –/0 N/A

24. Result Overview: Memory ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ 0/+ N/A MTBDD size varies heavily almost independent from model size

25. Experiment 2 Bounded Reachability Tandem Queueing Network: P <0.01 (true U ≤2 full ) Is the probability that the system gets full in 2 time units small?

27. Analysis ETMCCslow, out of memory PRISMsparse=faster, hybrid=smaller MRMCfast for small models VESTA ok if you can afford statistical errors YMER best choice if you can afford statistical errors

28. Result Overview: Timing ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ –/0 N/A bounded U –+0/++/+++++

29. Result Overview: Memory ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ 0/+ N/A bounded U –++/++++++

30. Experiment 3 Steady State Property Tandem Queuing Network S >0.2 ( P >0.1 (X 2nd queue full ) ) In equilibrium, the probability to satisfy is > 0.2 P >0.1 (X 2nd queue full ) P >0.1 (X... )

32. Simulating Steady State? simulation of bounded reachability has clear stopping criterion simulation of unbounded reachability  reachability with very large bound simulation of steady state?  never stops

33. Analysis ETMCCslow, out of memory PRISM sparse=faster hybrid=slightly smaller MRMCfastest VESTAnot implemented YMERnot implemented

34. Result Overview: Timing ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ –/0 N/A bounded U –+0/++/+++++ steady state –++0/++ N/A

35. Result Overview: Memory ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ 0/+ N/A bounded U –++/++++++ steady state –++/+++ N/A

36. Nested Formulas Birth–Death Process: P ≥0.8 (P ≥0.9 (true U ≤100 n 70 ) U n 50 ) The probability to reach n 50 (while the probability to reach n 70 in 100 steps never drops <0.9) is ≥0.8

38. Result Overview: Timing ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ –/0 N/A bounded U –+0/++/+++++ steady state –++0/++ N/A nested –++0/++– –– – N/A did not terminate

39. Result Overview: Memory ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ 0/+ N/A bounded U –++/++++++ steady state –++/+++ N/A nested –++/+++ N/A

40. Conclusions ETMCCworst, only small models MRMCfastest for small models PRISM hybrid fast if MTBDD is small PRISM sparse fast VESTArather slow, no S, statistical errors YMER slim & very fast, only U ≤t, few statistical errors