1. How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp 2, Mariëlle Stoelinga 2, Ivan Zapreev 1,2 1 MOVES Group, RWTH Aachen University 2 FMT Group, University of Twente, Enschede 3 ICIS, Radboud University, Nijmegen

2. Probabilistic Model Checker Probabilistic Model Checking Probabilistic SystemProbabilistic Requirement  Probabilistic ModelProbabilistic Formula YesNo Probability ETMCC MRMC PRISM (sparse) PRISM (hybrid) YMER VESTA

3. Why Are Probabilities Useful? system performance uncertainty in the environment randomized (networking) algorithms abstract from large populations

4. Probabilistic Model Checking... What is inside? –temporal logics + model checking –numerical and optimisation techniques from performance and operations research Where is it used? –powerful tools –applications: distributed systems, security, biology, quantum computing... Problem: Which tool to choose?

5. Probabilistic Model Checker Probabilistic Models Probabilistic SystemProbabilistic Requirement Probabilistic ModelProbabilistic Formula YesNo Probability Discrete time Markov chains Continuous time Markov chains automata transitions are probabilistic timing for CTMC: Prob(wait time ≤ t) = 1 – e – t

6. Synchronous Leader Election nodes in a ring elect a leader –each node selects random number as id –passes it around the ring (synchronously) –if  unique id, node with maximum unique id is leader [Itai & Rodeh 1990]

7. Synchronous Leader Election 1 3 2 1 5 5 4 2

8. 5 4 1 2 2 3 5 14 1 2 2 3 5 1 51 2 2 3 5 1 5 4 1 3 2 1 5 5 4 2

9. Models Discrete time Markov chain –transitions are fully probabilistic –timing is irrelevant Continuous time Markov chain –transitions are fully probabilistic –and timing also: Prob(wait time ≤ t) = 1 – e – t

10. Probabilistic Model Checker Probabilistic Formulas Probabilistic SystemProbabilistic Requirement Probabilistic ModelProbabilistic Formula YesNo Probability Reachability Bounded Reachability Steady-State Property extensions of CTL

11. Probabilistic Model Checker Probabilistic Model Checkers Probabilistic SystemProbabilistic Requirement Probabilistic ModelProbabilistic Formula YesNo Probability Choices made Three examples Overall evaluation

12. Tools ETMC C numerical CTMCJava MRMCDTMC + CTMCC PRISM hybrid DTMC + CTMC MTBDD for transition matrix C(++ ) and Java PRISM sparse DTMC + CTMC sparse transition matrix VESTA statist CTMC, reachabilityJava YMERCTMC, bounded reachC(++ )

13. Modelling PRISM model.tra format model PRISM YMER model ETMCCMRMCYMERVESTA model adapt syntax informal description

14. Selected Benchmarks Synchronous Leader Electiondiscrete time Randomized Dining Philosophersdiscrete time Birth–Death Processdiscrete time Tandem Queuing Networkcontinuous time Cyclic Server Pollingcontinuous time

15. Experiment Relevance Repeatable Verifiable Significant Encapsulated

16. Experiment 1 Reachability Cyclic Polling Server: server cycles over n stations and serves each one in turn –e.g. teacher walks through class, each pupil may ask a question busy 1  P ≥1 (true U poll 1 ) If station 1 is busy, the server will poll it eventually

18. Analysis ETMCCslow, out of memory PRISMonly symbolic  sparse=hybrid MRMCfastest tool for small models VESTAexcessive number of samples, slow YMERnot implemented

19. PRISM: MTBDD Size Multi-Terminal BDD = data structure for transition matrix size heavily depends on model large MTBDD  slow Model# states# MTBDD nodes Synchronous leader election 500.0001.000.000. Cyclic polling7.000.000< 3.000.

20. CPS versus SLE runtime 7.077.888 states 2.745 MTBDD nodes 458.847 states 1.131.806 MTBDD nodes

21. VESTA: simulation problem actual probability close to bound P ≥p (...) estimate is almost always in [p– ,p+  ] some irregularity stops the simulation 0.95  Prob(yes  actual Prob≥p)  Prob(actual Prob≥p  yes)

22. depends heavily on MTBDD size depends heavily on MTBDD size depends heavily on MTBDD size Result Overview: Timing ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ –/0 N/A

23. Result Overview: Memory ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ 0/+ N/A MTBDD size varies heavily almost independent from model size

24. Experiment 2 Bounded Reachability Tandem Queueing Network –two queues after each other P <0.01 (true U ≤2 full ) Is the probability that the system gets full in 2 time units small? checkin counter security check

26. Analysis ETMCCslow, out of memory PRISMsparse=faster, hybrid=smaller MRMCfast for small models VESTA ok if you can afford statistical errors YMER best choice if you can afford statistical errors

27. Result Overview: Timing ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ –/0 N/A bounded U –+0/++/+++++

28. Result Overview: Memory ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ 0/+ N/A bounded U –++/++++++

29. Experiment 3 Steady State Property Tandem Queuing Network S >0.2 ( P >0.1 (X 2nd queue full ) ) In equilibrium, the probability to satisfy is > 0.2 P >0.1 (X 2nd queue full ) P >0.1 (X... )

31. Analysis ETMCCslow, out of memory PRISM sparse=faster hybrid=slightly smaller MRMCfastest VESTAnot implemented YMERnot implemented

32. Simulating Steady State? simulation of bounded reachability has clear stopping criterion simulation of unbounded reachability  reachability with very large bound simulation of steady state?  never stops

33. Result Overview: Timing ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ –/0 N/A bounded U –+0/++/+++++ steady state –++0/++ N/A

34. Result Overview: Memory ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ 0/+ N/A bounded U –++/++++++ steady state –++/+++ N/A

35. Nested Formulas we also checked nested properties P ≥0.8 (P ≥0.9 (true U ≤100 n 70 ) U n 50 ) not detailed here

36. Result Overview: Timing ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ –/0 N/A bounded U –+0/++/+++++ steady state –++0/++ N/A nested –++0/++– –– – N/A based on a single property only: did not terminate

37. Result Overview: Memory ETMCCMRMCPRISM hybrid PRISM sparse VESTAYMER unb’nded U –++/++ 0/+ N/A bounded U –++/++++++ steady state –++/+++ N/A nested –++/+++ N/A

38. Conclusions ETMCCworst, only small models MRMCfastest for small models PRISM hybrid fast if MTBDD is small PRISM sparse fast VESTArather slow, statistical errors YMER slim & very fast, only bounded reach, few statistical errors