1. Parallelizing Incremental Bayesian Segmentation (IBS) Joseph Hastings Sid Sen

2. Outline Background on IBS Code Overview Parallelization Methods (Cilk, MPI) Cilk Version MPI Version Summary of Results Final Comments

3. Background on IBS

4. IBS Incremental Bayesian Segmentation [1] is an on-line machine learning algorithm designed to segment time-series data into a set of distinct clusters It models the time-series as the concatenation of processes, each generated by a distinct Markov chain, and attempts to find the most-likely break points between the processes

5. Training Process During the training phase of the algorithm, IBS builds a set of Markov matrices that it believes are most likely to describe the set of processes responsible for generating the time series

6. Code Overview

7. Hi-Level Control Flow main() loops through input file Runs break-point detection Foreach segment: check_out_process() Foreach existing matrix compute_subsumed_marginal_likelihood() Adds segment to set of matrices or subsumes

8. Parallelizable Computation compute_subsumed_marginal_likelihood() Depends on a single matrix and the new segment Produces a single score The index of the best score must be calculated

9. Code Status Original code (Java, LISP, Perl) C++ code C code

10. Parallelization Methods

11. MPI Library facilitating inter-process communication Provides useful communication routines, particularly MPI_Allreduce, which simultaneously reduces data on all nodes and broadcasts the result

12. Cilk Originally developed by the Supercomputing Technologies Group at the MIT Laboratory for Computer Science Cilk is a language for multithreaded parallel programming based on ANSI C that is very effective for exploiting highly asynchronous parallelism [3] (which can be difficult to write using message-passing interfaces like MPI)

13. Cilk Specify number of worker threads or “processors” to create when running a Cilk job No one-to-one mapping of worker threads to processors, hence the quotes Work-stealing algorithm When a processor runs out of work, asks another processor chosen at random for work to do Cilk’s work-stealing scheduler executes any Cilk computation in nearly optimal time Computation on P processors executed in time T p ≤ T 1 /P + O(T  )

14. Code Status Original code (Java, LISP, Perl) C++ code C code MPICilk

15. Cilk Version

16. Code Modifications Keywords: cilk, spawn, sync Convert any methods that will be spawned or that will spawn other (Cilk) methods into Cilk methods In our case: main(), check_out_process(), compute_subsumed_marginal_likelihood() Main source of parallelism comes from subsuming current process with each existing process and choosing subsumption with the best score spawn compute_subsumed_marginal_likelihood(proc, get(processes,i), copy_process_list(processes));

17. Code Modifications When updating global_score need to enforce mutual exclusion between worker threads Cilk_lockvar score_lock;... Cilk_lock(score_lock);... Cilk_unlock(score_lock);

18. Cilk Results Optimal performance achieved using 2 processors (trade-off between overhead of Cilk and parallelism of program)

19. Adaptive Parallelism Real intelligence is in the Cilk runtime system, which handles load balancing, paging, and communication protocols between running worker threads Currently have to specify the number of processors to run a Cilk job on Goal is to eventually make the runtime system adaptively parallel by intelligently determining how many threads/processors to use Fair and efficient allocation among all running Cilk jobs Cilk Macroscheduler [4] uses steal rate of worker thread as a measure of its processor desire (if a Cilk job spends a substantial amount of its time stealing, then the job has more processors than it desires)

20. MPI Version

21. Code Modifications check_out_process() first broadcasts the segment using MPI_Bcast() Each process loops over all matrices, but only performs subsumption if (I % np == rank) Each process computes best score, and MPI_Allreduce() is used to reduce this information to the globally best score Each process learns the index of the best matrix and performs the identical subsumption

22. MPI Results Big improvement from 1 to 2 processors; levels off for 3 or more

23. Summary of Results

24. MPI vs. Cilk

25. Final Comments

26. MPI vs. Cilk MPI version much more complicated, involved more lines of code, and much more difficult to debug Cilk version required thinking about mutual-exclusion, which MPI avoids Cilk version required few code changes, but conceptually more complicated to think about

27. References (Presentation) [1] Paola Sebastiani and Marco Ramoni. Incremental Bayesian Segmentation of Categorical Temporal Data. 2000. [2] Wenke Lee and Salvatore J. Stolfo. Data Mining Approaches for Intrusion Detection. 1998. [3] Cilk 5.3.2 Reference Manual. Supercomputing Technologies Group, MIT Lab for Computer Science. November 9, 2001. Available online: http://supertech.lcs.mit.edu/manual-5.3.2.pdf. http://supertech.lcs.mit.edu/manual-5.3.2.pdf [4] R. D. Blumofe, C. E. Leiserson, B. Song. Automatic Processor Allocation for Work-Stealing Jobs. (Work in progress)