Distributed Process Discovery From Large Event Logs Sergio Hernández de Mesa { Eindhoven, The Netherlands 12th March, 2015

Distributed Process Discovery From Large Event Logs Distributed Process Discovery A Framework for Distributed Computing Summary and Future Work Outline Sergio Hernández de Mesa 12th March,

Distributed Process Discovery From Large Event Logs Distributed Process Discovery A Framework for Distributed Computing Summary and Future Work Outline Sergio Hernández de Mesa 12th March,

Distributed Process Discovery From Large Event Logs Distributed Process Discovery Sergio Hernández de Mesa 12th March, 2015 Big Data 4

Distributed Process Discovery From Large Event Logs Distributed Process Discovery Sergio Hernández de Mesa 12th March, 2015 The 3 V’s of Big Data 5

Distributed Process Discovery From Large Event Logs Distributed Process Discovery Sergio Hernández de Mesa 12th March, 2015 Big Data and process discovery 6 XES logs CSV files Offline analysis Real-time analysis Data streams TB GB MB

Distributed Process Discovery From Large Event Logs Distributed Process Discovery Sergio Hernández de Mesa 12th March, 2015 Actual problem 7

Distributed Process Discovery From Large Event Logs Distribute/Parallelize process discovery techniques - Inductive Miner - Alpha Miner - Heuristics Miner - … Take advantage of HPC infrastructures and parallel programming models - Clusters, grids and clouds - MapReduce Distributed Process Discovery Performance improvement opportunities Sergio Hernández de Mesa 12th March,

Distributed Process Discovery From Large Event Logs No computing resources No computing resources but money Hadoop Cluster HPC infrastructure Sergio Hernández de Mesa 12th March, “Classical” ProM Amazon Elastic MapReduce MapReduce model Distributed approach Distributed Process Discovery Execution scenarios

Distributed Process Discovery From Large Event Logs MapReduce - Programming model for data-oriented applications - Proposed by Google - Map: (k 1, v 1 )  list (k 2,v 2 ) - Reduce: (k 2, list(v 2 ) )  list (v 3 ) Hadoop - Software for reliable, scalable and distributed computing - Developed by Apache - Core components: Hadoop Distributed File System (HDFS) Hadoop MapReduce Hadoop YARN Sergio Hernández de Mesa 12th March, Distributed Process Discovery MapReduce and Hadoop

Distributed Process Discovery From Large Event Logs Distributed Process Discovery Sergio Hernández de Mesa 12th March, 2015 Step 1 Directly-Follows Graph DFG Process Model XES log XES log Optimized version of Step 1 – Reading data as a stream (SAXParser) – HashMaps to efficiently count frequencias Example XES log: 100 million traces (40 activities) – Size: 218 GB – Step 1: XES to DFG: ~ 2-3 hours – Step 2: DFG to Process Model: ~ few seconds Step 2 Motivational example: Inductive Miner 11

Distributed Process Discovery From Large Event Logs HDFS (Hadoop Distributed File System) HDFS (Hadoop Distributed File System) HDFS (Hadoop Distributed File System) XES Logs Block 1 Block 2 Block N … ….. ….. … MAP 1 MAP 2 MAP N … … … … … … … … … … … DFG 1 DFG 2 DFG N REDUCEREDUCE REDUCEREDUCE FINAL DFG Split phase Distributed Process Discovery Computing DFG: Hadoop/MapReduce approach …… Sergio Hernández de Mesa 12th March,

Distributed Process Discovery From Large Event Logs XES Logs XES sublog 1 XES sublog 2 XES Sublog N XES Sublog N … XES2DFG (MAP) XES2DFG (MAP) XES2DFG (MAP) XES2DFG (MAP) XES2DFG (MAP) XES2DFG (MAP) … DFG 1 DFG 2 DFG N REDUCE_DFGSREDUCE_DFGS REDUCE_DFGSREDUCE_DFGS FINAL DFG … Distributed Process Discovery Computing DFG: Distributed/HPC approach Sergio Hernández de Mesa 12th March,

Distributed Process Discovery From Large Event Logs Distributed Process Discovery A Framework for Distributed Computing Summary and Future Work Outline Sergio Hernández de Mesa 12th March,

Distributed Process Discovery From Large Event Logs A Framework for Distributed Computing Sergio Hernández de Mesa 12th March, Scientific computing

Distributed Process Discovery From Large Event Logs A Framework for Distributed Computing Sergio Hernández de Mesa 12th March, Heterogeneous Execution Environments

Distributed Process Discovery From Large Event Logs A Framework for Distributed Computing Sergio Hernández de Mesa 12th March, Challenges of scientific computing in HPC Strong coupling between applications and execution environments Lifecycle management Using multiple computing infrastructures

Distributed Process Discovery From Large Event Logs A Framework for Distributed Computing Sergio Hernández de Mesa 12th March, Framework architecture

Distributed Process Discovery From Large Event Logs Amazon EC2 Mediator A Framework for Distributed Computing Sergio Hernández de Mesa 12th March, 2015 HERMES Mediator Message bus HERMES Meta-scheduler Fault Management User application JSDL Message ✓ Selecting a computing infrastructure Job execution ✘ Selecting fault handling policy Resubmission Alternative infrastructure Aborting job execution 19 Framework operation Job execution ✓

Distributed Process Discovery From Large Event Logs Distributed Process Discovery A Framework for Distributed Computing Summary and Future Work Outline Sergio Hernández de Mesa 12th March,

Distributed Process Discovery From Large Event Logs Summary and Future Work Summary Sergio Hernández de Mesa 12th March, Inductive Miner Alpha Miner Heuristics Miner …

Distributed Process Discovery From Large Event Logs New ProM plugin No computing resources No computing resources but money Hadoop Server HPC infrastructure Sergio Hernández de Mesa 12th March, “Classical” ProM Amazon Elastic MapReduce MapReduce model Distributed approach Summary and Future Work Execution scenarios

Distributed Process Discovery From Large Event Logs Summary and Future Work Solution approach Sergio Hernández de Mesa 12th March,

Distributed Process Discovery From Large Event Logs Process discovery from Large Event Logs - “Sequential” way: Time-consuming - Solution approach: MapReduce and Distributed computing Current state - Code developed for distributed computing DFGs - Setting up Hadoop Cluster Future Work - Integration with the distributed computing framework - Development of a ProM plugin Sergio Hernández de Mesa 12th March, Summary and Future Work Conclusions

Distributed Process Discovery From Large Event Logs Sergio Hernández de Mesa { Eindhoven, The Netherlands 12th March, 2015