Presentation is loading. Please wait.

Presentation is loading. Please wait.

Ubiquity Generator (UG) Framework: Introduction

Similar presentations


Presentation on theme: "Ubiquity Generator (UG) Framework: Introduction"— Presentation transcript:

1 Ubiquity Generator (UG) Framework: Introduction
Yuji Shinano Zuse Institute Berlin Firste intenational UG workshop 2019, Berlin, Germany

2 Firste intenational UG workshop 2019, Berlin, Germany
Outline What is Ubiquity Generator (UG) Framework Brief history of the development Computational behavior of state-of-the-art Branch-and-bound based solvers Example: Mixed Integer Linear Program (MILP) solver Anomalies of Parallel Branch-and-Bound UG and how UG does parallel tree search Ramp-up mechanism Dynamic load balancing Checkpoint and restart mechanism Deterministic mode implementation What is ug[SCIP, *] libraries UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

3 Firste intenational UG workshop 2019, Berlin, Germany
Outline What is Ubiquity Generator (UG) Framework Brief history of the development Computational behavior of state-of-the-art Branch-and-bound based solvers Example: Mixed Integer Linear Program (MILP) solver Anomalies of Parallel Branch-and-Bound UG and how UG does parallel tree search Ramp-up mechanism Dynamic load balancing Checkpoint and restart mechanism Deterministic mode implementation What is ug[SCIP, *] libraries UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

4 What is Ubiquity Generator Framework
A generic framework to parallelize branch-and-bound based solvers Exploits powerful performance of state-of-the-art base solvers The Free Lunch is Over: A Fundamental Turn Toward Concurrency in Software by Herb Sutter, 2005 State-of-the-art (LP based) branch-and-bound solvers (base solvers) Mixed Integer Program solvers: SCIP, Xpress, Gurobi, CPLEX Steiner Tree Problem solver: SCIP-Jack Traveling Salesman Problem solver: Concorde : Algorithmic improvements >> Parallelization Development of massively parallel branch-and-bound based solvers can solve instances that cannot be solved by state-of-the-art solvers keeps catching up performance improvements state-of-the-art solvers Purpose Parallel solving with mathematically supercharged algorithms Firste intenational UG workshop 2019, Berlin, Germany

5 Brief history of the UG framework development
~mid 90s A parallel branch-and-bound was realized for a specific problem and on a specific parallel computer mid 90s~ Standardization of the message passing interface PVM(Parallel Virtual Machine), MPI(Message Passing Interface) Parallel branch-and-bound software frameworks PPBB-Lib, BOB, ZRAM, PUBB, …, COIN-OR projects(open source) B&B tree data structure is managed by the framework B&B solvers need to develop with the framework from scratch Firste intenational UG workshop 2019, Berlin, Germany

6 Brief history of the UG framework development
2003~: CPLEX solver parallelization on distributed memory computing environments with PUBB2 Performance was good in terms of solving hard instances Too tricky design for the parallelization 2007: ParaLEX: a specialized implementation for CPLEX Firste intenational UG workshop 2019, Berlin, Germany

7 Brief history of the UG framework development
2008: Redesigned ParaLEX 2009: Generalize ParaLEX to a software framework and use it to develop ParaSCIP CPLEX parallelization was planning Framework name: PUBB-ZIB PUBB: Parallelization Utility for Branch-and-Bound Need to corporate with state-of-the-art solver developers! Rename Ubiquity Generator (UG) Framework Firste intenational UG workshop 2019, Berlin, Germany

8 Firste intenational UG workshop 2019, Berlin, Germany
Outline What is Ubiquity Generator (UG) Framework Brief history of the development Computational behavior of state-of-the-art Branch-and-bound based solvers Example: Mixed Integer Linear Program (MILP) solver Anomalies of Parallel Branch-and-Bound UG and how UG does parallel tree search Ramp-up mechanism Dynamic load balancing Checkpoint and restart mechanism Deterministic mode implementation What is ug[SCIP, *] libraries UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

9 Firste intenational UG workshop 2019, Berlin, Germany
Computational behavior of state-of-the-art Branch-and-bound based solvers: An MILP solver as an example Mixed Integer Linear Programs (MILP) Preprocessing (presolving) Reformulate the original problem instance Original instance Presolved instance A: Original A’: Presolved Firste intenational UG workshop 2019, Berlin, Germany

10 Firste intenational UG workshop 2019, Berlin, Germany
Computational behavior of state-of-the-art Branch-and-bound based solvers: An MILP solver as an example Root node computation takes much longer time Root node: all procedures before the first branch preprocessing, adding cutting planes, … root comp. time = 5.4 (sec), (comp. time)/node=28.88/770≒0.375(sec) Firste intenational UG workshop 2019, Berlin, Germany

11 Firste intenational UG workshop 2019, Berlin, Germany
Computational behavior of state-of-the-art Branch-and-bound based solvers: An MILP solver as an example B&B tree can be highly unbalanced Firste intenational UG workshop 2019, Berlin, Germany

12 Firste intenational UG workshop 2019, Berlin, Germany
Computational behavior of state-of-the-art Branch-and-bound based solvers: An MILP solver as an example Performance variability Changes in performance measures for the same problem that are caused by seemingly performance-neutral changes in the environment or the input format The best ten permutations need less than 3 min, but nearly 20% hit the time limit of 10 h. Thorsten Koch, Tobias Achterberg, Erling Andersen, Oliver Bastert, Timo Berthold, Robert E. Bixby, Emilie Danna, Gerald Gamrath, Ambros M. Gleixner, Stefan Heinz, Andrea Lodi, Hans Mittelmann, Ted Ralphs, Domenico Salvagnin, Daniel E. Steffy, Kati Wolter, MIPLIB2010, Mathematical Programming Computation, 3:2 (2011) Firste intenational UG workshop 2019, Berlin, Germany

13 Solving techniques involved in SCIP
Firste intenational UG workshop 2019, Berlin, Germany

14 SCIP (solving constraint integer programs)
- plugin structure - over 800, lines of C code Firste intenational UG workshop 2019, Berlin, Germany

15 Firste intenational UG workshop 2019, Berlin, Germany
Outline What is Ubiquity Generator (UG) Framework Brief history of the development Computational behavior of state-of-the-art Branch-and-bound based solvers Example: Mixed Integer Linear Program (MILP) solver Anomalies of Parallel Branch-and-Bound UG and how UG does parallel tree search Ramp-up mechanism Dynamic load balancing Checkpoint and restart mechanism Deterministic mode implementation What is ug[SCIP, *] libraries UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

16 Anomalies in parallel branch-and-bound
Speedup anomaly Sequential 15 9 5 6 5 9 9 6 9 9 10 Objective function value 9 13 9 11 9 5 Sequential Parallel 13 12 12 11 10 10 1 2 3 4 5 6 7 8 9 10 Time generated processing To simplify, assume that all nodes are solved by a unit of time Incumbent Pruned Firste intenational UG workshop 2019, Berlin, Germany

17 Anomalies in parallel branch-and-bound
Speedup anomaly faster more than 2 times Sequential Parallel: solve 2 nodes at the same time 5 10 15 9 5 6 5 9 6 9 Objective function value 9 9 Sequential Parallel 1 2 3 4 5 6 7 8 9 10 Time generated processing To simplify, assume that all nodes are solved by a unit of time Incumbent Pruned Firste intenational UG workshop 2019, Berlin, Germany

18 Anomalies in parallel branch-and-bound
Detrimental anomaly Sequential 15 5 8 10 9 5 8 9 10 9 9 8 10 Objective function value 5 Sequential Parallel 1 2 3 4 5 6 7 8 9 10 Time generated processing To simplify, assume that all nodes are solved by a unit of time Incumbent Pruned Firste intenational UG workshop 2019, Berlin, Germany

19 Anomalies in parallel branch-and-bound
Detrimental anomaly Sequential Parallel: solve 2 nodes at the same time Slow down in parallel 15 5 6 9 10 5 6 9 6 9 10 11 14 15 12 13 10 Objective function value 10 9 10 9 10 11 14 15 12 13 5 Sequential Parallel 14 15 12 13 12 13 1 2 3 4 5 6 7 8 9 10 Time generated processing To simplify, assume that all nodes are solved by a unit of time Incumbent Pruned Firste intenational UG workshop 2019, Berlin, Germany

20 Firste intenational UG workshop 2019, Berlin, Germany
Outline What is Ubiquity Generator (UG) Framework Brief history of the development Computational behavior of state-of-the-art Branch-and-bound based solvers Example: Mixed Integer Linear Program (MILP) solver Anomalies of Parallel Branch-and-Bound UG and how UG does parallel tree search Ramp-up mechanism Dynamic load balancing Checkpoint and restart mechanism Deterministic mode implementation What is ug[SCIP, *] libraries UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

21 UG Ubiquity Generator Framework - 1
Existing (sequential) solvers can be used Subproblem data need to be extracted from the solver by API branching variables, branching constraints, local cuts, variable statistics, branching history, .. Software Framework: A set of C++ base classes Goal: To identify the necessary APIs that are needed in base solvers to be parallelized in a way that UG provides Naming of instantiated solver by UG ug[base solver name, parallelization library name] Design policy: Simple, small and intuitive interfaces with enough flexibility Firste intenational UG workshop 2019, Berlin, Germany

22 UG Ubiquity Generator Framework - 2
Abstractions of base solvers and parallelization libraries For Xpress For SCIP scipInitiator scipSolver scipInstance scipDiffSubproblem scipSolution Subproblem Data Parallelization framework for B&B (A load balancing mechanism, ramp-ups, checkpoint & restart) Initiator Solver Instance DiffSubproblem Solution ParaInitiator ParaSolver ParaInstance ParaSolution Communicator For MPI World CommMpiWorld For MPI Intercomm, Pthreads, GRID, P2P, Cloud computing etc. Firste intenational UG workshop 2019, Berlin, Germany

23 UG Ubiquity Generator Framework - 3
UG framework LoadCorrdinator Loads are coordinated by a special process or thread Base solver I/O , presolve : Initiator Base solver Base solver Base solver (UG) Solver Using API to control solving algorithms Using MPI or C++11 for communications shared memory ug[SCIP, C++11]: FiberSCIP ug[Xpress, C++11]: FiberXpress Parallel Solver Instantiation distributed memory ug[SCIP, MPI]: ParaSCIP ug[Xpress, MPI]: ParaXpress External parallelization Run on PC Run on PC clusters and supercomputers Firste intenational UG workshop 2019, Berlin, Germany

24 How UG initializes all solvers
[Initialization] Original LoadCoordinator waiting: running: Base solver I/O , presolve P A: Original (sub-)problem A’: Presolved Solver 1 Solver 2 Solver 3 Solver 4 presolved instance Presolved instance is distributed Solver n All Solvers keep the presolved instance Firste intenational UG workshop 2019, Berlin, Germany

25 How UG does ramp-up (start with a single tree search)
[Normal Ramp-up] Original LoadCoordinator waiting: running: Base solver I/O , presolve Solver 1 presolved instance Solver 2 Solver 3 Solver 4 Presolve again with added bound changes Solver n This procedure last until all solvers become busy Firste intenational UG workshop 2019, Berlin, Germany

26 How UG does ramp-up (start with a single tree search)
[Normal Ramp-up] Original LoadCoordinator waiting: running: Base solver I/O , presolve All feasible solutions need to be converted back for the original instance Solver 1 presolved instance Solver 2 Solver 3 Solver 4 All transfer data need to be converted back for the presolved instance Presolve again with added bound changes Solver n Firste intenational UG workshop 2019, Berlin, Germany

27 How UG does ramp-up (start with multiple trees search)
[Ramp-up(Racing)] Original LoadCoordinator waiting: running: Base solver I/O , presolve Winner Solver 1 Solver 2 Solver 3 Solver 4 Solver n All Solvers start solving immediately, trying to generate different search trees ug[SCIP,*]: work with distributed domain propagation Firste intenational UG workshop 2019, Berlin, Germany

28 How UG does ramp-up (start with multiple trees search)
[Ramp-up(Racing)] Original LoadCoordinator waiting: running: Base solver I/O , presolve Winner Solver 1 Solver 2 Solver 3 Solver 4 All nodes are not always collected ot LC Solver n When the racing terminated without enough open nodes, automatically the ramp-up is continued with normal ramp-up Firste intenational UG workshop 2019, Berlin, Germany

29 Parameters to control racing termination
General parameters (red : default value) NEvaluationSolversToStopRacing -1 stops at all of the solvers, 0 stop at half of the solvers, n > 0 RacingRampUpTerminationCriteria 0 - stop at the number of nodes left reached 1 - stop at time limit, 2: - stop at the Solver with the best dual bound value has a certain number  of nodes 3 - adaptive(node first) 4 - adaptive (time first) StopRacingNumberOfNodesLeft (300) StopRacingTimeLimit (720 sec.) NMaxRacingBaseParameters (in case RacingParamsDirPath is specified) ug[SCIP,*] oriented parameters CommunicateTighterBoundsInRacing (true) : default racing parameters Combination of meta-parameters + random seeds Firste intenational UG workshop 2019, Berlin, Germany

30 Firste intenational UG workshop 2019, Berlin, Germany
How UG does dynamic load balancing during primal single tree search in parallel p LoadCoordinator Open nodes: Solver 1 Solver 2 Solver 3 Solver 4 Solver 5 Solver n Try to keep p open nodes in LoadCoordinator Notification message: best lower bound, # nodes remain, # nodes solved - Send periodically and asynchronously - Interval is specified by a parameter LoadCoordinator makes selected Solvers in collecting mode Expected to have heavy nodes: large subtree underneath Firste intenational UG workshop 2019, Berlin, Germany

31 Solver which has best lower bound node
How UG does dynamic load balancing during primal single tree search in parallel p*mp p p LoadCoordinator Open nodes: Solver 1 Solver 2 Solver 3 Solver 4 Solver 5 Solver n Collecting mode Solver Changes search strategy to best dual bound first Sends requested number of nodes Solver which has best lower bound node Firste intenational UG workshop 2019, Berlin, Germany

32 Solver which has best lower bound node
How UG does dynamic load balancing during primal single tree search in parallel p LoadCoordinator Open nodes: Solver 1 Solver 2 Solver 3 Solver 4 Solver 5 Solver n Collecting mode Solver The # of solvers at a time is restricted Starts from 1 Dynamically switching The number is increased by at most even if run with 80,000 Solvers Solver which has best lower bound node Firste intenational UG workshop 2019, Berlin, Germany

33 Parameters to control dynamic load balancing
LoadCoordinator p p`*mp NotificationInterval (1.0 sec) NChangeIntoCollectingMode (1) MultiplierForCollectingMode (2.0) MinNumberOfCollectingModeSolvers (1) MaxNumberOfCollectingModeSolvers(-1: nulimited) TimeToIncreaseCMS(10.0 sec.) CMS: Collecting Mode solvers If node pool is empty more than this time, increase # of CMS CollectingModeInteraval (10.0 sec.) Interval time between collecting modes less than this time, increase the number (+1) : NOTE: Nodes are collected only from solvers which have heavy nodes Solver dose not always send nodes (in case the root takes too long time) Open nodes: p mp p Firste intenational UG workshop 2019, Berlin, Germany

34 What does the dynamic load balancing do
Searches hardest part of sub-trees Assign more computing resources to the sub-trees Apply more presolving and cutting planes to sub-MIPs in the sub-tree. Firste intenational UG workshop 2019, Berlin, Germany

35 Firste intenational UG workshop 2019, Berlin, Germany
Layered presolving Original A: Original (sub-)problem A’: Presolved A’: Original (sub-)problem A’’: Presolved UG causes algorithmic changes to the base solver Firste intenational UG workshop 2019, Berlin, Germany

36 Firste intenational UG workshop 2019, Berlin, Germany
Check pointing Only the essential nodes are saved depending on run-time situation Parameters for checkpoint Checkpoint (FALSE) CheckpointInterval(3600 sec.) Usually, 1800 sec. is used Firste intenational UG workshop 2019, Berlin, Germany

37 Firste intenational UG workshop 2019, Berlin, Germany
Restarting Only the essential nodes are saved depending on run-time situation Huge trees might be thrown away, but the saved nodes’ lower bound values are calculated more precisely. Firste intenational UG workshop 2019, Berlin, Germany

38 Implementation of deterministic mode
LoadCoordinator Base solver I/O , presolve Solver 1 presolved instance Solver 2 presolved instance Solver 3 presolved instance Solver 4 presolved instance All communications are done between LoadCoordinator and Solvers => The messages between them are serialized => Ordering the messages so that they are always performed the same order Firste intenational UG workshop 2019, Berlin, Germany

39 Implementation of deterministic mode
LoadCoordinator Base solver I/O , presolve Solver 1 presolved instance Solver 2 presolved instance Solver 3 presolved instance Solver 4 presolved instance Token Tools: Deterministic timer (a counter) and a token Circulate the token to all Solvers via LoadCoordinator When a Solver reached to a communication interval time in the deterministic time, only the Solver that has token can starts communication, the others wait for the token Firste intenational UG workshop 2019, Berlin, Germany

40 Instantiated parallel solvers by UG
Single thread base solver ParaSCIP: ug[SCIP, MPI], FiberSCIP: ug[SCIP, Pthreads/C++11] Multi-threaded base solver ParaXpress: ug[Xpress, MPI], FiberXpress: ug[Xpress, Pthreads/C++11] 80,000 (MPI processes) x 24 (threads) = 1,920,000 cores ParaNUOPT: ug[NUOPT, MPI], FiberNUOPT: ug[NUOPT, Pthreads/C++11] Only tested with single thread NUOPT Distributed base solver ug[PIPS-SBB, MPI] PIPS-SBB: a distributed memory solver for Stochastic MIP 80,000 (MPI processes) x 100 (MPI processes) = 8,000,000 cores Concorde: TSP solver ug[Concorde, MPI], ug[Concorde, Pthreds/C++11] A new subproblem transfer mechanism via file A new check-pointing and restarting mechanism With Concorde inside parallelization Firste intenational UG workshop 2019, Berlin, Germany

41 What is ug[SCIP,*] libraries
SCIP (Solving Constraint Integer Programs) A software framework for LP based B&B algorithms plugin based design ug[SCIP,*] libraries allow SCIP users to parallelize their own customized SCIP solvers with minimal effort. ug[SCIP, C++11/pthreads](=FiberSCIP) library: A software library to make a customized SCIP solver run on a shared memory environment in parallel libugscip linux.x86 64.gnu.opt.cpp11.a libugscip linux.x86 64.gnu.opt.pth.a ug[SCIP, MPI](=ParaSCIP) library: A software library to make a customized SCIP solver run on a distributed memory environment in parallel libugscip linux.x86 64.gnu.opt.mpi.a User plugins are installed appropriately to FiberSCIP or ParaSCIP automatically. ug[SCIP,*] libraries are general purpose parallel B&B libraries. Firste intenational UG workshop 2019, Berlin, Germany

42 Firste intenational UG workshop 2019, Berlin, Germany
Outline What is Ubiquity Generator (UG) Framework Brief history of the development Computational behavior of state-of-the-art Branch-and-bound based solvers Example: Mixed Integer Linear Program (MILP) solver Anomalies of Parallel Branch-and-Bound UG and how UG does parallel tree search Ramp-up mechanism Dynamic load balancing Checkpoint and restart mechanism Deterministic mode implementation What is ug[SCIP, *] libraries UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

43 User needs to add glue code
LoadCoordinator SCIP solver I/O , presolve FiberSCIP and ParaSCIP as libraries - Use “ScipUserPlugin class” SCIP solver Solver 1 class StpUserPlugins : public ScipUserPlugins { void operator()(SCIP *scip) {       /* include stp pricer */       // SCIP_CALL_ABORT( SCIPincludePricerStp(scip) );       /* include steiner tree reader */       SCIP_CALL_ABORT( SCIPincludeReaderStp(scip) );       /* include event handler for printing primal solution development */       SCIP_CALL_ABORT( SCIPincludeEventHdlrBestsol(scip) );       /* include branching rule */       SCIP_CALL_ABORT( SCIPincludeBranchruleStp(scip) );       /* include propagator */       SCIP_CALL_ABORT( SCIPincludePropStp(scip) ); : } }; Copy from cmain.c Firste intenational UG workshop 2019, Berlin, Germany

44 User can control algorithms flexibly
LoadCoordinator SCIP solver I/O , presolve presolving settings SCIP solver Solver 1 Each solver settings Global root solving settings Three independent parameter sets can be specified Firste intenational UG workshop 2019, Berlin, Germany

45 Built parallel solvers with ug[SCIP,*] libraries
ug[SCIP-Jack,*] Parallel solvers for solving Steiner Tree Problems ug[SCIP-SDP,*] Parallel solvers for solving Mixed Integer Semidefinite Programs New features (UG-0.8.6) The availability of C++11 threads for communication, customized racing ramp-up, which allows to use a set of user defined parameters for racing ramp-up, branching on constraints for ug[SCIP,*], user interfaces to initialize subproblems for ug[SCIP,*] more aggressive layered presolving Firste intenational UG workshop 2019, Berlin, Germany

46 The glue code would be small
Firste intenational UG workshop 2019, Berlin, Germany

47 Firste intenational UG workshop 2019, Berlin, Germany
Outline What is Ubiquity Generator (UG) Framework Brief history of the development Computational behavior of state-of-the-art Branch-and-bound based solvers Example: Mixed Integer Linear Program (MILP) solver Anomalies of Parallel Branch-and-Bound UG and how UG does parallel tree search Ramp-up mechanism Dynamic load balancing Checkpoint and restart mechanism Deterministic mode implementation What is ug[SCIP, *] libraries UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

48 Firste intenational UG workshop 2019, Berlin, Germany
UG Synthesizer (UGS) Run different solvers with different configurations in parallel as MPMD(Multiple Program, Multiple Data) model MPI program UGS ugs Xpress config1 ugs Xpress config2 ugs Gurobi config1 ugs CPLEX config1 ugs ParaXpress config1 ugs PAC_Xpress config1 Mediate solution sharing : B&B solvers can be distributed memory solvers : Heuristic solvers mpirun -np 1 ../bin/ugs test-run -ugsc ../config/test.cfg : -np 1 ../bin/ugs_gurobi1_1 ../IP/miplib2003/ds.mps.gz ugsc ../config/test.cfg -param grb1.prm : -np 1 ../bin/ugs_gurobi2_1 ../IP/miplib2003/ds.mps.gz ugsc ../config/test.cfg -param grb2.prm : np 1 ../bin/ugs_xpress1_1 ../IP/miplib2003/ds.mps.gz ugsc ../config/test.cfg -param xpress1.prm : -np /bin/paraxpress1_ /settings/ds.set /IP/miplib2003/ds.mps.gz –ugsc ../config/test.config -s paraxpress.set –sr paraxpress.set –sl paraxpress.set Firste intenational UG workshop 2019, Berlin, Germany

49 Firste intenational UG workshop 2019, Berlin, Germany
Concluding remarks Computational results for each parallel solver are presented during this workshop Firste intenational UG workshop 2019, Berlin, Germany


Download ppt "Ubiquity Generator (UG) Framework: Introduction"

Similar presentations


Ads by Google