1. Ubiquity Generator (UG) Framework: Introduction
Yuji Shinano
Zuse Institute Berlin
Firste intenational UG workshop 2019, Berlin, Germany

2. Firste intenational UG workshop 2019, Berlin, Germany
Outline
What is Ubiquity Generator (UG) Framework
Brief history of the development
Computational behavior of state-of-the-art Branch-and-bound based solvers
Example: Mixed Integer Linear Program (MILP) solver
Anomalies of Parallel Branch-and-Bound
UG and how UG does parallel tree search
Ramp-up mechanism
Dynamic load balancing
Checkpoint and restart mechanism
Deterministic mode implementation
What is ug[SCIP, *] libraries
UG synthesizer
Firste intenational UG workshop 2019, Berlin, Germany

3. Firste intenational UG workshop 2019, Berlin, Germany
Outline
What is Ubiquity Generator (UG) Framework
Brief history of the development
Computational behavior of state-of-the-art Branch-and-bound based solvers
Example: Mixed Integer Linear Program (MILP) solver
Anomalies of Parallel Branch-and-Bound
UG and how UG does parallel tree search
Ramp-up mechanism
Dynamic load balancing
Checkpoint and restart mechanism
Deterministic mode implementation
What is ug[SCIP, *] libraries
UG synthesizer
Firste intenational UG workshop 2019, Berlin, Germany

4. What is Ubiquity Generator Framework
A generic framework to parallelize branch-and-bound based solvers
Exploits powerful performance of state-of-the-art base solvers
The Free Lunch is Over: A Fundamental Turn Toward Concurrency in Software by Herb Sutter, 2005
State-of-the-art (LP based) branch-and-bound solvers (base solvers)
Mixed Integer Program solvers: SCIP, Xpress, Gurobi, CPLEX
Steiner Tree Problem solver: SCIP-Jack
Traveling Salesman Problem solver: Concorde :
Algorithmic improvements >> Parallelization
Development of massively parallel branch-and-bound based solvers
can solve instances that cannot be solved by state-of-the-art solvers
keeps catching up performance improvements state-of-the-art solvers
Purpose
Parallel solving with mathematically supercharged algorithms
Firste intenational UG workshop 2019, Berlin, Germany

5. Brief history of the UG framework development
~mid 90s
A parallel branch-and-bound was realized for a specific problem and on a specific parallel computer
mid 90s~
Standardization of the message passing interface
PVM(Parallel Virtual Machine), MPI(Message Passing Interface)
Parallel branch-and-bound software frameworks
PPBB-Lib, BOB, ZRAM, PUBB, …, COIN-OR projects(open source)
B&B tree data structure is managed by the framework
B&B solvers need to develop with the framework from scratch
Firste intenational UG workshop 2019, Berlin, Germany

6. Brief history of the UG framework development
2003~: CPLEX solver parallelization on distributed memory computing environments with PUBB2
Performance was good in terms of solving hard instances
Too tricky design for the parallelization
2007: ParaLEX: a specialized implementation for CPLEX
Firste intenational UG workshop 2019, Berlin, Germany

7. Brief history of the UG framework development
2008: Redesigned ParaLEX
2009: Generalize ParaLEX to a software framework and use it to develop ParaSCIP
CPLEX parallelization was planning
Framework name: PUBB-ZIB
PUBB: Parallelization Utility for Branch-and-Bound
Need to corporate with state-of-the-art solver developers!
Rename
Ubiquity Generator (UG) Framework
Firste intenational UG workshop 2019, Berlin, Germany

8. Firste intenational UG workshop 2019, Berlin, Germany
Outline
What is Ubiquity Generator (UG) Framework
Brief history of the development
Computational behavior of state-of-the-art Branch-and-bound based solvers
Example: Mixed Integer Linear Program (MILP) solver
Anomalies of Parallel Branch-and-Bound
UG and how UG does parallel tree search
Ramp-up mechanism
Dynamic load balancing
Checkpoint and restart mechanism
Deterministic mode implementation
What is ug[SCIP, *] libraries
UG synthesizer
Firste intenational UG workshop 2019, Berlin, Germany

9. Firste intenational UG workshop 2019, Berlin, Germany
Computational behavior of state-of-the-art Branch-and-bound based solvers: An MILP solver as an example
Mixed Integer Linear Programs (MILP)
Preprocessing (presolving)
Reformulate the original problem instance
Original instance
Presolved instance
A: Original
A’: Presolved
Firste intenational UG workshop 2019, Berlin, Germany

10. Firste intenational UG workshop 2019, Berlin, Germany
Computational behavior of state-of-the-art Branch-and-bound based solvers: An MILP solver as an example
Root node computation takes much longer time
Root node: all procedures before the first branch
preprocessing, adding cutting planes, …
root comp. time = 5.4 (sec),
(comp. time)/node=28.88/770≒0.375(sec)
Firste intenational UG workshop 2019, Berlin, Germany

11. Firste intenational UG workshop 2019, Berlin, Germany
Computational behavior of state-of-the-art Branch-and-bound based solvers: An MILP solver as an example
B&B tree can be highly unbalanced
Firste intenational UG workshop 2019, Berlin, Germany

12. Firste intenational UG workshop 2019, Berlin, Germany
Computational behavior of state-of-the-art Branch-and-bound based solvers: An MILP solver as an example
Performance variability
Changes in performance measures for the same problem that are caused by seemingly performance-neutral changes in the environment or the input format
The best ten permutations need less than 3 min, but nearly 20% hit the time limit of 10 h.
Thorsten Koch, Tobias Achterberg, Erling Andersen, Oliver Bastert, Timo Berthold, Robert E. Bixby, Emilie Danna, Gerald Gamrath, Ambros M. Gleixner, Stefan Heinz, Andrea Lodi, Hans Mittelmann, Ted Ralphs, Domenico Salvagnin, Daniel E. Steffy, Kati Wolter, MIPLIB2010, Mathematical Programming Computation, 3:2 (2011)
Firste intenational UG workshop 2019, Berlin, Germany

13. Solving techniques involved in SCIP
Firste intenational UG workshop 2019, Berlin, Germany

14. SCIP (solving constraint integer programs)
- plugin structure
- over 800, lines of C code
Firste intenational UG workshop 2019, Berlin, Germany

15. Firste intenational UG workshop 2019, Berlin, Germany
Outline
What is Ubiquity Generator (UG) Framework
Brief history of the development
Computational behavior of state-of-the-art Branch-and-bound based solvers
Example: Mixed Integer Linear Program (MILP) solver
Anomalies of Parallel Branch-and-Bound
UG and how UG does parallel tree search
Ramp-up mechanism
Dynamic load balancing
Checkpoint and restart mechanism
Deterministic mode implementation
What is ug[SCIP, *] libraries
UG synthesizer
Firste intenational UG workshop 2019, Berlin, Germany

16. Anomalies in parallel branch-and-bound
Speedup anomaly
Sequential 15 9 5 6 5 9 9 6 9 9 10
Objective function value 9 13 9 11 9 5
Sequential
Parallel 13 12 12 11 10 10 1 2 3 4 5 6 7 8 9 10
Time
generated
processing
To simplify, assume that
all nodes are solved by a unit of time
Incumbent
Pruned
Firste intenational UG workshop 2019, Berlin, Germany

17. Anomalies in parallel branch-and-bound
Speedup anomaly
faster more than 2 times
Sequential
Parallel: solve 2 nodes at the same time 5 10 15 9 5 6 5 9 6 9
Objective function value 9 9
Sequential
Parallel 1 2 3 4 5 6 7 8 9 10
Time
generated
processing
To simplify, assume that
all nodes are solved by a unit of time
Incumbent
Pruned
Firste intenational UG workshop 2019, Berlin, Germany

18. Anomalies in parallel branch-and-bound
Detrimental anomaly
Sequential 15 5 8 10 9 5 8 9 10 9 9 8 10
Objective function value 5
Sequential
Parallel 1 2 3 4 5 6 7 8 9 10
Time
generated
processing
To simplify, assume that
all nodes are solved by a unit of time
Incumbent
Pruned
Firste intenational UG workshop 2019, Berlin, Germany

19. Anomalies in parallel branch-and-bound
Detrimental anomaly
Sequential
Parallel: solve 2 nodes at the same time
Slow down
in parallel 15 5 6 9 10 5 6 9 6 9 10 11 14 15 12 13 10
Objective function value 10 9 10 9 10 11 14 15 12 13 5
Sequential
Parallel 14 15 12 13 12 13 1 2 3 4 5 6 7 8 9 10
Time
generated
processing
To simplify, assume that
all nodes are solved by a unit of time
Incumbent
Pruned
Firste intenational UG workshop 2019, Berlin, Germany

20. Firste intenational UG workshop 2019, Berlin, Germany
Outline
What is Ubiquity Generator (UG) Framework
Brief history of the development
Computational behavior of state-of-the-art Branch-and-bound based solvers
Example: Mixed Integer Linear Program (MILP) solver
Anomalies of Parallel Branch-and-Bound
UG and how UG does parallel tree search
Ramp-up mechanism
Dynamic load balancing
Checkpoint and restart mechanism
Deterministic mode implementation
What is ug[SCIP, *] libraries
UG synthesizer
Firste intenational UG workshop 2019, Berlin, Germany

21. UG Ubiquity Generator Framework - 1
Existing (sequential) solvers can be used
Subproblem data need to be extracted from the solver by API
branching variables, branching constraints, local cuts, variable statistics, branching history, ..
Software Framework: A set of C++ base classes
Goal: To identify the necessary APIs that are needed in base solvers to be parallelized in a way that UG provides
Naming of instantiated solver by UG ug[base solver name, parallelization library name]
Design policy: Simple, small and intuitive interfaces with enough flexibility
Firste intenational UG workshop 2019, Berlin, Germany

22. UG Ubiquity Generator Framework - 2
Abstractions of base solvers and parallelization libraries
For Xpress
For SCIP
scipInitiator
scipSolver
scipInstance
scipDiffSubproblem
scipSolution
Subproblem Data
Parallelization framework for B&B
(A load balancing mechanism, ramp-ups, checkpoint & restart)
Initiator
Solver
Instance
DiffSubproblem
Solution
ParaInitiator
ParaSolver
ParaInstance
ParaSolution
Communicator
For MPI World
CommMpiWorld
For MPI Intercomm, Pthreads, GRID, P2P, Cloud computing etc.
Firste intenational UG workshop 2019, Berlin, Germany

23. UG Ubiquity Generator Framework - 3
UG framework
LoadCorrdinator
Loads are coordinated by a special process or thread
Base solver
I/O , presolve
: Initiator
Base solver
Base solver
Base solver
(UG)
Solver
Using API to control
solving algorithms
Using MPI or C++11 for communications
shared memory
ug[SCIP, C++11]: FiberSCIP
ug[Xpress, C++11]: FiberXpress
Parallel
Solver
Instantiation
distributed memory
ug[SCIP, MPI]: ParaSCIP ug[Xpress, MPI]: ParaXpress
External
parallelization
Run on PC
Run on PC clusters and supercomputers
Firste intenational UG workshop 2019, Berlin, Germany

24. How UG initializes all solvers
[Initialization]
Original
LoadCoordinator
waiting:
running:
Base solver
I/O , presolve P
A: Original
(sub-)problem
A’: Presolved
Solver 1
Solver 2
Solver 3
Solver 4
presolved
instance
Presolved instance is distributed
Solver n
All Solvers keep the presolved instance
Firste intenational UG workshop 2019, Berlin, Germany

25. How UG does ramp-up (start with a single tree search)
[Normal Ramp-up]
Original
LoadCoordinator
waiting:
running:
Base solver
I/O , presolve
Solver 1
presolved
instance
Solver 2
Solver 3
Solver 4
Presolve again with added bound changes
Solver n
This procedure last until all solvers become busy
Firste intenational UG workshop 2019, Berlin, Germany

26. How UG does ramp-up (start with a single tree search)
[Normal Ramp-up]
Original
LoadCoordinator
waiting:
running:
Base solver
I/O , presolve
All feasible solutions need to be converted back for the original instance
Solver 1
presolved
instance
Solver 2
Solver 3
Solver 4
All transfer data need to be converted back for the presolved instance
Presolve again with added bound changes
Solver n
Firste intenational UG workshop 2019, Berlin, Germany

27. How UG does ramp-up (start with multiple trees search)
[Ramp-up(Racing)]
Original
LoadCoordinator
waiting:
running:
Base solver
I/O , presolve
Winner
Solver 1
Solver 2
Solver 3
Solver 4
Solver n
All Solvers start solving immediately, trying to generate different search trees
ug[SCIP,*]: work with distributed domain propagation
Firste intenational UG workshop 2019, Berlin, Germany

28. How UG does ramp-up (start with multiple trees search)
[Ramp-up(Racing)]
Original
LoadCoordinator
waiting:
running:
Base solver
I/O , presolve
Winner
Solver 1
Solver 2
Solver 3
Solver 4
All nodes are not always collected ot LC
Solver n
When the racing terminated without enough open nodes,
automatically the ramp-up is continued with normal ramp-up
Firste intenational UG workshop 2019, Berlin, Germany

29. Parameters to control racing termination
General parameters (red : default value)
NEvaluationSolversToStopRacing
-1 stops at all of the solvers, 0 stop at half of the solvers, n > 0
RacingRampUpTerminationCriteria
0 - stop at the number of nodes left reached
1 - stop at time limit,
2: - stop at the Solver with the best dual bound value has a certain number  of nodes
3 - adaptive(node first)
4 - adaptive (time first)
StopRacingNumberOfNodesLeft (300)
StopRacingTimeLimit (720 sec.)
NMaxRacingBaseParameters (in case RacingParamsDirPath is specified)
ug[SCIP,*] oriented parameters
CommunicateTighterBoundsInRacing (true) :
default racing parameters
Combination of meta-parameters + random seeds
Firste intenational UG workshop 2019, Berlin, Germany

30. Firste intenational UG workshop 2019, Berlin, Germany
How UG does dynamic load balancing during primal single tree search in parallel p
LoadCoordinator
Open nodes:
Solver 1
Solver 2
Solver 3
Solver 4
Solver 5
Solver n
Try to keep p open nodes in LoadCoordinator
Notification message: best lower bound, # nodes remain, # nodes solved
- Send periodically and asynchronously
- Interval is specified by a parameter
LoadCoordinator makes selected Solvers in collecting mode
Expected to have heavy nodes:
large subtree underneath
Firste intenational UG workshop 2019, Berlin, Germany

31. Solver which has best lower bound node
How UG does dynamic load balancing during primal single tree search in parallel
p*mp p p
LoadCoordinator
Open nodes:
Solver 1
Solver 2
Solver 3
Solver 4
Solver 5
Solver n
Collecting mode Solver
Changes search strategy to best dual bound first
Sends requested number of nodes
Solver which has best lower bound node
Firste intenational UG workshop 2019, Berlin, Germany

32. Solver which has best lower bound node
How UG does dynamic load balancing during primal single tree search in parallel p
LoadCoordinator
Open nodes:
Solver 1
Solver 2
Solver 3
Solver 4
Solver 5
Solver n
Collecting mode Solver
The # of solvers at a time is restricted
Starts from 1
Dynamically switching
The number is increased by at most even if run with 80,000 Solvers
Solver which has best lower bound node
Firste intenational UG workshop 2019, Berlin, Germany

33. Parameters to control dynamic load balancing
LoadCoordinator p
p`*mp
NotificationInterval (1.0 sec)
NChangeIntoCollectingMode (1)
MultiplierForCollectingMode (2.0)
MinNumberOfCollectingModeSolvers (1)
MaxNumberOfCollectingModeSolvers(-1: nulimited)
TimeToIncreaseCMS(10.0 sec.) CMS: Collecting Mode solvers
If node pool is empty more than this time, increase # of CMS
CollectingModeInteraval (10.0 sec.)
Interval time between collecting modes less than this time, increase the number (+1) :
NOTE:
Nodes are collected only from solvers which have heavy nodes
Solver dose not always send nodes (in case the root takes too long time)
Open nodes: p mp p
Firste intenational UG workshop 2019, Berlin, Germany

34. What does the dynamic load balancing do
Searches hardest part of sub-trees
Assign more computing resources to the sub-trees
Apply more presolving and cutting planes to sub-MIPs in the sub-tree.
Firste intenational UG workshop 2019, Berlin, Germany

35. Firste intenational UG workshop 2019, Berlin, Germany
Layered presolving
Original
A: Original
(sub-)problem
A’: Presolved
A’: Original
(sub-)problem
A’’: Presolved
UG causes algorithmic changes to
the base solver
Firste intenational UG workshop 2019, Berlin, Germany

36. Firste intenational UG workshop 2019, Berlin, Germany
Check pointing
Only the essential nodes are saved
depending on run-time situation
Parameters for checkpoint
Checkpoint (FALSE)
CheckpointInterval(3600 sec.) Usually, 1800 sec. is used
Firste intenational UG workshop 2019, Berlin, Germany

37. Firste intenational UG workshop 2019, Berlin, Germany
Restarting
Only the essential nodes are saved
depending on run-time situation
Huge trees might be thrown away,
but the saved nodes’ lower bound values are calculated more precisely.
Firste intenational UG workshop 2019, Berlin, Germany

38. Implementation of deterministic mode
LoadCoordinator
Base solver
I/O , presolve
Solver 1
presolved
instance
Solver 2
presolved
instance
Solver 3
presolved
instance
Solver 4
presolved
instance
All communications are done between LoadCoordinator and Solvers
=> The messages between them are serialized
=> Ordering the messages so that they are always performed the same order
Firste intenational UG workshop 2019, Berlin, Germany

39. Implementation of deterministic mode
LoadCoordinator
Base solver
I/O , presolve
Solver 1
presolved
instance
Solver 2
presolved
instance
Solver 3
presolved
instance
Solver 4
presolved
instance
Token
Tools: Deterministic timer (a counter) and a token
Circulate the token to all Solvers via LoadCoordinator
When a Solver reached to a communication interval time in the deterministic time, only the Solver that has token can starts communication, the others wait for the token
Firste intenational UG workshop 2019, Berlin, Germany

40. Instantiated parallel solvers by UG
Single thread base solver
ParaSCIP: ug[SCIP, MPI], FiberSCIP: ug[SCIP, Pthreads/C++11]
Multi-threaded base solver
ParaXpress: ug[Xpress, MPI], FiberXpress: ug[Xpress, Pthreads/C++11]
80,000 (MPI processes) x 24 (threads) = 1,920,000 cores
ParaNUOPT: ug[NUOPT, MPI], FiberNUOPT: ug[NUOPT, Pthreads/C++11]
Only tested with single thread NUOPT
Distributed base solver
ug[PIPS-SBB, MPI]
PIPS-SBB: a distributed memory solver for Stochastic MIP
80,000 (MPI processes) x 100 (MPI processes) = 8,000,000 cores
Concorde: TSP solver
ug[Concorde, MPI], ug[Concorde, Pthreds/C++11]
A new subproblem transfer mechanism via file
A new check-pointing and restarting mechanism
With Concorde inside parallelization
Firste intenational UG workshop 2019, Berlin, Germany

41. What is ug[SCIP,*] libraries
SCIP (Solving Constraint Integer Programs)
A software framework for LP based B&B algorithms
plugin based design
ug[SCIP,*] libraries allow SCIP users to parallelize their own customized SCIP solvers with minimal effort.
ug[SCIP, C++11/pthreads](=FiberSCIP) library: A software library to make a customized SCIP solver run on a shared memory environment in parallel
libugscip linux.x86 64.gnu.opt.cpp11.a
libugscip linux.x86 64.gnu.opt.pth.a
ug[SCIP, MPI](=ParaSCIP) library: A software library to make a customized SCIP solver run on a distributed memory environment in parallel
libugscip linux.x86 64.gnu.opt.mpi.a
User plugins are installed appropriately to FiberSCIP or ParaSCIP automatically.
ug[SCIP,*] libraries are general purpose parallel B&B libraries.
Firste intenational UG workshop 2019, Berlin, Germany

42. Firste intenational UG workshop 2019, Berlin, Germany
Outline
What is Ubiquity Generator (UG) Framework
Brief history of the development
Computational behavior of state-of-the-art Branch-and-bound based solvers
Example: Mixed Integer Linear Program (MILP) solver
Anomalies of Parallel Branch-and-Bound
UG and how UG does parallel tree search
Ramp-up mechanism
Dynamic load balancing
Checkpoint and restart mechanism
Deterministic mode implementation
What is ug[SCIP, *] libraries
UG synthesizer
Firste intenational UG workshop 2019, Berlin, Germany

43. User needs to add glue code
LoadCoordinator
SCIP solver
I/O , presolve
FiberSCIP and ParaSCIP as libraries
- Use “ScipUserPlugin class”
SCIP solver
Solver 1
class StpUserPlugins : public ScipUserPlugins {
void operator()(SCIP *scip) {
      /* include stp pricer */
      // SCIP_CALL_ABORT( SCIPincludePricerStp(scip) );
      /* include steiner tree reader */
      SCIP_CALL_ABORT( SCIPincludeReaderStp(scip) );
      /* include event handler for printing primal solution development */
      SCIP_CALL_ABORT( SCIPincludeEventHdlrBestsol(scip) );
      /* include branching rule */
      SCIP_CALL_ABORT( SCIPincludeBranchruleStp(scip) );
      /* include propagator */
      SCIP_CALL_ABORT( SCIPincludePropStp(scip) ); : } };
Copy from
cmain.c
Firste intenational UG workshop 2019, Berlin, Germany

44. User can control algorithms flexibly
LoadCoordinator
SCIP solver
I/O , presolve
presolving settings
SCIP solver
Solver 1
Each solver settings
Global root solving settings
Three independent parameter sets
can be specified
Firste intenational UG workshop 2019, Berlin, Germany

45. Built parallel solvers with ug[SCIP,*] libraries
ug[SCIP-Jack,*]
Parallel solvers for solving Steiner Tree Problems
ug[SCIP-SDP,*]
Parallel solvers for solving Mixed Integer Semidefinite Programs
New features (UG-0.8.6)
The availability of C++11 threads for communication,
customized racing ramp-up, which allows to use a set of user defined parameters for racing ramp-up,
branching on constraints for ug[SCIP,*],
user interfaces to initialize subproblems for ug[SCIP,*]
more aggressive layered presolving
Firste intenational UG workshop 2019, Berlin, Germany

46. The glue code would be small
Firste intenational UG workshop 2019, Berlin, Germany

47. Firste intenational UG workshop 2019, Berlin, Germany
Outline
What is Ubiquity Generator (UG) Framework
Brief history of the development
Computational behavior of state-of-the-art Branch-and-bound based solvers
Example: Mixed Integer Linear Program (MILP) solver
Anomalies of Parallel Branch-and-Bound
UG and how UG does parallel tree search
Ramp-up mechanism
Dynamic load balancing
Checkpoint and restart mechanism
Deterministic mode implementation
What is ug[SCIP, *] libraries
UG synthesizer
Firste intenational UG workshop 2019, Berlin, Germany

48. Firste intenational UG workshop 2019, Berlin, Germany
UG Synthesizer (UGS)
Run different solvers with different configurations in parallel as MPMD(Multiple Program, Multiple Data) model MPI program …
UGS
ugs Xpress
config1
ugs Xpress
config2
ugs Gurobi
config1
ugs CPLEX
config1
ugs ParaXpress
config1
ugs PAC_Xpress
config1
Mediate
solution
sharing
: B&B solvers
can be distributed memory solvers
: Heuristic solvers
mpirun -np 1 ../bin/ugs test-run -ugsc ../config/test.cfg :
-np 1 ../bin/ugs_gurobi1_1 ../IP/miplib2003/ds.mps.gz ugsc ../config/test.cfg -param grb1.prm :
-np 1 ../bin/ugs_gurobi2_1 ../IP/miplib2003/ds.mps.gz ugsc ../config/test.cfg -param grb2.prm : np 1 ../bin/ugs_xpress1_1 ../IP/miplib2003/ds.mps.gz ugsc ../config/test.cfg -param xpress1.prm :
-np /bin/paraxpress1_ /settings/ds.set /IP/miplib2003/ds.mps.gz –ugsc ../config/test.config
-s paraxpress.set –sr paraxpress.set –sl paraxpress.set
Firste intenational UG workshop 2019, Berlin, Germany

49. Firste intenational UG workshop 2019, Berlin, Germany
Concluding remarks
Computational results for each parallel solver are presented during this workshop
Firste intenational UG workshop 2019, Berlin, Germany