1. Introduction to parallel computing in Mathematica
Radoslaw Jedynak, PhD
Poland, Technical University of Radom
Faculty of Teacher Training
Department of Mathematics

2. What is Mathematica
Mathematica is a computer program created by Wolfram Research Inc.(1988)
Mathematica is a powerful desktop computer program capable of doing algebraic calculations, numerical approximations, and computer graphics.
It is suitable for both numeric and symbolic work, and it has remarkable word-processing capabilities as well.

3. Background Created by Stephen Wolfram and his team Wolfram Research.
Version 1.0 was released in 1988.
Latest version is Mathematica 8.0 – released last year.
Stephen Wolfram: creator of Mathematica

4. Features of Mathematica
An interactive program with a vast range of uses:
Numerical Computation
Equation Solving
Symbolic calculation
Calculus
Graphics
Self-defined functions
List
Pattern Matching
Animation

5. Structure Mathematica has two parts: the Kernel and the Front End.
The Kernel is the main part of the system, which accepts Mathematica commands, processes them and sends back results. This is called evaluating the command.
The Front End is the part of the system that handles such things as screen display, printing and the creation of Mathematica documents.

6. Launching Mathematica
User sees the Notebook interface.
Palettes
A separate kernel is used for all mathematical processing

7. How does Mathematica work?
Mathematica documents are called Notebooks. A Notebook is a bit like a word-processor document; you can type and edit commands, send them to the Kernel for evaluation, display the results and save your work.

8. More on notebooks Notebooks are made up of cells.
There are different cell types e.g. “Title”, “Input”, “Output” with associated properties
To evaluate a cell, highlight it and then press shift-enter
To stop evaluation of code, in the tool bar click on Kernel, then Quit Kernel

9. Mathematica Syntax
Mathematica commands start with a CAPITAL letter, e.g. Sin[x]. If the command is really severalwords in English joined together then each one starts with a capital but WithNoSpacesInBetween.
Case Sensitive: the word Fun is different from the word fun which is different from the word fuN.

10. Mathematica Syntax
Mathematica uses a lot of brackets and all the different sorts of them, ( [ { } ] ).
Arguments of functions are given in square brackets [ ].
Curly brackets, called “braces”, {…} are used to make a list, usually to allow several objects to be treated as one.
Expression are cascaded by () pairs.

11. SPECIAL CONSTANTS
Mathematica protects these names; they cannot be used as names for other
quantities.

12. Mathematica as a calculator
Operation Symbol
Operation Name
Sample +
Add
3+5 -
Substruct
8-3 /
Divide
4/6
* or Space
Multiply
2*4 or 2 4 ^
Power
3^4

13. Some Mathematical Functions
Sqrt[x]
square root ( )
Exp[x]
exponential (ex)
Log[x]
natural logarithm lnx
Log[b, x]
logarithm to base b (logbx)
Sin[x],
Cos[x], Tan[x]
trigonometric functions (with arguments in radians) n!
factorial of n
Mod[n, m]
n modulo m
FactorInteger[n]
prime factors of n

14. Creating your own functions
A function can be defined by using the following form:
f[x_] := an expression involving x (no underbar)
Example: f[x_]:=x^2+3x
f[4]
f[a+1]
?f (This shows the definition you made for f. )

15. Do and If
Do[expr, {i, imin, imax, di}] evaluates expr with i successively taking the values imin through imax in steps of di.
If[condition, t, f, u] evaluates t if condition evaluates to True, f if it evaluates to False and u if it evaluates to neither.
Mention pretty printing.

16. Calculus See D to differentiate.
Can do both definite and indefinite integrals – see Integrate
For a numeric approximation to an integral use NIntegrate.

17. Equation solving
Use Solve to solve an equation with an exact solution, including a symbolic solution.
Use NSolve or FindRoot to obtain a numerical approximation to the solution.
Use DSolve or NDSolve for differential equations.
To use solutions need to use expr/ .x y.

18. Graphics
Mathematica incorporates a wide range of two-and three-dimensional graphics functions. The simplest is Plot, which generates two-dimensional Cartesian graphs
Plot[f, {x, xmin, xmax}]
generates a plot of f as a function of x from xmin to xmax.
Plot[{f1, f2, … }, {x, xmin, xmax}]
plots several functions fi.

19. Graphics (Example)

20. Dimensional Plotting Plot3D[ f, {x,xmin,xmax}, {y,ymin,ymax}]
will draw the graph of z = f(x,y) over the rectangle [xmin,xmax]×[ymin,ymax] in the xy-plane.

21. Optimisation Numeric: local – FindMinimum, FindMaximum
fitting - FindFit
global - NMinimize, NMaximize
Symbolic: Minimize, Maximize

22. Introduction to parallel computing
Parallel computing in Mathematica is based on launching and controlling multiple Mathematica kernel (worker) processes from within a single master Mathematica, providing a distributed-memory environment for parallel programming.
Every copy of Mathematica comes with all the components and tools to run and create parallel applications.

23. Connection Methods
Mathematica can run parallel workers in a number of different ways, locally on the same machine or remote on a network. In addition, the network might be a homogeneous grid controlled by some dedicated management application, or it might be a heterogenous grid. To run in these situations Mathematica needs to launch parallel workers and then communicate with them.
To carry out parallel computation in these different situations Mathematica provides a number of different connection methods, each of which is specialized to a particular style of usage.

24. Connection Methods Local Kernels
The Local Kernels connection method is used to run parallel workers on the same computer as the master Mathematica. It is suitable for a multi-core environment, and is the easiest way to get up and running with parallel computation.
The Lightweight Grid
The Lightweight Grid connection method is used to run parallel workers on different computers from the master Mathematica. It uses Wolfram Lightweight Grid technology to launch Mathematica on the remote machines. It is suitable for a heterogeneous network and where there is no management technology.

25. Connection Methods Cluster Integration
The Cluster Integration connection method is used to run parallel workers on different computers from the master Mathematica. It integrates with a large number of third party cluster management technologies.
Remote Kernels
The Remote Kernels connection method is used to run parallel workers on different computers from the master Mathematica. It relies on using a remote shell invocation technology for launching, and is typically harder to configure and maintain.

26. Configuring and Monitoring
Mathematica provides a number of tools for configuring and monitoring parallel computations. Some of these are accessed using menus from the Mathematica notebook front end.
Parallel Preferences
The default settings of Mathematica automatically configure a number of parallel kernels to use for parallel computation. Typically, on a multi-core machine you will get a number of worker kernels to match the number of cores (up to a license process limit). This default setting means that when you carry out a parallel computation for the first time the system is initialized.

27. Configuring and Monitoring
Status
You can monitor the operation of your parallel kernels with the status display. This can be opened from the front end menu bar, choose Evaluation > Parallel Kernel Status....
This shows a master and four worker kernels all running on the same computer.

28. Configuring and Monitoring
System Information
To get a detailed listing of the configuration of your system for parallel computation you can use the parallel tab of SystemInformation.

29. Launching and Connecting
Mathematica can run parallel kernels in a number of different ways; locally on the same machine or remote on other machines connected in a network. Furthermore, the network might be a homogeneous grid controlled by some dedicated management application, or it might be a heterogenous grid. To use all these different configurations Mathematica needs to launch the parallel kernels and then communicate with them. When working with remote machines it is complicated to launch the parallel kernels. However the parallel kernels were launched, communication always uses MathLink.
Typically, Mathematica launches parallel kernels automatically as they are needed. This uses settings from the ParallelPreferences, but you can also launch kernels manually with the command LaunchKernels.

30. Launching Kernels
Mathematica parallel computation will, typically, automatically launch kernels when a parallel command is executed. The kernels that are launched are those that are set in the Parallel Preferences. The default setting is to launch kernels that run on the same computer, using the Local Kernels connection method; for a two-core machine there will be two parallel kernels. You can see the kernels that will be automatically launched with $ConfiguredKernels; in the following there will be two local kernels.
In[1]:= $ConfiguredKernels
Out[1]= {<<2 local kernels>>}
If you use LaunchKernels, with no arguments this will launch the kernels that have been configured. Here, the two kernels are launched.
In[3]:= Kernels[]
Out[3]= {KernelObject[1, local], KernelObject[2, local]}

31. Launching Kernels
The number of processor cores is given by $ProcessorCount; if you with to change its value you can unprotect it and assign a new value. Here, its value is 2.
In[1]:= $ProcessorCount
Out[1]= 2
The number of parallel kernel licenses is given by $MaxLicenseSubprocesses which is controlled by the Mathematica license. Here the value is 4.
In[2]:= $MaxLicenseSubprocesses
Out[2]= 4
A setting of 4 means that you can launch up to 4 parallel kernels.

32. The Lightweight Grid
The Lightweight Grid connection method is used to run parallel workers on different computers from the master Mathematica. It uses Wolfram Lightweight Grid technology to launch Mathematica on the remote machines. It is suitable for a heterogeneous network and where there is no management technology.
Configuration of the Lightweight Grid connection method is done through the ParallelPreferences. The configuration panel should look similar to the following.
In this setup the master kernel has found three other computers running a Lightweight Grid Manager and each is configured to run 2 kernels.

33. Cluster Integration
The Cluster Integration connection method is used to run parallel workers on different computers from the master Mathematica. It integrates with a large number of third party cluster management technologies.
Mathematica supports the following cluster management technologies.
Windows Computer Cluster Server
Windows HPC Server 2008
Platform LSF
Altair PBS Professional
Sun Grid Engine
In this setup one cluster has been configured to use Windows Computer Cluster Server, with a head node of clusterboss. Each machine is setup to run 2 kernels.

34. Remote Kernels
The Remote Kernels connection method is used to run parallel workers on different computers from the master Mathematica. It relies on using remote shell invocation technology for launching, and is typically harder to configure and maintain.
In this configuration two remote machines are configured each to provide four parallel kernels.

35. LogTM: Log-based Transactional Memory
11/12/2018
Recall Amdahl’s Law
Begins with Simple Software Assumption (Limit Arg.)
Fraction F of execution time perfectly parallelizable
No Overhead for
Scheduling
Communication
Synchronization, etc.
Fraction (1 – F) Completely Serial
UW-Madison Architecture Seminar

36. LogTM: Log-based Transactional Memory
11/12/2018
Recall Amdahl’s Law [1967]
Time on 1 core = (1 – F) / 1 + F / 1 = 1
Time on N cores = (1 – F) / 1 + F / N
Amdahl’s Speedup = 1 +
1 - F F N
For mainframes, Amdahl expected 1 - F = 35%
For a 4-processor speedup = 2
For infinite-processor speedup < 3
UW-Madison Architecture Seminar

37. Amdahl’s law Parallelization
In the special case of parallelization, Amdahl's law states that if F is the fraction of a calculation that is sequential (i.e. cannot benefit from parallelization), and (1 − F) is the fraction that can be parallelized, then the maximum speedup that can be achieved by using N processors is

38. Example 1
95% of a program’s execution time occurs inside a loop that can be executed in parallel. What is the maximum speedup we should expect from a parallel version of the program executing on 8 CPUs?

39. Example 2
20% of a program’s execution time is spent within inherently sequential code. What is the limit to the speedup achievable by a parallel version of the program?

40. Illustration of the Amdahl Effect

41. What is Gustafson’s law [1988]
Gustafson’s law states that, with increasing data size, the speedup obtained through parallelization increases, because the parallel work increases with data size
The speedup factor is F + N ( 1 – F )

42. Gustafson’s Law
Linear speedup of Gustafson compared to “diminishing return” speedup of Amdahl with 50% of code available for parallelization.

43. Comparison the two laws
Amdahl’s presumption of fixed data size is obviously a restriction which does not map into reality for many problems
Both laws are in fact different perspective over the same truth – one sees data size as fixed and the other sees the relation as a function of data size

44. Examples of parallel commands
ParallelEvaluate[expr] evaluates the expression expr on all available parallel kernels and returns the list of results obtained.
ParallelEvaluate[$KernelID]
{1,2,3,4,5,6,7,8}
Obtain each parallel kernel's unique ID number.

45. Examples of parallel commands
Table[expr,{imax}] generates a list of imax copies of expr.
Table[i^2,{i,10}]
{1,4,9,16,25,36,49,64,81,100}
ParallelTable[expr,{ imax}] generates in parallel a list of imax copies of expr.
ParallelTable[i^2,{i,10}]

46. Examples of parallel commands
Map[f, expr] or
applies f to each element on the first level in expr.
Map[f,{a,b,c,d,e}]
{f[a],f[b],f[c],f[d],f[e]}
ParallelMap[f,expr]
ParallelMap is a parallel version of Map, which automatically distributes different applications of f among different kernels and processors.
ParallelMap[N[#,50]&,{E,Pi,Catalan}]

47. Examples of parallel commands
ParallelSum[expr,{i,imax}] evaluates in parallel the sum .
ParallelSum[i^2,{i,1000}]
ParallelProduct[expr,{i,imax}] evaluates the product in parallel.
ParallelProduct[i,{i,100}]

48. Examples of parallel commands
ParallelDo[expr,{i,imin,imax},{j,jmin,jmax},…] evaluates expr looping in parallel over different values of j, etc. for each i.
ParallelDo is a parallel version of Do which automatically distributes different evaluations of expr among different kernels and processors.
ParallelDo[Print[i],{i,4}] 1 2 3 4

49. Examples of parallel calculation
Random Number Generation - Compute 16 million random numbers
On a single core
t1=AbsoluteTime[];
data=Table[RandomReal[],{i,1,16 106}];
timeLocal=AbsoluteTime[]-t1

50. Examples of parallel calculation
Random Number Generation - Compute 16 million random numbers
On eight cores
ParallelEvaluate[$KernelID]
{1,2,3,4,5,6,7,8}
t1=AbsoluteTime[];
data=ParallelTable[RandomReal[],{j,1,16},{i,1,106}];
timeParallel=AbsoluteTime[]-t1
Resulting in a slow-down by a factor of more than 2.6.
Trival computation but lots of communication overhead

51. Examples of parallel calculation
Tasks that are good candidates for parallelization include those where you have multiple independent expressions and seek to apply a function to each one, and then combine the results in the end.
An example computed with serial processing:
Table[FactorInteger[Floor[E^i*Log[i/Sin[i]]]], {i, 1, 60}];
//AbsoluteTiming
{ ,Null}

52. Examples of parallel calculation
The same example using parallel processing:
ParallelTable[FactorInteger[Floor[E^i*Log[i/Sin[i]]]], {i, 1, 60}];//
AbsoluteTiming
{ ,Null}
Here we apply a function where we compute the prime factorization (or just factor for short) the integer part of Exp(x)*Log(x/Sin(x)) to every integer in the range of 1 to 60.

53. Examples of parallel calculation
An example computed with serial processing:
t=AbsoluteTime[];
Map[N[#,1000]&,{Catalan,Glaisher,GoldenRatio,E,EulerGamma,Khinchin,Pi}];
timing=AbsoluteTime[]-t

54. Examples of parallel calculation
The same example using parallel processing:
t=AbsoluteTime[];
ParallelMap[N[#,1000]&,{Catalan,Glaisher,GoldenRatio,E,EulerGamma,Khinchin,Pi}];
timing=AbsoluteTime[]-t

55. Computation of the value of π
Let us consider the computation of the value of π with monte-carlo method. The standard technique is to generate random numbers from uniform distribution [0, 1] as pair of coordinates ( xi , yi) of points in plane. The approximated value of p ca be computed as,
where Np is the number of the points having distance from the origin less or equal with 1., and N is the number of the all randomly generated points.
The Mathematica function to compute this simulation can be,

56. Computation of the value of π
Les us carry out the computation for 10 millions random points, we get
PI[ ]//AbsoluteTiming
{ , }
Now let us try to employ explicit parallel mode using a modified code, with ParallelTable instead of Table,
PIParallelNaive[ ]//AbsoluteTiming
{ , }

57. Computation of the value of π
Now we devide the total work into two portions and assigned them to the workers. We should slightly modify out function,
This function should be available for all workers,
DistributeDefinitions[PIParallel]
We formulate the parallel evaluation as,
t=AbsoluteTime[];
job1=ParallelSubmit[PIParallel[ ]];
job2=ParallelSubmit[PIParallel[ ]];
{a1,a2}=WaitAll[{job1,job2}];
pi=(a1+a2)/ ;
time2=AbsoluteTime[]-t pi

58. Computation of the value of π
Let us simplify our code as,
PIParallelReasonable[n_,m_]:=
Apply[Plus,WaitAll[Map[ParallelSubmit[PIParallel[#]]&,Table[n/m,{m}]]]]
and let is run with 8 portions,
PIParallelReasonable[ ,8]//AbsoluteTiming
{ , }

59. Thank you for your attention