1. Data Parallel SPMD Programming Environments: Fortran to Java
Han-Ku Lee
Department of Computer Science
Florida State University

2. Outline
Background - historical review of data-parallel languages, message-passing frameworks, and high-level libraries for distributed arrays
HPspmd programming language model – HPJava
The compilation strategies for HPJava
Related systems
Conclusions

3. Acknowledgements
This work was supported in part by the National Science Foundation (NSF ) Division of Advanced Computational Infrastructure and Research
Contract number –

4. Research Objectives
Data-parallel programming and languages have played a major role in high-performance computing
HPF – difficult (compilation)
Library-based lower-level SPMD programming – successful
HPspmd programming language model – a flexible hybrid of HPF-like data-parallel language and the popular, library-oriented, SPMD style
Base-language for HPspmd model should be clean and simple object semantics, cross-platform portability, security, and popular – Java
To power up Java in data-parallel SPMD environment

5. Data Parallel Languages
Large data-structures, typically arrays, are split across nodes
Each node performs similar computations on a different part of the data structure
SIMD – Illiac IV and ICL DAP introduced a new concept, distributed arrays
MIMD – asynchronous, flexible, hard to program
SPMD – loosely synchronous model (SIMD+MIMD)
Each node has its own local copy of program

6. HPF (High Performance Fortran)
By early 90s, value of portable, standardized languages universally acknowledged.
Goal of HPF Forum – a single language for High Performance programming. Effective across architectures—vector, SIMD, MIMD, though SPMD a focus.
HPF - an extension of Fortran 90 to support the data parallel programming model on distributed memory parallel computers
Supported by Cray, DEC, Fujitsu, HP, IBM, Intel, Maspar, Meiko, nCube, Sun, and Thinking Machines

7. HPF
Ideal data distribution
Processors
Memory Area
Multi-processing and data distribution – communication and load-balance
Introduced processor arrangement and Templates
Data Alignment

8. Message-passing for HPC
Processes explicitly communicate through messages on some classes of parallel machines with distributed memory
Early Message-Passing Frameworks – p4, PARMACS, PVM, and Express
Message Passing Interface Forum established a standard API for message-passing library routines – MPI
Portability and scalability

9. High Level Libraries for Distributed Arrays
Distributed Array – a collective object shared by a number of processes
PARTI, The Global Array (GA) Toolkit
Adlib
high-level runtime library, designed to support translation of data-parallel languages
Implemented 1994 in the shpf project at Southampton University and much improved during the Parallel Compiler Runtime Consortium (PCRC) project at Syracuse University
Initially invented for HPF
Currently used in the HPJava project at the Florida State University and Indiana University

10. Adlib Built-in model of distributed arrays and sections.
Equivalent to HPF 1.0 model, plus ghost extensions and general block distribution from HPF 2.0
Collective communication library.
Direct support for array section assignments, ghost region updates, F90 array intrinsics, general gather/scatter.
Implemented on top of MPI.
Adlib kernel implemented in C++.
Object-based distributed array descriptor (DAD)
Interfaces – shpf Fortran interface, PCRC Fortran interface, ad++ interface, and HPJava interface

11. Features of HPJava
A language for parallel programming, especially suitable for massively parallel, distributed memory computers.
Takes various ideas from High Performance Fortran.
HPJava has a distributed array model very similar to the HPF model.
Almost identical set of distribution and alignment options.
In other respects of HPJava is a lower level parallel programming language than HPF.
Programming model is explicit SPMD, needing explicit calls to communication libraries such as MPI
The HPJava system is built on Java technology.
The HPJava programming language is an extension of the Java programming language.

12. Benefits of HPspmd Model
Translators are much easier to implement than HPF compilers. No compiler magic needed
Attractive framework for library development, avoiding inconsistent parameterizations of distributed array arguments
Better prospects for handling irregular problems – easier to fall back on specialized libraries as required
Can directly call MPI functions from within an HPspmd program

13. HPspmd Architecture

14. Multidimensional Arrays
Java is an attractive language, but needs to be improved for large computational tasks
Java provides an array of arrays => disadvantage
Time consumption for out-of bounds checking
The ability to alias rows of an array
The cost of accessing an element
HPJava introduces true multidimensional arrays and regular section
For example
int [[*,*]] a = new int [[5, 5]] ;
for (int i=0; i<4; i++) a [i, i+1] = 19 ;
foo ( a [[:, 0]] ) ;

15. Processes
Proces2 p = new Procs(2, 3) ;
on (p) {
Range x = new BlockRange(N, p.dim(0)) ;
Range y = new BlockRange(N, p.dim(1)) ;
float [[-,-]] a = new float [[x, y]] ;
float [[-,-]] b = new float [[x, y]] ;
float [[-,-]] c = new float [[x, y]] ;
… initialize ‘a’, ‘b’
overall (i=x for :)
overall (j=y for :)
c [i, j] = a [i, j] + b [i, j]; }
An HPJava program is concurrently started on all members of some process collection – process groups
on construct limits control to the active process group (APG), p
The class BlockRange is a subclass of Range, representing an index range block-distributed over the process dimension passed to its constructor 1 2 1 p

16. Distributed arrays The most important feature of HPJava
A collective object shared by a number of processes
Elements of a distributed array are distributed
True multidimensional array
Forms a regular section of an array
When N = 8 in the previous example code, the distributed array, ‘a’ is distributed like:

17. Overall construct overall (i = x for l : u : s) { … }
A distributed parallel loop
i – distributed index whose value is Location, which is a particular element of a particular distributed range
Index triplet represents a lower bound, an upper bound, and a step – all of which are integer expressions
The step is optional – the default step is 1
The lower bound may be omitted – the default is 0
The upper bound may be omitted – the default is N-1
An HPJava range object => a collection of location
With a few exception, the subscript of a distributed array must be a distributed index, and the location should be an element of the range associated with the array dimension
This restriction is an important feature, ensuring that referenced array elements are locally held

18. At construct at (i =x [4]) { … }
HPJava defines a distributed index when we want to update or access a single element of a distributed array rather than accessing a whole set of elements in parallel
When we want to update a [1, 4]
float [[-,-]] a = new float [[x, y]] ;
// a [1, 4] = 19 ; <---- Not allowed since 1 and 4 are not distributed indices,
// therefore, not legal subscripts …
at (i = x [1])
at (j = y[4])
a [i, j] = 19;
The operational semantics of at construct is similar to that of on construct
i` - the back quote symbol is used as a postfix operator on a distributed index

19. Distribution format
Range
BlockRange
CyclicRange
ExtBlockRange
IrregRange
CollapsedRange
Dimension
HPJava provides further distribution formats for dimensions of distributed arrays without further extensions to the syntax
Instead, the Range class hierarchy is extended
BlockRange, CyclicRange, IrregRange, Dimension
ExtBlockRange – a BlockRange distribution extended with ghost regions
CollapsedRange – a range that is not distributed, i.e. all elements of the range mapped to a single process

20. Ghost regions
Ghost region – extra space “around the edges” of the locally held block of distributed array elements
These extra space can cache some of the element values properly belonging to adjacent processors
With ghost regions, the inner loop of algorithms for stencil updates can be written in a simple way, since the edges of the block don’t need special treatment in accessing neighboring elements
Shifted indices can locate the proper values cached in the ghost region
e.g … a [i, j+1] …

21. Array Sections
HPJava supports subarrays modeled on the array sections of Fortran 90
Whereas an element reference is a variable, an array section is an expression that represents a new distributed array object
The new array section is a subset of the elements of the parent array
Triplet subscript
The rank of an array section is equal to the number of triplet subscripts
e.g. float [[-,-]] a = new float [[x, y]] ;
float [[-]] b = a [[0, :]] ;
Subrange – the range of an array section
e.g. Range u = x [0 : N-1 : 2] ;

22. Distributed Array Type
Type signature of a distributed array
T [[attr0, …, attrR-1]] bras
where R is the rank of the array and each term attrr is either a single hyphen, - or a single asterisk, *, the term bras is a string of zero or more bracket pairs, []
T can be any Java type other than an array type. This signature represents the type of a distributed array whose elements have Java type
T bras
A distributed array type is not treated as a class type
It means that a distributed array cannot be an element of an ordinary Java array, nor can a distributed array reference be stored in a standard library class like Vector, which expects an Object
If we say “distributed arrays have a class”, it would commits us to either extending the definition of class in Java language, or creating genuine Java classes for each type of HPJava array that might be need – impractical

23. HPspmd classes and APG
HPJava translator try to distinguish HPJava code from Java code
It introduces a special interface, hpjava.lang.HPspmd, which must be implemented by any class that uses the special syntax
An HPspmd class is a class that implements the hpjava.lang.HPspmd interface. Any other class is a non-HPspmd class
Many of the special operations in HPJava rely on the active process group – the APG
APG is changed during the course of the program as distributed control constructs limit control to different subsets of the processors
In the current HPJava translator, the value of APG is passed as a hidden argument to methods and constructors of HPspmd classes (like “this” reference)

24. Basic Translation Scheme
The HPJava system is not exactly a high-level parallel programming language – more like a tool to assist programmers generate SPMD parallel code
This suggests the transformations the system applies should be relatively simple and well-documented, so programmers can exploit the tool more effectively
We don’t expect the generated code to be human readable or modifiable, but at least the programmer should be able to work out what is going on
The HPJava specification defines the basic translation scheme as a series of schema

25. Translation of a distributed array declaration
Source: T [[attr0, …, attrR-1]] a ;
TRANSLATION: T [] a ’dat ;
ArrayBase a ’bas ;
DIMENSION_TYPE (attr0) a ’0 ; …
DIMENSION_TYPE (attrR-1) a ’R-1 ;
where DIMENSION_TYPE (attrr) ≡ ArrayDim if attrr is a hyphen, or
DIMENSION_TYPE (attrr) ≡ SeqArrayDim if attrr is a asterisk
e.g.
float [[-,*]] var ;  float [] var__$DS ;
ArrayBase var__$bas ;
ArrayDim var__$0 ;
SeqArrayDim var__$1 ;

26. Translation of the overall construct
SOURCE: overall (i = x for e lo : e hi : e stp) S
TRANSLATION: Block b = x.localBlock(T [e lo], T [e hi], T [e stp]) ;
Group p = ((Group) apg.clone()).restrict(x.dim()) ;
for (int l = 0; l < b.count; l ++) {
int sub = b.sub_bas + b.sub_stp * l ;
int glb = b.glb_bas + b.glb_stp * l ;
T [S | p] }
where: i is an index name in the source program,
x is a simple expression in the source program,
e lo, e hi, and e stp are expressions in the source,
S is a statement in the source program, and
b, p, l, sub and glb are names of new variables

27. Important features of translation scheme
From the last slide, the basic translation scheme reduces overall constructs to simple local for loops
Inside these loops, the only overheads relative to hand-coded local for loops is a proliferation of references to fields of simple classes like Block and ArrayDim
These things can easily be lifted outside loops, strength reduction optimizations can be applied to the local subscript expressions, loops can be unrolled, remove redundant checks (run-time checks) etc
These things can all be done easily by a slightly more optimized form of the translator

28. Optimization Strategies
Here we only consider strength reduction optimizations on the index expression
Consider the nested overall and loop constructs
overall (i=x for :)
overall (j=y for :) {
float sum = 0 ;
for (int k=0; k<N; k++)
sum += a [i, k] * b [k, j] ;
c [i, j] = sum ; }

29. A correct but naive translation
Block bi = x.localBlock() ;
for (int lx = 0; lx<bi.count; lx ++) {
Block bj = y.localBlock() ;
for (int ly = 0; ly<bj.count; ly ++) {
float sum = 0 ;
for (int k = 0; k<N; k ++)
sum += a.dat() [a.bas() + (bi.sub_bas + bi.sub_stp * lx) * a.str(0) +
k * a.str(1)] *
b.dat() [b.bas() + (bj.sub_bas + bj.sub_stp * ly) * b.str(1) +
k * b.str(0)] ;
c.dat() [c.bas() + (bi.sub_bas + bi.sub_stp * lx) * c.str(0) +
(bj.sub_bas + bj.sub_stp * ly) * c.str(1)] = sum; }

30. Strength-Reduction Optimization
The complexity of the associated terms in the subscript expressions
The subscript expressions can be greatly simplified by application of strength-reduction optimization
Eliminate complicated expressions involving multiplication from expressions in inner loops by introducing the induction variables:
vai_ ≡ a.bas() + (bi.sub_bas + bi.sub_stp * lx) * a.str(0)
vci_ ≡ c.bas() + (bi.sub_bas + bi.sub_stp * lx) * c.str(0)
vb_j ≡ b.bas() + (bj.sub_bas + bj.sub_stp * ly) * b.str(1)
vcij ≡ c.bas() + (bj.sub_bas + bj.sub_stp * ly) * c.str(0) +
bj.sub_bas + bj.sub_stp * ly) * c.str(1)
Which can be computed efficiently by increasing at suitable points with the induction increments:
sia0 ≡ bi.sub_stp * a. str(0) sic0 ≡ bi.sub_stp * c. str(0)
sjb0 ≡ bj.sub_stp * b. str(1) sjc1 ≡ bj.sub_stp * c. str(1)

31. Block bi = x.localBlock() ;
Translation of overall after applying strength reduction to distributed index subscript expression
Block bi = x.localBlock() ;
int vai_ = a.bas() + bi.sub_bas * a.str(0) ;
int vci_ = c.bas() + bi.sub_bas * c.str(0) ;
final int sia0 = bi.sub_stp * a.str(0), sic0 = bi.sub_stp * c.str(0);
for (int lx = 0; lx < bi.count; lx ++) {
Block bj = y.localBlock() ;
int vb_j = b.bas() + bj.sub_bas * b.str(1) ;
int vcij = vci_ bj.sub_bas * c.str(1) ;
final int sjb1 = bj.sub_stp * b.str(1), sjc1 = bj.sub_stp * c.str(1);
for (int ly = 0; ly < bj.count; ly ++) {
float sum = 0;
for (int k = 0 ; k < N; k ++)
sum += a.dat() [vai_ + k * a.str(1)] * b.dat() [vb_j + k * b.str(0)] ;
c.dat() [vcij] = sum ;
vb_j += sia0 ; vcij += sjc1 ; }
vai_ += sia0 ; vci_ += sic0 ;

32. Related Systems (1) Co-Array Fortran (formerly called F--) ZPL
A simple and small set of extensions to Fortran 95 for SPMD processing
The logical model of communication is built-in
HPJava follows MPI philosophy i.e. no communication primitives
ZPL
An array programming language designed from first principles for fast execution on both sequential and parallel computers
A := A + B ; (where A and B are two dimensional arrays)
Parallelism and communication is more implicit than HPJava
HPJava provides lower-level access to parallel machine using mpiJava

33. Related Systems (2) Spar STAPL
A Java-based programming language for semi-automatic array-parallel programming
Multidimensional arrays, array sections and parallel loop
Similar in syntax, but semantically different to HPJava
Suitable to shared memory computing systems
HPJava targets massively parallel distributed memory computing
STAPL
A parallel C++ library designed as a super set of the ANSI C++ STL and executed on uni- or multi- processors for SPMD programming
While STAPL and HPJava share a SPMD programming model, HPJava is more naturally suited to distributed memory systems since it is using the philosophy of distributed arrays

34. Java Performances
Benchmarked on Linux Red Hats 7.2 (Pentium IV 1.5 GHZ)
Linpack
Compared Java with GNU cc and Fortran77
Seems like we don’t need loop unrolling for Java
MFLOPS
Unrolled
Rolled
cc –O5
403.68
376.45
g77 -O5
218.83
215.76
IMB Developer Kit 1.3
13.78
233.81
Sun JDK 1.4 beta
193.35
175.73
Blackdown JDK 1.3.1
199.11
158.08

35. Why Fortran is slower than C ?
Could say “performance of Fortran and C” are same
But, depends upon compilers
GNU Fortran 77 compiler generates more machine codes than GNU cc compiler does for main loop in Linpack

36. Conclusions Historical review of data-parallel languages such as HPF
Message-passing frameworks – p4, PARMACS, PVM and MPI standard
High-level libraries for distributed arrays – PARTI, GA and Adlib
HPspmd programming language model – SPMD framework for using libraries based on distributed arrays
Specific syntax, new control constructs, basic translation schemes, and basic optimization strategies for HPJava
Related systems – Co-Array Fortran, ZPL, Spar, and STAPL
Current status of HPJava
Collaborated with Bryan Carpenter, Geoffrey Fox, Guansong Zhang, Sang Lim and Zheng Qiang
The first fully functional HPJava translator (written in Java) is now operational
Parser – JavaCC and JTB tools
Has been tested and debugged against small test suite and 800-line multigrid code
Next stage – implement the optimization