1. Computer and Automation Research Institute Hungarian Academy of Sciences The P-GRADE Visual Parallel Programming Environment Péter Kacsuk Laboratory of Parallel and Distributed Systems MTA SZTAKI Research Institute kacsuk@sztaki.hu www.lpds.sztaki.hu

2. Problems of Developing Parallel Programs High-Speed Switch Observing? Programming?

3. Our Solution: P-GRADE  P-GRADE is a parallel programming environment which supports the whole life-cycle of parallel program development  For non-specialist programmers it provides a complete solution for efficient and easy parallel program development  Fast reengineering of sequential programs for parallel computers  Unified graphical support in program design, debugging and performance analysis  Portability on supercomputers and heterogeneous workstation/PC clusters based on PVM and MPI

4. Tools of P-GRADE GRAPNEL: Hybrid Parallel Prog. Language –Graphics to express parallelism –C/C++ to describe sequential parts GRED: Graphical Editor GRP2C: Pre-compiler to (C/C++)+(PVM/MPI) DIWIDE: Integrated distributed debugger and animation system GRM: distributed monitoring system PROVE: Integrated visualisation tool

5. Parallel Program Design GRAPNE L GRE D Mapping User mapping GRP file Pre-compilation GRP2C C source code, Cross-ref file, Make file Building executables C compiler, linker GRP-PVM GRM Library PVM Library GRP-MPI GRM Library MPI Library executables Trace file Monitoring GRM Visualisation PROVE Life-cycle of Parallel Program Development and its support in P-GRADE GRP file Debugging DIWIDE

6. Design Goals of GRAPNEL Graphical interface –to define all parallel activities –Strong support for hierarchical design –Visual abstractions to hide the low level details of message-passing C/C++ (or Fortran) to describe sequential parts –Strong support for parallelizing sequential applications –Support for programming in large –No steep learning curve GRAPNEL = (C/C++) + graphics

7. GRAPNEL: GRaphical Process NEt Language Programming paradigm: message-passing –component processes run in parallel and can interact only by means of sending and receiving messages Communication model: –point-to-point, synchronous/asynchronous –collective (e.g. multicast, scatter, reduce, etc.) Process model: –single processes –process groups –predefined process communication templates

8. Three layers of GRAPNEL

9. GRAPNEL  Hierarchical design levels:  Graphics used at application level:  Defines interprocess communication topology  Port protocols  Graphics hides PVM/MPI function calls  Support for SPMD programming style  Predefined communication patterns  Automatic scaling of parallel programs

10. Communication Templates Pre-defined regular process topologies –process farm –pipeline –2D mesh –tree User defines: –representative processes –actual size Automatic scaling

11. Mesh Template

12. Tree Template

13. The process farm parallelisation approach Master Send work packages send(); Collect results recv(); Slave1Slave2SlaveN spawn(N); The code of each slave is the same.

14. Parallelising the Mandelbrot set computation

15. Draw process output

16. Compute process input

17. Compute process output

18. Draw process input

19. Process Groups Hierarchical design (subgraph abstraction) Collective communication (group ports) –multicast –scatter –gather –reduce

20. GRAPNEL  Hierarchical design levels:  Graphics used at process internal level  C/C++ used at the text level  Synch/asynch. comm.  Programming in large:  Any C/C++ library call can be included in text blocks  Graphical support for object- based programming

21. GRAPNEL  Structuring facility by macro graphs

22. multicast gather Userdef (grp_in) reduce scatter Point-point gather scatter Userdef (grp_out) GRAPNEL

23. Parallelising the Mandelbrot set computation

27. GRED Editor  Supports the creation of all the elements of GRAPNEL  Drag-and-drop style of drawing  Cut/copy/paste/move on graphical objects  Automatic port positioning with minimal lengths and crossing of communication channels

28. GRED Editor  Extremely easy and fast construction of process graph  Automatic arrange of the process graph  Automatic resizing of process windows  Cut/copy/paste on graphical objects  Macro graph construction at arbitrarily nested level  C/C++ code can be edited by any standard text editor

29. GRP2C Pre-compiler Automatic generation of PVM and MPI calls based on GRAPNEL graphics GRP2C C/C++ graphics GRAPNEL Automatic code instrumentation for debugging and performance monitoring C/C++ PVM/MPI Generated code

30. Debugging Parallel Programs High-Speed Switch Observing?

31. Principle of sequential program debugging Reproducibility - determinism –For the same input set the sequential program delivers always the same output set (even if the program is incorrect) Used technique: cyclic debugging –breakpoints –step-by-step execution

32. Problem of parallel program debugging Non-reproducibility (non-determinism) –For the same input set the incorrect parallel program can deliver different output sets Cyclic debugging cannot be used –breakpoints –step-by-step execution

33. Classification of parallel debuggers Parallel running seq. debuggers Replayable debuggers Monitor&replay Control&replay

34. DIWIDE Debugger  Graphical and C/C++ level debug support (breakpoints, variable inspection, etc.)  3 kinds of “step by step execution”, according to the programmer’s demand:  Instruction by instruction,  Graphical item by graphical item,  Macrostep by macrostep  Visualisation and animation support

35. Hierarchical Debugging by DIWIDE

36. Classification of parallel debuggers Parallel running seq. debuggers Replayable debuggers Monitor&replay Control&replay

37. Classification of parallel breakpoints Local breakpoints Global breakpoints Individual breakpoints Collective breakpoints

38. Principle of Macrostep Debugging  Parallel debugging is as easy as debugging traditional sequential programs. Macrosteps Collective Breakpoints M 0 = {S 1 -> A 1, S 2 -> A 2, S 3 -> A 3 } A 1 A 2 A 3 M 1 = {A 1 -> B 1, A 2 -> B 2, A 3 -> B 3 } B 1 B 2 B 3 M 2 = {B 1 -> B 1, B 2 -> C 2, B 3 -> B 3 } B 1 C 2 B 3 M 3 = {B 1 -> B 1, C 2 -> D 2, B 3 -> E 3 } B 1 D 2 E 3 M 4 = {B 1 -> E 1, D 2 -> E 2 } E 1 E 2 where S i = Start i and E i = End i P1P1 P2P2 P3P3 S1S1 A1A1 B1B1 E1E1 S2S2 S3S3 A2A2 A3A3 B2B2 C2C2 D2D2 E2E2 B3B3 E3E3

39. Macrostep Debugging  Support for systematic debugging to handle non- deterministic behaviour of parallel applications  Systematic and automatic generation of Execution Trees  Testing parallel programs for all time conditions  Replay technique with collective breakpoints

40. Automatic Deadlock Detection by Macrostep Debugging

41. Integration of Macrostep Debugging and PROVE

42. Performance monitoring and analysis of Parallel Programs High-Speed Switch Observing?

43. Visualisation Systems Scientific (Data Oriented) Visualisation Program Visualisation Problem Visualisation (Alg. Animation) Correctness Debugging Performance (Debugging) Visualisation Combined Visualisation Goal of visualisation? What to visualise?

44. Program Visualisation Correctness Debugging Performance Visualisation Combined Visualisation Goal of visualisation? Off-line On-line Semi On-line When to visualise?

45. Phases of Performance Visualisation Source Code Instrumentation (GRAPNEL/GRED) Runtime Monitoring (GRM) Visualisation (PROVE) Data Analysis (PROVE)

46. Performance Visualisation Scalability (Data handling) Scalability (Data handling) Source Code Instrumentation Source Code Instrumentation Versatility (Visualisation) Versatility (Visualisation) Evaluation Criteria

47. Source Code Instrumentation Manual or Automatic Monitoring modes Filtering Click-back facility Selectable program units Individual Events On/off facility Statistics

48. Scalability Data Acquisition Data Analysis & Display Turning tracing on/off Filtering Zooming Filtering Interactive Non-Interactive VISTOPNupshot

49. Versatility Interoperate with other tools Different views Event views No Statistics views Yes

50. Standalone Performance Analysis Tools VAMPIR Pablo ParaGraph AIMS Paradyn

51. VAMPIR

52. Integrated Performance Analysis Tools VISTOP (TOPSYS) PVMVis (EDPEPPS) PROVE (GRADE)

53. Source Code Instrumentation Automatic Monitoring modes Filtering Click-back facility Selectable program units Individual Events On/off facility Statistics

54. Source Code Instrumentation Automatic Monitoring modes Filtering Click-back facility Selectable program units Individual Events On/off facility Statistics

55. Source code click-back facility and click-forward facility

56. Scalability Data Acquisition Data Analysis & Display Turning tracing on/off Filtering Zooming Filtering Interactive Non-Interactive

57. Scalability Data Acquisition Data Analysis & Display Turning tracing on/off Filtering Zooming Filtering Interactive Non-Interactive

58. Behaviour Window of PROVE  Scrolling visualisation windows forward and backwards  User controlled focus on processors, processes and messages  Zooming, event filtering facilities

59. Scalability Data Acquisition Data Analysis & Display Turning tracing on/off Filtering Zooming Filtering Interactive Non-Interactive

60. Filtering in PROVE

61. Versatility Interoperate with other tools Different views Event views No Statistics views Yes

62. PROVE Performance analyser Various views for displaying performance information  Synchronised multi-window visualisation

63. PROVE Summary Windows  Various views for displaying summary information  Synchronised multi-window visualisation

64. PROVE Statistics Windows  Profiling based on counters  Analysis of very long running programs is enabled

65. Versatility Interoperate with other tools Different views Event views No Statistics views Yes P-GRADE

66. The GRM Monitor Off-line monitoring (GRADE) –stores trace events in a (local or global) storage and –makes it available after execution for post-mortem processing. Semi-on-line monitoring (P-GRADE) –stores trace events in a storage but –makes it available for the visualisation tool any time during execution if the user asks for it –interactive usage of PROVE –user can remove already inspected part of the trace –evaluation of long-running programs –macrostep debugging in P-GRADE with execution visualisation

67. Application-level monitor Tracing + statistics collection Semi-on-line GRM monitor

68. Buffer is full (to a certain threshold) Trace collection MM LM Process 1Process 2Process 3 Trace file Process notifies LM LM notifies MM MM asks all LMs to stop application MM for each LM: asks each LM to send trace sets timestamps to a global time writes trace into the trace file receives trace from LM MM asks LMs to continue application Trace file

69. Portability Supported Hardware/Software Platforms  Workstation clusters  SGI MIPS  / IRIX  5.x/6.x (MTA SZTAKI, Univ. of Vienna)  Sun  UltraSPARC  / Solaris  2.x (Univ. of Athens)  Intel  x86 / Linux (MTA SZTAKI)  Supercomputers  Hitachi  SR2201 / HI-UX/MPP  (Polish-Japanese School, Warsaw)  Cray  T3E  / UNICOS  (Jülich, Germany)

70. International installations Current –UK –Austria –Spain –Portugal –Poland –Germany –Slovakia –Greece –Japan –Mexico –USA Planned –Australia –Korea

71. Further Developments Family of parallel programming environments P-GRADEVisualMPVisualGrid - checkpointing - dynamic load balancing - fault tolerance - grid resource management - grid monitoring - mobile processes

72. Conclusion Current applications in physics –Efficency lost due to high level graphical programming is less than 2 % Weather forecast application under development Download version: –www.lpds.sztaki.hu P-GRADE (Professional GRADE) –Project with Silicon Graphics Hungary –Current developments to support SPMD style programming Object based programming

73. Thank You... ?