1. K T A U Kernel Tuning and Analysis Utilities Department of Computer and Information Science Performance Research Laboratory University of Oregon

2. University of Oregon Performance Research Lab Agenda Motivations KTAU Overview ZeptoOS - KTAU - TAU on BG/L KTAU - TAU on Linux Cluster

3. University of Oregon Performance Research Lab What is a process is doing inside a kernel? Solution: Context-of-Execution Based profile/trace We can analyze the execution path of a process, and store the data local to a process.

4. University of Oregon Performance Research Lab What about other processes on the system? Solution: System-wide performance analysis By aggregating performance of each process in the system (all or selectively), we can capture interactions among processes.

5. University of Oregon Performance Research Lab Profiling or Tracing? Answer: Why not doing both? Profile A summarized view of performance data, with the advantage of compact data size. Trace A detail view of process execution timeline, with a disadvantage of large data size.

6. University of Oregon Performance Research Lab Why do we need another kernel profiling/tracing tool? Answer: Why not? LTT Oprofile KernInst

7. University of Oregon Performance Research Lab KTAU Design Goals Fine-grained kernel-level performance measurement –Parallel applications –Support both profiling and tracing Both process-centric and system-wide view Merge user-space performance with kernel-space Detailed program-OS interaction data Analysis and visualization compatible with existing tools

8. University of Oregon Performance Research Lab KTAU Method Instruments Linux kernel source with KTAU profiling API Maintains performance data for each kernel routine (per process) Performance data accessible via /proc filesystem Instrumented application maintains data in user- space Post-execution performance data analysis

9. University of Oregon Performance Research Lab KTAUFrameworkKTAUFramework

10. KTAU Architecture 5 modules - KTAU Instrumentation - KTAU Profiling/Tracing Infrastructure - KTAU Proc Interface - KTAU User-API Library - KTAU-D

11. University of Oregon Performance Research Lab Kernel Profiling Issues on BG/L I/O node kernel Linux kernel approach Compute node kernel No daemon processes Single address space –single performance database –single callstack across user/kernel Keeps track of one process only (optimization) Instrumented compute node kernel

12. University of Oregon Performance Research Lab KTAU on BG/L I/O Node

13. University of Oregon Performance Research Lab KTAU on BG/L Current status –IO Node ZeptoOS kernel profiling/tracing –KTAU integrated into ZeptoOS build system –Detailed IO Node kernel observation now possible –KTAU-Daemon (KTAU-D) on IO Node monitors system-wide and individual process more than what strace allows –Visualization of trace/profile of ZeptoOS and CIOD Vampir/JumpShot (trace), and Paraprof (profile),

14. University of Oregon Performance Research Lab KTAU Usage Models for BG/L IO-Node Daemon-based monitoring (KTAU-D) –Use KTAU-D to monitor (profile/trace) a single process (e.g., CIOD) or entire IO-Node kernel –No access to source code of user-space program –CIOD kernel-activity available though CIOD source N/A Self monitoring –A user-space program can be instrumented (e.g., with TAU) to access its OWN kernel-level trace/profile data –ZIOD (ZeptoOS IO-D) source (when available) can be instrumented –Can produce MERGED user-kernel trace/profile

15. University of Oregon Performance Research Lab More on KTAU-D A daemon running on BG/L IO-node that periodically accesses kernel profile/trace data and outputs to filesystem Configuration done through ZeptoOS configuration tool KTAU-D, configuration file, and necessary scripts are integrated into the ZeptoOS runtime environment.

16. University of Oregon Performance Research Lab KTAU-D Configuration in ZeptoOS-1.2

17. University of Oregon Performance Research Lab KTAU-D Profile Data KTAU-D can be used to access profile data (system- wide and individual process) of BGL IO-Node Data is obtained at the start and stop of KTAUD, and then the resulting profile is generated Currrently flat profiles with inclusive/exclusive times and Function call counts are produced –(Future work: Call-graph profiles). Profile data is viewed using the ParaProf visualization tool

18. University of Oregon Performance Research Lab Example of Operating System Profile on I/O Nodes Running Flash3 on 32 compute-node Ciod Kernel Profile

19. University of Oregon Performance Research Lab KTAU-D Trace KTAU-D can be used to access system-wide and individual process trace data of BGL IO-Node Trace from KTAU-D is converted into TAU trace- format which then can be converted into other formats –Vampir, Jumpshot Trace from KTAU-D can be used together (merged) with trace from TAU to monitor both user and kernel space activities –(Work in progress)

20. University of Oregon Performance Research Lab Exp 1: Observe activities on the IO node Set up: –KTAU: Enable all instrumentation points Number of kernel trace entries per process = 10K –KTAU-D: System-wide tracing Accessing trace every 1 second and dump trace output to a file in users home directory through NFS –IOTEST: An mpi-based benchmark (open/write/read/close) Running with default parameters (block-size = 16MB) on NFS.

21. University of Oregon Performance Research Lab Read Time Write Time Write Seek Time Read Seek Time Main IOTEST with TAU instrumentation

22. University of Oregon Performance Research Lab sys_write() / sys_read() KTAU Trace of CIOD running 2, 4, 8, 16, 32 nodes As the number of compute node increase, CIOD has to handle larger amount of sys_call being forwarded.

23. University of Oregon Performance Research Lab Zoomed View of CIOD Trace (8 compute nodes)

24. University of Oregon Performance Research Lab Can Correlate CIOD Activity with RPC-IOD? Activity within a BG/L ionode system switching from CIOD to rpciod during a sys_write system call rpciod performs socket_send and interrupt handling before switching back rpciod ciod

25. University of Oregon Performance Research Lab Exp 2: Correlating multiple traces from Compute-node and IO-node Set up: –Running IOTEST with TAU instrumentation on 64 compute nodes –Running ZeptoOS-1.2 with KTAU on 2 io-node –Reduced set of kernel instrumentation. No TCP stack and schedule() –10K entries of ring-trace buffer –Using PVFS2 (Note: Trace of 64 compute-node and 2 io-node)

26. University of Oregon Performance Research Lab read() @ 12:678 sec write() @ 3:283 sec TAU Trace

27. University of Oregon Performance Research Lab sys_open() @ 53:1sys_read() @1:05:545sys_write() @ 56:6 sys_open() @ 53:2sys_write() @ 56:85sys_read() @ 1:05:778 ciod on ionode23 ciod on ionode47 pvfs2-client on ionode23 pvfs2-client on ionode47

28. University of Oregon Performance Research Lab Exp 3: Analyze system-wide performance Set up: –2 runs of IOTEST with TAU instrumentation on 32 compute nodes NFS PVFS –Running ZeptoOS-1.2 with KTAU on 1 io-node –Analyzing both profile and trace data

29. University of Oregon Performance Research Lab write() @ 39:00 read() @ 47.804 write() @ 42:99read() @ 54:61 pvfs2-client ciod rpciod ciod