HOMME Trace Analysis Fabrice Mizero Mentor: Dr. John Dennis Collaborators: Prof. Malathi Veeraraghavan (University of Virginia) Prof. Robert D. Russell (University of New Hampshire) Qian Liu(University of New Hampshire) Aug 1, 2014

Roadmap Motivation Background Methodology Results Conclusion and Solutions Future Work

Big Picture Understanding the causes of poor performance of CESM on Yellowstone: a 5-step approach Experimental execution and data collection HOMME trace analysis IBMgtSim: routing study Network simulation Integrated simulation

3-level 2-hop 4-hop 6-hop *Credit: Dr. John Dennis Zhengyang Liu

Suspected Causes Network Congestion OS Jitter “…OS noise, shape of the allocated partition, and interference from other jobs.” Abhinav Bhatele et al. SC13 Network Congestion Head of Line Blocking Credit-Based Flow Control OS Jitter Kernel Interrupts Application Interference: Self-Interference Interference with others (Neighborhood Effect) Competition against OS Daemons, Timer Interrupts, buffer-cache synchronization, etc.

Congestion Head of Line Blocking (HOL) Worst Case Scenario: Congestion Spreading due to HOL H1 Victim Flow Out of Buffer Space!! H4 Out of Buffer Space!! S1 S2 H2 H5 Stuck!!! H3 H6 H7

OS Jitter Each compute node runs its own OS - RHEL Interference caused by OS routines Timer interrupts OS Daemons Hardware interrupts Competition for CPU resources. Example: Line Printer Daemon

3 Questions How does congestion impact network latency? How important is OS Jitter to network latency? What has a bigger impact to message latency: OS Jitter or Congestion?

Experimental Set-Up Congestion: OS Jitter: 2 Platforms Jellystone: Non-production machine Yellowstone: production machine Different message sizes & Hop distance OS Jitter: Linux Transparent Huge Pages (THP)

Extrae Trace Collection Methodology Extrae Trace Collection Clock Skew Correction Hop, Size Hop, Size Wilcoxon Rank Sum Test

Extrae Tracing tool Developed at BSC Chronologic event, state, communications records One way communication delays – Visuals with Paraver MPI-Isend Start Time End

Clock Skew Same size, Same Hop-Count, host-pair level Host A Ca(t1) Ideally, CAB= Cb(t2) – Ca(t1) Host B Cb(t2) In reality, Offset = Ca(t) – Cb(t) != 0 Skew = Ca’(t) - Cb’(t) != 0 Same size, Same Hop-Count, host-pair level Min delay: best approximation of offset CAB(t) – min( CAB(t)) + minpingpong

Statistical Methods Wilcoxon Rank Sum Test: Non-parametric significance test Compare the means of two independent populations Tests: OS Jitter? Jellystone: no THP <=> with THP Congestion? Yellowstone: 0-Hop delays  4-Hop Delays Jellystone: THP  Yellowstone: THP

Perfquery Perfquery: IB performance counters query tool. PortXmitWait: Port congestion monitoring Credit-Based Flow control TOR Switch Credits? PortXmitWait No Yes Host A

Results How important is OS Jitter to network latency? Jellystone::0-Hop::NoTHP vs. Jellystone::0-Hop::THP Intranode communications delays with THP enabled are slower than without THP. Msg size Sample size p-Value Interpretation 488B 54624::45727 <0.001, <0.001,1 NoTHP is faster than with THP 1952B 9503::7950 2440B 102120::85468 2928B 47504::39764

Results What has a bigger impact to message latency: OS Jitter or Congestion? Comparing: Yellowstone: 0-Hop delays, 4-Hop delays For all considered message sizes, intranode communications delays can outweigh internode delays Msg size Sample size p-Values Interpretation 488B 54325::23621 <0.001, <0.001,1 4-Hop is faster than 0-Hop 2440B 101581::16529 2928B 47243::21259 4880B 49603::4720

Conclusion OS Jitter can cause performance degradation or variability. Inter-job interference can lead to application performance variability. Solutions Congestion: Dynamic Allocation of Virtual Lanes to redirect victim flows around congested ports. OS Jitter: Linux Tickless Kernel MPI-3 for better control over share-memory communications.

Future Work Further study on the Dynamic Virtual Lanes assignment solution Plan and collect new HOMME traces with PortXmitWait monitored and LSF Logs saved. Study intra-job interference More efficient algorithm of correcting Clock Skew

Fabrice Mizero fm9ab@virginia.edu Thank You Fabrice Mizero fm9ab@virginia.edu