Presentation is loading. Please wait.

Presentation is loading. Please wait.

ISC14 BoF: Drilling Down: Understanding User–Level Activity on Today’s Supercomputers ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's.

Similar presentations


Presentation on theme: "ISC14 BoF: Drilling Down: Understanding User–Level Activity on Today’s Supercomputers ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's."— Presentation transcript:

1 ISC14 BoF: Drilling Down: Understanding User–Level Activity on Today’s Supercomputers ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

2 Outline Brief presentation Open discussion Demo ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

3 Robert McLay TACC ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

4 My Passions Protect new user but stay out of vet's way Make staff support efficient and effective Automate detection, correction, prevention Make the repeat tickets go away! ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

5 Making a difference… Maintain consistent, compatible software environment $ module swap mvapich2 impi Inactive Modules: 1) vasp Due to MODULEPATH changes the following have been reloaded: 1) fftw3/3.3.2 $ module load mvapich2 Lmod Error: You can only have one MPI module loaded at a time. You already have impi loaded. Lmod and related tools ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

6 Making a difference… Detect potential problems and alert users TACC: Starting up job ****************************************************** WARNING: Your MPI Environment is: mvapich2/1.9a2 Your executable was built with: impi/ ****************************************************** Lariat and related tools ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

7 Making a difference… Job-level usage data on libraries and applications ALTD (Mark Fahey -- NICS) ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

8 Joining forces… Detect potential problems and alert users Lariat Job-level usage data on libraries and applications ALTD XALT TACC: Starting up job ****************************************************** WARNING: Your MPI Environment is: mvapich2/1.9a2 Your executable was built with: impi/ ****************************************************** ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

9 My own not-so-hidden agenda... Looking for XALT beta users Hungry for ideas, needs, feedback Wanting to begin conversation with kindred souls ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

10 Mark Fahey UTK ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

11 ALTD What it does – Intercepts linker (ln) and job launcher (aprun) – Uses linker tracemap option to get all libraries – Stores all of this in a database What it gets – Full path of the executable – Static and dynamic libraries used by the executable What it can be used for – Which executables use the largest number of core hours? Are they managed by center? Do they use the system efficiently? – Which libraries, applications, or tools are being used? Are there libraries we should remove? Are there libraries we should install? – What percentage of executables are scripts? Are these scripts being used because the job starter isn’t sophisticated enough? – Are there any executables with modification times older than 1 year? Should we ask the user to recompile? ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

12 What Does NERSC Collect? ALTD – Track library usage both at compile and run time Torque Logs – Job information, accounting ALPS Logs – Track applications run time data and options on the Cray systems Darshan – IO profiling data IPM – MPI profiling data Performance Monitoring – Monitoring system performance over the life time of the machines LMT – Lustre data ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

13 ALTD is enabled on all major computing platforms at NERSC ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

14 Applications of ALTD Understanding current library usage and plan for future software need Providing usage statistics to developers and vendors Restoring the program environment where user applications were built Assisting with debugging system issues An ALTD tool to restore the build environment for an application: linkinfo.sh /global/homes/a/aryal/bin/gvasp5.3.2 User : zz217 Linked on : Executable Name: vasp Libraries Used : //usr/lib64/libhugetlbfs.a../vasp.5.lib/libdmy.a /opt/cray/atp/1.6.0/lib//libAtpSigHCommData.a /opt/cray/atp/1.6.0/lib//libAtpSigHandler.a /opt/cray/libsci/ /cray/81/sandybridge/lib/libsci_cray_mp.a /opt/fftw/ /x86_64/lib/libfftw3.a /opt/cray/mpt/5.6.0/gni/mpich2-cray/74/lib/libmpich_cray.a /opt/cray/mpt/5.6.0/gni/mpich2-cray/74/lib/libmpl.a /opt/cray/xpmem/ ari/lib64/libxpmem.a /opt/cray/pmi/ ari/lib64/libpmi.a /opt/cray/ugni/ ari/lib64/libugni.a /opt/cray/udreg/ ari/lib64/libudreg.a /opt/cray/alps/ ari/lib64/libalpslli.a /opt/cray/alps/ ari/lib64/libalpsutil.a /opt/cray/cce/8.1.2/craylibs/x86-64/libpgas-dmapp.a /opt/cray/cce/8.1.2/craylibs/x86-64/libu.a /opt/cray/dmapp/ ari/lib64/libdmapp.a /opt/cray/pmi/ ari/lib64/libpmi.a /opt/cray/cce/8.1.2/craylibs/x86-64/libfi.a /opt/gcc/4.4.4/snos/lib64/libstdc++.a /opt/gcc/4.4.4/snos/lib/gcc/x86_64-suse-linux/4.4.4/libgcc_eh.a /opt/cray/cce/8.1.2/craylibs/x86-64/libf.a /opt/cray/cce/8.1.2/craylibs/x86-64/libcraymath.a /opt/cray/cce/8.1.2/craylibs/x86-64/libcraymp.a /opt/cray/cce/8.1.2/craylibs/x86-64/libu.a /opt/cray/cce/8.1.2/craylibs/x86-64/libcsup.a //usr/lib64/librt.a /opt/cray/cce/8.1.2/craylibs/x86-64/libtcmalloc_minimal.a //usr/lib64/libpthread.a //usr/lib64/libc.a /opt/gcc/4.4.4/snos/lib/gcc/x86_64-suse-linux/4.4.4/libgcc_eh.a //usr/lib64/libm.a /opt/gcc/4.4.4/snos/lib/gcc/x86_64-suse-linux/4.4.4/libgcc.a ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

15 ALTD at CSCS In production at CSCS since 2011 Rock solid: just a single downtime in two years – Rosa (Cray XE6) since March K compilations, 2.8M jobs – Todi (Cray XK6/XK7) since October K compilations, 500K jobs – Daint (Cray XC30) since March K compilations, 550K jobs We’ve added an additional SQL table “accounting” which logs more data about the application execution – number of cores used, number of cores claimed, number of threads, MPI processes, processes per node, … We want to be able to detect situations like the use of a buggy or non-performant library ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

16 How we mine data: a hypothetic situation A critical bug has been identified in FFTW version , affecting code correctness ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

17 First, find which users have linked this library mysql> select distinct username from altd_rosa_link_tags,altd_rosa_linkline where altd_rosa_link_tags.linkline_id=altd_rosa_linkline.linking_inc and exit_code=0 and linkline like '%fftw/ /%' ; | username | | tkachenn | | boswald | | liang | | robinson | | yunding | | zilia | rows in set (4.33 sec) –Querying the ALTD database reveals that several users have applications linked to the buggy library ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

18 Now, check if they are using the buggy application And it’s confirmed that user “robinson” is running the application linked to the buggy library It’s now up to the user services group to contact the user and recommend relinking their applications against the newer version of FFTW, which has fixed the bug mysql> select altd_rosa_jobs.* from altd_rosa_link_tags,altd_rosa_linkline,altd_rosa_jobs where altd_rosa_jobs.tag_id=altd_rosa_link_tags.tag_id and altd_rosa_link_tags.linkline_id=altd_rosa_linkline.linking_inc and exit_code=0 and linkline like '%fftw/ /%' and altd_rosa_jobs.username="robinson"; | run_inc | tag_id | executable | username | run_date | job_id | build_machine | | | | | /users/robinson/mycode | robinson | | | rosa | | | | /users/robinson/mycode | robinson | | | rosa | | | | /users/robinson/mycode | robinson | | | rosa | | | | /users/robinson/mycode | robinson | | | rosa | | 4 rows in set (0.65 sec) ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

19 This methodology is clearly unmanageable! Ideally, user support specialists would be alerted automatically to “situations of interest” Users running applications linked to legacy, less-performant, or buggy libraries Users running legacy versions of applications Users building code with legacy compilers Users making use of their own libs or apps, when more optimized versions are available centrally How can we automate the processes of data mining, reporting and alerting? ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

20 Porting ALTD on IBM x86 and BGP Neser x_86, – ALTD worked instantly for linking stage ( Intel and GNU compiler) – Postponed “temporarily” tracking the executable ( several OpenMPI versions 1.4.3, 1.5.4, 1.6.4,1.6.5, 1.7.3, 1.8.1) [x2 for Intel and GNU] More effort needed for BGP – On this machine at least 4 sets of compilers are available : Native IBM or GNU For the front (64 bits) or core nodes (32 bits) – Every one of them has its dedicated linker  only patching/wrapping ld and mpirun was no more possible or sufficient XL_LINKER environment available ( force linker to use ALTD like for Cray compilers) XL_LINKER –F ALTD_PATH+'/alias/dev/xlf_linker/xlf_xx._xx.cfg – Need to specify the appropriate ld with the corresponding.cfg file ISC14 : BoF ALTD at KAUST Supercomputin g Lab 20

21 New implementation of ALTD based on bash aliases Each alias points to a unique python script $ALTD_ENV/rerouting/altd_alias.py – an analysis of the command line gives the name of the original command along with some interesting parameters (name of the compiler, executable or job file, requested number of processors) – depending on these information it eventually executes commands to add new information to the database, modifies of the running environment, or patches a job file before submitting it – runs the same command line, this time in an environment unaware of the aliases. – in case of error, the original command runs in an environment unaware of the aliases. The error is logged in a file and a mail is sent to ALTD maintainers. It gathers the encountered exception or error as well as numerous information on the users environment at that time Known Issue: threaded IBM compiler, mpixf90_r, mpixlc_r – Error at runtime about “Insufficient memory to start application” – disabled aliases for mpi…._r compilers ISC14 : BoF ALTD at KAUST Supercomputin g Lab 21

22 ALTD Early mining ( 2months in full production) ALTD fully ported for x86 and BGP Identified: – few compiled codes, (our users are mostly black-box users or in productions) – Enabled storing jobs even if no ALTD header ( we store tag_id=0) Linking – many users linked with libraries built in their home directories Execution – Climate modeling ( mainly pre-processing) – Combustion ( S3D, NGA) – Molecular dynamic ( VASP, NAMD,…) On going – Relink major software installed by the staff to track them better – additional tracking of other statistics (number of requested nodes vs actual used ) – work on a web-interface for automatic mining. ISC14 : BoF ALTD at KAUST Supercomputin g Lab 22

23 TACC_Stats Job-level transparent performance monitoring from HPC compute nodes – CPU performance counters – IB statistics – Lustre statistics – Scheduler job statistics – Host data – OS statistics Analyses integrate available Lariat data (XALT in the future) ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

24 XALT: Understanding the Software Needs of High End Computer Users Newly NSF funded project Is combining the best of Lariat and ALTD Collecting job-level and link-time level data and subsequent analytics Building a community around analytics – potentially one of many tools Will make it available to the community – Optional interface to XDMod/SUPREMME ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

25 XALT Goals Goal is a census of libraries and applications and automatic filtering of user issues – what additional user problems can we detect and report (perhaps correct) automatically? – How can we leverage lessons learned by the tacc stats team to implement additional automatic filtering? – Plan to add tracking of function calls as well Want to balance the need for portability with support for site-specific capabilities Want to simplify the processes system administrators use to install, configure, and manage ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

26 XALT Agenda New tracking infrastructure – XALT Alpha version available today – Deployed at NICS and TACC – LANL and CSCS testing it Some new functionality still to add – Detect function calls – Check runtime environment versus compile time env – Want feedback, hungry for ideas ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

27 Thanks to Richard Gerber and Zhengji Zhao, NERSC Tim Robinson, CSCS Bill Barth, TACC Bilel Hadri, KAUST Julius Westerman, LANL ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

28 Contact Info Mark R. Fahey – Robert McLay – ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

29 Background Slides ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

30 Lariat User #1 Bill Barth Director of HPC, TACC Co-PI SUPreMM ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

31 TACC Stats Job-level transparent performance monitoring from HPC compute nodes – CPU performance counters – IB statistics – Lustre statistics – Scheduler job statistics – Host data – OS statistics Analyses integrate available Lariat data ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

32 Nightly Analyses Automatically analyzes jobs nightly Highlights jobs worth looking at Tries to provide a one-stop view of a job for – Support staff – Sysadmins – And soon, users ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

33 Current Reports High levels of imbalance Low Flops (but other activity) Idle hosts Catastrophic performance drop ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

34

35

36 Richard Gerber Zhengji Zhao NERSC User Services NERSC Job Data ISC14 BoF: Drilling Down: Understanding User- Level Activity on Today's Supercomputers

37 What Does NERSC Collect? ALTD – Track library usage both at compile and run time Torque Logs – Job information, accounting ALPS Logs – Track applications run time data and options on the Cray systems Darshan – IO profiling data IPM – MPI profiling data Performance Monitoring – Monitoring system performance over the life time of the machines LMT – Lustre data ISC14 BoF: Drilling Down: Understanding User- Level Activity on Today's Supercomputers

38 Expose job data via the web We try to make as much data available as possible via the web – For users to track usage – For users to check resource utilization – For users to monitor performance – For staff to help debug jobs – For summary reports The following are web screen shots – All data collection is transparent to users ISC14 BoF: Drilling Down: Understanding User- Level Activity on Today's Supercomputers

39

40

41

42

43

44

45

46 ALTD is enabled on all major computing platforms at NERSC ISC14 BoF: Drilling Down: Understanding User- Level Activity on Today's Supercomputers

47 Applications of ALTD Understanding current library usage and plan for future software need Providing usage statistics to developers and vendors Restoring the program environment where user applications were built Assisting with debugging system issues An ALTD tool to restore the build environment for an application: linkinfo.sh /global/homes/a/aryal/bin/gvasp5.3.2 User : zz217 Linked on : Executable Name: vasp Libraries Used : //usr/lib64/libhugetlbfs.a../vasp.5.lib/libdmy.a /opt/cray/atp/1.6.0/lib//libAtpSigHCommData.a /opt/cray/atp/1.6.0/lib//libAtpSigHandler.a /opt/cray/libsci/ /cray/81/sandybridge/lib/libsci_cray_mp.a /opt/fftw/ /x86_64/lib/libfftw3.a /opt/cray/mpt/5.6.0/gni/mpich2-cray/74/lib/libmpich_cray.a /opt/cray/mpt/5.6.0/gni/mpich2-cray/74/lib/libmpl.a /opt/cray/xpmem/ ari/lib64/libxpmem.a /opt/cray/pmi/ ari/lib64/libpmi.a /opt/cray/ugni/ ari/lib64/libugni.a /opt/cray/udreg/ ari/lib64/libudreg.a /opt/cray/alps/ ari/lib64/libalpslli.a /opt/cray/alps/ ari/lib64/libalpsutil.a /opt/cray/cce/8.1.2/craylibs/x86-64/libpgas-dmapp.a /opt/cray/cce/8.1.2/craylibs/x86-64/libu.a /opt/cray/dmapp/ ari/lib64/libdmapp.a /opt/cray/pmi/ ari/lib64/libpmi.a /opt/cray/cce/8.1.2/craylibs/x86-64/libfi.a /opt/gcc/4.4.4/snos/lib64/libstdc++.a /opt/gcc/4.4.4/snos/lib/gcc/x86_64-suse-linux/4.4.4/libgcc_eh.a /opt/cray/cce/8.1.2/craylibs/x86-64/libf.a /opt/cray/cce/8.1.2/craylibs/x86-64/libcraymath.a /opt/cray/cce/8.1.2/craylibs/x86-64/libcraymp.a /opt/cray/cce/8.1.2/craylibs/x86-64/libu.a /opt/cray/cce/8.1.2/craylibs/x86-64/libcsup.a //usr/lib64/librt.a /opt/cray/cce/8.1.2/craylibs/x86-64/libtcmalloc_minimal.a //usr/lib64/libpthread.a //usr/lib64/libc.a /opt/gcc/4.4.4/snos/lib/gcc/x86_64-suse-linux/4.4.4/libgcc_eh.a //usr/lib64/libm.a /opt/gcc/4.4.4/snos/lib/gcc/x86_64-suse-linux/4.4.4/libgcc.a ISC14 BoF: Drilling Down: Understanding User- Level Activity on Today's Supercomputers

48 Monitoring Software Usage at CSCS Dr Tim Robinson CSCS Drilling Down: Understanding User-Level Activity on Today’s Supercomputers

49 We support many, many libs, tools, apps, compilers…

50 How can we tell if software is actually being used? Support staff want to answer questions like –is anyone using a legacy version of a certain library or application? Research teams, software developers, vendors, and funding agencies need to know –is their software being used? Where should they place effort for future development? User surveys? Counting “module loads”? A better solution: the Automatic Library Tracking Database (Fahey, Jones, and Hadri, Cray User Group meeting, 2010)

51 The Automatic Library Tracking Database ALTD records information every time an application is linked and every time the resulting executable is launched on the compute nodes This is done by intercepting the GNU linker and the job launcher –ALTD records the entire link line so it can be used to determine ancillary information about the compilation, such as which compiler suite was used to build the application Extremely lightweight – essentially no overhead at compile/launch time Only tracks libraries that are actually used in the application, not all libraries that appear on a user’s link line

52 ALTD at CSCS In production at CSCS since 2011 –Rock solid: just a single downtime in two years Rosa (Cray XE6) since March 2011 –600K compilations, 2.8M jobs Todi (Cray XK6/XK7) since October 2012 –470K compilations, 500K jobs Daint (Cray XC30) since March 2013 –100K compilations, 550K jobs We’ve added an additional SQL table “accounting” which logs more data about the application execution – number of cores used, number of cores claimed, number of threads, MPI processes, processes per node, … We want to be able to detect situations like the use of a buggy or non- performant library

53 How we mine data: a hypothetic situation A critical bug has been identified in FFTW version , affecting code correctness

54 First, find which users have linked this library mysql> select distinct username from altd_rosa_link_tags,altd_rosa_linkline where altd_rosa_link_tags.linkline_id=altd_rosa_linkline.linking_inc and exit_code=0 and linkline like '%fftw/ /%' ; | username | | tkachenn | | boswald | | liang | | robinson | | yunding | | zilia | rows in set (4.33 sec) –Querying the ALTD database reveals that several users have applications linked to the buggy library

55 Now, check if they are using the buggy application –And it’s confirmed that user “robinson” is running the application linked to the buggy library –It’s now up to the user services group to contact the user and recommend relinking their applications against the newer version of FFTW, which has fixed the bug mysql> select altd_rosa_jobs.* from altd_rosa_link_tags,altd_rosa_linkline,altd_rosa_jobs where altd_rosa_jobs.tag_id=altd_rosa_link_tags.tag_id and altd_rosa_link_tags.linkline_id=altd_rosa_linkline.linking_inc and exit_code=0 and linkline like '%fftw/ /%' and altd_rosa_jobs.username="robinson"; | run_inc | tag_id | executable | username | run_date | job_id | build_machine | | | | | /users/robinson/mycode | robinson | | | rosa | | | | /users/robinson/mycode | robinson | | | rosa | | | | /users/robinson/mycode | robinson | | | rosa | | | | /users/robinson/mycode | robinson | | | rosa | | 4 rows in set (0.65 sec)

56 This methodology is clearly unmanageable! Ideally, user support specialists would be alerted automatically to “situations of interest” –Users running applications linked to legacy, less-performant, or buggy libraries –Users running legacy versions of applications –Users building code with legacy compilers –Users making use of their own libs or apps, when more optimized versions are available centrally How can we automate the processes of data mining, reporting and alerting?

57 Drilling Down: Understanding User-Level Activity on Today's Supercomputers Mark R. Fahey SC BoF November 20, 2013 ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

58 XALT: Understanding the Software Needs of High End Computer Users Newly NSF funded project Will be combining the best of Lariat and ALTD Collecting job-level and link-time level data and subsequent analytics Building a community around analytics – potentially one of many tools Will make it available to the community – Optional interface to XDMod/SUPREMME ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

59 XALT Goals Goal is a census of libraries and applications and automatic filtering of user issues – what additional user problems can we detect and report (perhaps correct) automatically? – How can we leverage lessons learned by the tacc stats team to implement additional automatic filtering? – Plan to add tracking of function calls as well Want to balance the need for portability with support for site-specific capabilities Want to simplify the processes system administrators use to install, configure, and manage ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

60 Related Tool tacc_stats is a related component in a collection of related initiatives/tools – measures performance at the level of the individual job – every job – Also has an automated process of identifying issues that deserve attention/resources These tools can work together ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

61 Mark’s not so hidden agenda Do you know what libraries are being used? Do you know how many users have trouble with runtime environment matching compile time? Would you have strong opposition to intercepting the linker "ld"? Anyone willing to be a beta tester for our before and after study? Do you have any issues with dropping dot files in user home directories? Do you want to track library function calls? Will be forming a mailing list; we want you to join – Want feedback, hungry for ideas ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

62 Contact Info Mark R. Fahey – Robert McLay – ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers


Download ppt "ISC14 BoF: Drilling Down: Understanding User–Level Activity on Today’s Supercomputers ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's."

Similar presentations


Ads by Google