1. Research Computing at Virginia Tech
Advanced Research Computing

2. Outline ARC Overview ARC Resources Training & Education
Getting Started

3. ARC overview

4. Terascale Computing Facility
2200 Processor - Apple G5 Cluster
10.28 teraflops; 3 on 2003 Top500 list
ICAM
Virginia Tech

5. Advanced Research Computing (ARC)
Unit within the Office of the Vice President of Information Technology
Office of Vice President for Research
Provide centralized resources for:
Research computing
Visualization
Staff to assist users
Website:

6. Goals Advance the use of computing and visualization in VT research
Centralize resource acquisition, maintenance, and support for research community
HPC Investment Committee
Provide support to facilitate usage of resources and minimize barriers to entry
Enable and participate in research collaborations between departments

7. Personnel Terry Herdman, Associate VP for Research Computing
BD Kim, Deputy Director, HPC
Nicholas Polys, Director, Visualization
Computational Scientists
Justin Krometis
James McClure
Gabriel Mateescu
User Support GRAs

8. ARC Resources

9. Computational Resources
Blue Ridge – Large scale Linux cluster
Hokie Speed – GPU cluster
Hokie One – SGI UV SMP machine
Athena – Data Analysis and Viz cluster
Ithaca – IBM iDataPlex
Dante – Dell R810
Other resources for individual research groups

10. Blue Ridge Large Scale Cluster
Resources for running jobs
318 dual-socket nodes with 16 cores/node
socket is an eight-core Intel Sandy Bridge-EP Xeon
4 GB/core, 64 GB/node
total: 5,088 cores, 20 TB memory
Two login nodes and two admin nodes
128 GB/node
Interconnect: Quad-data-rate (QDR) InfiniBand
Top500 #402 (November 2012)
Requires allocation to run (only ARC system)
Released to users on March 20, 2013

11. Allocation System Like a bank account for system units
Jobs run are deducted from allocation account
Project PIs (i.e., faculty) request allocation for research project
Based on research output of project (papers, grants) and type of computing/software used
Once approved, add other users (faculty, researchers, students)
Only applies to BlueRidge (no allocation required to run on other ARC systems)

12. HokieSpeed – CPU/GPU Cluster
206 nodes, each with:
Two 6-core 2.40-gigahertz Intel Xeon E5645 CPUs and 24 GB of RAM
Two NVIDIA M2050 Fermi GPUs (448 cores/socket)
Total: 2,472 CPU cores, 412 GPUs, 5 TB of RAM
Top500 #221, Green500 #43 (November 2012)
14-foot by 4-foot 3D visualization wall
Intended Use: Large-scale GPU computing
Available to NSF Grant Co-PIs

13. HokieOne - SGI UV SMP System
492 Intel Xeon 7542 (2.66GHz) cores
Two six-way sockets per blade (12 cores/blade)
41 blades for apps; one blade for system + login
2.6TB of Shared Memory (NUMA)
64 GB/blade, blades connected with NUMAlink
SUSE Linux 11.1
Recommended Uses:
Memory-heavy applications
Shared-memory (e.g. OpenMP) applications

14. Athena – Data Analytics Cluster
42 AMD 2.3GHz Magny Cours quad-socket, octa-core nodes (Total: 1,344 cores, 12.4 TFLOP peak)
32 NVIDIA Tesla S2050 (quad-core) GPUs
6 GB GPU memory
Memory: 2 GB/core (64 GB/node, 2.7 TB Total)
Quad-data-rate (QDR) InfiniBand
Recommended uses:
GPU Computations
Visualization
Data intensive applications

15. Ithaca – IBM iDataPlex
84 dual-socket quad-core Nehalem 2.26 GHz nodes (672 cores in all)
66 nodes available for general use
Memory (2 TB Total):
56 nodes have 24 GB (3 GB/core)
10 nodes have 48 GB (6 GB/core)
Quad-data-rate (QDR) InfiniBand
Recommended uses:
Parallel Matlab
ISV apps needing x86/Linux environment

16. Dante (Dell R810) 4 octa-socket, octa-core nodes (256 cores in all)
64 GB RAM
Intel x86 64-bit, Red Hat Enterprise Linux 5.6
No queuing system
Recommended uses:
Testing, debugging
Specialty software

17. Visualization Resources
VisCube: 3D immersion environment with three 10′ by 10′ walls and a floor of 1920×1920 stereo projection screens
DeepSix: Six tiled monitors with combined resolution of 7680×3200
Athena GPUs: Accelerated rendering
ROVR Stereo Wall
AISB Stereo Wall

18. Education & Training

19. Spring 2013 (Faculty Track)
Intro to HPC (13 Feb)
Research Computing at VT (20 Feb)
Shared-Memory Prog. in OpenMP (27 Feb)
Distributed Memory Prog. using MPI (6 Mar)
Two session courses:
Visual Computing (25 Feb, 25 Mar)
Scientific Programming with Python (1 Apr, 8 Apr)
GPU Programming (10 Apr, 17 Apr)
Parallel MATLAB (15 Apr, 22 Apr)

20. Workshops Offered last: January 2013, August 2012 Two days, covering:
High-performance computing concepts
Introduction to ARC’s resources
Programming in OpenMP and MPI
Third-party libraries
Optimization
Visualization
Next offered: Summer 2013?

21. Other Courses Offered
Parallel Programming with Intel Cilk Plus (Fall 2012)
MATLAB Optimization Toolbox (ICAM
Others being considered/in development:
Parallel R

22. Graduate Certificate (Proposed)
Certificate Requirements (10 credits)
2 core-coursework: developed and taught by ARC computational scientists
Introduction to Scientific Computing & Visualization (3 credits)
Applied Parallel Computing for Scientists &Engineers (3 credits)
A selection of existing coursework (3 credits - list provided in proposal draft)
HPC&V seminar (1 credit)
Interdisciplinary coursework (3 credits – optional)
Administration
Steering/Admissions Committee
Core faculty: develop the courseware and seminar, PhD committee member
Affiliate faculty: instruct existing courses, guest lectures, etc.

23. Proposed Core Courses & Content
Introduction to Scientific Computing & Visualization
Programming environment in HPC
Numerical Analysis
Basic parallel programming with OpenMP and MPI
Visualization tools
Applied Parallel Computing for Scientists &Engineers
Advanced parallelism
Hybrid programming with MPI/OpenMP
CUDA/MIC programming
Optimization and scalability of large-scale HPC applications
Parallel & remote visualization and data analysis
Programming - how much?
How do the skills evolve over 2 semesters?

24. Getting Started on ARC’s Systems

25. Getting Started Steps Apply for an account (all users)
Apply for an allocation (PIs only for projects wishing to use BlueRidge)
Log in (SSH) into the system
System examples
Compile
Submit to scheduler
Compile and submit your own programs

26. Resources ARC Website: http://www.arc.vt.edu
ARC Compute Resources & Documentation:
Allocation System:
New Users Guide:
Training:

27. Research Projects at VT Interdisciplinary Center for Applied mathematics
Terry L. Herdman
Associate Vice President for Research Computing
Director Interdisciplinary Center for Applied Mathematics
Professor Mathematics
Virginia Tech

28. AGILITY - INGENUITY - INTEGRITY
ICAM History
Founded in 1987 to promote and facilitate interdisciplinary
research and education in applied and computational
mathematics at Virginia Tech. Currently, ICAM has 45 members
from 10 departments, 2 colleges, VBI and ARC.
Named SCHEV Commonwealth Center of Excellence in 1990.
Named DOD Center of Research Excellence & Transition in 1996.
Received more than $25 Million in external funding from federal sources and numerous industrial partners.
Received several MURI and other large center grants.
leader of the VT effort on Energy Efficient Building HUB (EEB)
AGILITY - INGENUITY - INTEGRITY
DON’T OVER PROMISE
KEEP SCIENTIFIC CREDIBILITY & REPUTATION
BUILD EXCELLENT WORKING RELATIONSHIPS WITH INDUSTRY AND NATIONAL LABORATORIES
MATHEMATICAL MODELS FOR MANY DIFFERENT PROBLEMS

29. Industry Funding Sources
Sources of ICAM’s Funding
Department of Defense
AIR FORCE OFFICE OF SCIENTIFIC RESEARCH - AFOSR
DEFENSE ADVANCED RESEARCH PROJECT AGENCY – DARPA
ARMY RESEARCH OFFICE - ARO
OFFICE OF NAVAL RESEARCH - ONR
environmental technology demonstration & validation program - ESTCP
VARIOUS AIR FORCE RESEARCH LABS – AFRL
Flight Dynamics Lab - Weapons Lab Munitions Lab
Other Agencies
NATIONAL SCIENCE FOUNDATION – NSF
NATIONAL AERONAUTICS AND SPACE ADMINISTRATION – NASA
Federal Bureau of InvestigatioN – fbi
DEPARTMENT OF HOMELAND SECURITY – DHS
DEPARTMENT OF ENERGY – DOE EERE, ORNL
NATIONAL INSITUTES OF HEALTH – NIH (ID IQ CONTRACT PROPOSAL)
Industry Funding Sources
AEROSOFT, INC. - BABCOCK & WILCOX - BOEING AEROSPACE - CAMBRIDGE HYDRODYNAMICS - COMMONWEALTH SCIENTIFIC CORP. - HONEYWELL - HARRIS CORP. - LOCKHEED - SAIC - TEKTRONIX - UNITED TECHNOLOGIES - SOTERA DEFENSE SOLUTIONS…

30. Industry-National Lab Partners
Nestles
(Ludwigsburg)
Germany
Deutsche Bank
(Frankfurt)
Boeing
(Seattle)
Tektronix
(Beaverton)
Lockheed
(Los Angeles)
AeroSoft
(Blacksburg)
Babcock & Wilcox
(Lynchberg)
Air Force
AFRL
(Albuquerque)
Harris Corp.
(Melbourne
United Technologies
(Hartford)
Sandia
NASA
(Langley
LLNL
DOE Lab
(Livermore)
LBNL
(Berkeley)
Air Force Flight Dynamics
(Dayton)
ORNL
(Oak Ridge)
NREL
(Golden)
(Ames)
AEDC
(Tullahoma)
Air Force Munitions Lab
(Eglin)
Honeywell
(Minneapolis)
SAIC
(McLean)

31. International Collaborations

32. CURRENT ASSOCIATE MEMBERS
ICAM Team
10 Academic Departments
2 Colleges
VBI
ARC - IT
FACULTY
DEPARTMENT
COLLEGE
Ball, Joseph A.
Mathematics
Science
Baumann, William T.
Electrical Engineering
Engineering
Beattie, Christopher
Borggaard, Jeff
Broadwater, Robert
Burns, John A.
Ball, Ken
Mechanical Engineering
Cliff, Eugene M.
Aerospace Engineering
Day, Martin V.
Raffaella De Vita
Engr. Science & Mechanic
Diplas, Panayiotis
Civil Engineering
S. Gugercin
Hagedorn, George A.
Herdman, Terry L.
Iliescu, Traian
Inman, Daniel J.
Kapania, Rakesh K.
Kim, Jong U.
Kohler, Werner E.
Laubenbacher, Reinhard
Bioinformatics Institute
VBI
Lin, Tao
Lindner, Douglas K.
Marathe, Madhav
Neu, Wayne L.
Pierson, Mark
Polys, Nichalos
Research Computing
Information Technology
Prather, Carl L.
Puri, Ishwar
Engr. Science and Mechanics
Renardy, Michael
Renardy, Yuriko
Ribbens, Calvin
Computer Science
Rogers, Robert C.
Russell, David
Sachs, Ekkehard
Santos, Eunice
Shinpaugh, Kevin
Spanos, Aris
Economics
Sun, Shu-Ming
Tyson, John J.
Biology
Vick, Brian
Watson, Layne T.
Wheeler, Robert L.
Williams, Michael
L. Zietsman
CORE MEMBERS*
FACULTY
DEPARTMENT
COLLEGE/INSTITUTE
J. T. Borggaard
Mathematics
Science
J. A. Burns
E. M. Cliff
Aerospace & Ocean Engr.
Engineering
T. L. Herdman
S. Gugercin
T. Iliescu
D. J. Inman
Mechanical Engineering
Reinhard Laubenbacher
Discrete Modeling
VBI
Madhav Marathe
Simulation
Henning Mortveit
Nicholas Polys
Visualization
ARC- IT
Kevin Shinpaugh
HPC
ARC - IT
L. Zietsman
* DEPENDS ON CURRENT PROJECTS & FUNDING
CURRENT ASSOCIATE MEMBERS
1 staff person: Misty Bland

33. ICAM History of Interdisciplinary Projects
Homeland Security
Space Platforms
Design of Jets
Advanced Control
H1N1
CANCER
HIV
IMMUNE
Life Sciences
Nano Technology
Energy Efficient
Buildings
HPC - CS & E

34. Good News / Bad News Good News Bad News
Every IBG Science Problem has a Mathematics Component
Bad News
No IBG Science Problem has only a Mathematics Component
W.R. Pulleyblank
Director, Deep Computing Institute Director, Exploratory Server Systems IBM Research

35. Two Applications to Aerospace
Past Application / New Application
Airfoil Flutter
New Application
Next Generation Large Space Systems
ICAM
Virginia Tech

37. Stealth Began as an unclassified project at DARPA in the early ’70’s
Proved that physically large objects could still have miniscule RCS (radar cross section)
Challenge was to make it fly!

38. CRASH CAUSED BY FLUTTER MURI TOPIC: CONTROL OF AIR FLOWS
ICAM History of Interdisciplinary Projects
DARPA - $1.4 M
An Integrated Research Program for the Modeling, Analysis and Control of Aerospace Systems
TEAM
VT- ICAM NASA USAF
USAF - $2.76 M
Optimal Design And Control of Nonlinear Distributed Parameter Systems
University Research Initiative Center Grant
MURI
TEAM
VT - ICAM Boeing USAF
NC STATE Lockheed
X - 29
F – 117A
DARPA ALSO PROVIDED FUNDS FOR THE RENOVATION OF WRIGHT HOUSE – ICAM’s HOME SINCE 1989
09/14/97: F-117A
CRASH CAUSED BY FLUTTER
MURI TOPIC: CONTROL OF AIR FLOWS

39. Mathematical Research
motivated by problems of interest to industry, business, and government organizations as well as the science and engineering communities.
Mathematical framework: both theoretical and computational
Projects require expertise in several disciplines
Projects require HPC
Projects require Computational Science: Modeling, analysis, algorithm development, optimization, visualization.

40. University Research Team
John Burns
Dennis Brewer
Herman Brunner
Gene Cliff
Yanzhao Cao
Harlan Stech
Janos Turi
Dan Inman
Kazifumi Ito
Graciela Cerezo
Elena Fernandez
Brian Fulton
Z. Liu
Hoan Nguyen
Diana Rubio
Ricardo Sanchez Pena
8 Undergraduate Students
10 Graduate Students
ICAM
Virginia Tech

41. Research Support and Partners
AFOSR
DARPA ACM and SPO
NASA- LaRC
NIA
Flight Dynamics Lab, WPAFB
Lockheed Martin
ICAM
Virginia Tech

42. Build Math Model ICAM start simple use and keep the Physics (Science)
use and keep Engineering Principles
do not try to be an expert in all associated disciplines – interdisciplinary team
learn enough so that you can communicate
know the literature
computational/experimental validation
ICAM
Virginia Tech

43. Spring Mass System ICAM Virginia Tech h(t) plunge α(t) Pitch Angle
β(t) Flap Angle
ICAM
Virginia Tech

44. Pitching, Plunging and Flap Motions of Airfoil
ICAM
Virginia Tech

45. Note: Lift depends on past history
Force: Lift
Note: Lift depends on past history
ICAM
Virginia Tech

46. Evolution Equation for Airfoil Circulation:
ICAM
Virginia Tech

47. Mathematical Model ICAM Virginia Tech
change 2nd order ODE to 1st order system
couple ODE with evolution equation
past history of circulation function provides part of the initial conditions
ICAM
Virginia Tech

48. Complete Mathematical Model
A is a singular 8 by 8 matrix : last row zeros
A(s) : A8i=0 i=1,2,…,7
A88(s)=[(Us-2)/Us]1/2, U constant
B constant matrix, B(s) is smooth
Non Atomic Neutral Functional Differential Equation
ICAM
Virginia Tech

49. Non Atomic NFDE Need Theory of Non Atomic NFDE
Well Posedness results
Approximation Techniques
Parameter Identification
Validation of the Model

50. Abstract Cauchy Problem
ICAM
Virginia Tech

52. ISAT Innovative Space Based Radar Antenna Technology
Canister
Inflatable booms
Antenna surface
300 m long truss structure, 1000 m2 antenna
fly in 2009
Launched in container the size of a small SUV

53. Next Generation Space Systems
develop and deploy large space antennas
take advantage of new materials
take advantage of inflatable technology
joint effort DARPA, NASA LaRC, NIA and Virginia Tech ICAM, Boeing, Lockheed Martin, JPL, Harris Corp., AFRL and others
ICAM – build physics based mathematical models for simulation and control (after deployment)
NASA/AFRL – experiments, testing, development, packaging and deployment
ICAM
Virginia Tech

54. Build Math Model Dr. Joe Guerci, SPO, DARPA
Remember: Obey all Physics (Science) laws
need experts in all associated disciplines – interdisciplinary team
Must communication across disciplines and organizations
know the literature
computational/experimental validation

55. New Mathematical Models
Including Thermal Effects Changes Everything
ADD THERMAL
EQUATIONS
MORE ACCURATE – MORE COMPLEX – MORE DIFFICULT

56. Necessary Components for Success
research expertise in many areas – interdisciplinary team
experience (knowledge of what may work)
MATHEMATICS
external support
state of the art computing facilities
GRAs and young research faculty (new ideas)
ICAM
Virginia Tech