Vijay K. Agarwala Senior Director, Research Computing and Cyberinfrastructure The Pennsylvania State University University Park, PA USA Research Computing: Critical Needs and Opportunities at a University Based Academic Computing Center HPC User Forum October , 2008 High Performance Computing Center Stuttgart (HLRS),Germany October 16 th, 2008 Imperial College, London

Human Resources Vice Provost for Information Technology and CIO Associate Vice Pro vost Financial Services Marketing and Communication Research Computing and Cyberinfrastructure (RCC) Consulting and Support Services (CSS) Security Operations and Services (SOS) Teaching and Learning with Technology (TLT) Digital Library Technologies (DLT) Administrative Information Services (AIS) Telecommunications and Networking Services (TNS) Research Security Libraries Instruction CommunicationBusiness Support ITS Organization

High Performance Computing Systems Domain Expertise and Consulting Support Software Development and Programming Support Visualization and Telecollaborative Systems Senior Director Faculty Advisory Committee on Research Computing and Cyberinfrastructure Information Technology Services Research Computing and Cyberinfrastructure Meets the high-end computing technology needs of scholars in their research and teaching endeavors. The group partners with faculty members and collaborates with technology companies and other research organizations.

Research Computing and Cyberinfrastructure Provide systems services by researching current practices in operating system, file system, data storage, job scheduling as well as computational support related to compilers, parallel computations, libraries, and other software support. Also supports visualization of large datasets by innovative means to gain better insight from the results of simulations. Enable large-scale computations and data management by building and operating several state-of-the art computational clusters and machines with a variety of architectures. Consolidate and thus significantly increase the research computing resources available to each faculty participant. Faculty members can frequently exceed their share of the machine to meet peak computing needs Provide support and expertise for using programming languages, libraries, and specialized data and software for several disciplines. Investigate emerging visual computing technologies and implement leading-edge solutions in a cost-effective manner to help faculty better integrate data visualization tools and immersive facilities in their research and instruction. Investigate emerging architectures for numerically-intensive computations and work with early-stage companies. For example: interconnects, networking, and graphics processors and FPGA for computations. Help build inter- and intra-institutional research communities using cyberinfrastructure and grid technologies. Maintain close contacts with NSF and DoE funded national centers, and help faculty members with porting and scaling of codes across multiple architectures.

Compute Engines LION-XJ 144 nodes 288 quad-core procs Memory-16GB /node IB interconnect LION-XK 144 nodes 288 quad-core procs Memory-32GB /node IB interconnect Pleiades 170 nodes 340 processors 340 GB RAM 35.1 TB storage LION-XO 132 nodes 368 processors 1280 GB RAM Silverstorm Infiniband interconnect Hammer/LION-XD 16 nodes 64 processors Memory-128GB /node Infiniband interconnect LION-XB 16 nodes 128 processors 512 GB RAM Pathscale Infinipath Infiniband interconnect LION-XC 140 nodes 560 processors 1664 GB RAM Unisys ES nodes 64 processors 192 GB RAM Myrinet interconnect HPC Storage Farm 100 TB of disk needs to be 1000 TB

Programs, Libraries, and Application Codes in Support of Computational Research Compilers and Debuggers: AbsoftProFortran, GNU Pascal, IBM XLF, IBM XLC/C++, Intel Fortran, Intel C/C++, Lahey/Fujitsu Fortran 95 Pro, Portland PGI Compilers and Tools, Java, PathScale EKO compiler suite, TotalView, DDT, Valgrind Computational Biology: BLAST, Blastall, Cister, ClustalW, ClustalX, Dotter, FASTA, fastDNAml, GeneMachine, GENSCAN, HMMgene, MrBayes, MZEF, PHRED/PHRAP/Consed, PHYLIP, ReadSeq, RepeatMasker, SEG, sim4, Sputnik, Treetool, wuBlast Computational Chemistry and Material Science: FHI98MD, Gamess, Gaussian 03, GaussView, SemiChem, NWChem, Jaguar, Maestro, CHARMM, WIEN2K, VASP, ThermoCalc, Accelrys Material Studio, ADF, tmolex, Amber, Gromacs, NAMD, WxDragon, Molden, CPMD, Rosetta, CCP4 Finite Element Solvers: ABAQUS, ANSYS, FLUENT, GAMBIT, FIELDVIEW, LS-DYNA, MD/Nastran, OpenFOAM Mathematical and Statistical Libraries and Applications: ATLAS, BLAS, ESSL, IMSL, LaPack, ScaLaPack, MASS, GOTO, Intel MKL, AMD ACML, Mathematica, MATLAB, Maple, Distributed MATLAB, PETSc, NAG, StarP, Watson Sparse Matrix Solver Solid Modeling: MD/Patran Statistics: R, SAS Parallel Libraries: MPICH, Optimized versions of MPICH for high-performance cluster interconnects, Parallel IMSL, Distributed MATLAB, StarP, Distributed Maple Optimization: GAMS / CPLEX, Csim, Tomlab Multiphysics: Comsol All software installations are driven by faculty. The software stack on every system is customized and entirely shaped by faculty needs.

Joint research and education initiative (NSF, DOE, PSU) focused on understanding molecular issues related to environmental chemical kinetics, geochemical cycling of elements, fate and transport of contaminants, and carbon sequestration. (Dr. Susan L. Brantley, Professor of Geosciences) Virtual center that integrates genetic, immunological, ecological and other studies to understand how disease processes work, and how they inter-relate across time and length scales. (Dr. Ottar Bjornstad, Associate Professor of Entomology and Biology; Dr. Bryan Grenfell, Alumni Professor of Biology ) Center for Gravitational Wave Physics (NSF, PSU) fosters research of a truly interdisciplinary character linking the highest caliber astrophysics, gravitational wave physics and experimental gravitational wave detection in the pursuit of the scientific understanding of gravity. (Dr. Lee S. Finn, Professor of Physics, Astronomy and Astrophysics) Interdisciplinary center (NSF, NIH, PSU) focused on identifying issues in statistics, research design, and measurement emerging in the prevention and treatment of problem behaviors, particularly drug abuse. (Dr. Linda M. Collins, Professor of Human Development and Family Studies, Statistics) Participating Research Centers

Collaborative effort (NSF, PSU, Georgia Tech) aimed to educate the next generation of scientists and engineers in the emerging field of materials design. (Dr. Zi-Kui Liu, Professor of Materials Science and Engineering) State Large collaboration (NSF, DOE, PSU, and others) of about 150 scientists working to use the AMANDA and IceCube telescopes to detect ultra-high energy neutrinos. (Dr. Douglas F. Cowen, Professor of Physics, Astronomy and Astrophysics) Center aimed at describing, modeling, and understanding the Earth's climate system. (Dr. Michael E. Mann, Associate Professor of Meteorology) A hub to connect experimental and simulation activities through the organization of collaborative projects, short courses and workshops. (Dr. Jorge O. Sofo, Associate Professor of Physics, Astronomy and Astrophysics) Participating Research Centers

2-layer ONIOM Method HF/6-311+G(d,p)/HF3-21G* Low levelHigh level-90.4 ppm HPC

Performance of LS-DYNA Case Study – Blast Loading Blast load using ConWep algorithm 450,000 dofs, spherical blast for 11ms Study of mesh convergence for plastic strain stability Professor: Ashok D.Belegundu Student : Vikas Argod Dept of Mechanical and Nuclear Engineering

Visualization Services Recent support areas include: Visualization system design and deployment 3D modeling and geometry exchange (e.g. FormZ) Visualization development applications and programming toolkits (e.g. OpenDX, VTK ) VR development and device libraries (e.g. VRPN, VMRL, JAVA3D, OpenGL, OpenSG, CaveLib) Domain specific visualization tools (e.g. VMD, SCIRun) Telecollaborative tools and facilities support (e.g. Access Grid, VNC) Parallel graphics and online visualization (e.g. Paraview, DCV) Programming for graphics (e.g. C/C++, JAVA3D, Tcl/Tk, Qt) Staff members provide consulting, teach seminars, assist faculty and support facilities for visualization and VR.

Visualization Facilities Locating facilities strategically across campus for convenient access by targeted disciplines and user communities Leveraging existing applications and workflows so that visualization and VR can be natural extensions to existing work habits for the users being served Providing outreach seminars, end-user training and ongoing staff support in use of the facilities Working on an ongoing basis with academic partners to develop and adapt these resources more precisely to fit their needs Helping to identify and pursue funding opportunities for further enhancement of these efforts as they take root Our goal is to foster more effective use of visualization and VR techniques in research and teaching across colleges and disciplines via strategic deployment of facilities and related support.

Immersive Environments Lab (IEL) in partnership with School of Architecture and Landscape Architecture 208 Stuckeman Family Building Focused on teaching and research in the experiential understanding of design spaces by architecture and landscape architecture students 3-screen 3D-stereo multi-OS display offers multi-modal immersive environment Experiential design review in design studio and digital media courses Telecollaborative studio using standard definition video and 3D application sharing with Carleton University, Spring 2007 (lab also supports Access Grid) Research into immersive and collaborative tools for design professions Develop application and data integration workflows for ARCH/LARCH (Building Information Modeling, energy and structural analyses, land use planning, etc.) in conjunction with ICON lab under internal Bowers support

Immersive Construction Lab (ICon) in partnership with Architectural Engineering 306 Engineering Unit C Focused on research and teaching in the use of immersive visualization and VR techniques for planning and management of large construction projects 3-screen, 3D-stereo, Windows desktop, immersive information environment VR extensions to commercial construction planning applications, custom development of “VR-like” teaching modules Industry partners provide real world use cases for studying the practical application of tools under development Linked SMART Board allows dynamic VR updates from scheduling applications, etc. Laptop display sharing facilitates group collaboration among students Building upon ICON lab, IEL and related work, Penn State hosted CONVR 2007 international conference on construction applications of virtual reality

Visualization/VR Lab, 215 Osmond in partnership with Materials Simulation Center VR facility for central campus science community (Materials Science, Molecular Biology, Physics and more) 8’ x 8’ active stereo display Tracked devices for user interaction Linux console workstation Complement of open source data visualization tools (VMD, VTK, Paraview - can be built upon in response to user needs) Seminars for teaching graduate students on use of VR tools Initial users in Materials Science and Molecular Biology

Visualization/VR Lab, 336 IST Building in partnership with Computer Science and Engineering Facility targets compute-intensive applications in science, engineering and related disciplines Large-format 3D stereo display (6.75 x 9 ft., 1400 x 1050 pixel) for VR applications 2 x 2 tiled display (6 x 8 ft., 2800 x 2100 pixel) for high-resolution applications Linux console workstations Interactive device support Initial complement of data visualization tools (VMD, VTK, Paraview, SCIRun) to be built upon in response to user needs Opened in October 2007 Seminars and outreach activity underway

Sports Medicine VR Lab assisting a partnership between Kinesiology, Athletics and HMC Lasch Football Building Special purpose lab supports study of perception action disruptions in posture and balance related to mild traumatic brain injury (e.g. concussions), elderly populations, etc. Motion in VR display is synchronized with measurement from EEG, postural tracking and force plate instrumentation Enhanced two-screen lab (wall and floor) in development for Recreation Building for broader use by faculty in kinesiology, psychology, engineering science and mechanics, SSRI and Hershey Medical Center

Collaborative Tools in Research Telecollaboration –Adobe Connect (site license) –Vyew –Access Grid (many-to-many) Document / Source Sharing –Subversion –Wiki MediaWiki (public domain) Confluence Wiki (commercial) Web-Based Science Gateways –Materials Simulation Center Gateway (with Materials Research Institute)

ACCESS Grid Teleconferencing Facility Room 140 Computer Building Allows faculty to participate in Access Grid events with international academic community Scalable, multi-group telecollaboration (voice, video, application sharing) using multicast internet connections. Small number of highly satisfied users Needs greater awareness and adoption Ongoing research collaboration: Dean Snow (ANTHY), Craig Cameron (BMB) Research reporting: Richard Alley (GEOSCI), Donald Bryant (BMB), Mark Gahegan (GEOG) Virtual conferences: Genomics and Bioinformatics, SC Global, SC Desktop

Industry Outreach in partnership with Institute for Computational Science (ICS) and Industrial Research Office (IRO) "... to out-compete, we must out-compute.... " Putting Pennsylvania based companies at a competitive advantage by helping meet their large-scale computing needs. Many small-to-medium sized companies, and even larger ones, do not have enough in-house expertise and resources for large-scale computations and as a result have not been able to use simulation and analysis tools with far greater frequency to help them innovate faster and become more competitive. Helping develop stronger relationship between faculty and industry by providing computational services; may lead to more alignment between faculty research areas and industry needs and positive impact on economic development along the I-99 corridor and throughout the commonwealth.