1. High-Performance Computing, Computational Science, and NeuroInformatics Research Allen D. Malony Department of Computer and Information Science NeuroInformatics Center (NIC) Computational Science Institute University of Oregon

2. April 29, 2004PNNL UO Visit Outline  High-performance computing research  Interactions and funding  Project areas  TAU parallel performance system  Computational science at UO  Projects  Computational Science Institute  Neuroinformatics research  NeuroInformatics Center (NIC)  ICONIC Grid

3. April 29, 2004PNNL UO Visit High-Performance Computing Research  Strong associations with DOE national laboratories  Los Alamos National Lab  Lawrence Livermore National Lab  Sandia National Lab (Livermore)  Argonne National Lab  National Energy Research Supercomputing Center  DOE funding  Office of Science, Advance Scientific Computing  ASCI/NNSA  NSF funding  Academic Research Infrastructure  Major Research Instrumentation

4. April 29, 2004PNNL UO Visit Project Areas  Parallel performance evaluation and tools  Parallel language systems  Tools for parallel system and software interaction  Source code analysis  Parallel component software  Computational services  Grid computing  Parallel modeling and simulation  Scientific problem solving environments

5. Allen D. Malony Sameer S. Shende Department of Computer and Information Science Computational Science Institute University of Oregon TAU Parallel Performance System

6. April 29, 2004PNNL UO Visit Parallel Performance Research  Tools for performance problem solving  Empirical-based performance optimization process characterization Performance Tuning Performance Diagnosis Performance Experimentation Performance Observation hypotheses properties Instrumentation Measurement Analysis Visualization Performance Technology

7. April 29, 2004PNNL UO Visit Complexity Challenges for Performance Tools  Computing system environment complexity  Observation integration and optimization  Access, accuracy, and granularity constraints  Diverse/specialized observation capabilities/technology  Restricted modes limit performance problem solving  Sophisticated software development environments  Programming paradigms and performance models  Performance data mapping to software abstractions  Uniformity of performance abstraction across platforms  Rich observation capabilities and flexible configuration  Common performance problem solving methods

8. April 29, 2004PNNL UO Visit General Problems How do we create robust and ubiquitous performance technology for the analysis and tuning of parallel and distributed software and systems in the presence of (evolving) complexity challenges? How do we apply performance technology effectively for the variety and diversity of performance problems that arise in the context of complex parallel and distributed computer systems? 

9. April 29, 2004PNNL UO Visit TAU Performance System  Tuning and Analysis Utilities  Performance system framework for scalable parallel and distributed high-performance computing  Targets a general complex system computation model  nodes / contexts / threads  Multi-level: system / software / parallelism  Measurement and analysis abstraction  Integrated toolkit for performance instrumentation, measurement, analysis, and visualization  Portable performance profiling and tracing facility  Open software approach with technology integration  University of Oregon, Forschungszentrum Jülich, LANL

10. April 29, 2004PNNL UO Visit TAU Performance System Architecture EPILOG Paraver

11. April 29, 2004PNNL UO Visit TAU Performance System Status  Computing platforms  IBM SP / Power4, SGI Origin 2K/3K, ASCI Red, Cray T3E / SV-1 / X-1, HP (Compaq) SC (Tru64), HP Superdome (HP-UX), Sun, Hitachi SR8000, NEX SX- 5/6, Linux clusters (IA-32/64, Alpha, PPC, PA-RISC, Power, Opteron), Apple (G4/5, OS X), Windows  Programming languages  C, C++, Fortran 77/90/95, HPF, Java, OpenMP, Python  Communication libraries  MPI, PVM, Nexus, shmem, LAMPI, MPIJava  Thread libraries  pthreads, SGI sproc, Java,Windows, OpenMP

12. April 29, 2004PNNL UO Visit TAU Performance System Status (continued)  Compilers  Intel KAI (KCC, KAP/Pro), PGI, GNU, Fujitsu, Sun, Microsoft, SGI, Cray, IBM (xlc, xlf), Compaq, Hitachi, NEC, Intel  Application libraries (selected)  Blitz++, A++/P++, PETSc, SAMRAI, Overture, PAWS  Application frameworks (selected)  POOMA, MC++, ECMF, Uintah, VTF, UPS, GrACE  Performance technology integrated with TAU  PAPI, PCL, DyninstAPI, mpiP, MUSE/Magnet  TAU full distribution (Version 2.x, web download)  TAU performance system toolkit and user’s guide  Automatic software installation and examples

13. April 29, 2004PNNL UO Visit Computational Science Computer Science Biology Neuroscience Psychology Paleontology Geoscience Math  Integration of computer science in traditional science disciplines  Third model of scientific research  Application of high-performance computation, algorithms and networking  Parallel computing  Grid computing

14. April 29, 2004PNNL UO Visit Computational Science Projects at UO  Geological science  Model coupling for hydrology  Bioinformatics  Zebrafish Information Network (ZFIN)  Evolution of gene families  Oregon Bioinformatics Tool  Neuroinformatics  Electronic notebooks  Domain-specific problem solving environments  Dinosaur skeleton and motion modeling  Computational Science Institute

15. April 29, 2004PNNL UO Visit Computational Science  Cognitive Neuroscience  Computational methods applied to scientific research  High-performance simulation of complex phenomena  Large-scale data analysis and visualization  Understand functional activity of the human cortex  Multiple cognitive, clinical, and medical domains  Multiple experimental paradigms and methods  Need for coupled/integrated modeling and analysis  Multi-modal (electromagnetic, MR, optical)  Physical brain models and theoretical cognitive models  Need for robust tools: computational & informatic

16. April 29, 2004PNNL UO Visit Brain Dynamics Analysis Problem  Identify functional components  Different cognitive neuroscience research contexts  Clinical and medical applications  Interpret with respect to physical and cognitive models  Requirements: spatial (structure), temporal (activity)  Imaging techniques for analyzing brain dynamics  Blood flow neuroimaging (PET, fMRI)  good spatial resolution  functional brain mapping  temporal limitations to tracking of dynamic activities  Electromagnetic measures (EEG/ERP, MEG)  msec temporal resolution to distinguish components  spatial resolution sub-optimal (source localization)

17. April 29, 2004PNNL UO Visit Integrated Electromagnetic Brain Analysis Individual Brain Analysis Structural / Functional MRI/PET Dense Array EEG / MEG Constraint Analysis Head Analysis Source Analysis Signal Analysis Response Analysis Experiment subject temporal dynamics neural constraints Cortical Activity Model Component Response Model spatial pattern recognition temporal pattern recognition Cortical Activity Knowledge Base Component Response Knowledge Base good spatial poor temporal poor spatial good temporal neuroimaging integration

18. April 29, 2004PNNL UO Visit Experimental Methodology and Tool Integration source localization constrained to cortical surface processed EEG BrainVoyager BESA CT / MRI EEG segmented tissues 16x256 bits per millisec (30MB/m) mesh generation EMSE Interpolator 3D NetStation

19. April 29, 2004PNNL UO Visit NeuroInformatics Center (NIC)  Application of computational science methods to cognitive and clinical neuroscience problems  Understand functional activity of the brain  Help to diagnosis brain-related disorders  Utilize high-performance computing and simulation  Support large-scale data analysis and visualization  Advance techniques for integrated neuroimaging  Coupled modeling (EEG/ERP and MR analysis)  Advanced statistical factor analysis  FDM/FEM brain models (EEG, CT, MRI)  Source localization  Problem-solving environment for brain analysis

20. April 29, 2004PNNL UO Visit NIC Organization  Director, Allen D. Malony  Associate Professor, Computer and Information Science  Associate Director, Don M. Tucker  Professor, Psychology; CEO, EGI  Computational Scientist, Kevin Glass  Ph.D., Computer Science; B.S., Physics  Computational Physicist, Sergei Turovets  Ph.D., Computer Science; B.S., Physics  Computer Scientist, Sameer S. Shende  Ph.D., Computer Science; parallel computing specialist  Mathematician, Bob Frank  M.S., Mathematics

21. April 29, 2004PNNL UO Visit Funding Support  BBMI federal appropriation  DoD Telemedicine Advanced Technology Research Command (TATRC)  Initial budget of approximately $750K  Oct. 1, 2002 through March 31, 2004  NSF Major Research Instrumentation  ICONIC Grid, awarded  New proposal opportunities  NIH Human Brain Project Neuroinformatics  NSF ITR

22. April 29, 2004PNNL UO Visit NIC Approaches  Optimize spatial resolution  MRI structural information  Measurement of skull conductivity  Convergence / co-recording with MEG and fMRI  Optimize temporal resolution  Use EEG/MEG time course for fMRI signal extraction  Decomposition of component analysis (ICA, PCA)  Single-trial analysis  Computational brain models  Boundary and finite element brain models  Brain information databases and atlases

23. April 29, 2004PNNL UO Visit EEG/ERP Methodology  Electroencephalogram (EEG)  Event-Related Potential (ERP)  Stimulus-locked measures of brain dynamics  Generated from subject- and trial-based analysis  Raw EEG datasets processed and analyzed  Segmentation to time series waveforms  Blink removal and other cleaning  ERP analysis  Averaging for increasing signal to noise  Characterization with respect to trial conditions  Results visualization  Source localization

24. April 29, 2004PNNL UO Visit EGI Geodesics Sensor Net  Electrical Geodesics Inc.  Dense-array sensor technology  64/128/256 channels  256-channel geodesics sensor net  AgCl plastic electrodes  Carbon fiber leads  Future optical sensors  EGI + LANL

25. April 29, 2004PNNL UO Visit EEG/ERP Experiment Management System  Support EEG-based cognitive neuroscience research  Based on experiment model  Experiment type  Subjects measured for trial types  Management of experiment data  Raw and processed datasets and derived statistics  Per experiment/subject/trial database  Secure protection and storage with selective access  Analysis tools and workflows  Generation of results (across experimental variables)  Analysis processes with multi-tool workflows

26. April 29, 2004PNNL UO Visit EEG/ERP Experiment Analysis Environment …… raw processed datasets / derived results analysis workflow storage resources virtual services compute resources

27. April 29, 2004PNNL UO Visit Source Localization  Mapping of scalp potentials to cortical generators  Single time sample and time series  Requirements  Accurate head model and physics  High-resolution 3D structural geometry  Precise tissue identification and segmentation  Correct tissue conductivity assessment  Computational head model formulation  Finite element model (FEM)  Finite difference model (FDM)  Forward problem calculation  Dipole search strategy

28. April 29, 2004PNNL UO Visit Advanced Image Segmentation  Native MR gives high gray-to-white matter contrast  Edge detection finds region boundaries  Segments formed by edge merger  Color depicts tissue type  Investigate more advance level set methods and hybrid methods

29. April 29, 2004PNNL UO Visit Building Finite Element Brain Models  MRI segmentation of brain tissues  Conductivity model  Measure head tissue conductivity  Electrical impedance tomography  small currents are injected between electrode pair  resulting potential measured at remaining electrodes  Finite element forward solution  Source inverse modeling  Explicit and implicit methods  Bayesian methodology scalp CSF skull cortex

30. April 29, 2004PNNL UO Visit Conductivity Modeling Governing Equations ICS/BCS Discretization System of Algebraic Equations Equation (Matrix) Solver Approximate Solution Continuous Solutions Finite-Difference Finite-Element Boundary-Element Finite-Volume Spectral Discrete Nodal Values Tridiagonal ADI SOR Gauss-Seidel Gaussian elimination  (x,y,z,t) J (x,y,z,t) B (x,y,z,t)

31. April 29, 2004PNNL UO Visit Source Localization Analysis Environment …… raw storage resources virtual services compute resources

32. April 29, 2004PNNL UO Visit NIC Computational Cluster (“Neuronic Cluster”)  Dell computational cluster  16 dual-processor nodes  2.8 MHz Pentium Xeon  4 Gbyte memory  36 Gbyte disk  Dual Gigabit ethernet adaptors  2U form factor  Master node (same specs)  2 Gigabit ethernet switches  Brain modeling  Component analysis

33. April 29, 2004PNNL UO Visit NIC Relationships Psychology CIS BDLBEL CSI OHSU / OGI Utah UCSD USC AcademicLabs / Centers LANLArgonne NCSA Internet2 EGI Industry IntelIBM NIC UO Departments UO Centers/Institutes BBMI CDSI CNI Physics NSI Sandia

34. April 29, 2004PNNL UO Visit NSF MRI Proposal  Major Research Instrumentation (MRI)  “Acquisition of the Oregon ICONIC Grid for Integrated COgnitive Neuroscience Informatics and Computation”  PIs  Computer Science: Malony, Conery  Psychology: Tucker, Posner, Nunnally  Senior personnel  Computer Science: Douglas, Cuny  Psychology: Neville, Awh, White  Approximately $1.2M over three years

35. April 29, 2004PNNL UO Visit SMP Server IBM p655 Graphics SMP SGI MARS SAN Storage System Gbit Campus Backbone NICCIS Internet 2 Shared Memory IBM p690 Distributed Memory IBM JS20 CNI Distributed Memory Dell Pentium Xeon NIC 4x816 2x82x16 graphics workstationsinteractive, immersive vizother campus clusters ICONIC Grid 5 Terabytes

36. April 29, 2004PNNL UO Visit Cognitive Neuroscience and ICONIC Grid  Common questions to be explored  Identifying brain networks  Critical periods during normal development  Network involvement in psychopathologies  Training interventions in network development  Research areas  Development of attentional networks  Brain plasticity in normal development and deprived  Attention and emotion regulation  Spatial working memory and selective attention  Attention and psychopathology

37. April 29, 2004PNNL UO Visit Computer Science and ICONIC Grid  Scheduling and resource management  Assign hardware resources to computation tasks  Scheduling of workloads for  PSEs for computational science  Provide scientists an entrée to the computational and data management power of the infrastructure without requiring specialized knowledge of parallel execution  Marine seismic tomograph, molecular evolution  Interactive / immersive three-dimensional visualization  Explore multi-sensory visualization  Merge 3D graphics with force-feedback haptics  Parallel performance evaluation