Presentation on theme: "Providing HPC Resources for Philips Research and Partners Ronald van Driel Senior Technologist Research ICT April 23 rd, 2007."— Presentation transcript:
Providing HPC Resources for Philips Research and Partners Ronald van Driel Senior Technologist Research ICT April 23 rd, 2007
2 Research Agenda Introduction Philips and Research Research Interest in HPC HPC and Grid Related Activities Vision on Future Directions
3 Research Royal Philips Electronics One of the largest global electronics company with sales in –2006 of EUR 26,976 million –Q4 2006 of EUR 8,128 million Founded in 1891 Multinational workforce of 121,700 employees (January 2007) Active in the areas of Healthcare, Lifestyle and Technology Manufacturing sites in 28 countries, sales outlets in 150 countries R&D expenditures EUR 1,619 mln (2006) Headquarters: Amsterdam, The Netherlands
4 Research Significant new product introductions in 2006/2007 Medical SystemsLighting Consumer Electronics DAP EP Navigator Gemini Time-of-Flight PET/CT SureSigns VM BrightView SPECT Aluminum Juicer Wardrobe care Williams F1 shaver Wake up light VOIP phone Portable Media Devices Ambient Experience HD Category ActiLume Edore Living Colours Luxeon ® K2LED Cosmopolis
9 Research World-class technology centre of high tech companies working together in the development of new technologies 910,000 sq.m Over 50 nationalities 7,000-8,000 people by 2008 more than EUR 500 million investment by Philips Comprises a broad set of world- class facilities and expertise, that are shared with partners: Micro-and Nanotechnology infrastructure and services (MiPlaza), HD Studio, EMC 3, LSF, etc.
10 Research 10 High Tech Campus Eindhoven Strength through diversity & networks Corporate innovators Consultancy & services Research institutes Economic development companies
11 Research Global Research ICT Organization CIO Patrick Stemkens Adaptive & Dedicated ICT (ADICT) Global Infrastructures Infrastructure Development Infrastructure Management Business Information Systems Information Services
12 Research Agenda Introduction Philips and Research Research Interest in HPC HPC and Grid Related Activities Vision on Future Directions
13 Research History of Central Compute Facilities at Philips Research Eindhoven 198019851990199520002005 IBM Mainframe – VM/CMS & MVS VAX/VMS Apollo Workstations GRID Central Compute Cluster (NXA)HP-UXLinux Silicon Graphics – Video/Audio/Graphics
14 Research Observations I Developments in application fields e.g. Medical Imaging and Bioinformatics –Many sensors and high-resolution sensors: data explosion –Correlation of data from different sources (e.g. SPECT/CT) Open Innovation –Need to cooperate to get resources and/or knowledge in the right place at the right time: “Support the technology chain” Infrastructure –Cope with the “data explosion” and “large scale data analysis” –Large computations, but much can be parallelized or distributed –Must facilitate secure collaboration with partners From model driven to data driven experiments
15 Research Observations II Cannot be done by a single entity Must share and cooperate Big growth in application complexity. Resource needs of some application fields may grow orders of magnitude Will lead to huge growth in data storage and compute capacity needs. And today's business innovation is done in an Open Innovation setting: Collaboration in virtual teams is the way of working. (Industry, Universities & Institutes) Needs to be supported by applications and IT infrastructures.
16 Research IT Resources Importance I Storage: The ability to cope with very large and often distributed datasets. Visualization: "Virtual reality" techniques are becoming an increasingly important part of applications or workflows.
17 Research IT Resources Importance II Compute: A substantial increase in demand for computer power and the ability to run data intensive tasks. Networks: Enabling technology for providing fast access to distributed resources.
18 Research Research Interest in HPC? A number of program, project and application fields at Philips Research will require (or may benefit from) high performance computing: –Medical imaging to improve image quality and accuracy. –Molecular Medicine and Bioinformatics will be faced with an explosive growth of data. –Large simulations and inverse problems Lighting armature optics for solid state lighting –System in Package (SIP) and Solid State Lighting may need to introduce 3D models and simulations. HPC capabilities might yield attractive added value for potential partners of Philips Research. –Important to be part of the ECO system
19 Research Agenda Introduction Philips and Research Research Interest in HPC HPC and Grid Related Activities Vision on Future Directions
20 Research Research ICT HPC and Grid Activities Philips Research ICT is trying to close the gap between “demanding” jobs that researchers have and the supporting IT infrastructures. Translate upcoming IT technology in new strategic options Developed various HPC/grid demonstrator aimed at helping researchers to do their calculations faster… Installed a cluster that is connect to DutchGrid to understand and investigate the technology –Access via “gLite” or “UNICORE” grid middleware software –Allows for more demanding, jobs to be sent to other (possibly external) “clusters” Participate in “Virtual Laboratory for e-Science” (VL-e) and “BIG-GRID” projects
21 Research Example: Researchers Workflow Short cycles require maximum flexibility: PC/WS or local cluster Secure access to large data and compute resources: shared on a site. Debug possibilities Usually executed on a “Site Cluster” Full simulations using specialized data, compute and visualization infrastructures. E.g. medical imaging, bioinformatics simulation and search, 3D multi-physics modeling, system in package simulation, etc. Many projects do not follow the complete flow, but stay at one or more stages. Application dependent solutions e.g. dedicated computers or a bunch of chips Workflow View Experimenting with algorithm or numerical set-up Validation and regression tests Pre-production tests with full implementation Production implementation
22 Research Example: Researchers Workflow Short cycles require maximum flexibility: PC/WS or local cluster Secure access to large data and compute resources: shared on a site. Debug possibilities This “Site Cluster” is part of “DutchGrid”! Full simulations using specialized data, compute and visualization infrastructures. E.g. medical imaging, bioinformatics simulation and search, 3D multi-physics modeling, system in package simulation, etc. Many projects do not follow the complete flow, but stay at one or more stages. Application dependent solutions e.g. dedicated computers or a bunch of chips Workflow View Experimenting with algorithm or numerical set-up Validation and regression tests Pre-production tests with full implementation Production implementation Site (small) cluster at HTC Eindhoven Part of VL-e infrastructure Infrastructure provider for VL-e Maximum control and fast feedback Priority setting and security can be guaranteed
23 Research Grid Demonstrator – Aachen / Eindhoven PET System Simulation Monte Carlo simulation of the scanner up to the detection of scintillation light –~32 CPU days on 2.8 GHz Xeon –100 x 1 GB output System-level simulation of the detector electronics (Mona/Lisa) –2 phase post processing Simulation can be highly parallelized
24 Research PET Simulation Workflow Schematic motron I2O2 motron I..O.. motron I1O1 I Motron SIN S Mona MCA C2 MCA C.. MCA C1 1 GB 80 MB 100 MB < 1 KB
25 Research Cluster/Server/WorkStation Meta-Scheduler / Scheduler OpenPBS, SGE, LSF, Condor, Others (SDK) GridAgent GridXpert Synergy : Architecture GridManager (J2EE) PKI RDBMS GridXpert Synergy: GridManager –Full J2EE standards –Embedded Certificate Authority / PKI for all resources and users –API available Enterprise portal (EIP) Import/export XML modelization GridAgent –Job Management –3rd party data transfer between GridAgent (GridFTP) –Based on Globus development –Integration with lead schedulers Web Browser Jdbc HTTPS / RSL Globus HTTPS / HTML Soap Application XML CLI Meta-Scheduler / Scheduler OpenPBS, SGE, LSF, Condor, Others (SDK) GridAgent
26 Research Philips jobs at NIKHEF SPECT Simulation 20,000 jobs of ~3.2 CPU hours each Executed at NIKHEF and SARA clusters Access via LCG-2 Grid middleware software Single Photon Emission Computed Tomography
27 Research Grid Portal for PET System Simulation Positron Emission Tomography Application developed by Researcher in Aachen. Design (graphical/web) portals –Core services too complex to present to scientists Based on GridSphere web portal technology (JSR-168) Initial version developed with assistance from GridwiseTech
28 Research Simulate Full HDTV Data Processing Chain Picture quality enhancement Picture rate-up conversion Pixel plus … InputOutputAlgorithm 1Algorithm n ~0.5 TB~1500 CPU hours~2.2 TB Original (Frame repetition) Measured > 50 MB/s data transfer rates between Amsterdam and Eindhoven over GigaPort connection
29 Research GAMA Research Healthcare Systems Architecture research group Windows IDL, Pride Linux Globus Gateway Client LAN DutchGrid Resources The GAMA architecture: computational Grids Architecture for solving compute-intensive medical applications Minimally invasive: Running on Grid as alternative, easy fall back to local versions Simultaneously provides different sets of services to multiple users and applications Adaptive to various healthcare applications Example: Brain imaging, the High Angular Fiber tracking (HAFT) application Clinical trials of HAFT software at AMC are being executed on VL-e cluster at HTC in Eindhoven.
30 Research VL-e Cluster HTC Eindhoven Super Compute Centers BIG GRID, Sara, Jülich, etceteras Fast Internet Bioinformatics infrastructure Internet DutchGrid Secure Environment extern Philips Compute Cluster Cluster Load Balancing Grid Access Layer External Oracle Database Compute servers
31 Research UNICORE Uniform Interface to Computing Resources Speedup Monte Carlo based simulation package developed on Microsoft Windows platform Ported application to Linux Allows for easy parallelization using loosely coupled jobs Deploy UNICORN client on Microsoft Windows platform Python script to create XML file that is loaded in OpenMolGRID Project defined to developed application specific UNICORE plug-in
32 Research Agenda Introduction Philips and Research Research Interest in HPC HPC and Grid Related Activities Vision on Future Directions
33 Research Vision: Shared Resources and Infrastructure End-user view Applications “flow through infrastructure” and can be run and used anywhere: Boosts sharing and collaboration Users decide whether to keep data in internal secure environment or to put in shared space Applications might be split in secure and non- secure parts
34 Research Vision: Shared Resources and Infrastructure Infrastructure view Grid is next generation shared infrastructure comprising distributed data storage capacity, (visualization) equipment and compute clusters: –Capitalize on our extensive experiences with centralized compute and data clusters –Other equipment will shift to the shared infrastructure too (e.g. VL-e)
35 Research Vision: Shared Resources and Infrastructure Success factors End-user guidance and support is essential!! Deploy workbenches for application fields Continuous development, be a leader Standardisation of middleware layer –e.g. gLite, UNICORE and Globus QoS and AAA are key in e-Science production infrastructures Basic building blocks are data storage and (low latency) compute clusters