30 Years of Navy Modeling and Supercomputers: an Anecdotal History Tom Rosmond Marine Meteorology Division Naval Research Laboratory Monterey, California USA

Outline Before my Time: My Early Years: The Golden Years of NWP: late 1970’s-early 1980’s Up to the Present Conclusions and Thoughts for the Future

Before my Time: : Capt Wolff sets up shop on NPS Campus, Monterey CDC 1604: Seymour Cray’s first design with CDC (first supercomputer?) 1961: Establishment of Fleet Numerical Weather Facility (FNWF) 1964: Routine dissemination of numerical products Successive correction analyses of SLP, upper air heights NH Barotropic and thickness advection models THE BEGINNING

Before my Time: : FNWC procurement of CDC 6500 to replace 1604 Dual-processors, comparable to 386 microprocessor in FP performance Supercomputer of time was Seymour Cray’s CDC 6600 (NMC in 1966) 1969: Second 6500 acquired 1970: Northern Hemisphere Primitive Equation (NHPE) model operational on 4 processors. World’s first multi-processor production code CDC 6500’s and NHPE

Before my Time: Developed by P. Kessel and F. Winninghoff Processors shared model data through extended core storage (ECS) OS modifications to allow processor synchronization Similarities to both shared memory (OpenMP) and distributed memory (MPI) programming models Parallel efficiency = ~ 75% An impressive achievement that is certainly underappreciated An example of the struggle it was to fit models onto these early systems 4-processor 6500 NHPE

My Early Years: My organization: EPRF NEPRF NOARL NRL-Monterey 1974: I joined the Environmental Prediction Research Facility (EPRF) 1 Little involvement in FNWC 2 model development Access of FNWC computer systems (6500’s, 1604) 1976: NEPRF Numerical Modeling Department formed at request of FNWC CO Capt. R. Hughes He realized that FNWC was unable to maintain R & D continuity needed for NWP system development. Need for a global NWP capability was major motivation, and he was committed to getting computer system to support it. 1976: FNWC acquired CDC CYBER 175 (similar to CDC 7600) 2 Fleet Numerical: FNWF FNWC FNOC FNMOC

My Early Years: NEPRF Global NWP System Development 1976: Navy Operational Global Atmospheric Prediction System (NOGAPS) UCLA General Circulation Model Barnes Successive Corrections Global Analysis Variational Initialization with Balance Equation Constraint 1977: We also had spectral model dynamical core UCLA GCM physics Sat on shelf for several years in favor of UCLA GCM based system 1980: Prototype NOGAPS running on CYBER 175

My Early Years: : Benchmarking for FNWC procurement NCAR Cray-1, first system outside Los Alamos? UCLA GCM x 3.0 0, 6 levels First introduction to vector programming Crude compiler + non-vectorizable code = poor performance Overly conservative target performance, CYBER 203 could compete Allowed subsequent CDC success with CYBER : CYBER 203 delivered to FNOC Heroic work by CDC to get UCLA model to run fast 1982: CYBER 203 replaced with CYBER 205

The Golden Years of NWP 1 st half: late 70’s – mid 80’s Supercomputers were cheapest computers you could buy (price/performance) Cray: Cray-1, XMP, etc CDC: CYBER 203/205, ETA-10 IBM: VP Development of global NWP forecast systems Dominance of spectral models Increased realization of importance of data assimilation Establishment of ECMWF “Raised the stakes” in operational NWP Accelerated progress Provided gold standard

The Golden Years of NWP Navy operational models NOGAPS UCLA-GCM x3.00 x L9 : 1-pipe/8Mbyte C205 : ~25min/fcstday Spectral – T47L18 : 2-pipe/32Mbyte C205 : ~6min/fcstday Spectral – T79L18 : 2-pipe/32Mbyte C205 : ~25min/fcstday (32bit) Spectral – T79L24 : 4-pipe/64Mbyte C205 : ~12min/fcstday (32bit) NORAPS (developed by Rich Hodur, NEPRF) Globally relocatable regional model 6-8 areas run operationally Varying resolution, domain and grid sizes e.g. ran over South Atlantic during Falklands war (1980)

Some special comments about the CYBER 205 CDC aggressively pursued meteorology market, both operations and research FNOC, NMC, UKMO, GSFC, GFDL, etc CRAY systems were better general purpose, but 205 excelled on our applications Initially 205 was difficult to use, compiler/user software was rudimentary, but Language extension showed how machine worked Rich array of exotic vector hardware instructions System software matured 32 bit/64 bit floating point support Easy mixing of Fortran and explicit vector instructions, we could get “close to the hardware” Spectacular percentage of peak performance possible

The Golden Years of NWP 2 nd half: mid 80’s – early 90’s Introduction of multi-processor systems Divide and conquer X/MP, Y/MP, ETA-10 Multi-tasking/vectorization programming model (parallel/vector) Spelled the end of single processor supercomputers, e.g. CYBER 205 Important changes in technology Price/performance advantage of supercomputers ending Introduction of desktop workstations Many people didn’t need supercomputers, just cheap computing Shrinking supercomputer market

Up to the Present: early 1990’s 1990: Introduction of CRAY C90 The “best” supercomputer ever? Run by a higher percentage of NWP operational/research centers than any system before or since. Parallel/vector programming model very user friendly Easy to get high percentage of peak performance 1991: 8 processor C90 at FNMOC NOGAPS: T159L24 – 6 processors : 10 min/fcstday 1996: 8 and16 processor C90’s at FNMOC NOGAPS: T159L24 – 12 processors : 6 min/fcstday 1997: COAMPS TM replaced NORAPS as Navy’s regional forecast system

Up to the Present Mid to Late 1990’s Beginnings of dramatic changes in supercomputer industry Scalable commodity based architectures appearing, e.g. CRAY T3E Powerful workstations replacing supercomputers for many applications End of domination by American supercomputer vendors CDC/ETA were long gone CRAY sold to SGI MTTB (mean time to bankruptcy) very short for new companies Rise of Japanese vendors to dominate NWP marketplace, at least outside U.S. Fujitsu NEC

Up to the Present 2000 to today Proliferation of commodity based, scalable architectures for NWP applications Many T3E’s still in use SGI: Origin 2000, Origin 3000 IBM: SP3, SP4 Linux clusters are also viable alternatives, especially when price/performance is overriding issue But, rumors of the demise of vector architectures are exaggerated 2000: NEC only vendor with traditional vector architecture, SX-6, but 2002: Resurrected CRAY, Inc introduced X-1 Price/performance vs ultimate performance is central question concerning long-term prospects of these systems

Up to the Present 2000 to today 2001: FNMOC replaced C90’s with O3000’s 1152 total processors NOGAPS: T239L30 – 120 processors: 6 min/fcstday COAMPS TM : 8-11 areas – processors: min/fcstday 2003: Operational NAVDAS/NOGAPS (3DVAR): 60 processors 2004: Direct radiance retrievals with NAVDAS 2004: Currently under development at NRL, Monterey T479L54 semi-Lagrangian NOGAPS NAVDAS-AR: 4DVAR extension of NAVDAS COAMPS TM /WRF: more areas, higher resolution Clearly computational requirements are never satisfied!!!

Conclusions and thoughts for the future Vector architectures will continue to be viable candidates for NWP applications Sharing some features with commodity systems, e.g. caches When ultimate performance overrides price/performance Will be part of heterogeneous computing environments Scientific computing ( and therefore NWP) is now niche market Be thankful that video game applications share many of our requirements Consumer based industry has driven some hardware costs to astonishingly low levels, e.g. hard drives We must work with vendors to ensure that our requirements are not forgotten Fortran compilers High-performance interconnects