Download presentation

Presentation is loading. Please wait.

Published byScott Delaware Modified about 1 year ago

1
Designing a cluster for geophysical fluid dynamics applications Göran Broström Dep. of Oceanography, Earth Science Centre, Göteborg University.

2
Our cluster (me and Johan Nilsson, Dep. of Meterology, Stockholm University) Grant from the Knut & Alice Wallenberg foundation (1.4 MSEK) 48 cpu cluster Intel P Ghz 500 Mb 800Mhz Rdram SCI cards Delivered by South Pole Run by NSC (thanks Niclas & Peter)

3
What we study

4
Geophysical fluid dynamics Oceanography Meteorology Climate dynamics

5
Thin fluid layers Large aspect ratio

6
Highly turbulent Gulf stream: Re~10 12

7
Large variety of scales Parameterizations are important in geophysical fluid dynamics

8
Timescales Atmospheric low pressures: 10 days Seasonal/annual cycles: years Ocean eddies:0.1-1 year El Nino: 2-5 years. North Atlantic Oscillation: 5-50 years. Turnovertime of atmophere: 10 years. Anthropogenic forced climate change: 100 years. Turnover time of the ocean: years. Glacial-interglacial timescales: years.

9
Some examples of atmospheric and oceanic low pressures.

10
Timescales Atmospheric low pressures: 10 days Seasonal/annual cycles: years Ocean eddies:0.1-1 year El Nino: 2-5 years. North Atlantic Oscillation: 5-50 years. Turnovertime of atmophere: 10 years. Anthropogenic forced climate change: 100 years. Turnover time of the ocean: years. Glacial-interglacial timescales: years.

11
Normal state

12
Initial ENSO state

13
The ENSO state

14

15
Timescales Atmospheric low pressures: 10 days Seasonal/annual cycles: years Ocean eddies:0.1-1 year El Nino: 2-5 years. North Atlantic Oscillation: 5-50 years. Turnovertime of atmophere: 10 years. Anthropogenic forced climate change: 100 years. Turnover time of the ocean: years. Glacial-interglacial timescales: years.

16
Positive NAO phase Negative NAO phase

17

18
Positive NAO phase Negative NAO phase

19

20
Timescales Atmospheric low pressures: 10 days Seasonal/annual cycles: years Ocean eddies:0.1-1 year El Nino: 2-5 years. North Atlantic Oscillation: 5-50 years. Turnovertime of atmophere: 10 years. Anthropogenic forced climate change: 100 years. Turnover time of the ocean: years. Glacial-interglacial timescales: years.

21
Temperature in the North Atlantic

22
Timescales Atmospheric low pressures: 10 days Seasonal/annual cycles: years Ocean eddies:0.1-1 year El Nino: 2-5 years. North Atlantic Oscillation: 5-50 years. Turnovertime of atmophere: 10 years. Anthropogenic forced climate change: 100 years. Turnover time of the ocean: years. Glacial-interglacial timescales: years.

23
Ice coverage, sea level

24
What model will we use?

25
MIT General circulation model

26
General fluid dynamics solver Atmospheric and ocean physics Sophisticated mixing schemes Biogeochemical modules Efficient solvers Sophisticated coordinate system Automatic adjoint schemes Data assimilation routines Finite difference scheme F77 code Portable

27
MIT General circulation model Spherical coordinates“Cubed sphere”

28
MIT General circulation model General fluid dynamics solver Atmospheric and ocean physics Sophisticated mixing schemes Biogeochemical modules Efficient solvers Sophisticated coordinate system Automatic adjoint schemes Data assimilation routines Finite difference scheme F77 code Portable

29
MIT General circulation model

30

31

32

33

34

35

36

37
Some computational aspects

38
Some tests in INGVAR (32 AMD 900 Mhz cluster)

39
Experiments with 60*60*20 grid points

40

41

42
Experiments with 120*120*20 grid points

43
MM5 Regional atmospheric model

44

45

46
Choosing cpu’s, motherboard, memory, connections

47
Specfp (swim)

48
Run time on different nodes

49
Choosing interconnection (requires a cluster to test) Based on earlier experience we use SCI from Dolphinics (SCALI)

50
Our choice Named Otto SCI cards P GHz (single cpus) 800 Mhz Rdram (500 Mb) Intel motherboards (the only available) 48 nodes NSC (nicely in the shadow of Monolith)

51
Otto (P GHz)

52
Scaling Otto (P GHz) Ingvar (AMD 900 MHz)

53
Why do we get this kind of results?

54
Time spent on different “subroutines” 60*60*20120*120*20

55
Relative time Otto/Ingvar

56
Some tests on other machines INGVAR: 32 node, AMD 900 MHz, SCI Idefix: 16 node, Dual PIII 1000 MHz, SCI SGI 3800: 96 Proc. 500 MHz Otto: 48 node, P Mhz, SCI ? MIT, LCS: 32 node, P Mhz, MYRINET

57
Comparing different system (120*120*20 gridpoints)

58

59
Comparing different system (60*60*20 gridpoints)

60
SCI or Myrinet? 120*120*20 gridpoints

61
SCI or Myrinet? 120*120*20 gridpoints (60*60*20 gripoints) (ooops, I used the ifc Compiler for these tests)

62
SCI or Myrinet? 120*120*20 gridpoints (60*60*20 gripoints) (ooops, I used the ifc Compiler for these tests) (1066Mhz rdram?)

63
SCI or Myrinet? (time spent in pressure calc.) 120*120*20 gridpoints (60*60*20 gripoints) (ooops, I used the ifc Compiler for these tests) (1066Mhz rdram?)

64
Conclusions Linux clusters are useful in computational geophysical fluid dynamics!! SCI cards are necessary for parallel runs >10 nodes. For efficient parallelization: >50*50*20 grid points per node! Few users - great for development. Memory limitations, for 48 proc. a’ 500 Mb, 1200*1200*30 grid points is maximum (eddy resolving North Atlantic, Baltic Sea). For applications similar as ours, go for SCI cards + cpu with fast memory bus and fast memory!!

65
Experiment with low resolution (eddies are parameterized)

66

67

68

69
Thanks for your attention

Similar presentations

© 2016 SlidePlayer.com Inc.

All rights reserved.

Ads by Google