Download presentation
Presentation is loading. Please wait.
Published byVerity Shelton Modified over 8 years ago
1
1 Opportunities and Challenges of Modern Communication Architectures: Case Study with QsNet CAC Workshop Santa Fe, NM, 2004 Sameer Kumar* and Laxmikant V. Kale Parallel Programming Laboratory University of Illinois at Urbana Champaign
2
2 Outline Processor virtualization QsNet Opportunities Performance Evaluation of QsNet Challenges of QsNet Summary
3
3 Processor Virtualization Basic idea of processor virtualization User specifies interaction between objects (VPs) RTS maps VPs onto physical processors Typically, # virtual processors > # processors Embodied in Charm++ and AMPI User View System implementation
4
4 QsNet Popular interconnect from Quadrics Several parallel systems in top500 use QsNet Pittsburgh ’ s Lemieux (6TF) ASCI-Q (20TF) Elite network Elan adaptor
5
5 Elite Network 320 MB/s each way after protocol Reliable fat-tree network Multiple routes provides fault tolerance Adaptive worm hole routing 35 ns per hop
6
6 Elan Network Adaptor Features Low latency (4.5 μs for MPI) High bandwidth (320MB/s/node) Components Sparc processor DMA Engine 64 MB RAM On chip cache
7
7 Low CPU Overhead CPU Overhead is small and does not change much with the message size
8
8 Traditional Message Passing Time P0P0 P1P1 Send OverheadReceive Overhead Idle Time Traditional Message Passing does not utilize low CPU overhead of Elan
9
9 Adaptive Overlap VP 0 VP 1 VP 0 VP 1 Time P0P0 P1P1 Send OverheadReceive Overhead Processor Virtualization takes full advantage of the low CPU overhead of Elan
10
10 Benefit of Adaptive Overlap Problem setup: 3D stencil calculation of size 240 3 run on Lemieux. Shows AMPI with virtualization ratio of 1 and 8.
11
11 Charm++ Message Driven Execution Handler Scheduler Pump Garbage Collection Send Tport Send Post Receives Receive Message
12
12 NAMD: A Production MD System Written in Charm++ Fully featured program NIH-funded development Distributed free of charge (5000+ downloads so far) Binaries and source code Installed at NSF centers Large published simulations (e.g., aquaporin simulation featured in keynote)
13
13 Scaling NAMD Several QsNet challenges had to be overcome to scale NAMD
14
14 QsNet Challange: Latency Applications need to post receives for messages of different sizes
15
15 Latency Bottlenecks Latency Slow NIC processor with a 100Mhz clock Cache size only 8KB Traversing a large loop flushes it 186017 592475 9103037 13174060 171008003 Cache Misses vs Number of Receives Posted
16
16 Managing Latency: Message Combining Organize processors in a 2D (virtual) Mesh Phase 1: Processors send messages to row neighbors Message from (x1,y1) to (x2,y2) goes via (x1,y2) Phase 1: Processors send messages to column neighbors 2* messages instead of P-1
17
17 NAMD PME Performance Performance of Namd with the Atpase molecule. PME step in Namd involves an a 192 X 144 processor collective operation with 900 byte messages
18
18 QsNet Challenge: Bandwidth MB/s One Way290 Two Way128 PCI/DMA Contention restricts bandwidth on Alpha servers QsNet Network Bandwidth 320 MB/s
19
19 Improving Bandwidth Main-MainElan-MainElan-Elan One Way290305319 Two Way128305319 Sending messages from Elan memory is faster Node bandwidth (MB/s) for different placements of source and destination
20
20 QsNet Challenge: Stretched Handlers Stretched Sends Green superscripts Similar stretches observed in the middle of entry methods NAMD Timeline Time Processors Force compute Integrate
21
21 Stretching Solution Stretched Sends Elan Isend blocked when the rendezvous for the previous Isend to any destination had not been acknowledged Solved the problem by closely working with Quadrics and obtaining a patch Isend only blocks on the rendezvous of the previous message to the same destination
22
22 Stretching Solution Contd. Stretches in the middle of entry methods Caused by OS daemons Using blocking receives minimized these stretches Daemons can be scheduled when processor is idle
23
23 NAMD With Blocking Receives Processors Time Blocking Receives
24
24 NAMD Performance on Lemieux
25
25 Summary QsNet and excellent network NIC co-processor ideal for message driven execution Programming guidelines Send messages from Elan memory Post limited number of receives and before the sends Blocking receives to avoid stretching
26
26 Future Work One sided communication Barrier? Persistent one sided communication Reserve buffers on destination
27
27 Fat Tree Topology a)b) c) 4-ary 1-tree 4-ary 2-tree 4-ary 3 tree
28
28 Elan3 Adapter DMA Engine Thread Processor On chip shared cache 64 bit 66 Mhz PCI interface 64 MB RAM
29
29 Object Based Communication Framework Application AMPI Charm++ Comm. Framework Object Layer Converse Comm. Framework Processor Layer Communication Layer Performs Object Level Optimizations Optimizes Inter-Processor Communication Strategy
30
30 AAPC Processor Overhead Mesh Completion Time Direct CPU Overhead Lower CPU overhead enables applications using Mesh to perform better even for large messages Direct Completion Time Mesh CPU Overhead
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.