© 2003 Mercury Computer Systems, Inc. Data Reorganization Interface (DRI) Kenneth Cain Jr. Mercury Computer Systems, Inc. High Performance Embedded Computing (HPEC) Conference September 2003 On behalf of the Data Reorganization Forum

© 2003 Mercury Computer Systems, Inc. 2 Status update for the DRI-1.0 standard since Sep publication Outline, Acknowledgements l DRI Overview. l Highlights of First DRI Demonstration. wCommon Imagery Processor (Brian Sroka, MITRE). l Vendor Status. wMercury Computer Systems, Inc. wMPI Software Technology, Inc. (Anthony Skjellum). wSKY Computers, Inc. (Stephen Paavola).

© 2003 Mercury Computer Systems, Inc. 3 What is DRI? l Partition for data-parallel processing wDivide multi-dimensional dataset across processes wWhole, block, block-cyclic partitioning wOverlapped data elements in partitioning wProcess group topology specification l Redistribute data to next processing stage wMulti-point data transfer with single function call wMulti-buffered to enable communication / computation overlap wPlanned transfers for higher performance Standard API that complements existing communication middleware

First DRI-based Demonstration Common Imagery Processor (CIP) Conducted by Brian Sroka of The MITRE Corporation

© 2003 Mercury Computer Systems, Inc. 5 CIP and APG-73 Background CIP l The primary sensor processing element of the Common Imagery Ground/Surface System (CIGSS) l Processes imagery data into exploitable image, outputs to other CIGSS elements l A hardware independent software architecture supporting multi-sensor processing capabilities l Prime Contractor: Northrop Grumman, Electronic Sensor Systems Sector l Enhancements directed by CIP Cross- Service IPT, Wright Patterson AFB APG-73 SAR component of F/A-18 Advanced Tactical Airborne Reconnaissance System (ATARS) Imagery from airborne platforms sent to TEG via Common Data Link Source: “HPEC-SI Demonstration: Common Imagery Processor (CIP) Demonstration -- APG73 Image Formation”, Brian Sroka, The MITRE Corporation, HPEC 2002

© 2003 Mercury Computer Systems, Inc. 6 APG-73 Data Reorganization (1) Block data distribution P0P0 P1P1 Range Cross Range %16 P N-1 PNPN P0P0 P1P1 PNPN Block data distribution Source No overlap Blocks of cross-range Full range Destination No overlap Full cross-range Mod-16 length blocks of range Source: “CIP APG-73 Demonstration: Lessons Learned”, Brian Sroka, The MITRE Corporation, March 2003 HPEC-SI meeting

© 2003 Mercury Computer Systems, Inc. 7 APG-73 Data Reorganization (2) %16 Range Cross-Range P0P0 P1P1 PNPN Block data distribution Source No overlap Full cross-range Mod-16 length blocks of range cells Destination 7 points right overlap Full cross-range Blocked portion of range cells P0P0 P1P1 PNPN Block with overlap Source: “CIP APG-73 Demonstration: Lessons Learned”, Brian Sroka, The MITRE Corporation, March 2003 HPEC-SI meeting

© 2003 Mercury Computer Systems, Inc. 8 DRI Use in CIP APG-73 SAR Simple transition to DRI l #pragma splits loop over global data among threads l DRI: loop over local data Demonstration completed Demonstrations underway MITRE DRI MPI * Application Mercury PAS/DRI Application SKY MPICH/DRI Application * MPI/Pro (MSTI) and MPICH demonstrated Range compression Inverse weighting for Azimuth compression Inverse weighting for Side-lobe clutter removal Amplitude detection Image data compression for DRI-2: Overlap exchange DRI-1: Cornerturn DRI Implementations Used Source: “HPEC-SI Demonstration: Common Imagery Processor (CIP) Demonstration -- APG73 Image Formation”, Brian Sroka, The MITRE Corporation, HPEC 2002 workshop

© 2003 Mercury Computer Systems, Inc. 9 Source Lines Of Code (SLOCs) l 5% SLOC increase for DRI includes code for: w2 scatter / gather reorgs w3 cornerturn data reorg cases w3 overlap exchange data reorg cases wmanaging interoperation between DRI and VSIPL libraries Portability: SLOC Comparison ~1% Increase~5% Increase VSIPLDRI VSIPLSequential Shared Memory VSIPL Source: “HPEC-SI Demonstration: Common Imagery Processor (CIP) Demonstration -- APG73 Image Formation”, Brian Sroka, The MITRE Corporation, HPEC 2002 workshop 1 for interleaved complex + 2 for split complex data format Using DRI requires much less source code than manual distributed-memory implementation

© 2003 Mercury Computer Systems, Inc. 10 CIP APG-73 DRI Conclusions Source: “HPEC-SI Demonstration: Common Imagery Processor (CIP) Demonstration -- APG73 Image Formation”, Brian Sroka, The MITRE Corporation, HPEC 2002 Source: “High Performance Embedded Computing Software Initiative (HPEC-SI)”, Dr. Jeremy Kepner, MIT Lincoln Laboratory, June 2003 HPEC-SI meeting l Applying DRI to operational software does not greatly affect software lines of code l DRI greatly reduces complexity of developing portable distributed-memory software (shared-memory transition easy) l Communication code in DRI estimated 6x smaller SLOCs than if implemented with MPI manually l No code changed to retarget application (MITRE DRI on MPI) l Features missing from DRI: wSplit complex wDynamic (changing) distributions wRound-robin distributions wPiecemeal data production / consumption wNon-CPU endpoints Future needs

Vendor DRI Status Mercury Computer Systems, Inc. MPI Software Technology, Inc. SKY Computers, Inc.

© 2003 Mercury Computer Systems, Inc. 12 Mercury Computer Systems (1/2) l Commercially available in PAS (Jul-03) wParallel Acceleration System (PAS) middleware product wDRI interface to existing PAS features The vast majority of DRI-1.0 is supported Not yet supported: block-cyclic, toroidal, some replication l Additional PAS features compatible with DRI wOptional: applications can use PAS and DRI APIs together l Applications can use MPI & PAS/DRI wExample: independent use of PAS/DRI and MPI libraries by the same application is possible (libraries not integrated)

© 2003 Mercury Computer Systems, Inc. 13 Mercury Computer Systems (2/2) DRI Adds No Significant Overhead DRI Achieves PAS Performance! l PAS communication features: wUser-driven buffering & synchronization wDynamically changing transfer attributes wDynamic process sets wI/O or memory device integration wTransfer only a Region of Interest (ROI) l PAS communication features: wUser-driven buffering & synchronization wDynamically changing transfer attributes wDynamic process sets wI/O or memory device integration wTransfer only a Region of Interest (ROI) Standard DRI_Distribution Object Standard DRI_Blockinfo Object Hybrid use of PAS and DRI APIs Built on Existing PAS Performance

© 2003 Mercury Computer Systems, Inc. 14 MPI Software Technology, Inc. l MPI Software Technology has released its ChaMPIon/Pro (MPI-2.1 product) this spring l Work now going on to provide DRI “in MPI clothing” as add-on to ChaMPIon/Pro l Confirmed targets are as follows: wLinux clusters with TCP/IP, Myrinet, InfiniBand wMercury RACE/RapidIO Multicomputers l Access to early adopters: 1Q04 l More info available from: (Tony Skjellum)

© 2003 Mercury Computer Systems, Inc. 15 SKY Computers, Inc. (1/2) Initial Implementation l Experimental version implemented for SKYchannel l Integrated with MPI l Achieving excellent performance for system sizes at least through 128 processors

© 2003 Mercury Computer Systems, Inc. 16 Sky Computers, Inc. (2/2) SKY’s Plans l Fully supported implementation with SMARTpac l Part of SKY’s plans for standards compliance l Included with MPI library l Optimized InfiniBand performance