1. Prabal Dutta March 12, 2003Networks of Workstations1 Prabal Dutta dutta.4@osu.edu Electrical Engineering 864 Advanced Computer Design

2. Prabal Dutta March 12, 2003Networks of Workstations2 Introduction Much of this class has been about the details of advanced and parallel computing. –Interconnection Networks –Routing. –Pipelining. –Scheduling. –Algorithms. The purpose of this presentation is to look at the trends in parallel computing with a focus on networking… And to understand the technical and economic forces driving changes in parallel computing today.

3. Prabal Dutta March 12, 2003Networks of Workstations3 Taxonomy Massively Parallel Processors (MPP) –Built with specialized networks –Specifically intended for use as parallel computers Clusters –Clusters: small number of processors/node. –Constellations: large number of processors/node. Networks of Workstations (NOW) –Scalable interconnection networks. –Clusters of commodity PCs/workstations and high-performance network gear. –Beowulf = Linux + Bonded Ethernet or Myrinet. –UCB NOW: HP9000 + FDDI, ATM, or Myrinet. Other –Single Instruction Multiple Data: Connection Machine 2 –Vector Processors: Cray Let’s take a look at some of the once-high-flyers of MPPs…

4. Prabal Dutta March 12, 2003Networks of Workstations4 Bankrupt, Acquired, Spun Out, Sold Off!

5. Prabal Dutta March 12, 2003Networks of Workstations5 What Happened? In the 1990s, the supercomputing market had completely changed: "The cold war was over, the commercial world was so competitive, and companies couldn’t afford the machines. Meanwhile, microprocessors became more powerful. It was the beginning of commodity technology instead of proprietary technology. There was the move to vertically integrate everything. We were in a new age. Supercomputers were too expensive. They were not only hard to build, but they took a long time to build.” - Steve Chen, Ph.D. 1975, UIUC Chief Architect of the Cray X-MP

6. Prabal Dutta March 12, 2003Networks of Workstations6 Granularity: Computation-Communications Tradeoff Communications Computations MPP Constellations Clusters Distributed Computing Desktop Computing SMP $$$ $ $$ Workstation Computing Networks of Workstations

7. Prabal Dutta March 12, 2003Networks of Workstations7 Top 500 Supercomputer Sites (Metric: GFlops) Source data: http://www.top500.org/

8. Prabal Dutta March 12, 2003Networks of Workstations8 Top 500 Supercomputer Sites (Metric: Processor Arch) Source data: http://www.top500.org/

9. Prabal Dutta March 12, 2003Networks of Workstations9 Why are MPPs Falling Behind? Massively Parallel Processors –Cost too much. –Required over-engineering due to high integration. –Focused on niche applications with small markets. –Did not take into account networking commoditization. –Always a year or two behind the current state of the art. On the other hand, Networks of Workstations –Were marginally more expensive than workstations. –Leveraged emerging fast, scalable, and switched LANs –Leveraged high-volume, low-cost, super-pipelined commodity processors. –Extended existing operating systems with global process management and shared memory/message passing abstractions.

10. Prabal Dutta March 12, 2003Networks of Workstations10 Emergence of the “Killer” Network Massively Parallel Processing took advantage of: –Switched Networks. –Scalable bandwidth. –Low-latency communications. –Low processor overhead. These technologies are becoming available in LANs: –Myrinet. Used in 140 of the top 500 supercomputers. –Asynchronous Transfer Mode (ATM). –Fiber Distributed Data Interconnect (FDDI). –Fast Ethernet/Gigabit Ethernet.

11. Prabal Dutta March 12, 2003Networks of Workstations11 Fast, Scalable, Switched Networks: Myrinet Myrinet is based on the networks from MPPs and is used in 140 of the top 500 supercomputers. By decoupling network from workstation, allows each to evolve at its own pace. General Features –Full-duplex 2x2Gbps data rate (250MBps). –Self-initializing. –Low-latency. –Cut-through crossbar switches. –Direct communications between user processes and network. –Scales to tens of thousands of hosts. –Provide alternative communication paths between hosts. –X-Y Dimension-order routing. –Flow control on every link obviates packet buffering. –Error control and "heartbeat" continuity monitoring every link. –Linux,Windows,Solaris,Mac OSX,True64, FreeBSB,VxWorks

12. Prabal Dutta March 12, 2003Networks of Workstations12 Fast, Scalable, Switched Networks: Myrinet Sustained One-Way Data Rates

13. Prabal Dutta March 12, 2003Networks of Workstations13 Fast, Scalable, Switched Networks: Myrinet Example Myrinet Topologies

14. Prabal Dutta March 12, 2003Networks of Workstations14 Fast, Scalable, Switched Networks: Myrinet PCI Host Interface Card: $995 Modular Switch Enclosure: $12,800 8-port Fiber Switch Line Card: $2,400 Affordable !!!

15. Prabal Dutta March 12, 2003Networks of Workstations15 The “Killer” Workstation (and Desktop) Workstations have become extraordinarily powerful and affordable. Example: Intel Pentium 4 - 3.06GHz System: <$1,200

16. Prabal Dutta March 12, 2003Networks of Workstations16 The I/O Bottleneck Processors are getting faster and disks are improving mostly in capacity, not performance. So…we’ve reached a point of diminishing returns –More and more time spent waiting for I/O. –Little benefit for the end user. Fast networks enable aggregate DRAM as a giant cache for disks. Network DRAM access time is 1/10 of local disk. I/O can be striped across multiple nodes like RAID.

17. Prabal Dutta March 12, 2003Networks of Workstations17 Key Lessons Never bet against the power of commoditization. Market forces are unstoppable. Frequently, we must decide between novel research or customer needs. Companies formed to commercialize innovative research without compelling market economics are doomed –There are plenty of good research problems. –There are plenty of good business opportunities. –There are far fewer good research problems that are also good business opportunities.

18. Prabal Dutta March 12, 2003Networks of Workstations18 Closing Thoughts

19. Prabal Dutta March 12, 2003Networks of Workstations19 References 1.T. Anderson, D. Culler, and D. Patterson, “A Case of NOW (Networks of Workstations),” IEEE Micro, Vol. 15, Issue 1, February 1995. 2.T. Anderson, D. Culler, and D. Patterson, “The Berkeley Networks of Workstations (NOW) Project,” CompCon ’95, March 1995. 3.N. Boden, D. Cohen, R. Feldman, A. Kulawik, C. Seitz, J. Seizovic, and W. Su, “Myrinet: A Gigabit-per-Second Local Area Network,” IEEE Micro, Vol. 15, Issue 1, February 1995. 4.Douglas P. Ghormley, David Petrou, Steven H. Rodrigues, Amin M. Vahdat, and Thomas E. Anderson, “GLUnix: A Global Layer Unix for a Network of Workstations,” Software Practice and Experience, Special Issue on Experience with Distributed Systems, 1998. 5.http://www.myrinet.comhttp://www.myrinet.com 6.http://www.columbusmicro.com/http://www.columbusmicro.com/