Presentation is loading. Please wait.

Presentation is loading. Please wait.

This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License.

Similar presentations


Presentation on theme: "This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License."— Presentation transcript:

1 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License

2 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License CCF CISGFS Rethink the nature of computing at extreme scale, from alternative, quantum computing models, through the transformative effects of manycore parallelism on programming systems and architectures, through massive cloud computing infrastructure designs. eXtreme Computing Division Dan Reed, CVP Microsoft Research ab initio research and development on cloud hardware and software infrastructure. Investigate cloud computing for research empowerment with worldwide government & academic partnerships. Cloud Computing Futures Group

3 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Data Center landscape Rise of the cloud computing platform. Data intensive research, role of cloud computing Key takeaways… Data centers and HPC, like twins separated at birth [Dan Reed] Data centers evolving at a blistering pace, driven by economics The Application Model for Cloud Computing Is Evolving Economic landscape increasingly favors pay as you go There are many obstacles, but economic forces will dominate the obstacles Emergence of the Fourth Paradigm, synergistic with cloud computing

4 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License

5 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License www.top500.org

6 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License www.top500.org

7 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Internet CR AR … SS LB Data Center Layer 3 Internet S S AAA … S S AAA … … Layer 2 Key: CR (L3 Border Router) AR (L3 Access Router) S (L2 Switch) LB (Load Balancer) A (20 Server Rack/TOR) Source: Albert Greenberg and Cisco GigE 10 GigE Monsoon network with Valiant routing

8 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License www.nersc.gov ~60 PB capacity

9 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Slide by Dan Reed

10 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License AttributeHPCCloud ProcessorHigh-end x86x86 Memory1-8 GB8 GB+ Local DiskScratch onlyPermanent storage SAN StorageCommonRare Tertiary StorageCommonRare InterconnectInfiniband or 10 GigE1 GigE/10GigE NetworkFlatHierarchical Physical PlantTraditionalOptimized

11 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Resource utilization pool concrete resources Decouples concrete resources enables migration Extend existing abstractions e.g. LUN expansion Hardware via existing ISA, memory mapped ports, etc. Storage via SCSI LUN or other disk interface Application via underlying API

12 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License LBL LANL ORNL ANL 38,640 cores 150,152 cores 163,840 cores ~130,000 cores

13 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Each data center is 11.5 times the size of a football field TechnologyCost in Medium-sized Data Center Cost in Very Large Data Center Ratio Network $95 per Mbps/ month $13 per Mbps/ month 7.1 Storage $2.20 per GB/ month $0.40 per GB/ month 5.7 Administration ~140 servers/ Administrator >1000 Servers/ Administrator 7.1 James Hamilton, LADIS 08 Economies of Scale Electricity Put Datacenters at Cheap Power Network Put Datacenters on Main Trunks Operations Standardize and Automate Ops Hardware Containerized Low-Cost Servers

14 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License

15 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License

16 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License

17 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License

18 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License

19 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Created by Navendu Jain, CJ Williams, Dan Reed and Jim Larus

20 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Server Capacity Generation 1 Data Center Collocation Rack Density and Sustainability Generation 2Containers Scalability Thousands of Servers Generation 3 Generation 4 (future) Modular Data Center Right Time to Market, Lower TCO (PUE) Scalable Data Centers Pre-Assembled Components Microsofts Data Center Evolution Deployment Scale Unit Data Center Evolution

21 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License …data as a service… …software as a service… cloud computing journal reports that… …everything as a service...

22 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Using a remote data center to manage scalable, reliable, on-demand access to application services and data. Scalable means Possibly millions of simultaneous users of app. Exploiting thousand-fold parallelism in the app. Reliable means on-demand means 5 nines available right now

23 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Server Hosted Server Cloud Fabric Windows (or Linux) Hosted OS Compute Fabric DB Server Hosted DB Server Storage Fabric …… What Youve Been Using So Far Hosted version of what you have been using so far Requires few changes if any to what you know and do New capabilities New cost structure Requires embracing a specific app model

24 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License

25 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Azure Services (storage) Load Balancer Public Internet Worker Role(s) Front-end Web Role Front-end Web Role Switches Highly-available Fabric Controller Highly-available Fabric Controller In-band communication – software control Load-balancers Abstract Programming Model

26 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License

27 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Roles are a mostly stateless process running in a Windows Server 2008 VM on one or more cores Web Roles provide web service access to app Web roles generate tasks for worker roles Worker Roles do heavy lifting and manage data in tables/blobs Communication is through queues. The number of instances can scale with load. A Scalable architecture is critical to take advantage of scalable infrastructure Queues decouple different parts of app, making it easier to scale app parts independently; Flexible resource allocation, different priority queues and separation of backend servers to process different queues. Queues mask faults in worker roles.

28 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Blobs can be bigup to 50 gigabytes each They can also have associated metadata

29 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License

30 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License

31 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Competitive Advantage Strategic Motive: Competitive Advantage Utilitarian Motive: Resource & Cost Optimization Innovation introduced by third firm Innovation introduced by second firm Innovation introduced by first firm Time Competitive advantage AND economics

32 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Assume Our Service Peaks at 500 Servers at Noon Trough Requires 100 Servers at Midnight Average Utilization Is 300 Servers Actual Utilization 300 × 24 = 7200 Server Hours / Day Provisioned Resources 500 × 24 = 12000 Servers Hours / Day Pay as You Go Break-Even Point 12000 = 7200 × 1.667 Cheaper When Pay as You Go Servers Are Less than 1.667 Times Purchased Servers Elasticity May Be More Cost-Effective Even with a Higher Per-Hour Charge! E-Commerce Peaks December Photo-Sharing Peaks January Takes Weeks to Acquire and Install Equipment Seasonal Demands Require Significant Provisioning This Example Underestimates the Benefits of Elasticity

33 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Research Funding 1.Have good idea 2.Write proposal 3.Wait 6 months 4.If successful, wait 3 months to get $$$ 5.Install Computers 6.Start Work Science Start-ups 1.Have good idea 2. Write Business Plan 3.Ask VCs to fund 4.If successful... 5.Install Computers 6.Start Work Cloud Computing Model 1.Have good idea 2.Grab nodes from Cloud provider 3.Start Work 4.Pay for what you actually used The Cloud Empowers the Long Tail of Research Slide compliments of Paul Watson, University of Newcastle (UK) Poised to reach a broad class of new users

34 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License With thanks to Jim Gray Astronomy was one of the first disciplines to embrace data-intensive science with the Virtual Observatory (VO), enabling highly efficient access to data and analysis tools at a centralized site. The image shows the Pleiades star cluster form the Digitized Sky Survey combined with an image of the moon, synthesized within the WorldWide Telescope

35 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Typical job, 10 – 20 CPU hours, extreme jobs require 1K – 2K CPU hours – Requires a large number of test runs for a given job (1 – 10M tests) – Highly compressed data per job ( ~100 KB per job) Science Example PhyloD as an Azure Service Cover of PLoS Biology November 2008 Statistical tool used to analyze DNA of HIV from large studies of infected patients PhyloD was developed by Microsoft Research and has been highly impactful Small but important group of researchers 100s of HIV and HepC researchers actively use it 1000s of research communities rely on results

36 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License BLAST user selects DBs and input sequence Blast Web Role Blast Web Role Input Splitter Worker Role Input Splitter Worker Role BLAST Execution Worker Role #n BLAST Execution Worker Role #n ….…. Combiner Worker Role Combiner Worker Role Genome DB 1 Genome DB K BLAST DB Configuration Azure Blob Storage BLAST Execution Worker Role #1 BLAST Execution Worker Role #1 Basic Map-Reduce - 2 GB database per worker - 500 MB input file.

37 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License

38 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License

39 This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License Q & AQ & AQ & AQ & A


Download ppt "This work is licensed under a Creative Commons Attribution 3.0 United States License.Creative Commons Attribution 3.0 United States License."

Similar presentations


Ads by Google