1. Inside Windows Azure Storage Name Title Microsoft Corporation

3. Geographically Distributed across 3 Regions Thousands of services/applications Anywhere at Anytime Access to your data Durability and Scalability 70 Petabytes raw storage today Grows to >200 Petabytes by start of 2012

4. Telemetry for Kinect Game Saves in Cloud Microsoft Zune Media Storage and Delivery Facebook and Twitter Near Real-Time Search

5. Bing Realtime facebook/twitter search ingestion engine Bing Ingestion Engine (Azure Service) VM Windows Azure Queues Index Facebook/Twitter data within 15 seconds of update Windows Azure Tables User postings Status updates ………… peak 40,000 Requests/sec 2~3 billion Requests per day Took 1 dev 2 months to design, build and release to production

6. What’s new for Blobs, Tables and Queues

7. Abstractions Blobs File system in the cloud Tables Massively scalable structured storage Queues Reliable storage and delivery of messages Drives Durable NTFS volumes for Windows Azure applications

8. User creates a globally unique storage account name Choose the primary location to host storage account

9. Account Table Entities Queue Messages Container Blobs https://.blob.core.windows.net/ https://.table.core.windows.net/ https://.queue.core.windows.net/

10. A highly scalable and durable file system in the cloud

13. Windows Azure Tables – What is new? Insert entity Update entity Merge Replace Delete entity Query entity Entity Group Transactions Query Projection ($select) Project only selected columns Upsert Entity InsertOrReplace InsertOrMerge

14. Windows Azure Tables - Projection public class Customer { public string PartitionKey { get; set; } // Customer Name public string RowKey { get; set; }// Customer Phone Number public DateTime CustomerSince { get; set; } public double TotalPurchase { get; set; } public string State { get; set; } // 100 more properties including profile picture etc.… } // Partial entity defined here public class CustomerDiscount { public string PartitionKey { get; set; } public string RowKey { get; set; } public double TotalPurchase { get; set; } }

15. Windows Azure Tables - Projection

16. Windows Azure Tables - Upsert

17. Provides reliable message delivery Programming semantics – Ensures that a message can be processed at least once Put message into the queue Get message makes the message invisible in queue for a specified invisibility timeout Delete message once done processing to remove message from queue If worker crashes, message becomes visible for another worker to process

19. Azure Queue Work items Web Role Worker Role Current time 7:00 Get Message with 5 minutes visibility timeout Expires @ 7:05AM 7:04 Extend visibility timeout with another 5 minutes Expires @ 7:09AM Periodically store progress information in message content 7:07 7:09 Retrieve progress from queue message and resume

20. Windows Azure Storage Analytics

21. Goal Capabilities Enable customers to understand and debug their usage of storage

22. Provides ability to answer commonly asked questions: Did a specific request make it to the storage service and how long did it take? What Client IP issued a “Delete container” request and when? How many requests were issued by a specific client or to a specific set of objects? list goes on and on…

24. The following are some of the fields logged for each record: Log Version Accessing Account Owner Account Service Type Request URL Object Key Request ID Operation Number Request Version Operation Type Start Time Application End to End Latency Storage Server Latency Authentication Type Request Status HTTP Status Code Client IP User Agent Referrer Client Request ID ETag LMT Request Packet Size Request Header Size Response Packet Size Response Header Size Request MD5 Server MD5 Conditions Used

25. Log Entry Example Log Version: 1.0 Start Time: 2011-07-28T18:02:40.6271789Z Operation Type: PutBlob Status: Success HTTP Status Code: 201 Application E2E Latency (milliseconds): 28 Storage Server Latency (milliseconds): 21 Accessing Account: sally Owner Account: sally Service Type: blob Request URL: PUT http://sally.blob.core.windows.net/thumbnails/lake.jpg Object Key: /sally/thumbnails/lake.jpg Request ID: fb658ee6-6123-41f5-81e2-4bfdc178fea3 Operation Number: 0 Request Version: 2009-09-19 Client IP: 201.9.10.20 Client Request ID: req12345 Log Entry in Blob: 1.0;2011-07- 28T18:02:40.6271789Z;PutBlob;Success;201;28;21;authenticate d;sally;sally;blob;"http://sally.blob.core.windows.net/thumbnail s/lake.jpg?timeout=30000";"/sally/thumbnails/lake.jpg";fb658 ee6-6123-41f5-81e2-4bfdc178fea3;0;201.9.10.20;2009-09- 19;438;100;223;0;100;;"66CbMXKirxDeTr82SXBKbg==";"0x8CE 1B67AD25AA05";Thursday, 28-Jul-11 18:02:40 GMT;;;;"req12345“

26. Provides ability to answer commonly asked questions: How many transactions did my service issue per hour over the past week? How many anonymous Get Blob requests were issued to my storage account? My application is not performing as expected, what is the availability and performance of storage for a given time period? list goes on and on…

28. Client application running in cloud start experiencing slow table access Compare Application E2E latency with Storage Server latency Storage Server Latency Application E2E Latency

30. Total Transactions

32. Logs Stored as blobs in separate Blob Container in the storage account being monitored http://account.blob.core.windows.net/$logs/ Metrics Stored as entities in a separate metrics Azure Tables in the storage account being monitored http://account.table.core.windows.net/$Metricshttp://account.table.core.windows.net/$Metrics* Isolation $logs and $Metrics have separate resource limits and throttling from the rest of the storage account traffic

33. Geo-Replication

34. East Asia South East Asia Europe West North Europe South Central US North Central US Geo-replication

36. North Central US Azure DNS http://account.blob.core.windows.net/ DNS lookup Data access HostnameIP Address account.blob.core.windows.netNorth Central US Failover Update DNS South Central US Existing URL works after failover Failover Trigger – failover would only be used if primary could not be recovered Asynchronous Geo-replication – may lose recent updates during failover Typically geo-replicate data within minutes, though no SLA guarantee Geo-replication South Central US

37. Windows Azure Storage Internals

38. Design Goals Highly Available Storage with Strong Consistency Provide access to data in face of hardware failures Durability Replicate data several times within and across data centers Scalability Need to scale to exabytes and beyond Automatically load balance data to meet peak traffic demands Provide a global namespace to access data around the world Additional details can be found in this white paper: “Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency”, ACM Symposium on Operating System Principals (SOSP), Oct. 2011 http://go.microsoft.com/fwlink/?LinkID=234565

39. Storage Stamp LB Storage Location Service Access blob storage via the URL: http://.blob.core.windows.net/ Data access Inter-stamp (Geo) replication Partition Layer Front-Ends DFS Layer Intra-stamp replication Storage Stamp LB Partition Layer Front-Ends DFS Layer Intra-stamp replication

40. M DFS Servers Paxos M M Distributed File System (DFS) Layer

41. Partition Server M Paxos M M Partition Server Partition Server Partition Server Partition Master Lock Service Partition Layer DFS Layer DFS Servers

42. Partition Server M Paxos Front End Layer FE M M Partition Server Partition Server Partition Server Partition Master FE Lock Service Partition Layer DFS Layer DFS Servers

43. Partition Server M Paxos Front End Layer FE Incoming Write Request M M Partition Server Partition Server Partition Server Partition Master FE Lock Service Ack Partition Layer DFS Layer DFS Servers

46. Account Name Container Name Blob Name aaaa aaaaa …….. zzzz zzzzz Storage Stamp Partition Server Partition Server Account Name Container Name Blob Name richardvideostennis ……… zzzz zzzzz Account Name Container Name Blob Name harrypicturessunset ……… richardvideossoccer Partition Server Partition Master Front-End Server PS 2 PS 3 PS 1 A-H: PS1 H’-R: PS2 R’-Z: PS3 A-H: PS1 H’-R: PS2 R’-Z: PS3 Partition Map Blob Index Partition Map Account Name Container Name Blob Name aaaa aaaaa ……… harrypicturessunrise A-H R’-Z H’-R

47. DFS Layer Partition Server 1 Partition Server 2 Partition Server 3 Partition Server 4 - RangePartition - Server Load Legend PM

48. 1.Scalability targets of a single storage account 2.Scalability targets for Blobs, Table Entities and Queues within a storage account

49. Scalability targets of a single storage account Account Scalability Targets Capacity – Up to 100 TBs Transactions – Up to 5000 entities per second Bandwidth – Up to 3 gigabits per second Partition data across storage accounts to go beyond these targets

50. Scalability targets for Blobs, Table Entities and Queues within a storage account Single Blob – up to 60MBytes per second Single PartitionKey in a Table – up to 500 entities per second Single Queue - up to 500 messages per second

52. Feedback and questions http://forums.dev.windows.com http://forums.dev.windows.com Session feedback http://bldw.in/SessionFeedback http://bldw.in/SessionFeedback

53. Thank You

54. © 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. © 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.