Presentation is loading. Please wait.

Presentation is loading. Please wait.

Windows Azure Storage Deep Dive

Similar presentations


Presentation on theme: "Windows Azure Storage Deep Dive"— Presentation transcript:

1 Windows Azure Storage Deep Dive
Tech Ed North America 2010 4/1/ :25 AM Required Slide SESSION CODE: COS310 Windows Azure Storage Deep Dive Jai Haridas Software Engineer Microsoft Corporation © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2 AGENDA Introduction to Storage Storage Abstractions Summary
Blobs Drives Tables Queues Summary Tribune Company Questions?

3 WINDOWS AZURE STORAGE Enables developers, apps and users to access cloud storage Scalable, durable, and available Anywhere at anytime access Only pay for what the service uses Easy to use REST and Client Libraries Blobs, tables, queues Existing NTFS APIs Drives

4 STORAGE ABSTRACTIONS Blobs – Provide a simple interface for storing named files along with metadata for the file Drives – Provides durable NTFS volumes for Windows Azure applications to use Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application

5 WINDOWS AZURE STORAGE ACCOUNT
User creates a globally unique storage account name Can choose geo-location to host storage account US – “North Central” and “South Central” Europe – “North” and “West” Asia – “East” and “Southeast” Can co-locate storage account with compute account Receive a 512 bit secret key when creating account You get two keys which can be used to rotate

6 BLOB STORAGE CONCEPTS Account Container Blob images videos PIC01.JPG
Account Container Blob PIC01.JPG images PIC02.JPG cohowinery videos VID1.AVI

7 BLOB FEATURES AND FUNCTIONS
Programming Interfaces PutBlob Insert blob and Update blob GetBlob Get whole blob or a specific range DeleteBlob CopyBlob SnapshotBlob LeaseBlob Associate Metadata with Blob Standard HTTP metadata (Cache-Control, Content-Encoding, Content-Type, etc.) Metadata is <name, value> pairs, up to 8KB per blob

8 BLOB CLIENT LIBRARY EXAMPLE
CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudBlobClient blobClient = new CloudBlobClient( account.BlobEndpoint, account.Credentials); // Create Container CloudBlobContainer cloudContainer = blobClient.GetContainerReference(containerName); bool hasCreated = cloudContainer.CreateIfNotExist(); // Access Blob in the Container CloudBlob cloudBlob = cloudContainer.GetBlobReference(blobName); //BlobRequestOptions has retry policy, timeout etc. BlobRequestOptions options = new BlobRequestOptions(); //Upload the local file to Blob service cloudBlob.UploadFile(uploadFileName, options); //Download to local file name cloudBlob.DownloadToFile(downloadFileName, options);

9 TWO TYPES OF BLOBS UNDER THE HOOD
Block Blob Targeted at streaming workloads Each blob consists of a sequence of blocks 2 Phase commit: Blocks are uploaded and then separately committed Size limit 200GB per blob Page Blob Targeted at random read/write workloads Each blob consists of an array of pages Each page range write is committed on PUT Size limit 1TB per blob

10 BLOCK BLOB Blob Streaming Workload w/ Random Reads + Committed Writes
Uploading a large blob Blob blobName = “blob.wmv”; PutBlock(blobName, blockId1, block1Bits); PutBlock(blobName, blockId2, block2Bits); ………… PutBlock(blobName, blockIdN, blockNBits); PutBlockList(blobName, blockId1, blockId2…,blockIdN); 10 GB Movie Block Id 1 Block Id 2 Block Id 3 Block Id N Benefit Update the blob via blocks any way you want Efficient continuation and retry Parallel and out of order upload of blocks Windows Azure Storage blob.wmv blob.wmv

11 PAGE BLOB – RANDOM READ/WRITE
Create MyBlob Specify Blob Size = 10 GB Fixed Page Size = 512 bytes Random Access Operations PutPage[512, 2048) PutPage[0, 1024) ClearPage[512, 1536) PutPage[2048,2560) GetPageRange[0, 4096) returns valid data ranges: [0,512) , [1536,2560) GetBlob[1000, 2048) returns All 0 for first 536 bytes Next 512 bytes are data stored in [1536,2048) 10 GB Address Space 512 1024 1536 2048 2560 10 GB

12 WINDOWS AZURE CONTENT DELIVERY NETWORK
Scenario Frequently accessed blobs Blobs accessed from many geographic locations Windows Azure Content Delivery Network (CDN) Cache and serve your Windows Azure Blobs from our network 18 locations globally (US, Europe, Asia, Australia and South America), and growing Benefit Better experience for users far from blob service Provide high-bandwidth content delivery, around the world, for popular events

13 WINDOWS CDN DETAILS Registering for CDN in Windows Azure Dev Portal
Enable storage account to use CDN Get back a domain name and can specify your own custom domain name to use Windows Azure CDN URL: Custom Domain Name for CDN: Windows Azure Storage URL: Using Windows Azure CDN for Blob Access Specify blobs HTTP Cache-Control policy as TTL Make blob’s container(s) publically accessible Reference blobs using CDN domain name The request is routed to our closest edge location On miss or TTL expiration blobs are retrieved from blob service and cached in the CDN Allows the blobs to be served from our CDN caches

14 SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs, but do not want to distribute their secret key Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs https://cohowinery.blob.core.windows.net/images/pic1.jpg? Valid time range st=Start time (optional) se=End time Two resource levels of access to grant c=Container | b=Blob Four types of permissions (or any combination) r=Read | w=Write | d=Delete | l=List Signed Identifier (optional) Allows time range and permissions to be stored in the blob service for the SAS Provides instant revocation of SAS st= T08:49Z &se= T09:49Z &sr=c &sp=rw &sig=3OSeIHP8haK%2fle9%2bBK3BX1DsdMM%3d &si=foo

15 BLOB TIPS For high throughput clients
Set ServicePointManager.DefaultConnectionLimit to allow parallel connections to cloud service Default value is 2 Upload/Download multiple files in parallel ParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading Parallel uploads used when size >= 32MB BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW Timeout = Size in KB/ExpectedThroughputInKBps Use retry and exponential backoff for timeouts/server busy

16 SUMMARY OF WINDOWS AZURE BLOBS
Blob Access Patterns Block Blobs – streaming + commit-based writes Page Blobs – random read/write Blob Operations Copy, Snapshot, and Lease work for both types Ways of Accessing and Serving Blob Content Content Delivery Network access Shared Access Signatures (Signed URLs) Custom Domain Names

17 STORAGE ABSTRACTIONS Blobs – Provide a simple interface for storing named files along with metadata for the file Drives – Provides durable NTFS volumes for Windows Azure applications to use Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application

18 WINDOWS AZURE DRIVE Provides a durable NTFS volume for Windows Azure applications to use Use existing NTFS APIs to access a network attached durable drive Benefits Enables existing applications using NTFS to more easily migrate to the cloud Durability and survival of data on application failover or hardware failure A Windows Azure Drive is a Page Blob Mounts Page Blob over the network as an NTFS drive All flushed and unbuffered writes to drive are made durable to the Page Blob

19 WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD) Drives can be up to 1TB A Page Blob can only be mounted by one VM at a time for read/write A VM can dynamically mount up to 16 drives Remote Access via Page Blob Can upload the VHD to a Page Blob using the blob interface, and then mount it as a Drive Can download the Drive through the Page Blob interface

20 HOW WINDOWS AZURE DRIVES WORKS
VM Drive is a formatted page blob stored in blob service Mount obtains a blob lease Mount specifies amount of local storage for cache NTFS flushed/unbuffered writes commit to blob store before returning to app NTFS reads can be served from local cache or from blob store (cache miss) Application Drive X: OS Local Cache Lease Windows Azure Blob Service DemoBlob

21 CLOUD DRIVE CLIENT LIBRARY SAMPLE
//Create Local Storage resource and initialize the local cache for drives CloudDrive.InitializeCache(localCacheDir, cacheSizeInMB); CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); //Create a cloud drive (PageBlob) CloudDrive drive = account.CreateCloudDrive(pageBlobUri); drive.Create(1000 /* sizeInMB */); //Mount the network attached drive on the local file system string pathOnLocalFS = drive.Mount(cacheSizeInMB, DriveMountOptions.None); //Use NTFS APIs to Read/Write files to drive //Snapshot drive while mounted to create backups Uri snapshotUri = drive.Snapshot(); //Unmount the drive drive.Unmount();

22 WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud A Page Blob can only be mounted by one VM at a time for read/write access All NTFS non-buffered writes and flushes are made durable to the Page Blob Can backup and create versions of a drive with Snapshot drive Can mount a drive snapshot as a read-only VHD with many VMs at the same time

23 STORAGE ABSTRACTIONS Blobs – Provide a simple interface for storing named files along with metadata for the file Drives – Provides durable NTFS volumes for Windows Azure applications to use Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application

24 WINDOWS AZURE TABLES Provides Massively Scalable Structured Storage
Table can have billions of entities (rows) and TBs of data Familiar and Easy to use API WCF (ADO.NET) Data Services .NET classes and LINQ REST – with any platform or language

25 TABLE STORAGE CONCEPTS
Account Table Entity Name =… = … customers Name =… = … cohowinery Photo ID =… Date =… winephotos Photo ID =… Date =…

26 ENTITY PROPERTIES Each entity can have up to 255 properties
Mandatory Properties for every entity PartitionKey & RowKey Uniquely identifies an entity Defines the sort order Timestamp Optimistic Concurrency No fixed schema for rest of properties Each property is stored as a <name, typed value> pair No schema stored for a table Properties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double

27 PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKey It is the first property (column) of your Table Used to group entities in the Table into partitions A Table Partition All entities in a Table with the same partition key value RowKey provides uniqueness within a partition PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables

28 PURPOSE OF THE PARTITIONKEY
Entity Locality Entities in the same partition will be stored together Efficient querying and cache locality Entity Group Transactions Atomically perform multiple Insert/Update/Delete over entities in same partition in a single transaction Table Scalability System monitors the usage patterns of partitions Automatically load balance partitions Each partition can be served by a different storage node Scale to meet the traffic needs of your table

29 PARTITIONS AND PARTITION RANGES
PartitionKey (Category) RowKey (Title) Timestamp ReleaseDate Action Fast & Furious 2009 The Bourne Ultimatum 2007 Animation Open Season 2 The Ant Bully 2006 Comedy Office Space 1999 SciFi X-Men Origins: Wolverine War Defiance 2008 PartitionKey (Category) RowKey (Title) Timestamp ReleaseDate Action Fast & Furious 2009 The Bourne Ultimatum 2007 Animation Open Season 2 The Ant Bully 2006 Server B Table = Movies [Comedy - MaxKey) Server A [MinKey - Comedy) Server A Table = Movies PartitionKey (Category) RowKey (Title) Timestamp ReleaseDate Comedy Office Space 1999 SciFi X-Men Origins: Wolverine 2009 War Defiance 2008

30 TABLE OPERATIONS Table Entities Create, Query, Delete Insert Update
Merge – Partial update Replace – Update entire entity Delete Query Entity Group Transactions

31 TABLE SCHEMA Define the schema as a .NET class
[DataServiceKey("PartitionKey", "RowKey")] public class Movie { /// Movie Category is the partition key public string PartitionKey { get; set; } /// Movie Title is the row key public string RowKey { get; set; } public DateTime Timestamp { get; set; } public int ReleaseYear { get; set; } public double Rating { get; set; } public string Language { get; set; } public bool Favorite { get; set; } }

32 TABLE CLIENT LIBRARY SAMPLE
CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudTableClient tableClient = new CloudTableClient( account.TableEndpoint, account.Credentials); // Create Movie Table string tableName = “Movies“; tableClient.CreateTableIfNotExist(tableName); TableServiceContext context = tableClient.GetDataServiceContext(); // Add movie context.AddObject(tableName, new Movie("Action", “White Water Rapids Survival")); context.SaveChangesWithRetries(); // Query movie var q = (from movie in context.CreateQuery<Movie>(tableName) where movie.PartitionKey == "Action" && movie.Rating > 4.0 select movie).AsTableServiceQuery<Movie>(); foreach (Movie movieToUpdate in q) { movieToUpdate.Favorite = true; context.UpdateObject(movieToUpdate); } context.SaveChangesWithRetries( ); SaveChangesOptions.Batch

33 TABLE TIPS Partition your data appropriately
Scale Queries Entity Group Transactions Avoid “Append only” write patterns based on PartitionKey values Avoid using monotonically increasing suffix with a constant prefix Example: using only the current timestamp as PartitionKey If needed, add varying prefix to PartitionKey

34 TABLE TIPS Avoid large table scans when performance is critical
Restructure your schema if required Concatenate different keys to form appropriate index Most Optimal: PartitionKey == “SciFi” and RowKey == “Star Wars” Scans: Expect continuation tokens PartitionKey == “SciFi” and “Sphere” ≤ RowKey ≤ “Star Wars” “Action” ≤ PartitionKey ≤ “Thriller” PartitionKey == “Action” || PartitionKey == “Thriller” - currently scans entire table “Cars” ≤ RowKey ≤ “Star Wars” - scans entire table

35 TABLE TIPS Default .NET HTTP connections is set to 2
ServicePointManager.DefaultConnectionLimit = X; Use retries and exponential back-off SaveChangesWithRetries – add/update/delete AsTableServiceQuery – queries Handle Conflict errors on Inserts and NotFound errors on Delete from retry Prior try could have completed

36 STORAGE ABSTRACTIONS Blobs – Provide a simple interface for storing named files along with metadata for the file Drives – Provides durable NTFS volumes for Windows Azure applications to use Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application

37 QUEUE STORAGE CONCEPTS
Account Queue Message customer ID order ID cohowinery order processing customer ID order ID

38 LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between roles Load work in a queue Producer can forget about message once it is in queue Many workers consume the queue Worker Role Web Role Worker Role Azure Queue Input Queue (Work Items) Web Role Worker Role Web Role Worker Role

39 QUEUE’S RELIABLE DELIVERY
Azure Queue Input Queue (Work Items) Web Role Worker Role Guarantee delivery/processing of messages (two-step consumption) Worker Dequeues message and it is marked as Invisible for a specified “Invisibility Time” Worker Deletes message when finished processing it If Worker role crashes, message becomes visible after the invisibility time for another Worker to process

40 QUEUE CLIENT LIBRARY SAMPLE
CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudQueueClient queueClient = new CloudQueueClient( account.QueueEndpoint, account.Credentials); //Create Queue CloudQueue queue = queueClient.GetQueueReference(queueName); queue.CreateIfNotExist(); //Add Message CloudQueueMessage message = new CloudQueueMessage(“some content"); queue.AddMessage(message); //Get Message message = queue.GetMessage(TimeSpan.FromMinutes(3) /*Invisibility timeout*/); // Process Message within the Invisibility Timeout //Delete Message queue.DeleteMessage(message);

41 QUEUE TIPS Messages can be up to 8KB
Use blob to store large messages, and store blob ref in message A message may be processed more than once Make message processing idempotent Work should be repeatable and can be done multiple times Assume messages put into queue can be processed in any order For higher throughput Batch multiple work items into a single message or into a blob Use multiple queues Use DequeueCount to remove poison messages Enforce threshold on message’s dequeue count Monitor message count to dynamically increase/reduce workers

42 WINDOWS AZURE STORAGE TAKEAWAYS
Fundamental data abstractions to build your applications Blobs – Files and large objects Drives – NTFS APIs for migrating applications Tables – Massively scalable structured storage Queues – Reliable delivery of messages Easy to use via the Storage Client Library Get going at the Windows Azure website More info on Windows Azure Storage at:

43 CUSTOMER Tribune Company
Tech Ed North America 2010 4/1/ :25 AM Tribune Company Jerome Schulist Solutions Architect Tribune Company CUSTOMER © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

44 What is the Tribune Company?

45 What has the Tribune Company done?
Tribune Company (LA Times, Chicago Tribune, WGN news, etc.) Reinventing themselves to stay relevant in a digital age Storage Needs 100 TB of content produced annually 25 years of digital historical content 137 years of additional history content to digitize

46 What is the value of Windows Azure storage?
Scale elastically and dynamically (out of the disk business) Improve customer experience through reliability and improved performance Quicker to market Easy operational management

47 What did they build? 2 3 1 Image Processor (Queue)
Content Ingestion Services 3 1 Image Processor (Worker Role) Content (Blob Storage)

48 Tech Ed North America 2010 4/1/ :25 AM Track Resources Windows Azure, Microsoft SQL Azure, Windows Azure platform AppFabric: Learn More: Visit the Windows Azure Boot Camp, Room 396 © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

49 Resources Learning Required Slide www.microsoft.com/teched
Tech Ed North America 2010 4/1/ :25 AM Required Slide Resources Learning Sessions On-Demand & Community Microsoft Certification & Training Resources Resources for IT Professionals Resources for Developers © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

50 Complete an evaluation on CommNet and enter to win!
Tech Ed North America 2010 4/1/ :25 AM Required Slide Complete an evaluation on CommNet and enter to win! © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

51 Sign up for Tech·Ed 2011 and save $500 starting June 8 – June 31st
You can also register at the North America 2011 kiosk located at registration Join us in Atlanta next year

52 Tech Ed North America 2010 4/1/2017 11:25 AM
© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

53 Tech Ed North America 2010 4/1/2017 11:25 AM
© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.


Download ppt "Windows Azure Storage Deep Dive"

Similar presentations


Ads by Google