1. Windows Azure Storage Deep Dive
Tech Ed North America 2010
4/1/ :25 AM
Required Slide
SESSION CODE: COS310
Windows Azure Storage Deep Dive
Jai Haridas
Software Engineer
Microsoft Corporation
© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. AGENDA Introduction to Storage Storage Abstractions Summary
Blobs
Drives
Tables
Queues
Summary
Tribune Company
Questions?

3. WINDOWS AZURE STORAGE
Enables developers, apps and users to access cloud storage
Scalable, durable, and available
Anywhere at anytime access
Only pay for what the service uses
Easy to use REST and Client Libraries
Blobs, tables, queues
Existing NTFS APIs
Drives

4. STORAGE ABSTRACTIONS
Blobs – Provide a simple interface for storing named files along with metadata for the file
Drives – Provides durable NTFS volumes for Windows Azure applications to use
Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties
Queues – Provide reliable storage and delivery of messages for an application

5. WINDOWS AZURE STORAGE ACCOUNT
User creates a globally unique storage account name
Can choose geo-location to host storage account
US – “North Central” and “South Central”
Europe – “North” and “West”
Asia – “East” and “Southeast”
Can co-locate storage account with compute account
Receive a 512 bit secret key when creating account
You get two keys which can be used to rotate

6. BLOB STORAGE CONCEPTS Account Container Blob images videos PIC01.JPG
Account
Container
Blob
PIC01.JPG
images
PIC02.JPG
cohowinery
videos
VID1.AVI

7. BLOB FEATURES AND FUNCTIONS
Programming Interfaces
PutBlob
Insert blob and Update blob
GetBlob
Get whole blob or a specific range
DeleteBlob
CopyBlob
SnapshotBlob
LeaseBlob
Associate Metadata with Blob
Standard HTTP metadata (Cache-Control, Content-Encoding, Content-Type, etc.)
Metadata is <name, value> pairs, up to 8KB per blob

8. BLOB CLIENT LIBRARY EXAMPLE
CloudStorageAccount account =
CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount");
CloudBlobClient blobClient = new CloudBlobClient(
account.BlobEndpoint, account.Credentials);
// Create Container
CloudBlobContainer cloudContainer = blobClient.GetContainerReference(containerName);
bool hasCreated = cloudContainer.CreateIfNotExist();
// Access Blob in the Container
CloudBlob cloudBlob = cloudContainer.GetBlobReference(blobName);
//BlobRequestOptions has retry policy, timeout etc.
BlobRequestOptions options = new BlobRequestOptions();
//Upload the local file to Blob service
cloudBlob.UploadFile(uploadFileName, options);
//Download to local file name
cloudBlob.DownloadToFile(downloadFileName, options);

9. TWO TYPES OF BLOBS UNDER THE HOOD
Block Blob
Targeted at streaming workloads
Each blob consists of a sequence of blocks
2 Phase commit: Blocks are uploaded and then separately committed
Size limit 200GB per blob
Page Blob
Targeted at random read/write workloads
Each blob consists of an array of pages
Each page range write is committed on PUT
Size limit 1TB per blob

10. BLOCK BLOB Blob Streaming Workload w/ Random Reads + Committed Writes
Uploading a large blob
Blob
blobName = “blob.wmv”;
PutBlock(blobName, blockId1, block1Bits);
PutBlock(blobName, blockId2, block2Bits);
…………
PutBlock(blobName, blockIdN, blockNBits);
PutBlockList(blobName,
blockId1, blockId2…,blockIdN);
10 GB Movie
Block Id 1
Block Id 2
Block Id 3
Block Id N
Benefit
Update the blob via blocks any way you want
Efficient continuation and retry
Parallel and out of order upload of blocks
Windows Azure Storage
blob.wmv
blob.wmv

11. PAGE BLOB – RANDOM READ/WRITE
Create MyBlob
Specify Blob Size = 10 GB
Fixed Page Size = 512 bytes
Random Access Operations
PutPage[512, 2048)
PutPage[0, 1024)
ClearPage[512, 1536)
PutPage[2048,2560)
GetPageRange[0, 4096) returns valid data ranges:
[0,512) , [1536,2560)
GetBlob[1000, 2048) returns
All 0 for first 536 bytes
Next 512 bytes are data stored in [1536,2048)
10 GB Address Space
512
1024
1536
2048
2560
10 GB

12. WINDOWS AZURE CONTENT DELIVERY NETWORK
Scenario
Frequently accessed blobs
Blobs accessed from many geographic locations
Windows Azure Content Delivery Network (CDN)
Cache and serve your Windows Azure Blobs from our network
18 locations globally (US, Europe, Asia, Australia and South America), and growing
Benefit
Better experience for users far from blob service
Provide high-bandwidth content delivery, around the world, for popular events

13. WINDOWS CDN DETAILS Registering for CDN in Windows Azure Dev Portal
Enable storage account to use CDN
Get back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL:
Custom Domain Name for CDN:
Windows Azure Storage URL:
Using Windows Azure CDN for Blob Access
Specify blobs HTTP Cache-Control policy as TTL
Make blob’s container(s) publically accessible
Reference blobs using CDN domain name
The request is routed to our closest edge location
On miss or TTL expiration blobs are retrieved from blob service and cached in the CDN
Allows the blobs to be served from our CDN caches

14. SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs, but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key
Then give out the SAS to provide time based access to blobs
Valid time range
st=Start time (optional)
se=End time
Two resource levels of access to grant
c=Container | b=Blob
Four types of permissions (or any combination)
r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)
Allows time range and permissions to be stored in the blob service for the SAS
Provides instant revocation of SAS
st= T08:49Z &se= T09:49Z
&sr=c
&sp=rw
&sig=3OSeIHP8haK%2fle9%2bBK3BX1DsdMM%3d
&si=foo

15. BLOB TIPS For high throughput clients
Set ServicePointManager.DefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2
Upload/Download multiple files in parallel
ParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size >= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KB/ExpectedThroughputInKBps
Use retry and exponential backoff for timeouts/server busy

16. SUMMARY OF WINDOWS AZURE BLOBS
Blob Access Patterns
Block Blobs – streaming + commit-based writes
Page Blobs – random read/write
Blob Operations
Copy, Snapshot, and Lease work for both types
Ways of Accessing and Serving Blob Content
Content Delivery Network access
Shared Access Signatures (Signed URLs)
Custom Domain Names

17. STORAGE ABSTRACTIONS
Blobs – Provide a simple interface for storing named files along with metadata for the file
Drives – Provides durable NTFS volumes for Windows Azure applications to use
Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties
Queues – Provide reliable storage and delivery of messages for an application

18. WINDOWS AZURE DRIVE
Provides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
Benefits
Enables existing applications using NTFS to more easily migrate to the cloud
Durability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page Blob
Mounts Page Blob over the network as an NTFS drive
All flushed and unbuffered writes to drive are made durable to the Page Blob

19. WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for read/write
A VM can dynamically mount up to 16 drives
Remote Access via Page Blob
Can upload the VHD to a Page Blob using the blob interface, and then mount it as a Drive
Can download the Drive through the Page Blob interface

20. HOW WINDOWS AZURE DRIVES WORKSVM
Drive is a formatted page blob stored in blob service
Mount obtains a blob lease
Mount specifies amount of local storage for cache
NTFS flushed/unbuffered writes commit to blob store before returning to app
NTFS reads can be served from local cache or from blob store (cache miss)
Application
Drive X: OS
Local Cache
Lease
Windows Azure Blob Service
DemoBlob

21. CLOUD DRIVE CLIENT LIBRARY SAMPLE
//Create Local Storage resource and initialize the local cache for drives
CloudDrive.InitializeCache(localCacheDir, cacheSizeInMB);
CloudStorageAccount account =
CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount");
//Create a cloud drive (PageBlob)
CloudDrive drive = account.CreateCloudDrive(pageBlobUri);
drive.Create(1000 /* sizeInMB */);
//Mount the network attached drive on the local file system
string pathOnLocalFS = drive.Mount(cacheSizeInMB, DriveMountOptions.None);
//Use NTFS APIs to Read/Write files to drive …
//Snapshot drive while mounted to create backups
Uri snapshotUri = drive.Snapshot();
//Unmount the drive
drive.Unmount();

22. WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes
Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for read/write access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time

23. STORAGE ABSTRACTIONS
Blobs – Provide a simple interface for storing named files along with metadata for the file
Drives – Provides durable NTFS volumes for Windows Azure applications to use
Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties
Queues – Provide reliable storage and delivery of messages for an application

24. WINDOWS AZURE TABLES Provides Massively Scalable Structured Storage
Table can have billions of entities (rows) and TBs of data
Familiar and Easy to use API
WCF (ADO.NET) Data Services
.NET classes and LINQ
REST – with any platform or language

25. TABLE STORAGE CONCEPTS
Account
Table
Entity
Name =…
= …
customers
Name =…
= …
cohowinery
Photo ID =…
Date =…
winephotos
Photo ID =…
Date =…

26. ENTITY PROPERTIES Each entity can have up to 255 properties
Mandatory Properties for every entity
PartitionKey & RowKey
Uniquely identifies an entity
Defines the sort order
Timestamp
Optimistic Concurrency
No fixed schema for rest of properties
Each property is stored as a <name, typed value> pair
No schema stored for a table
Properties can be the standard .NET types
String, binary, bool, DateTime, GUID, int, int64, and double

27. PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKey
It is the first property (column) of your Table
Used to group entities in the Table into partitions
A Table Partition
All entities in a Table with the same partition key value
RowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model
Allows application to control the granularity of the partitions and scalability of its tables

28. PURPOSE OF THE PARTITIONKEY
Entity Locality
Entities in the same partition will be stored together
Efficient querying and cache locality
Entity Group Transactions
Atomically perform multiple Insert/Update/Delete over entities in same partition in a single transaction
Table Scalability
System monitors the usage patterns of partitions
Automatically load balance partitions
Each partition can be served by a different storage node
Scale to meet the traffic needs of your table

29. PARTITIONS AND PARTITION RANGES
PartitionKey
(Category)
RowKey
(Title)
Timestamp
ReleaseDate
Action
Fast & Furious …
2009
The Bourne Ultimatum
2007
Animation
Open Season 2
The Ant Bully
2006
Comedy
Office Space
1999
SciFi
X-Men Origins: Wolverine
War
Defiance
2008
PartitionKey
(Category)
RowKey
(Title)
Timestamp
ReleaseDate
Action
Fast & Furious …
2009
The Bourne Ultimatum
2007
Animation
Open Season 2
The Ant Bully
2006
Server B
Table = Movies
[Comedy - MaxKey)
Server A
[MinKey - Comedy)
Server A
Table = Movies
PartitionKey
(Category)
RowKey
(Title)
Timestamp
ReleaseDate
Comedy
Office Space …
1999
SciFi
X-Men Origins: Wolverine
2009
War
Defiance
2008

30. TABLE OPERATIONS Table Entities Create, Query, Delete Insert Update
Merge – Partial update
Replace – Update entire entity
Delete
Query
Entity Group Transactions

31. TABLE SCHEMA Define the schema as a .NET class
[DataServiceKey("PartitionKey", "RowKey")]
public class Movie {
/// Movie Category is the partition key
public string PartitionKey { get; set; }
/// Movie Title is the row key
public string RowKey { get; set; }
public DateTime Timestamp { get; set; }
public int ReleaseYear { get; set; }
public double Rating { get; set; }
public string Language { get; set; }
public bool Favorite { get; set; } }

32. TABLE CLIENT LIBRARY SAMPLE
CloudStorageAccount account =
CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount");
CloudTableClient tableClient = new CloudTableClient(
account.TableEndpoint, account.Credentials);
// Create Movie Table
string tableName = “Movies“;
tableClient.CreateTableIfNotExist(tableName);
TableServiceContext context = tableClient.GetDataServiceContext();
// Add movie
context.AddObject(tableName, new Movie("Action", “White Water Rapids Survival"));
context.SaveChangesWithRetries();
// Query movie
var q = (from movie in context.CreateQuery<Movie>(tableName)
where movie.PartitionKey == "Action" && movie.Rating > 4.0
select movie).AsTableServiceQuery<Movie>();
foreach (Movie movieToUpdate in q) {
movieToUpdate.Favorite = true;
context.UpdateObject(movieToUpdate); }
context.SaveChangesWithRetries( );
SaveChangesOptions.Batch

33. TABLE TIPS Partition your data appropriately
Scale
Queries
Entity Group Transactions
Avoid “Append only” write patterns based on PartitionKey values
Avoid using monotonically increasing suffix with a constant prefix
Example: using only the current timestamp as PartitionKey
If needed, add varying prefix to PartitionKey

34. TABLE TIPS Avoid large table scans when performance is critical
Restructure your schema if required
Concatenate different keys to form appropriate index
Most Optimal:
PartitionKey == “SciFi” and RowKey == “Star Wars”
Scans: Expect continuation tokens
PartitionKey == “SciFi” and “Sphere” ≤ RowKey ≤ “Star Wars”
“Action” ≤ PartitionKey ≤ “Thriller”
PartitionKey == “Action” || PartitionKey == “Thriller” - currently scans entire table
“Cars” ≤ RowKey ≤ “Star Wars” - scans entire table

35. TABLE TIPS Default .NET HTTP connections is set to 2
ServicePointManager.DefaultConnectionLimit = X;
Use retries and exponential back-off
SaveChangesWithRetries – add/update/delete
AsTableServiceQuery – queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed

36. STORAGE ABSTRACTIONS
Blobs – Provide a simple interface for storing named files along with metadata for the file
Drives – Provides durable NTFS volumes for Windows Azure applications to use
Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties
Queues – Provide reliable storage and delivery of messages for an application

37. QUEUE STORAGE CONCEPTS
Account
Queue
Message
customer ID order ID
cohowinery
order processing
customer ID order ID

38. LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between roles
Load work in a queue
Producer can forget about message once it is in queue
Many workers consume the queue
Worker Role
Web Role
Worker Role
Azure Queue
Input Queue (Work Items)
Web Role
Worker Role
Web Role
Worker Role

39. QUEUE’S RELIABLE DELIVERY
Azure Queue
Input Queue (Work Items)
Web Role
Worker Role
Guarantee delivery/processing of messages (two-step consumption)
Worker Dequeues message and it is marked as Invisible for a specified “Invisibility Time”
Worker Deletes message when finished processing it
If Worker role crashes, message becomes visible after the invisibility time for another Worker to process

40. QUEUE CLIENT LIBRARY SAMPLE
CloudStorageAccount account =
CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount");
CloudQueueClient queueClient = new CloudQueueClient(
account.QueueEndpoint, account.Credentials);
//Create Queue
CloudQueue queue = queueClient.GetQueueReference(queueName);
queue.CreateIfNotExist();
//Add Message
CloudQueueMessage message = new CloudQueueMessage(“some content");
queue.AddMessage(message);
//Get Message
message = queue.GetMessage(TimeSpan.FromMinutes(3) /*Invisibility timeout*/);
// Process Message within the Invisibility Timeout
//Delete Message
queue.DeleteMessage(message);

41. QUEUE TIPS Messages can be up to 8KB
Use blob to store large messages, and store blob ref in message
A message may be processed more than once
Make message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughput
Batch multiple work items into a single message or into a blob
Use multiple queues
Use DequeueCount to remove poison messages
Enforce threshold on message’s dequeue count
Monitor message count to dynamically increase/reduce workers

42. WINDOWS AZURE STORAGE TAKEAWAYS
Fundamental data abstractions to build your applications
Blobs – Files and large objects
Drives – NTFS APIs for migrating applications
Tables – Massively scalable structured storage
Queues – Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure website
More info on Windows Azure Storage at:

43. CUSTOMER Tribune Company
Tech Ed North America 2010
4/1/ :25 AM
Tribune Company
Jerome Schulist Solutions Architect Tribune Company
CUSTOMER
© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

44. What is the Tribune Company?

45. What has the Tribune Company done?
Tribune Company (LA Times, Chicago Tribune, WGN news, etc.)
Reinventing themselves to stay relevant in a digital age
Storage Needs
100 TB of content produced annually
25 years of digital historical content
137 years of additional history content to digitize

46. What is the value of Windows Azure storage?
Scale elastically and dynamically (out of the disk business)
Improve customer experience through reliability and improved performance
Quicker to market
Easy operational management

47. What did they build? 2 3 1 Image Processor (Queue)
Content Ingestion Services 3 1
Image Processor (Worker Role)
Content (Blob Storage)

48. Tech Ed North America 2010
4/1/ :25 AM
Track Resources
Windows Azure, Microsoft SQL Azure, Windows Azure platform AppFabric:
Learn More: Visit the Windows Azure Boot Camp, Room 396
© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

49. Resources Learning Required Slide www.microsoft.com/teched
Tech Ed North America 2010
4/1/ :25 AM
Required Slide
Resources
Learning
Sessions On-Demand & Community
Microsoft Certification & Training Resources
Resources for IT Professionals
Resources for Developers
© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

50. Complete an evaluation on CommNet and enter to win!
Tech Ed North America 2010
4/1/ :25 AM
Required Slide
Complete an evaluation on CommNet and enter to win!
© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

51. Sign up for Tech·Ed 2011 and save $500 starting June 8 – June 31st
You can also register at the North America 2011 kiosk located at registration Join us in Atlanta next year

52. Tech Ed North America 2010 4/1/2017 11:25 AM
© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

53. Tech Ed North America 2010 4/1/2017 11:25 AM
© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.