Windows Azure Tables: Programming Cloud Table Storage 4/1/2017 2:37 PM ES07 Windows Azure Tables: Programming Cloud Table Storage  Niranjan Nilakantan Development Lead Microsoft Corporation  Pablo Castro Software Architect Microsoft Corporation © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Windows Azure Windows Azure is the foundation of Microsoft’s Cloud Platform It is an “Operating System in the Cloud” and provides Essential Services for the cloud Virtualized Computation Scalable Storage Automatic Management Developer SDK

Fundamental Data Abstractions Blobs – Provide a simple interface for storing named files along with metadata for the file Tables – Provide structured storage; A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application

Windows Azure Tables Provides Structured Storage Massively Scalable Tables Billions of entities (rows) and TBs of data Can use thousands of servers as traffic grows Highly Available Can always access your data Durable Data is replicated several times Familiar and Easy to use API ADO.NET Data Services – .NET 3.5 SP1 .NET classes and LINQ REST – with any platform or language

Agenda Data model Demo – creating tables Basic CRUD using .NET and REST Demo – a sample application Advanced features

Data Model Table Data is stored in Tables Entity A Storage Account can create many tables Table name is scoped by Account Data is stored in Tables A Table is a set of Entities (rows) An Entity is a set of Properties (columns) Entity Two “key” properties that together are the unique ID of the entity PartitionKey – enables scalability RowKey – uniquely identifies the entity within the partition

Partitioning Partition Key Document Row Key Version Property 3 ChangedOn Property 4 Description Examples Doc V1.0 8/2/2007 Committed version V2.0.1 9/28/2007 Alice’s working version FAQ Doc 5/2/2007 V1.0.1 7/6/2007 V1.0.2 8/1/2007 Sally’s working version Partition 1 Partition 2 Table Partition – all entities in table with same partition key value Application controls granularity of partition

Partitioning Guidelines Performance Use a PartitionKey that is common in your queries Entities with same partition key value are clustered Scalability We monitor partition traffic Automatically load balance partitions Each partition can potentially be served by a different storage node Scale to meet the traffic needs of your application More partitions – makes it easier to balance load

Entity Locality Getting all versions of FAQ is fast Partition Key Document Row Key Version Property 3 ChangedOn Property 4 Description Examples Doc V1.0 8/2/2007 Committed version V2.0.1 9/28/2007 Alice’s working version FAQ Doc 5/2/2007 V1.0.1 7/6/2007 V1.0.2 8/1/2007 Sally’s working version Partition 1 Partition 2 Getting all versions of FAQ is fast Access single partition Get all documents before 5/30/2008 takes longer Have to traverse all partitions

Entities and Properties Each Entity can have up to 255 properties Every Entity has fixed key properties Partition key Row key No fixed schema for rest of properties 2 entities in the same table can have different properties Properties stored as <Name, TypedValue> pairs Each entity has a system maintained version

Property Types Partition key and Row key Other properties String (up to 64KB) Other properties Binary (up to 64KB) Bool DateTime GUID Int Int64 Double

Sample Scenario Micro-blogging application Scalability challenges Post messages to channels Subscribe to channels Retrieve channel messages Scalability challenges Large volume of new messages daily Need to retrieve recent messages quickly Messages permanently addressable Name – PK Author CreatedDate Channels Channel – PK PostedDate – RK Text Rating Messages Name – PK Email EmailUpdates Channels Users

MicroBlogging Application 4/1/2017 2:37 PM demo MicroBlogging Application © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Table Schema Define the schema as a .NET class [DataServiceKey("PartitionKey", "RowKey")] public class Message { // ChannelName public string PartitionKey { get; set; } // PostedDate public string RowKey { get; set; } // User defined properties public string Text { get; set; } public int Rating { get; set; } }

Insert Entity using .NET Create a new Message and Insert into Table Message message = new Message { PartitionKey = "Channel9", // ChannelName RowKey = DateTime.UtcNow.ToString(), // PostedDate Text = "Hello PDC", Rating = 3 }; serviceUri = new Uri("http://<account>.table.core.windows.net"); var context = new DataServiceContext(serviceUri); context.AddObject("Messages", message); DataServiceContext response = context.SaveChanges();

Insert Entity using REST Create an Atom XML payload and POST it POST http://<Account>.table.core.windows.net/Messages ... <!– Atom envelope --> <m:properties> <d:PartitionKey>Channel9</d:PartitionKey> <!-- ChannelName --> <d:RowKey>Oct-29</d:RowKey> <!-- PostedDate --> <d:Text>Hello PDC</d:Text> <d:Rating>3</d:Rating> </m:properties>

Create Table Every Account has a table called “Tables” To use a table it has to be inserted into “Tables” [DataServiceKey("TableName")] public class TableStorageTable { public string TableName { get; set; } } // Service Uri is “http://<Account>.table.core.windows.net/” DataServiceContext context = new DataServiceContext(serviceUri); TableStorageTable table = new TableStorageTable("Messages"); context.AddObject("Tables", table); DataServiceResponse response = context.SaveChanges();

Query Entities .NET – LINQ REST serviceUri = new Uri("http://<account>.table.core.windows.net"); DataServiceContext context = new DataServiceContext(serviceUri); var messages = from message in context.CreateQuery<Message>("Messages") where message.Rating == 3 select message; foreach (Message message in messages) { } REST GET http://<serviceUri>/Messages?$filter= Rating eq 3

Update a property in an entity Using the .NET API Message message = (from message in context.CreateQuery<Message>("Messages") where message.PartitionKey == "Channel9"   && message.RowKey == "Oct-29" select message).FirstOrDefault(); message.Text = "Hi there"; context.UpdateObject(message); DataServiceResponse response = context.SaveChanges(); Using the REST API PUT http://<serviceUri>/Messages(‘Channel9’, ‘Oct-29’) <m:properties> <d:Text>Hi there</d:Text> <!-- Other properties are the same --> </m:properties>

Delete an Entity Using the .NET API Using the REST API // Get the Message object for ("Channel9", "Oct-29") context.DeleteObject(message); DataServiceResponse response = context.SaveChanges(); Using the REST API DELETE http://.../Messages(‘Channel9’, ‘Oct-29’)

.NET vs. REST ADO.NET Data Services REST Interface 4/1/2017 2:37 PM .NET vs. REST ADO.NET Data Services REST Interface Client included in .NET Framework 3.5 SP1 Data represented as .NET objects DataServiceContext methods for updates Use LINQ to define queries Use any HTTP stack Data represented in Atom (XML) Standard HTTP verbs for updates Use URLs to define queries © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

MicroBlogging Application 4/1/2017 2:37 PM demo MicroBlogging Application © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Windows Platform Integration Reuse .NET skills Fully compatible with ADO.NET data services .NET client included in .NET 3.5 SP1 LINQ support ASP.NET integration for website authoring Sample data source control for data binding ASP.NET dynamic data for instant front-ends ASP.NET providers for membership, roles, etc.

Advanced Topics Concurrent updates Pagination Table consistency

Concurrent Updates ClientB Client A 1: Ch9, Jan-2, 5 2: Ch9, Jan-2, 5 Version Rating 1: Ch9, Jan-2, 4 Error: 412 5 : Ch9, Jan-1, 3 If-Match: 1 Ch9, Jan-2, 4 If-Match: 1 Ch9, Jan-2, 5 1 : Ch9, Jan-2, 2 1 : Ch9, Jan-2, 2 2: Ch9, Jan-2, 5 1 : Ch9, Jan-2, 2 9 : Ch9, Jan-3, 6 Use standard HTTP mechanisms – Etag and If-Match Get entity – get system maintained version as ETag Update Entities Locally – change rating Send Update with version check - IF-Match with Etag Success if version matches, and update version on Client-A Precondition failed (412) if version does not match

Getting the Top N entities .NET: LINQ Take(N) function serviceUri = new Uri("http://<account>.table.core.windows.net"); DataServiceContext context = new DataServiceContext(serviceUri); var allMessages = context.CreateQuery<Message>("Messages"); foreach (Message message in allMessages.Take(100)) { Console.WriteLine(message.Name); } REST: $top=N query string option GET http://<serviceUri>/Messages?$top=100

Pagination – Continuation Tokens Send a request GET http://<serviceUri>/Messages?$filter=...&$top=100 Get continuation token in response header x-ms-continuation-NextPartitionKey: Channel9 x-ms-continuation-NextRowKey: Date101 Messages 100 Ch9, Date1, Ch9, Date2, Ch9, … Set HTTP query parameters Ch9,Date100, GET http://<Uri>/Messages?$filter=...&$top=100 &NextPartitionKey=Channel9 &NextRowKey=Date101 Ch9,Date101, Ch9, …

Single Table Consistency ACID transactions for single entity CUD Insert/update/delete Snapshot isolation for query within a single partition Consistent view from start time of the query No dirty (uncommitted) reads Does not block concurrent updates No snapshot isolation across partitions No snapshot isolation across different continuations of a query

Cross Table Consistency Application is responsible for maintaining consistency Example When a channel is deleted, delete all the messages for that channel Failures can occur in the middle Example - Application fails after deleting some messages Use Windows Azure Queues to help ensure completion of operation

Cross Table Consistency Delete channel Delete messages worker Front end Worker Queue Messages Ch1, Msg1 Del Ch5 Del Ch11 Del Ch1 Del Ch1 2 Ch1, Msg2 Ch1, Msg3 Front End Ch2, Msg1 Ch2, Msg2 Channels Ch1,… Ch3, Msg1 Ch2,… 1. Dequeue DelCh1 3. Delete from Messages 4. Delete queue entry 2. Delete Ch1 from Channels

Resuming After Failure Delete channel Delete messages worker Front end Worker 1 Del Ch1h Queue Messages Ch1, Msg1 Del Ch5 Del Ch11 Del Ch1 Del Ch1 2 Ch1, Msg2 Ch1, Msg3 Front End Ch2, Msg1 Worker2 Ch2, Msg2 Channels Ch1,… Ch3, Msg1 Ch2,… 2. Fails after deleting Ch1 and Msg1 3. DelCh1 is visible again 1. Dequeue DelCh1 and start delete 4. Dequeue DelCh1 again 5. Repeat delete operations

Future Windows Azure Table Support At PDC Single Index Query and retrieve results sorted by PartitionKey and RowKey Future Support for Secondary Indexes Query and retrieve results sorted by other properties Single Entity Transactions Single Entity Insert, Update, or Delete Atomically Entity Groups Atomic transactions across multiple entities within same partition

Summary Azure tables are Simple familiar API Massively Scalable Highly Available Simple familiar API Use .NET –- ADO.NET Data Services and LINQ Or use REST Leverage your .NET expertise ASP .NET integration for instant front ends

Resources Links Windows Azure ADO.NET Data Services http://www.azure.com/windows ADO.NET Data Services http://blogs.msdn.com/astoriateam

Evals & Recordings www.microsoftpdc.com Please fill out your evaluation for this session at: This session will be available as a recording at: www.microsoftpdc.com

Please use the microphones provided 4/1/2017 2:37 PM Q&A Please use the microphones provided © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4/1/2017 2:37 PM © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4/1/2017 2:37 PM © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.