Scaling out and in with Azure SQL DB Elastic Scale DBA-203 Warner Chaves, MCM/MVP, SQLTurbo.com, Pythian.com

SQL Server DBA for 10 years. Previously an L3 DBA at HP in Costa Rica, now a Principal Consultant at Pythian in Ottawa Ontario. SQL Server MCM and MVP. Blog: SQLTurbo.com Company: Pythian.com About me 2

Agenda Goal of today: provide an overview of the elastic capabilities of the Azure SQL Database platform. Database elasticity intro Database Elastic Tools Database Elastic Query Database Elastic Pools Database Elastic Jobs 3

We’re going to focus on DbaaS, Azure SQL Db. Building and shrinking database capacity on demand. Elastic scale focuses on scale OUT (“Sharding”), as opposed to just scaling UP. Assume one application, common schema on all dbs. Database layer elasticity 4

1.Hitting resource limits on a scale up solution (CPU-RAM-size). 2.Geographic considerations. 3.Security concerns require separate data. 4.Easier chargeback mechanism for client. 5.Easier to manage individually for a particular client. Why scale out? 5

Provisioning speed. Controlling cost. Shard data movement. Routing the application to the proper database. Querying the databases individually as well as easily querying the global set. Common challenges to scale out 6

A Db elastic solution architecture 7

Creating and working with an elastic data tier

The purpose of the library is to facilitate building applications with an elastic db layer on Azure SQL db. Main concepts: Shard ~ Database Shard Set Shardlet Shard Map Manager Reference tables Elastic db tools:.NET client library 9

Db Elastic Client Library 10

1.Create a database called a “Shard Map Manager”. 2.Register databases as shards in the SMM. 3.Setup the shard distribution method (lists of keys or ranges). 4.Request a connection from the library based on the sharding key (data dependent routing). 5.Use that connection for the db interaction of the shard. 6.Multi-shard queries are possible too. Workflow for the client library 11

The Split/Merge tool helps with rebalancing data in the shard set. You can SPLIT a db into multiple. You can MERGE multiple dbs into one. These operations allow to respond to changes in capacity needs. It’s delivered as a customer-hosted Azure service. Elastic db tools: Split/Merge tool 12

Elastic db tools: Split/Merge tool 13 Db1 DbN MERGESPLIT Split/Merge Service ShardMap SplitMergeDb

Elastic db tools: Split/Merge interface 14

Elastic db tools: Split/Merge interface 15

Allows querying a set of databases as a single entity through one connection string. This is useful for 3 rd party apps where we can’t insert the.NET elastic scale library (Excel, SSRS, etc). Elastic Database Query 16

1.A set of Azure SQL Dbs with a common schema (any tier). 2.A shard map and Standard or Premium tier db to be the query connection endpoint. 3.Creating a credential for the elastic query: Configuring Elastic Database Queries 17 CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'password'; CREATE DATABASE SCOPED CREDENTIAL WITH IDENTITY = ' ', SECRET = ' ';

4.Create an external data source on the endpoint db: Configuring Elastic Database Queries 18 CREATE EXTERNAL DATA SOURCE WITH (TYPE = SHARD_MAP_MANAGER, LOCATION = ' ', DATABASE_NAME = ' ', CREDENTIAL =, SHARD_MAP_NAME = ' ');

5.Create an external table: Configuring Elastic Database Queries 19 CREATE EXTERNAL TABLE [ database_name. [ dbo ]. | dbo. ] table_name ( { } [,...n ]) { WITH ( ) } )[;] ::= DATA_SOURCE =, DISTRIBUTION = SHARDED( ) | REPLICATED | ROUND_ROBIN

SP_EXECUTE_FANOUT: can consume a shard map to run a fanout query over the set of sharded databases and concatenate the results. TARGET=RDBMS: cross database queries are supported now by creating a external source that targets a single database. Other flavors of Elastic Db Queries 20

Sample App Demo

Managing elastic resources

Allows pooling Azure SQL resources for a set of dbs for a more efficient performance/cost ratio. The resources are quantified as eDTUs (elastic DTUs). Storage size is also tied to the eDTUs and depend on tier. Different pools per Service Tier. Elastic Database Pools 23

TierMax Pool Size Max DbseDTU pool range Max eDTU/db Max Size / db Basic120GB edtu 52GB Standard1200GB edtu GB Premium750GB edtu GB Database pool specifications 24

One db patternThe pooled pattern Elastic Database Pools 25

Fixed cost per eDTU (min 100, max 1500). Example: 16 S3 databases: 16x$150 per month = $2400, each db has static 100 DTU. 250GBx16 storage. One Standard pool: (800 eDTUx$2.23) per month=$1784, each db can go DTU. 800GB storage. Elastic Database Pools: pricing 26

Create a db pool Demo

Running jobs on an elastic data tier

Allows running jobs against a set of databases. Either a T-SQL script or a DACPAC. Can be a custom set, a shard set or a db pool. Scheduling and retry similar to the SQL Agent. Elastic Database Jobs 29

It’s a customer-hosted Azure solution. Cost is per component. Requires the following components setup during install: Cloud service (worker role to run tasks). SQL Db (control db to store metadata, S0 default). Service bus (used for work coordination). Azure storage (for diagnostic output). Setting up Elastic Database Jobs 30

How do Elastic Database Jobs work? 31 Control DB Controller Job Task Exec Portal/P$ A database set Job info Job tasks Execution and results

Using Elastic db jobs Demo

If you want easy scale out of a.NET app: elastic db tools. If you want to be flexible on resource consumption: elastic database pools. Recap 33

If you want queries that span multiple dbs: elastic db query. If you want to run jobs on 1 to N dbs: elastic database jobs. Recap 34

Q/A

Thank You!