1. Shard Attack - Horizontal Scaling in Azure SQL Database
David Postlethwaite
Abstract
Shard Attack - Horizontal Scaling in Azure SQL Database
Traditionally, when a server starts to reach its limit we have simply thrown more resources at it, either more CPU, memory or disk.
However there comes a point, especially in the cloud, where it is no longer possible to add more resources to a database. Here we need a different solution.
Instead of scaling up we must scale out, sometimes called horizontal scaling or sharding.
In this talk we will look at how to scale out in Microsoft Azure SQL database using the Azure Elastic Database tools
We will look at the requirements and options for horizontal scaling in Azure and then we will have a go at sharding an Azure SQL database and then querying and updating the different shards
We will be using t-sql, PowerShell and c# so come prepared for some serious coding
About the Author
David Postlethwaite has been a SQL Server and Oracle DBA for Liverpool Victoria in Bournemouth, England for the last 7 years.
He manages both Oracle and SQL including DBMS, SSIS, SSAS and Reporting Services.
Before that he was a .NET developer and way back in history a Windows and Netware administrator.
He is an occasional blogger on
Shard Attack - Horizontal Scaling in Azure SQL Database

2. BIG Thanks to SQL Sat Denmark sponsors
GOLD
SILVER
BRONZE

3. Raffle and goodbye Beer
Remember to visit the sponsors, stay for the raffle and goodbye beers  Join our sponsors for a lunch break session in : cust 0.01 and cust 1.06 We hope you’ll all have a great Saturday. Regis, Kenneth

4. David Postlethwaite Liverpool Victoria LV= SQL and Oracle DBA
MCSE 2014 Data Platform
MCITP 2008, 2005
Oracle OCA
25 years IT Experience
10 years as DBA
Blog: gethynellis.com
@postledm
Welcome
Good Afternoon.
My name is David Postlethwaite, I am a SQL DBA for a large financial services company on the south coast of England
I have been working as a DBA for the last 10 years
I currently manage both SQL and Oracle instances.
Previous to that I was a developer using .NET, SQL, Access, FoxPro and Oracle
And way back in time I was a Windows and NetWare administrator.
I am an occasional contributor to the blog on gethynellis.com  

5. The problem Your database is growing.
You need more disk space, more compute CPU and more memory
What can you do?
The problem
Here is our problem
You’ve built a SQL database and you are so successful that it keeps growing.
More people want access, the amount of data is growing, the queries are taking longer and longer
It needs more power otherwise it will be too slow and your customers might start looking elsewhere
What a great problem to have
What can we do to fix it?

6. The Solutions Vertical Scaling (Scale-up): Pros Cons
Adding more processors, RAM or disk space to our server
Pros
Generally less challenging to implement
No New licence costs
Small increase in power consumption
Less network hardware
Cons
Cost for large systems
Limit to how big a server can grow
Risk of hardware failure causing major outage
Doesn’t Support Geo Location
Vertical Scaling
Traditionally when we need more power we’ve just added more memory or bigger disks or extra cpu to our current server.
Worse case we’ve gone out and bought a bigger and more powerful server to replace our current one
It’s what we’ve done for the last 40 years, its easy to do, just plug the new bits in and off you go.
There’s little to go wrong, doesn’t tend to cost too much for small changes and it doesn't take much effort.
I suspect everyone here has done this in the past.
But buying a bigger and faster server can eventually become quite expensive.
And how big can you buy?
There comes a point when simply buying a bigger and faster server becomes impractical or doesn’t give the performance increase that you need.
And running everything on one box, putting all your eggs in one basket, can become risky
And what if we have customers all over the world?
Do we want them all coming back to one server, in one location.
Would it be better for them to write to a database near to where they are?

7. The Solutions Horizontal Scaling (Scale-out): Pros Cons
Adding more servers with similar CPU and RAM to compliment the original server.
Pros
Usually cheaper than scaling vertically
In theory can scale infinitely
Easier to run fault-tolerance
Easy to upgrade
Cons
More licence costs
Bigger footprint in your Data Center
More Power Consumption
More Infrastructure
Technically more complex
Horizontal Scaling
An alternative to just buying bigger is what is known as horizontal scaling, or scaling out.
Here you buy several smaller servers and link them together so even though your database is spread over several servers it looks and acts like one database.
This has been a common solution in the NoSQL world for a quite a long time
Buying several small servers is often cheaper than one massive server and in theory there is no limit to how big you can expand it.
Having lots of servers gives more resilience and in theory you can upgrade or apply patches much more easily
But more servers means more licence costs, more electrical power and more infrastructure and is technically a lot more complex to setup and manage
In SQL we can try and do this using replication but anyone who has used replication will know what fun that can be!

8. The Problem in Azure In Azure SQL Database BUT
Your database is growing. You need more disk space and more compute
BUT
Maximum Database Size is 1 - 4TB
Premium Tier is very, very expensive
Outgrowing one Azure SQL database
Need physical separation for security or compliance or geopolitical reasons
Keep as one logical database
Scaling out is a solution
Also know as Sharding
The Problem in Azure
In Azure we can have similar problems
Azure SQL Database has some serious limitations, the maximum size for standard database is only 1TB, which isn’t very big in this day and age
rising to 4TB for premium
but a premium database for either DTU or vCPU is very, very expensive
If your transaction throughput is out growing your Azure database
or if you want to separate your data into different locations around the world
or if you need to isolate different customers into separate databases for compliance, performance, or geopolitical reasons
but you still want to keep your data as one logical database for your own administration
then scaling out or horizontal scaling is the ideal way to achieve this.
* The splitting of data across multiple databases is also known as Sharding
And it is Microsoft’s implementation of Sharding in Azure SQL database which we are going to look at today

9. Terminology
Sharding - Splitting identically structured data across several independent databases
Terminology
Before we delve too deep let’s have a look at some of the terminology linked to Sharding.
Each database vendor uses slightly different wording for certain terms but these are the ones used by Microsoft.
Sharding is defined as a technique to distribute large amounts of identically structured data across a number of independent databases.
*The end users that use a shard are referred to as tenants.
* They will be defined by a shard key. Each tenant will have a unique shard key value
* The databases that contain our tenants are called Shards
* The shard key and hence each tenant will be linked to just one database or shard
* There will be a table called the Shard Map that keep track of which shard key belongs to which shard
* In Azure this is held on its own database called the Shard Map Manager
* All of the of shards that belong to the shard map is called a Shard Set.
* Tables that are sharded, in other words the tables that have data distributed across the shards based on their sharding key are called Sharded tables.
* Tables that are not sharded, tables that need to exist on each shard but don’t need to be split. Are called Reference tables:
All of the data associated with a single value of a sharding key on a shard is called a Shardlet
For sharding in Azure, to read and write to our shards we need an application that is built with the * Azure Elastic Database Client Library.
The client library is available for .NET and also for Java.
Finally there are two features within Azure which can also be used with sharded databases.
These are not part of the Client Library and can be used on normal Azure SQL databases as well as sharded database
Firstly
Elastic Database Query is bit a like a linked server. As you have probably found out Traditional Linked servers aren’t available in Azure SQL database.
This allows you to run queries against another Azure SQL database or Azure Data Warehouse.
Secondly
Elastic Database jobs : This is a scheduler within Azure SQL database that allows you run native t-sql scripts against multiple databases in one go.
So is very handy for your scheduled admin tasks that need to run on each shard
Shard Map Manager 1
Shard Map Manager 
DB Name
Country_ID
DB 1 2
DB 2 3
DB 3 4 2
Client App built with
Azure Elastic Database
Client Library
(.NET or Java)
Shard Map 3
Shardlet 4
Elastic Database Query
Sharded Table 
Address ID
Address 1 2
Address 2
Address 3 3
Reference Table 
Country ID
Denmark 2
Poland 3
Austria 4 5
Elastic Database Jobs
(Elastic Job Agent)
Shard Set
Tenant
Shard key
Shards
Sharded Tables
Reference Tables

10. Single and Multi Tenant
Tenant: A person who occupies land or property rented from a landlord.
Multi-tenant and single-tenant
When we come to split our data into our shards we have two different ways we can do this
The simplest approach is to create a separate database for each tenant.
This is know as  single tenant sharding 
It provides complete isolation for each tenant allowing easy backup and restore. You can also scale each tenant or take the database offline without affecting anyone else.
With single tenant, each database is associated with only one Shard Key.
The alternative to single-tenant is to place multiple tenants together on the same shard
This is called  multi-tenant sharding 
This is suitable for an application that manages large numbers of small tenants.
In multi-tenant sharding, the rows in the sharded database tables must have the sharding key in them to distinguish between the different tenants.
In single tenant sharding because there is only one tenant the rows don’t necessarily have to have the sharding key
In both cases the application tier is responsible for routing a tenant’s request to the appropriate database.

11. Multi Tenant. List and Range Mapping
List Mapping
Specific IDs in each database
Range Mapping
Range of IDs to each database
Multi Tenant. Range and List Mapping
There are two models for multi tenant sharding
Firstly List Mapping
Here we assign specific IDs to a particular database
This gives you the ability to control how many tenants go on each database. You would most likely use this when tenants vary a lot in size to ensure each shard has a similar work load.
Secondly Range Mapping
Here we assign a range of IDs to each database. This is generally used where each tenant is of a similar size

12. Growing the Shards
Use Horizontal scaling out to increase capacity of a shard. Use Vertical scaling out to increase the performance of a shard.
Growing the Shards
Naturally we can add more shards as our business grows or we can scale up a particular database to cope with the growth of one tenant.
So we can have vertical scaling inside horizontal scaling
((Have sip of wait so audience can read the slide))

13. Data Dependent Routing
Querying a particular shard using a specific sharding key.
Shard Map
Manager
Shard Key
Elastic Client Library
DB 1
GSM
cache
Data
Data Dependent Routing
Once we have created our shards we have two ways of querying our database.
Firstly Data Dependent Routing.
Here we query just one tenant using the sharding key
* Our client application, * written using the Elastic Client Library, will pass the * shard key of the tenant that it wants to read or write to, to the Sharp Map Manager.
The Shard Map Manager has a table called the * Global Shard Map.
It will look up the appropriate database name in this Sharp Map for the shard key that the client has sent and will pass the * database name back to the client application.
* The client app can now query the correct database.
The shard database will have its own shard map called the * Local Shard Map.
The shard will double check that you are querying the correct shard and that the shard is online for that shard key.
If all is well the database will run the query and * return the data to the client
If not, it will pass back an error
The Client API will * cache the shard details received from Shard Map Manager to avoid unwanted round trips on each request.
And this is why the Local Shard Map has to double check in case a shard has moved to a different database since the information was cached.
DB 1
GSM Global Shard Map
LSM Local Shard Map
LSM

14. Multi Shard Query Query multiple shards
Results sets are returned using UNION ALL
Shard Map
Manager
Elastic Client Library
SELECT count(*)
from addresses
UNION ALL result set
Multi Shard Query
Multi Shard Query, as the names suggests, queries multiple databases in one go. This will most likely be used for administration purposes
The * client application * written using the Elastic Client Library will query the * Shard Map Manager to find the names of all the shards *
It will then * query each in turn and return the results using the * UNION ALL operator
DB 1
DB 2
DB 3
DB 4
DB 5

15. Splitting and Merging Shards
Split: Move shardlets from one shard to another (typically new) shard.
Merge: Move shardlets from two shards to one shard
Shardlet Move: The act of moving a single shardlet to a different shard.
The Shard Map Manager must be updated with the change
Manually moving rows could lead to lost data
Split/Merge Service
Azure Web Service to move data between shards
Requires its own database
Can use web page to specify the shard to move
Or Write c# or PowerShell code into your application
Split/Merge Function
There will come a time when you need to move a tenant from one shard to another.
Perhaps a tenant in a multi tenant operation has become too big for its database and needs to moved to a new shard
Or maybe you need to consolidate some of your shards. Move some smaller tenants onto one shard
* If you do move tenants , then obviously its important that the shard map manager is updated with the change
You could move the data using t-sql and then use some PowerShell to update the shard map manually afterwards but this will almost certainly cause issues and lost data
* A better way is to use the Azure Split Merge Service.
This is an Azure web service that will manage the movement of the data, take the tenant offline whilst its being moved and update the shard maps
It requires another Azure SQL database called the split-merge database which will hold the status of each shard as tenants are being moved
so that the shard map manager knows if the tenant or shard is available.
You can use code, either PowerShell or c# to do this or you can manually type the information into the web page associated with the web service.
* The link on the screen gives basic instructions on how to build the Split Merge Azure service.
I have followed them to build the service. It is quite involved but not too difficult, even for a DBA!
(( but the instructions are slightly out of date in places because they use the MakeCert program which has been deprecated by Microsoft.
You should now be using PowerShell to create the certificates.
Fortunately I had access to a Windows 7 machine that had makecert still installed ))

16. Recovery Manager What if the GSM and LSM become out of sync?
Orphaned Sharp Maps
Shard is deleted
Shard is renamed
A geo-failover occurs.
A shard or the ShardMapManager is restored
to an earlier point-in time
Part of the Elastic Client Library
Available Methods
DetachShard()
DetectMappingDifferences()
ResolveMappingDifferences()
AttachShard()
Recovery Manager
It is essential that the Shard Map Manager is accurate and kept up to date otherwise data is going to end up on the wrong shard.
If the Global shard map in the Shard Map Manager and the local Shard maps on each Shard become out of sync then you are going to have problems, you won’t be able to get to data in the shards
Keeping these two databases in line is essential to the safe running of a shard.
If you delete or rename a shard, either by accident or on purpose, you are going to end up with an orphaned shard mapping.
So the shard will either need to be removed from the Global Shard Map or the shard keys pointed to the correct shard
If you are using Geo replication and a failover occurs you will need have mechanisms in place to update your application to point to the new location of the Shard Map Manager.
You’ll then need to update the shard mapping details for all shards in your shard maps have failed over as well.
If you restore an Azure SQL database it will be restored with a different name. Again the Shard Map Manager will need to know the location of the restored Shard.
And if either a shard or the ShardMapManager database is restored to an earlier point-in time then it will no longer be in sync with the other databases.
In these cases you are going to have to have mechanisms tried and tested to recover quickly
* Microsoft have helped by producing a c# class called the Recovery Manager which is part of the Elastic Client Library. This has a number of methods to help resolve your mapping issues.
I haven’t actually tried these but if you are going to build a system you will need to build these into your administrative application

17. The Demo Shards Reference Table Shard Set Shard key
Europe Addresses
Address
City
Country
Country_ID
27 Mier
Antwerp
Belgium 2
Rue de l’Etuve
Brussels
Trangravsvej 14
Copenhagen
Denmark 4
Sankt Augustin
Bonn
Germany 8
Friesenstraße 30
Koln
Oudegracht 196
Utrecht
Netherlands 14
Aleja Korfantego
Katowice
Poland 15
Prečna ulica 6
Ljubliana
Slovenia 19
Shard key
Shards
Europe Addresses 
City
Country_ID
Antwerp 2
Brussels
Copenhagen 4
Europe Addresses 
City
Country_ID
Bonn 8
Utrecht 14
Europe Addresses
City
Country_ID
Plovdiv 15
Ljubljana 19
Reference Table
Countries 
Country
Country_ID
Austria 1
Belgium 2
Denmark 4
Germany 8
Netherland 14
Poland 15
Slovenia 19
Elastic Scale Elements
This is what we are going to attempt
I have a table full of addresses from 21 different countries across Europe which we want to split up
In our demo we we will place records for different countries into * different shards
We will use the * Country ID as the Shard Key
* Shard 1 will just have ID 1 - 7, * which in our subset of data will include Belgium and Denmark
* Shard 2 will have IDs 8-14 * Germany and the Netherlands
* Shard 3 will have ID * which will include Poland and Slovenia
* This is our Shard Set
* We will also have a reference table
In our case this is the countries table which we will join with the addresses table to get the country name
We could create a copy of this on each shard but we will and use * Azure External Tables and * Elastic Database Queries to reference the table that exists on the Europe Addresses database
We are going to use T-SQL and PowerShell to setup the shards and a c# application to query them for this demo, so I hope you’re ready
You can download a zip file of all the code from gethynellis.com. I’ll give you the address at the end.
Azure External Tables
Elastic Database Query
Shard 1
Range: ID 1 -7
Shard 2
Range: ID 8-14
Shard 3
Range: ID 15-21
Shard Set

18. DEMO Let’s see if we can get it all to work
We have a large databases of addresses
We want to split it into several new databases / shards
And query it using a .NET application
DEMO
We have a large databases of addresses in an Elastic Pool
We want to split it into several new databases / shards
And query it using a .NET application

19. Disadvantages of Sharding
More Complex Code Can circumvent the sharding and write straight to the table Row Level Security Cross Shard ACID transaction not supported Reverential Integrity Performance Failover Corruption – Recovery Manager
Disadvantages of Sharding
Finally now you’ve heard all the good news, what are the problems inherent to Sharding schemes, not just Azure
Now your data is spread over several databases there are several issues that come to light
Firstly. You’ve got to use some pretty complex code to ensure you talk to the correct shard. We’ve gone beyond just a simple T-SQL query to using .NET c# or Java code.
Get this wrong and you could end up with data in the wrong place, seen by the wrong people and then GDPR fun and games start
Of course, there is no easy way to stop people just writing straight to the shard and by passing the Shard Map Manager completely.
Again you can have data in the wrong shard and GDPR issues
You could improve security on your shards be using row-level security, which was introduced in SQL Serve 2016 to filter which rows each tenant can access
Since the data is now in different databases, if you want to run a multi shard query you can’t guarantee transactions completing across all the shards.
You will have find other ways to ensure your transactions across all shards complete successfully
It’s also not really possible to perform joins across multiple shards due to performance constraints since data has to be compiled from multiple servers and the additional complexity of performing such cross-server queries.
Trying to enforce data integrity constraints across shards isn’t going to work. You are going to have to use your application code to achieve this and probably have regular SQL jobs to clean up any missing references
All the extra round trips to the Shard map Manager could slow your application down. Something you will have to test
If you use Geo replication for high availability you will need code to deal with that.
And a corrupt shard or mapping will cause issues and possible downtime so, again, you will need more code using the recovery manger to help with this.
I think its safe to say that Sharding is not for the faint hearted.

20. Conclusion Links My Source Code
Hopefully you now have a better understanding of Horizontal Partitioning
Links
My Source Code
Any Questions
@postledm

21. BIG Thanks to SQL Sat Denmark sponsors
GOLD
SILVER
BRONZE

22. Raffle and goodbye Beer
Remember to visit the sponsors, stay for the raffle and goodbye beers  Join our sponsors for a lunch break session in : cust 0.01 and cust 1.06 We hope you’ll all have a great Saturday. Regis, Kenneth