1. Keep your data in sync between SQL Server and Azure SQL Database
11/24/2018 1:31 PM
Keep your data in sync between SQL Server and Azure SQL Database
Xiaochen Wu
Senior Program Manager (Microsoft)
Kun Cheng
Principal Program Manager (Microsoft)
Sathyan Narasingh
Engineering Manager (GEP)
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Agenda What is Data Sync Why and how people use Data Sync
11/24/2018 1:31 PM
Agenda
What is Data Sync
Why and how people use Data Sync
Inside Data Sync
Demo
Q&A
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. 11/24/2018 1:31 PM
What is Data Sync
Synchronize data across multiple Azure SQL databases and SQL Server instances, in uni-direction or bi-direction.
Cloud App
On-prem App
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. Build global-scale apps in Azure
11/24/2018 1:31 PM
Build global-scale apps in Azure
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. 40+ Build global-scale apps in Azure
Azure regions
Archive global scale, in local regions
Provide instantaneous response time and consistent experience

6. SATHYAN NARASINGH
Engineering Manager
@snarasingh

7. Who We Are and What We Do
We combine innovative technology and end-to-end services to drive realized savings to your bottom line.
End-to-End Services
Procurement
Strategy
Procurement
Software
Category
Management
Procure-to-Pay
(P2P)
Sourcing
Support Services
Faster time
to value
Increased visibility
and control
Reduced
operating costs
Improved
compliance
Superior customer
service
Mexico City
Costa Rica
Princeton, NJ
Clark, NJ
London
Prague
Mumbai
Hyderabad
Shanghai
Singapore
Sydney
São Paulo
Dublin
Toronto

8. Procurement on the move

9. Architecture in Azure Case Study: http://bit.ly/2dD8kWk VNET IaaS
Azure ACS
Content Delivery Network
Media Services
Users
VNET
B2B Integration
Customer’s IDP
IaaS
ERP
(SAP, Oracle, PeopleSoft, JD Edwards, Microsoft Dynamics GP, Lawson, Asset Suite etc.)
File
cXML
EDI
ADFS, Site Minder, Ping Identity, One Login, etc.,
HTTPS
HTTPS
HTTPS
Integration
FTP
HTTPS
AS2
HTTPS
API
ARR
Web
BI Reports
HTTPS
HTTPS
SaaS
Traffic Manager
Worker
FTP
HTTPS
TCP
TCP
Flat Files
Service Bus Relay
Worker
TCP
TCP
HTTPS
Workflow
HTTPS
Search
Web Services
HTTPS
Devices
Key-based Authentication
IP Firewall
TDS / SSL
TDS / SSL
TDE
OLTP
TDE
Analytics
Tenant 1
Tenant 2
Configuration
Tenant 1
Tenant 2
TABLE Storage
BLOB Attachments
Azure Redis Cache
Monitoring and Automation
SQL Database Elastic Pool
Internet Traffic
Internal Traffic
Case Study:

10. Database Architecture
Azure SQL Database
Multi-tenant app with Single-tenant database model for isolation and defined by performance level - Azure SQL DB Elastic pools (Standard, Premium)
Each customer has two databases, serving OLTP and DW needs
Geo-Replication configured across regions
Same DB architecture used in each region
Web
Worker
Config DB
Users
Multi-Tenant Code
Azure SQL Elastic Pool
ETL
Tenant 1
Tenant 2
Tenant N
Read-Scale Out
Azure SQL Elastic Pool
Tenant DW 1
Tenant DW 2
Tenant DW N
Report BI Tools
Read Scale-Out : Through PS: Set-AzureRmSqlDatabase -ResourceGroupName “RSG" -ServerName “DBS" -DatabaseName “Test" -ReadScale Enabled; For ODBC Connection String: “ApplicationIntent=ReadOnly”

11. Multi-Region Deployment
North Europe
Ireland
West Europe
Netherlands
East US
Virginia
West US
California
East US 2
Virginia
India West
Mumbai
SE Asia
Singapore
Prod DR
Dev
Azure Express Route

12. Global Performance
About the Graph
Screenshot from New Relic APM
Analyzed 100K page views served from US East Coast
Average page-load time increase as latency increase from North America  Europe  Australia
Average page-load time varies from 2.5 to 4.4 seconds across different regions
CDN distributed static content from locations closest to user
How to globally distribute the database (for key application transactions)?
Need for an multi-master database?

13. Azure Traffic Manager (Performance Based Profile)
Data Sync Architecture
Region A
Region B
Redis Cache
Elastic Search
Azure Traffic Manager (Performance Based Profile)
Web
Worker
Config DB
Code Async
Tenant DB
Azure Data Sync ( Bi-Directional )
CDN
Configuring Azure Traffic Manager with Performance based profile helped us route the traffic to the region closest to user
Bi-Directional Azure Data Sync allowed us to accomplish Read & Write in all the regional databases at the same time
Rules has been setup for conflict resolution between data modifications
No major code changes needed to include Azure Data Sync
Architecture and DB level changes needed to in-corporate some complex use cases
Architecture changes done for other data stores like Elastic & Redis

14. GEP DB Design Changes
0003
Jacob
0x455..
07076
Region A Hub Database
0003
Tom
0x265..
25467
User “Tom” created in Region A and “Jacob” created in Region B at the same time
It is a completely valid case in multi–master DB scenario
User_id
Username
Password
Zipcode
0001
Max
0xab7..
98052
0002
John
0x2ff..
32114
Sync
User_id
Username
Password
Zipcode
0001
Max
0xab7..
98052
0002
John
0x2ff..
32114
Now Azure Data Sync runs
It will create a conflict since there are two different rows with the same primary key “0003”
Based on Azure Data Sync conflict rule “Hub wins” the data in Region B DB will be replaced from Region A DB, resulting in wrong data update / losing data
Region B Member Database
Conclusion: Primary key values must be globally unique

15. GEP DB Design Changes
c41bae..
5003
Jacob
0x455..
07076
45fbcr..
0003
Tom
0x265..
25467
Region A Hub Database
User_GUID
User_id
Username
Password
Zipcode
27ccbb..
0001
Max
0xab7..
98052
0df831..
0002
John
0x2ff..
32114
45fbcr..
0003
Tom
0x265
25467
C41bae.
5003
Jacob
0x455..
07076
User_GUID
User_id
Username
Password
Zipcode
27ccbb..
0001
Max
0xab7..
98052
0df831..
0002
John
0x2ff..
32114
Introduced new column User_GUID (uniqueidentifier datatype) as a Primary Key
Default value is added by using newid() function
ALTER TABLE [dbo].[Table_Name] ADD CONSTRAINT [DF_Table_Name_UserGUID] DEFAULT (newid()) FOR [User_GUID]
If a user gets created in 2 regional DBs at the same time, Azure Date Sync will now be able to sync data without conflict
Sync
User_GUID
User_id
Username
Password
Zipcode
27ccbb..
0001
Max
0xab7..
98052
0df831..
0002
John
0x2ff..
32114
User_GUID
User_id
Username
Password
Zipcode
27ccbb..
0001
Max
0xab7..
98052
0df831..
0002
John
0x2ff..
32114
45fbcr..
0003
Tom
0x265
25467
C41bae..
5003
Jacob
0x455..
07076
Region B Member Database

16. Azure Data Sync Lessons Learned
Azure Data Sync Points To Consider
Azure Data Sync frequency
Minimum sync interval is 5 minutes
Choose the right use case; GEP also implemented async write for cases where we cannot wait for 5 min
Monitoring & Alerts
Azure portal is the first level of monitoring; GEP implemented its own business level monitoring
We cannot trigger sync on-demand through application code
Limitations on the number of tables supported in a sync group (500 tables)
Schema changes need to be considered as part of DevOps
Auto DR & Failover
Need to reconfigure Azure Data Sync
Compliance requirements
Temporal & Auditing: Need a way to identify the data changes happening through Azure Data Sync
Demo

17. 11/24/2018 1:31 PM
Gartner predicts that by 2020, 90 percent of organizations will adopt hybrid infrastructure management capabilities.
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Demo: Contoso University
11/24/2018 1:31 PM
Demo: Contoso University
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. Sony Crackle

20. Company Overview
Crackle, Inc. is a free to use multi-platform video entertainment network of full length movies, TV shows and original programming.
Migrated from on-prem to Azure 6 years ago.
Store meta data in SQL Database partitioned by region and language. Store media data in blob storage.

21. Crackle Solution on Azure

22. Client Layer
Integration Layer
Application Layer
Data Layer

23. Crackle Data Sync Scenario
Cloud vendor independence
Use AWS RDS as DR standby
Use Data Sync to transfer data changes to RDS
Azure DC
AWS DC
Application Services
Application Services
SQL database
SQL Data Sync
RDS
SQL
BACPAC
BLOB S3
BLOB S3

24. Crackle Data Sync Scenario Cont.
Traffic Manager
Cloud vendor independence
Could use Traffic Manager to route traffic between Azure and AWS
Instead decided to go with SQL Database Geo- Replication
Azure DC
AWS DC
Application Services
Application Services
SQL database
replica
Geo Repl
SQL database
SQL Data Sync
RDS
SQL
BLOB S3
BLOB S3

25. Distribute data across regions to extend your business globally
11/24/2018 1:31 PM
Extend your on-premises assets and build hybrid applications before fully migrating to Azure.
Distribute data across different services, applications or environments
Power BI
Sales
System
Customer
Portal
Azure
Distribute data across regions to extend your business globally
Synchronize you data between databases hosted by different cloud vendors for cloud vendor independency
AWS (DR Site)
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. Flexibility and agility
Efficiency
Flexibility and agility
Peace of mind

27. 11/24/2018 1:31 PM
Inside Data Sync
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. Data Sync Concepts Sync Database Sync Group A Member Database
(West US)
Sync Group A
Sync Database
(West US)
Metadata & log
Member Database
(West US)
Member Database
Hub Database
Sync
Sync
Metadata & log
Cloud
Sync
On Premises
Local Sync Agent
Member Database (on-prem)
Sync Group B
(West US)

29. Tracking Data Changes Primary Key INSERT Trigger INSERT Trigger
UPDATE Trigger
DELETE Trigger
INSERT Trigger
Create triggers
UPDATE Trigger
User_id
Username
Password
Zipcode
0001
Max
0xab7..
98052
0002
John
0x2ff..
32114
0003
Tom
0x265..
90876
DELETE Trigger
0x762..
Create tracking table
Primary Key
User_id
Last_change_datetime
Sync_row_is_tombstoned ……
0004
Jack
0x29c..
25467
0004
6:01 9/24/17 ……
6:08 9/24/17 1
0002
6:05 9/24/17 ……

30. Sync Initialization Sync 0001 Max 0xab7.. 98052 9/1/2017 0002 John
User_id
Username
Password
Zipcode
LastUpdate
0001
Max
0xab7..
98052
9/1/2017
0002
John
0x2ff..
32114
1/5/2016
0003
Tom
0x265..
90876
1/3/2015
User_id
Username
Password
Zipcode
0001
Max
0xab7..
98052
0002
John
0x2ff..
32114
0003
Tom
0x265..
90876
User_id
Username
Password
Zipcode

31. Synchronize Data Hub Database Member Database 1 Member Database 2 0001
User_id
Username
Password
Zipcode
0001
Max
0xab7..
98052
0002
John
0x2ff..
32114
0003
Tom
0x265..
90876
0001
Max
0xab7..
25467
0001
Max
0xab7..
25467
0001
Max
0xab7..
25467
0003
Tom
0x265..
78654
Sync
Sync
User_id
Username
Password
Zipcode
0001
Max
0xab7..
98052
0002
John
0x2ff..
32114
0003
Tom
0x265..
90876
User_id
Username
Password
Zipcode
0001
Max
0xab7..
98052
0002
John
0x2ff..
32114
0003
Tom
0x265..
90876
0002
John
0x2ff..
37648
0003
Tom
0x265..
78654
0003
Tom
0x265..
78654
Member Database 1
Member Database 2

32. Resolve Conflicts Hub Database Conflict resolution = Hub Win
User_id
Username
Password
Zipcode
0001
Max
0xab7..
98052
0002
John
0x2ff..
32114
0003
Tom
0x265..
90876
Conflict resolution = Member Win
0x762..
0x762..
0x5A9..
0x762..
Sync
Sync
User_id
Username
Password
Zipcode
0001
Max
0xab7..
98052
0002
John
0x2ff..
32114
0003
Tom
0x265..
90876
User_id
Username
Password
Zipcode
0001
Max
0xab7..
98052
0002
John
0x2ff..
32114
0003
Tom
0x265..
90876
0x231..
0x231..
0x231..
0x5A9..
0x5A9..
0x5A9..
Member Database 1
Member Database 2

33. Demo: Setup a Sync Group
11/24/2018 1:31 PM
Demo: Setup a Sync Group
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

34. Demo: Monitor Sync Group in OMS
11/24/2018 1:31 PM
Demo: Monitor Sync Group in OMS
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

35. Data Sync & Other Data Integration Technologies
11/24/2018 1:31 PM
Data Sync & Other Data Integration Technologies
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

36. Data Sync & Active Geo Replication
11/24/2018 1:31 PM
Data Sync & Active Geo Replication
Data Sync
Active Geo Replication
Pros
Active-active support
Sync selected tables and columns
Sync between on-prem and Azure SQL Database
Seconds level latency
Transactional consistency
Auto failover with failover group
Designed for DR or read-only scaling
Cons
5 min or more latency
No transactional consistency
Higher performance impact
Non-Writeable secondaries
Replicates the entire database
Secondary must use same edition
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

37. Data Sync & Transactional Replication
11/24/2018 1:31 PM
Data Sync & Transactional Replication
Data Sync
Transactional Replication
Pros
Active-active support
Bi-directional between on-prem and Azure SQL Database
Lower latency
Transactional consistency
Designed for on-prem to Azure DB replication or migration
Cons
5 min or more latency
No transactional consistency
Higher performance impact
On-prem/Azure SQL VM to Azure SQL Database only
High maintenance cost
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

38. Data Sync & SSIS Data Sync SSIS Pros Easy configuration
11/24/2018 1:31 PM
Data Sync & SSIS
Data Sync
SSIS
Pros
Easy configuration
Support transformation
Support more types of sources and destinations
Designed for ETL
Cons
Transformation is not supported
Domain knowledge required
Need extra hosted services (VM or SSIS PaaS)
Need additional change tracking technologies
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

39. 11/24/2018 1:31 PM
Takeaways
Azure SQL Database feature (in public preview) to sync data between on-prem SQL Server or Azure SQL Databases
Build global scale or hybrid data applications using Data Sync
How Data Sync works
Choose the right data integration technology for your use cases
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

40. References Online documentation: Powershell Samples: Other resources:
11/24/2018 1:31 PM
References
Online documentation:
Getting Started with Azure SQL Data Sync (Preview)
Sync data across multiple cloud and on-premises databases with SQL Data Sync
Powershell Samples:
Sync data between SQL databases
Sync data between SQL Database and SQL Server on-premises
Other resources:
Monitoring Azure SQL Data Sync using OMS Log Analytics
Azure SQL Database Blog
SQL Customer Advisory Team Blog
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

41. 11/24/2018 1:31 PM
Q&A
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.