Presentation is loading. Please wait.

Presentation is loading. Please wait.

Database Mirroring in the Real World

Similar presentations


Presentation on theme: "Database Mirroring in the Real World"— Presentation transcript:

1 Database Mirroring in the Real World
Craig Purnell

2 About Me Database Administrator with Baker Hostetler
Experience with SQL Server 7/2000 and up Previously consulted in the transportation industry, programming custom UNIX ERP solutions MCSE / MCSA / MCITP (DBA) etc.

3 Overview Mirroring is a high availability technology
Continuous stream of log records are sent to the mirror and “replayed” Mirror is unavailable for client connections Protects a single database at a time It is a software solution that increases database availability by providing a hot standby database on another instance of SQL Server. - Mirroring is log shipping on steroids! log shipping is a series of Backup/restores , mirroring is realtime / near realtime. -You can have only 1 mirrored copy of each database. - Mirroring always happens at the database level not instance. Unavailable but You can, however, create a snapshot on the mirror database MIRRORING should be just one of your tools in your H.A. toolkit.

4 Mirroring Demystified
Synchronous with Witness No data loss Auto detection of Failure / Failover Synchronous without Witness Possibility of downtime Asynchronous Manual failover Possibility of data loss. Synchronous operates in FULL transaction safety mode. The Key difference is that Sync writes to both sets of log drives. Async is a best effort. Having a witness gives you automatic failover capability. The witness arbitrates between the 2 servers to determine who is the Principal. - WE WILL BE FOCUSING MOSTLY ON ASYNC MIRRORING

5 Asynchronous Mirroring
The key point here is that in asynchronous mirroring, SQL does not wait for writes to complete on the mirror. The mirror server could be 2000 miles away. This is only available in Enterprise editon. Works great for geographic DR. Think of it as a poor man’s SAN replication. Not as integrated but a whole lot cheaper.

6 Synchronous Mirroring
In Synchronous mirroring, the principal waits for an achnowlegement from the Mirror before signaling the client that the tranaction is complete. Not recommended for WAN

7 Synchronous Mirroring with Witness
Microsoft’s SAP system Primary Data Center Log Shipping Disaster Recovery Data Center Principal Mirror Witness Synchronous Database Mirroring Log Shipping Secondary Replaces the clustering single point of failure! - Downside : You now need 2 SANs - Nice Side-effect: You now have to use Log Shipping for DR.

8 Mirroring with Certificates
Different domains without a trust (mergers, acquisitions, DMZ) Internet (partners, affiliates) NOT VERY COMMON Only devote 1 slide to this. Over 5 years of working with mirroring, I have never come across the need to use Certs. Common scenarios include domains without trust relationships. Internet – collocated server facility. OR – business partner , OR to a stand alone server

9 Mirroring at Baker Hostetler
11 Offices; All SQL is centralized Early adopter of the technology Was mirroring from Cleveland to Denver Currently in a state of transition…. Future: Eagan, MN to Cleveland -Early adopter: We’ve been using the technology since we went live on SQL 2005-SP1 . - We mostly use asynchronous mirroring – coupled with Failover Clustering for H.A. Our Sharepoint/SCCM instance uses synchronous mirroring.

10 Mirroring at Baker Hostetler
We have 5 instances of SQL on a 4 Node SQL 2008 cluster. This is our core LOB application : Document Management System. This is just one small part of our environment. will be mirroring it back to cleveland. Other instance include Sharepoint, General Purpose, Citrix family, and Legal specific. Hardware is DELL 710 with Netapp SAN with iSCSI to the Netapp

11 Endpoints (1) Required by mirroring
SQL Server Object that maps in TCP socket Authentication Encryption (RC-4 etc) See BOL for options - An endpoint is an object in SQL Server that maps in a TCP socket connection from the operating system. Database mirroring sessions use this as their communications PIPE. Endpoints are at the instance level, for all databases. We will review the specific syntax in the demo

12 Limitations (32bit) Best Practice: 10 mirrored databases per server.
Effects of mirroring on the target: SQL runs out of VAS (32 bit) Band-aid: -g512 (increase memtoleave) - Best Practice as noted in the DB mirroring white paper is 10 (32 bit) On one server, we had around 40. That is the limit. We have seen VAS memory exhaustion as a result of high # of databases specifically on 32 bit. - Memtoleave is outside of the buffer pool, (for Linked servers, extended stored procedures, CLR) -Mirroring is very thread intensive -> each thread uses memory buffers -g startup option – default is 256

13 Mirror Configuration Identical Hardware not required
Memory / Disk requirements Drive Letter layouts File growths/shrinks Unlike clustering identical hardware is not required - memory and disk space should be close however. It’s best to lay out your mirror server disks exactly the same – this makes things easier for restores. File growths / Dbcc shrinkfiles also happen on the mirror.. So watch out for your drive space!

14 Demo #1 (T-SQL) Megadata from I1 to I2
DISCUSS HOW THE DEMOS ARE SET UP DC / SERVERA / SERVERB We will be setting up Megadata using T-SQL Complete set up (I1-I2) Security Endpoints Restore DB / Restore T-LOG Mirror

15 WAN acceleration (1) Riverbed Steelhead (hardware based)
Turn off encryption and compression SQL 2008 Effects (native compression) Trace Flag -T1462 - We are using Riverbed steelhead for wan optimization . Operates at Layer 2 in the OSI Think of the Steelhead as ZIP file for the WAN. (26 a’s = 1a + checksum), - ~75% compression ratio on log stream (as reported by the Steelhead) - Comparison : SQL 2008 offers about 73% compression at a cost of higher CPU. - So it’s really nearly a wash. - T is available in SQL 2008 to disable compression - Explain the hazards of disabling compression in SQL 2008 (no compressed backups)

16 WAN Acceleration (2) Index Rebuilds : 70-75% compression
We have had good luck with the Steelheads, with the exception of problems with their dual WAN partitioning algorithm. Index Rebuilds : 70-75% compression

17 MPLS Network QOS QOS = Quality of Service
- Basically we tag the packets sent to AT&T and they are supposed to honor their priority. - Voice – IP phones - Interactive – Terminal services / Citrix - Business Critical – LOB applications - Default - try and put mirroring here --You really want to stay out of scavenger here. Can’t stress enough.

18 Mirroring Timeouts -What we believe finally was the issue was how the Steelhead operates in a dual-WAN environment. -How it partitions the bandwith.

19 Mirroring Timeouts We were able to reproduce this in the lab with a 10MB crossover and a big database. Test: 2 servers / 10Mbps crossover, dedicated endpoints

20 Demo #2 (GUI) Adventureworks from I1 to I2
GUI DEMO to MIRROR ADVENTUREWORKS START AT FULLY RESTORED VERSION OF ADVENTUREWORKS; CONFIGURE SECURITY DIALOG: AUTOMATICALLY CREATES THE ENDPOINTS IF THEY ARE NOT THERE. MAKE SURE WE SHOW THE BAD DIALOG. THEN: DEMONSTRATE FAILOVER WITH THE GUI SHOW HOW THE ROLES ARE REVERSED TRY AND QUERY THE MIRROR DATABASE THEN CREATE A SNAPSHOT ON THE MIRROR.

21 Mirror on a Dedicated IP
Assign IP to 2nd NIC Create Endpoint to listen only on that IP Repeat for mirror Have Network ops team configure routing This is something we were experimenting with. We have done this in Test by using a crossover cable. WHY do we care about this ? If you are using redundant WAN, this can allow Net ops to fine tune where your traffic goes.

22 Mirroring and Licensing (1)
Witness can be SQL Express Mirror server can be unlicensed – as long as you do not have anything running queries against the instance. No snapshots allowed. See SQL Licensing White paper for more details Important Disclaimer: I am not a licensing expert. - Unlicensed mirror, as long as no production databases on it. -WITNESS CAN ALSO BE SQL EXPRESS.

23 Mirroring and Licensing (2)
Must use Standard or Enterprise Synchronous vs. Asynchronous mirroring Can’t mix versions. Only enterprise edition can perform async mirroring.

24 Demo #3 (T-SQL) Rolling Upgrade of Northwind and Pubs
IN THE NEXT DEMO , WE HAVE NORTHWIND AND PUBS MIRRORED FROM A SQL 2005 INSTANCE TO SQL 2008 R2 AND THEN FAILOVER AND BREAK THE MIRROR. THE NET EFFECT WILL BE TO UPGRADE THE DATABASE QUICKLY ALSO WORKS FOR QUICKLY MOVING Database between Datacenters (or Servers) with low down time.

25 Mirroring: Lessons Learned
Not a substitute for T-Log backups Network latency definitely is a factor Timeouts due to “design flaw” in Riverbed dual WAN configuration Log Buildup on Primary QOS network rules Great way to shovel data from A to B - Synchronous mirroring on the WAN: Don’t try it. You still have to do Transaction Log backups! - The mirroring code inside the SQL engine is especially sensitive to network latency.(over 100ms is usually the breaking point) If a WAN link goes down, the log will fill up on your production log drive. - Index rebuilds place a heavy load on the log drives as well - Shoveling data : Rapid upgrades and Server Moves

26 Index Rebuilds Carefully Schedule production instances
Selective rebuild script really helps. Your mileage may vary. You want to Carefully schedule production rebuilds so they don’t interfere with each other on the WAN

27 Monitoring Mirroring (1)
System Monitor Counters T-SQL Mirroring Monitor GUI DMVs: sys.dm_db_mirroring_connections sys.database_mirroring sys.database_mirroring_witnesses sys.database_mirroring_endpoints sys.tcp_endpoints sys.server_principals sys.database_recovery_status There are multiple ways to monitor

28 Demo #4 Index Rebuilds and Monitoring
Effect on Send queue Usually leads to timeouts Show: multiple ways to monitor: LAUNCH PERFMON AND MIRRORING MONITOR BEFORE YOU KICK OFF THE REBUILD. Perfmon mirroring monitor. T-SQL

29 SQL Agent Alerting Configure SQL Agent to alert you of mirroring problems See Technical Article: Alerting on Mirroring Events (escape macros are wrong). Watch for the bug in the white paper also. This deals with how the escape macros are used in SQL Agent – macros changed at SQL 2005 SP1, and the paper is out dated. HOWEVER: there is really a better way. Configure a monitor job to look at your mirrored servers and you every 2 hours if they are behind. The correct code (macros) are below: DB Mirroring: Record State Changes Job Correct Syntax : INSERT INTO dbo.[DB Mirroring State Changes] ( [Event Time], [Event Description], [New State], [Database] ) VALUES ( '$(ESCAPE_SQUOTE(WMI(StartTime)))', '$(ESCAPE_SQUOTE(WMI(TextData)))', '$(ESCAPE_SQUOTE(WMI(State)))', '$(ESCAPE_SQUOTE(WMI(DatabaseName)))')

30 Error messages: T-SQL vs. GUI
Msg 1418, Level 16, State 1, Line 1 The server network address "TCP://MYMIRROREDSERVER.domainname.com:5022" can not be reached or does not exist. Check the network address name and that the ports for the local and remote endpoints are operational. 1) The GUI tends to give you better error messages. 2) You can’t mix mirroring between Standard and Enterprise 3) 1418 can manifest itself also as Endpoint configuration problems. Same error condition (mixing SQL versions)

31 Troubleshooting netstat –ano tcping hostname 5022
Verify endpoints are using the same encryption, authentication Are you running SQL as LocalSystem? Use the GUI to configure the endpoints for you Try and recreate the problem with the GUI Recreate it – T-SQL vs. the GUI. -if you are running SQL as localsystem, you must use certificates

32 Troubleshooting Verify the service accounts are Logins
Verify both service accounts has CONNECT rights on the endpoint(s) You must make a Backup and a T-Log backup AFTER the DB is in full recovery mode (watch for extraneous Tlog backups) Look in the Error log Windows Firewall must be configured IPV6! Netstat –ano Watch for IPV6

33 Other Issues Logins are NOT transferred
Use sp_help_revlogin to get an exact sid / SQL login list SQL Agent Jobs are not transferred Extended Stored Procedures Linked servers Basically anything that lives outside the user databases needs to be synced on a periodic basis. Keep the logins synced – could to be a weekly agent job

34 Key Takeaways Database Mirroring can eliminate the single point of failure of clustering (aka the shared disk problem) Database Mirroring, coupled with clustering can raise your business database uptime an order of magnitude. This technology can be implemented without the need of expensive SAN…just a little creativity on YOUR part All you need to do this is SQL Server (in the box). Mirroring is something do definitely want to consider. Mirroring , coupled with something else can really give your HA a boost. (4 9’s or even 5 9’s)

35 Any Questions? Thank you for your time.

36 Contact Information or

37 References (1) Tcping http://www.elifulkerson.com/projects/tcping.php
SQL Server Licensing White Paper Asynchronous Database Mirroring with Log Compression in SQL Server 2008 Database Mirroring White Paper Database Mirroring: Best Practices and Performance Considerations Mirroring with Certificates: Transferring Logins and Passwords between instances:

38 References (2) Alerting on Database Mirroring Events How to: Minimize Downtime for Mirrored Databases When Upgrading Server Instances Asynchronous Database Mirroring with Log Compression in SQL Server 2008 High Availability and Disaster Recovery for Microsoft’s SAP Data Tier: A SQL Server 2008 Technical Case Study

39 Resources SHunra VE Desktop – simulates latency
To simulate satellite latency in my test environment, I used a product from called VE Desktop. It is a simple to use application providing programmatic tweaks to the network adapter.

40 Review of H.A. technology
We grabbed this picture from SQL PASS , very informative

41 Monitoring Mirroring (3)
WHERE: @dbname – the mirrored database @interval - 0 = last row, 1 last two hours, 2 last four, 3 last eight, 4 last day, 5 last two days, 6 last 100, 7 last 500, 8 last 1000, 9 ALL sp_dbmmonitorupdate queries the underlying perfmon counters where we feel there is an underlying bug.

42 Supports automatic failover? Supports manual failover?
Mirroring Options Mode Alternate name Requires a witness? Supports automatic failover? Supports manual failover? Waits for mirror to write log block before client transaction is committed? High Availability Synchronous (with Witness) Yes Yes – if mirror not connected then no delay High Protection Synchronous No High Performance Asynchronous No, must be changed to Synchronous

43 Network Overview Craig –
We maintain a disaster recovery warm site at our office in Denver , CO The latency to Denver is about 65ms. So you see why we can’t do synchronous.

44 VLDB Considerations Multiple Log Restores (not in LSN window)
FEDEXnet (portable hard drive) Data safeguarding implications Disable Log Backups while attempting to resynchronize the mirror. In today’s day, any database backup is going out as encrypted. Don’t do re-indexing while drive is in flight. (copy massive amounts of data over the WAN.)

45 Timeouts OKB Sent -> SQL Refuses to transmit
Infinite to Send -> Can’t see mirror – why? Ping –l hostname (good) (better) Work closely with Network WAN team

46 Common Errors Msg 1412, Level 16, State 0, Line 1The remote copy of database “Adventureworks" has not been rolled forward to a point in time that is encompassed in the local copy of the database log. Msg 1408, Level 16, State 0, Line 1 The remote copy of database "Northwind" is not recovered far enough to enable database mirroring. Error 1416: DB is in standby or recovered mode. The last transaction log must be restored with NORECOVERY option. Errors 1418 and 1486 – these are communications related errors.

47 WAN Traffic Report

48 Mirroring Demystified
Operating Mode Transaction safety Transfer mechanism Quorum required Witness server Failover Type High Availability FULL Synchronous Y Automatic or Manual High Protection N Manual Only High Performance OFF Asynchronous N/A Forced Only MIRRORING should be just one of your tools in your H.A. toolkit. The Key difference is that Sync writes to both sets of log drives. Async is a best effort. Having a witness gives you automatic failover. The witness arbitrates between the 2 servers to determine who is the Principal. - WE WILL BE FOCUSING MOSTLY ON ASYNC MIRRORING


Download ppt "Database Mirroring in the Real World"

Similar presentations


Ads by Google