Jon Erdman, Senior Database Developer, Enova Financial

Similar presentations


Presentation on theme: "Jon Erdman, Senior Database Developer, Enova Financial"— Presentation transcript:

1 Jon Erdman, Senior Database Developer, Enova Financial
Using LVM2 to Provide Copies of Production Data for Testing Jon Erdman, Senior Database Developer, Enova Financial

2 Generated test data often doesn't represent production accurately
Generated test data often doesn't represent production accurately Full copies of prod use lots of space Refreshing multiple copies takes a long time Lots of developer contention for a single copy Production data is sometimes sensitive 2

3 Snapshots can solve many of these problems
Snapshots can solve many of these problems

4 LVM2 = Linux Volume Manager Block level snapshots Nearly instantaneous
LVM2 = Linux Volume Manager Block level snapshots Nearly instantaneous Use only the space of one copy + deltas Built into Linux 4

5 In LVM2, snapshots are read/write by default
What is an LVM2 snapshot? A new block device which presents an exact copy of a logical volume, frozen at some point in time In LVM2, snapshots are read/write by default 5

6 Snapshots work by creating an exception table, which is used to keep track of which blocks have been changed If a block is to be changed on the origin, it is first copied to the snapshot, marked as copied in the exception table, and then the new data is written to the original volume If data is written to the snapshot, that block is marked in the exception table as used, and never gets copied from the original volume 6

7 Overview of the procedure: Snap production volume (Checkpoint?)
Overview of the procedure: Snap production volume (Checkpoint?) Copy contents of snap to volume on test server Snap test server base volume and mount Start pg instance Rinse and repeat 7

8 Why is it “safe” to snap a running server?
Why is it “safe” to snap a running server? Instantaneous consistent view of entire data volume PITR / Crash Recovery 8

9 Uses no extra space until changes are made
Benefits: Uses no extra space until changes are made Copy process does not disturb existing dev instances Lots of copies New copies made quickly Copies reverted quickly Only one restore performed Only one copy to scrub

10 Additional IO load on master during copy Copies are entire clusters
Drawbacks: Additional IO load on master during copy Copies are entire clusters Each copy it's own instance, thus on it's own port / IP Second copy needed during refresh Refreshing affects all copies Not a good fit if devs write lots of data, rather than mostly DDL 10

11 Other approaches: Btrfs ZFS SAN/NAS

12 Demo / Discussion

13 We're hiring! We're looking for both DBAs and database developers and have opportunities in both Chicago and Austin, TX 13

14 Questions? jerdman@enovafinancial.com jon@progressivepractice.com
Q & A Questions? 14


Download ppt "Jon Erdman, Senior Database Developer, Enova Financial"

Similar presentations


Ads by Google