1© Copyright 2013 EMC Corporation. All rights reserved. Characterization of Incremental Data Changes for Efficient Data Protection Hyong Shim, Philip Shilane, & Windsor Hsu Backup Recovery Systems Division EMC Corporation

2© Copyright 2013 EMC Corporation. All rights reserved. Data Protection Environment SAN or LAN WAN Application Servers Primary Storage Data Protection Storage High I/O per sec. Medium Capacity Large Capacity Medium I/O per sec. Virtual Machines

3© Copyright 2013 EMC Corporation. All rights reserved. Contributions  Detailed analysis of data change characteristics from enterprise customers  Design for replication snapshots to lower overheads on primary storage.  Evaluation of overheads on data protection storage  Rules-of-thumb for storage engineers and administrators

4© Copyright 2013 EMC Corporation. All rights reserved. EMC Symmetrix VMAX Traces Trace Set#Volume# Storage Systems Duration hrs Estimated Capacity (GB) 1hr_1Wrt109, [78.3]71 [203] 1hr_1GBWrt16, [6.7]132 [262] 24hr_1GBWrt [1.2]318 [439]  Collected from enterprise customer sites

5© Copyright 2013 EMC Corporation. All rights reserved. Capacity and Write Footprint  Analysis for 1hr_1GBWrit  Not collected: applications using each volume

6© Copyright 2013 EMC Corporation. All rights reserved. I/O Properties Trace Set#Write reqs (1000s) Write size (GB) #Read reqs (1000s) Read size (GB) 1hr_1Wrt72 [510] 2 [31] 167 [1963] 5 [66] 1hr_1GBWrt429 [1270] 11 [80] 796 [4987] 25 [166] 24hr_1GBWrt1803 [4839] 51 [338] 7824 [23875] 242 [763]  X more read I/Os than write I/Os  X more GB read than written  High variability  More analysis in the paper

7© Copyright 2013 EMC Corporation. All rights reserved. Sequential vs. Random Write I/O  We measure how much data are written, on average, after seeking to a non-consecutive sector.  Selected most sequential and most random for analysis Storage Volume wwww Trace Timeline (w = Write I/O, r = Read I/O) rw Sequential Write I/O ( )/ 3 = 3

8© Copyright 2013 EMC Corporation. All rights reserved. rwwwrwrwwwwwwrw … Replication Interval 1 Transfer Period may require snapshot storage and I/O Trace Timeline (w = Write I/O, r = Read I/O) Storage Volume Sectors Replication Interval 2 Block Trace Analysis Methodology Create a snapshot to protect block data

9© Copyright 2013 EMC Corporation. All rights reserved. Replication Snapshot 0 Storage Volume state before transfer takes place 1234 Block: = Modified block to be transferred Trace Timeline (w = Write I/O)  Goal: Create a snapshot technique that is integrated with replication that decreases overheads on primary storage  Change block tracking records modified blocks for next replication interval, possibly with a bit vector.  A snapshot has to maintain block values against overwrites.

10© Copyright 2013 EMC Corporation. All rights reserved. Replication Snapshot  Baseline Snapshot: All writes cause copy-on-write 0 Storage Volume state before transfer takes place 1234 Block: = Modified block to be transferred Snapshot Area Trace Timeline (w = Write I/O) www Baseline Transfer in progress

11© Copyright 2013 EMC Corporation. All rights reserved. Replication Snapshot  Changed Block Replication Snapshot (CB): Only writes to tracked blocks cause copy-on-write Block: Snapshot Area www Baseline Transfer in progress CB

12© Copyright 2013 EMC Corporation. All rights reserved. Replication Snapshot  Changed Block with Early Release Replication Snapshot (CBER): Only writes to tracked blocks cause copy-on-write, and blocks are released once transferred Block: Snapshot Area www Baseline Transfer in progress CB CBER

13© Copyright 2013 EMC Corporation. All rights reserved. Replication Snapshot Block: Snapshot Area www Baseline CB CBER  Baseline Snapshot: All writes cause copy-on-write  Changed Block Replication Snapshot (CB): Only writes to tracked blocks cause copy-on-write  Changed Block with Early Release Replication Snapshot (CBER): Only writes to tracked blocks cause copy-on-write, and blocks are released once transferred = Modified block to be transferred

14© Copyright 2013 EMC Corporation. All rights reserved. Snapshot Storage Overheads Rule-of-thumb: Over-provision primary capacity by 8% for snapshots

15© Copyright 2013 EMC Corporation. All rights reserved. Snapshot I/O Overheads Rule-of-thumb: Over-provision primary I/O by 100% to support copy-on-write related write-amplification

16© Copyright 2013 EMC Corporation. All rights reserved. Snapshot I/O Overheads Rule-of-thumb: Over-provision primary I/O by 100% to support copy-on-write related write-amplification

17© Copyright 2013 EMC Corporation. All rights reserved. Transfer Size to Protection Storage Rule-of-thumb: 40% of written bytes are transferred to protection storage

18© Copyright 2013 EMC Corporation. All rights reserved. IOPS Requirements for Protection Storage Rule-of-thumb: Protection storage must support 20% of the I/O per second capabilities of primary storage

19© Copyright 2013 EMC Corporation. All rights reserved. Related Work  Trace analysis –Numerous publications Most closely related is Patterson [2002]  Snapshots –Common paradigm for storage but rarely integrated with incremental transfer techniques –Storage overheads Azagury [2002] and Shah [2006]  Synchronous Mirroring –Effective when change rates are low and geographic distance is small –We are focused on periodic, asynchronous replication

20© Copyright 2013 EMC Corporation. All rights reserved. Conclusion SAN or LAN WAN Application Servers Primary Storage Data Protection Storage High I/O per sec. Medium Capacity Large Capacity Medium I/O per sec.

21© Copyright 2013 EMC Corporation. All rights reserved. Conclusion  Trace analysis shows diversity of storage characteristics  Snapshot overheads on primary storage can be decreased by improved integration with network transfer  Sequential versus random access patterns affect incremental change patterns on both primary and protection storage

22© Copyright 2013 EMC Corporation. All rights reserved. Rules-of-Thumb  Over-provision primary capacity by 8% for snapshots  Over-provision primary I/O by 100% to support copy-on-write related write-amplification  A write buffer decreases snapshot I/O overheads but has little impact on storage overheads  40% of written bytes are transferred to protection storage  Schedule at least 6 hours between transfers to minimize clean data in transferred blocks  Schedule at least 12 hours between transfers to minimize peak network bandwidth requirements  Protection storage must support 20% of the I/O per second capabilities of primary storage

23© Copyright 2013 EMC Corporation. All rights reserved. Questions?

25© Copyright 2013 EMC Corporation. All rights reserved. Trace Analysis: Replication of Snapshots The amount of data to replicate drops in half with 12 hours between snapshots. 4KB results are compared to Patterson 2002.

26© Copyright 2013 EMC Corporation. All rights reserved. I/O Per Second (IOPS) Request Rate Trace SetAverage Write Rate Peak Write Rate 10ms Average Read Rate Peak Read Rate 10 ms 1hr_1Wrt0.7 [8] 1762 [2602] 2 [25] 1693 [2457] 1hr_1GBWrt15 [37] 4360 [4379] 29 [118] 3603 [4135] 24hr_1GBWrt20 [55] 9004 [8165] 89 [269] 5647 [7012] Peak Values: IOPS are calculated every 10ms period, and the peaks for each volume are averaged. More analysis in the paper

27© Copyright 2013 EMC Corporation. All rights reserved. Snapshot I/O Overheads Rule-of-thumb: Over-provision primary I/O by 100% to support copy-on-write related write-amplification