1. Veeam Backup & Replication: Tips and Tricks
SPO3292
Veeam Backup & Replication: Tips and Tricks
Anton Gostev Veeam Software @Gostev
Doug Hazelman Veeam Software @VMDoug
#vmworldsponsor

2. Quick Overview of v6 Architecture
Backup servers Backup proxy servers Backup repositories
“Automated everything”
Intelligent load balancing
Centralized management via Enterprise Manager

3. This illustrates how intelligent load balancing works

4. Scaling your backups
3 simple rules

5. Scaling your backups Keep the management server happy
Disable default proxy (requires 6.1 or later)
Allocate enough RAM for job manager processes
Keep concurrent (running) jobs under 100 per management server
Backup proxy servers: the more, the better?
No! Too much load on storage and network
Use max concurrent tasks limit on proxies or repositories
Watch for “job timed out waiting for resources” messages
Backup repository considerations
Be careful with the reversed incremental backup mode!
Limit concurrent jobs on backup repositories to a reasonable amount
Use ingest rate throttling for cross-SAN backups

6. A word on backup repositories
Don’t underestimate the importance of performance!
By far, the most commonly reported bottleneck
What makes the best backup repositories?
Windows or Linux server (can be same as backup proxy server)
Local storage, DAS or SAN mounted for physical server
pRDM disk (vSphere 5+), or iSCSI LUN connected via in-guest iSCSI for virtual server
Sub-optimal backup storage
NAS or network share
VMDK on VMFS (size and recoverability considerations)
RAID level
If you can afford it, use RAID10 (again, performance)

7. Processing Modes
All you need to know

8. Direct SAN Access: The good
Fastest processing mode
Least impact on production
Backup processing is fully offloaded to dedicated backup proxy servers
Backup traffic is isolated to the storage network (aka LAN-free)
Does not impact consolidation ratio, so cheapest too

9. Direct SAN Access: The bad
Supports block storage only
FC (fibre channel): physical backup proxy server only!
iSCSI: physical and virtual backup proxy servers both supported
Physical backup proxy server requirement for FC SAN
Might not go along well with your virtualization project
Consider repurposing older servers
Might be hard for beginners to setup
See Veeam Forums FAQ for step-by-step guide
Manual datastore mapping might be required
For certain SANs, B&R may not able to detect proxy connectivity

10. Direct SAN Access: The ugly
What’s the worst that could happen?
Windows re-signaturing your VMFS LUNs!
vSphere will no longer recognize datastores
Don’t panic, VMware Support should be able to fix
Three easy ways to get into trouble
Windows Explorer automounting new volumes (but not with Veeam)
Clicking Disk Management snap-in popup without reading
Giving Local Administrator rights to random people

11. Direct SAN Access: The safe way
Present VMFS LUNs to backup proxy server as read-only
Most SANs support it these days—chase your vendor if yours does not
Disable automount on your backup proxy servers
Do it the right way: use SANPolicy Windows setting!
Veeam backup proxy server setup does this automatically for you
Disable Disk Management snap-in with Group Policy
User Configuration > Administrative Templates Window Components > Microsoft Management Console > Restricted/Permitted snap-ins > Disk Management
Keep Local Administrator rights on backup proxy servers to yourself
Cannot really do this for default proxy due to FLR requirement
Another reason to use dedicated backup proxy server!
automount disable is deprecated, and causes issues!

12. Direct SAN Access: Tips & tricks
Got a fast SAN? Get a modern backup proxy server!
Multi-core CPU (compression) and fast RAM (inline deduplication)
Update firmware and drivers across the board
Disabling MPIO may increase performance
iSCSI SAN? Tweak TCP/IP on backup proxy
netsh interface tcp set global autotuning level = disable
Increase read-ahead buffer
Default is 4MB (optimal setting for most SANs)
To change, create the new value in bytes:
VddkPreReadBufferSize (DWORD)
Also, be sure to search Veeam forums for other people’s experiences. Some hardware is just bad, apparently.

13. Hot Add: The good Easy to setup—very little planning involved
Any Windows VM can be made a Hot Add backup proxy
Fast data transfers with any storage
Direct storage access (albeit through ESXi storage stack)
Supports all types of storage (including NFS)
Shared storage: at least 1 backup proxy server per vSphere cluster
Local storage or DAS: at least 1 backup proxy server per host
Use your existing Windows VMs (save on licensing)
Data processing engine process runs with lower priority (6.1)
Further CPU usage reduction in 6.5
Allows for 100% virtual deployment

14. Hot Add: The bad Not as mature as other modes
Affects your consolidation ratio
Backup proxy servers take host resources
Ultimately means more ESXi hosts, and more VMware licenses
Hot Add process itself is slooow
Can take up to 1–2 minutes to complete for each VM—adds up quickly!
Hot Add as a vSphere feature has a number of limitations
Good news—many are being removed as VDDK matures
See FAQ on Veeam forums FAQ for the complete list

15. Hot Add: The ugly Snapshot removal problems due to locks
Veeam B&R: multiple hooks in place to work around
CBT must be disabled on backup proxy VM
Prevents stun on Hot Add due to CBT initialization
NFS-specific issue
Extended VM stun on hot remove in some scenarios

16. Hot Add: Tips & tricks
Add extra virtual SCSI controller to backup proxy server
A single SCSI controller can have a maximum of 16 disks attached
Concurrent jobs on the same backup proxy server can result in more!
Keep vSphere and Veeam up to date
Single block size in VMFS5 removes the most common hot add issue
Latest Veeam Backup & Replication will have latest and greatest VDDK version
Try increasing read-ahead buffer
Seems to really help with certain NFS storage
Avoid cloning backup proxy VM
For example, to provision additional backup proxies

17. Network (NBD): The good
Easy to setup—in fact, no setup is required
Any existing server (physical or virtual) would do
Supports all types of storage, including NFS
Server placement does not matter (unlike with Hot Add)
Very quick to initialize data transfer
Can be quite fast—with 10Gb Ethernet

18. Network (NBD): The ugly
Painfully slow performance on 1Gb Ethernet
Average speed reported is MB/s
Leverages ESXi management interface

19. Network (NBD): Tips & tricks
1 Gb Ethernet
Use for sites with low change rate
Works faster than other processing modes in such conditions
Keep at least one Hot Add backup proxy server around
Full VM and virtual disk restores take forever over NBD
Keep in mind intelligent load balancing algorithms
Network backup proxy servers have lowest priority!
Upgrade to 6.1 or later
Improved network proxy location awareness

20. One last thing This hack significantly reduces supportability!
Cut up to 5 minutes of processing time per VM by disabling VDDK logging
Apply in stable environments only!
Create the new value and set to 1:
DisableVDDKNetworkOutput (DWORD)

21. Deduplicating Storage
Yes, you can afford it!

22. Deduplicating storage: The good
What gives? Global dedupe!
Deduplication across backup files from different jobs
Perfect for long-term backup archival
Top hardware appliances among Veeam users
EMC DataDomain
ExaGrid
HP StoreOnce
Top software appliances among Veeam users
ZFS-based appliances
StarWind
Windows Server 2012 dedupe is awesome

23. Deduplicating storage: The bad
Hardware appliances are expensive
Although they do provide excellent dedupe ratio
Software appliances are resource hogs
Both performance and dedupe ratio are sub-par, too
Windows Server 2012 dedupe is awesome
Included free of charge—start using it today!
Provides very decent dedupe ratio

24. Deduplicating storage: The ugly
Random access performance is lacking
A problem for all solutions featuring inline deduplication
Typically insufficient out-of-the-box for large-scale vPower usage
Exception: post-process deduplication
ExaGrid
Raw disk landing zone (full-speed vPower from recent backups)
Veeam-specific logic further optimizes performance
Windows Server 2012
Backups “land” on raw storage at full speed
Only old backup files are deduplicated—great for vPower
Decent speed even off already deduped backups

25. Deduplicating storage: Tips & tricks
Already own storage with inline deduplication?
Inline data “rehydration” process is what makes vPower slow
Reduce the block size in Advanced job settings
WAN (256KB) and LAN (512KB)
Reduced block size might impact backup performance
Use Linux-based backup repository
Large client cache, or even caching client file system (FS-cache), can significantly improve vPower performance

26. Deduplicating storage: Tips & tricks (continued)
Keep Veeam dedupe on
Use incremental backup mode
Choice of synthetic or active fulls depending on actual storage
For best dedupe ratio on device side…
Disable compression (significantly increases amount of data transferred from backup proxy server to backup repository over network)
For best backup performance and smallest window
Keep compression at default
If you like to avoid extremes…
Set compression to Low (dedupe-friendly)

27. Deduplicating storage: Tips & tricks (continued)
Got more than one deduplicating storage device?
Use internal replication to sync backups offsite!
Extremely traffic-efficient approach
Many customers use and report great success!
Keep the backups imported for easy DR
To automate repository refresh in DR site, use:
Get-VBRBackupRepository -Name "DR_Repository" | Sync-VBRBackupRepository

28. WAN Accelerators
Your WAN on steroids

29. WAN accelerators: The good
Two types of WAN accelerators
Caching WAN accelerators provide significant bandwidth savings with Veeam replication, but are typically quite expensive
Transport layer WAN optimizers are unlikely to offer significant bandwidth savings with Veeam replication, and are usually cheap
Both improve reliability of TCP
Long distance wireless or satellite links
IPsec rekey operations on a VPN tunnel
WAN links with high jitter, packet loss or occasional drops
Both allow long-running jobs to finish more consistently
For example, initial replication over network
Both improve WAN utilization for most workloads
Little bandwidth savings doesn’t mean these tools can't sometimes be useful. Some links can experience small "drops", for example, on long distance wireless or satellite links, or even simply unreliable internet links. In these situations a long replication cycle is likely to crash, but these transport layer optimizers can many times "hide" these short outages from the TCP layer by faking TCP retransmits/keepalives and filtering ICMP unreachable messages from reaching the backup proxy server. This can allow long-running replication jobs to finish more consistently in the presence of less-than-reliable WAN links.

30. WAN accelerators: What’s hot?
Top caching WAN accelerators among Veeam users
Cisco WAAS
Riverbed
SilverPeak
Top WAN optimizers among Veeam users
Hyper-IP

31. WAN optimizers: Tips & tricks
Veeam Backup & Replication leaves little room for bandwidth reduction by WAN optimizers
Built-in WAN optimizations in v6:
Multiple TCP streams to maximize throughput
Network traffic compression
Consider using on unreliable networks, but don’t expect them to add you extra bandwidth
Update to Backup & Replication 6.1 Patch 1 before evaluating
Veeam Backup & Replication not using all available bandwidth?
Increase the amount of TCP streams (default is 5)
DownloadStreamsNumber (DWORD)

32. Caching WAN accelerators: Tips & tricks
Multiple TCP streams can cause issues!
Disable multiple streams in B&R traffic throttling settings
Disable network traffic compression in Veeam
Low (dedupe-friendly) compression level might be a better option
If required, have network admins configure bypass on Veeam backup proxy servers to avoid polluting caches

33. Veeam Backup & Replication 6.5
We never stand still

34. What’s coming in 6.5 Veeam Explorer for Microsoft Exchange
Veeam Explorer for SAN Snapshots
VMware vSphere 5.1 support
Windows Server 2012 support

35. Veeam Explorer for Microsoft Exchange
Visibility into Exchange VM backups
Immediate: No need to provision storage, restore the VM or restore the mailbox store
Agentless
Requires no special backups or metadata collection—even works with existing Veeam backups (and SAN snapshots)
Free!
Included in all versions of Veeam Backup & Replication 6.5, including Free Edition
Eliminates need for expensive standalone tools licensed per-mailbox
Currently available in “exclusive beta”
Just restored a 145 GB #MSExchange Public Folder database in 2 min. Then restored a single item all under 10 min. SWEET!
The new #veeam explorer for exchange looks veeamy. That's right, I just made up a new word (it means awesomesauce)
Even the *beta* Exchange Explorer works a treat. Saved literally, hours of work.. and saved my bacon. #recommend
Really excited with the Explorer beta for #Microsoft Exchange VM backups - we have lots of interested customers ready for this!

36. Veeam Explorer for Microsoft Exchange (continued)
Capabilities
Browse: familiar Explorer-type interface
Search: familiar Outlook-like Find, including Advanced Find
Export: export to PST file, MSG file or attachment
Uses cases
E-discovery
Item-level restore: export and send to affected user
Mailbox archive
Supports Exchange Server 2010

37. Veeam Explorer for SAN Snapshots
Veeam restores from SAN snapshots
Supports tiered data protection strategy
Perform all restores through familiar, easy-to-use Veeam interface
Supports HP StoreVirtual VSA and HP LeftHand
SAN snapshots + Veeam restore = Best RPOs and RTOs for operational recovery

38. Veeam Explorer for SAN Snapshots (continued)
Fast: recover entire VM or individual items in < 2 minutes
Fully automated: clone & promote snapshot, present to vSphere, clean up
Restores directly from VM files on the SAN snapshot: no staging or intermediate restores required
Flexible
Specific VM
Individual guest files: Windows, Linux, et al
Individual Microsoft Exchange items
Free
Worry-free: automated process eliminates human errors and protects integrity of SAN snapshots and production LUNS
Agent-free: no agents to deploy on hosts or VMs
Literally free: included in all editions of Veeam Backup & Replication 6.5, including Free Edition

39. Questions? Comments?

40. Thanks for attending!

41. FILL OUT A SURVEY AT WWW.VMWORLD.COM/MOBILE
COMPLETE THE SURVEY WITHIN ONE HOUR AFTER EACH SESSION AND YOU WILL BE ENTERED INTO A DRAW FOR A GIFT FROM THE VMWARE COMPANY STORE

42. Veeam Backup & Replication: Tips and Tricks
SPO3292
Veeam Backup & Replication: Tips and Tricks
Anton Gostev Veeam Software @Gostev
Doug Hazelman Veeam Software @VMDoug
#vmworldsponsor