5 Scaling your backups Keep the management server happy Disable default proxy (requires 6.1 or later)Allocate enough RAM for job manager processesKeep concurrent (running) jobs under 100 per management serverBackup proxy servers: the more, the better?No! Too much load on storage and networkUse max concurrent tasks limit on proxies or repositoriesWatch for “job timed out waiting for resources” messagesBackup repository considerationsBe careful with the reversed incremental backup mode!Limit concurrent jobs on backup repositories to a reasonable amountUse ingest rate throttling for cross-SAN backups
6 A word on backup repositories Don’t underestimate the importance of performance!By far, the most commonly reported bottleneckWhat makes the best backup repositories?Windows or Linux server (can be same as backup proxy server)Local storage, DAS or SAN mounted for physical serverpRDM disk (vSphere 5+), or iSCSI LUN connected via in-guest iSCSI for virtual serverSub-optimal backup storageNAS or network shareVMDK on VMFS (size and recoverability considerations)RAID levelIf you can afford it, use RAID10 (again, performance)
8 Direct SAN Access: The good Fastest processing modeLeast impact on productionBackup processing is fully offloaded to dedicated backup proxy serversBackup traffic is isolated to the storage network (aka LAN-free)Does not impact consolidation ratio, so cheapest too
9 Direct SAN Access: The bad Supports block storage onlyFC (fibre channel): physical backup proxy server only!iSCSI: physical and virtual backup proxy servers both supportedPhysical backup proxy server requirement for FC SANMight not go along well with your virtualization projectConsider repurposing older serversMight be hard for beginners to setupSee Veeam Forums FAQ for step-by-step guideManual datastore mapping might be requiredFor certain SANs, B&R may not able to detect proxy connectivity
10 Direct SAN Access: The ugly What’s the worst that could happen?Windows re-signaturing your VMFS LUNs!vSphere will no longer recognize datastoresDon’t panic, VMware Support should be able to fixThree easy ways to get into troubleWindows Explorer automounting new volumes (but not with Veeam)Clicking Disk Management snap-in popup without readingGiving Local Administrator rights to random people
11 Direct SAN Access: The safe way Present VMFS LUNs to backup proxy server as read-onlyMost SANs support it these days—chase your vendor if yours does notDisable automount on your backup proxy serversDo it the right way: use SANPolicy Windows setting!Veeam backup proxy server setup does this automatically for youDisable Disk Management snap-in with Group PolicyUser Configuration > Administrative Templates Window Components > Microsoft Management Console > Restricted/Permitted snap-ins > Disk ManagementKeep Local Administrator rights on backup proxy servers to yourselfCannot really do this for default proxy due to FLR requirementAnother reason to use dedicated backup proxy server!automount disable is deprecated, and causes issues!
12 Direct SAN Access: Tips & tricks Got a fast SAN? Get a modern backup proxy server!Multi-core CPU (compression) and fast RAM (inline deduplication)Update firmware and drivers across the boardDisabling MPIO may increase performanceiSCSI SAN? Tweak TCP/IP on backup proxynetsh interface tcp set global autotuning level = disableIncrease read-ahead bufferDefault is 4MB (optimal setting for most SANs)To change, create the new value in bytes:VddkPreReadBufferSize (DWORD)Also, be sure to search Veeam forums for other people’s experiences. Some hardware is just bad, apparently.
13 Hot Add: The good Easy to setup—very little planning involved Any Windows VM can be made a Hot Add backup proxyFast data transfers with any storageDirect storage access (albeit through ESXi storage stack)Supports all types of storage (including NFS)Shared storage: at least 1 backup proxy server per vSphere clusterLocal storage or DAS: at least 1 backup proxy server per hostUse your existing Windows VMs (save on licensing)Data processing engine process runs with lower priority (6.1)Further CPU usage reduction in 6.5Allows for 100% virtual deployment
14 Hot Add: The bad Not as mature as other modes Affects your consolidation ratioBackup proxy servers take host resourcesUltimately means more ESXi hosts, and more VMware licensesHot Add process itself is slooowCan take up to 1–2 minutes to complete for each VM—adds up quickly!Hot Add as a vSphere feature has a number of limitationsGood news—many are being removed as VDDK maturesSee FAQ on Veeam forums FAQ for the complete list
15 Hot Add: The ugly Snapshot removal problems due to locks Veeam B&R: multiple hooks in place to work aroundCBT must be disabled on backup proxy VMPrevents stun on Hot Add due to CBT initializationNFS-specific issueExtended VM stun on hot remove in some scenarios
16 Hot Add: Tips & tricksAdd extra virtual SCSI controller to backup proxy serverA single SCSI controller can have a maximum of 16 disks attachedConcurrent jobs on the same backup proxy server can result in more!Keep vSphere and Veeam up to dateSingle block size in VMFS5 removes the most common hot add issueLatest Veeam Backup & Replication will have latest and greatest VDDK versionTry increasing read-ahead bufferSeems to really help with certain NFS storageAvoid cloning backup proxy VMFor example, to provision additional backup proxies
17 Network (NBD): The good Easy to setup—in fact, no setup is requiredAny existing server (physical or virtual) would doSupports all types of storage, including NFSServer placement does not matter (unlike with Hot Add)Very quick to initialize data transferCan be quite fast—with 10Gb Ethernet
18 Network (NBD): The ugly Painfully slow performance on 1Gb EthernetAverage speed reported is MB/sLeverages ESXi management interface
19 Network (NBD): Tips & tricks 1 Gb EthernetUse for sites with low change rateWorks faster than other processing modes in such conditionsKeep at least one Hot Add backup proxy server aroundFull VM and virtual disk restores take forever over NBDKeep in mind intelligent load balancing algorithmsNetwork backup proxy servers have lowest priority!Upgrade to 6.1 or laterImproved network proxy location awareness
20 One last thing This hack significantly reduces supportability! Cut up to 5 minutes of processing time per VM by disabling VDDK loggingApply in stable environments only!Create the new value and set to 1:DisableVDDKNetworkOutput (DWORD)
22 Deduplicating storage: The good What gives? Global dedupe!Deduplication across backup files from different jobsPerfect for long-term backup archivalTop hardware appliances among Veeam usersEMC DataDomainExaGridHP StoreOnceTop software appliances among Veeam usersZFS-based appliancesStarWindWindows Server 2012 dedupe is awesome
23 Deduplicating storage: The bad Hardware appliances are expensiveAlthough they do provide excellent dedupe ratioSoftware appliances are resource hogsBoth performance and dedupe ratio are sub-par, tooWindows Server 2012 dedupe is awesomeIncluded free of charge—start using it today!Provides very decent dedupe ratio
24 Deduplicating storage: The ugly Random access performance is lackingA problem for all solutions featuring inline deduplicationTypically insufficient out-of-the-box for large-scale vPower usageException: post-process deduplicationExaGridRaw disk landing zone (full-speed vPower from recent backups)Veeam-specific logic further optimizes performanceWindows Server 2012Backups “land” on raw storage at full speedOnly old backup files are deduplicated—great for vPowerDecent speed even off already deduped backups
25 Deduplicating storage: Tips & tricks Already own storage with inline deduplication?Inline data “rehydration” process is what makes vPower slowReduce the block size in Advanced job settingsWAN (256KB) and LAN (512KB)Reduced block size might impact backup performanceUse Linux-based backup repositoryLarge client cache, or even caching client file system (FS-cache), can significantly improve vPower performance
26 Deduplicating storage: Tips & tricks (continued) Keep Veeam dedupe onUse incremental backup modeChoice of synthetic or active fulls depending on actual storageFor best dedupe ratio on device side…Disable compression (significantly increases amount of data transferred from backup proxy server to backup repository over network)For best backup performance and smallest windowKeep compression at defaultIf you like to avoid extremes…Set compression to Low (dedupe-friendly)
27 Deduplicating storage: Tips & tricks (continued) Got more than one deduplicating storage device?Use internal replication to sync backups offsite!Extremely traffic-efficient approachMany customers use and report great success!Keep the backups imported for easy DRTo automate repository refresh in DR site, use:Get-VBRBackupRepository -Name "DR_Repository" | Sync-VBRBackupRepository
29 WAN accelerators: The good Two types of WAN acceleratorsCaching WAN accelerators provide significant bandwidth savings with Veeam replication, but are typically quite expensiveTransport layer WAN optimizers are unlikely to offer significant bandwidth savings with Veeam replication, and are usually cheapBoth improve reliability of TCPLong distance wireless or satellite linksIPsec rekey operations on a VPN tunnelWAN links with high jitter, packet loss or occasional dropsBoth allow long-running jobs to finish more consistentlyFor example, initial replication over networkBoth improve WAN utilization for most workloadsLittle bandwidth savings doesn’t mean these tools can't sometimes be useful. Some links can experience small "drops", for example, on long distance wireless or satellite links, or even simply unreliable internet links. In these situations a long replication cycle is likely to crash, but these transport layer optimizers can many times "hide" these short outages from the TCP layer by faking TCP retransmits/keepalives and filtering ICMP unreachable messages from reaching the backup proxy server. This can allow long-running replication jobs to finish more consistently in the presence of less-than-reliable WAN links.
30 WAN accelerators: What’s hot? Top caching WAN accelerators among Veeam usersCisco WAASRiverbedSilverPeakTop WAN optimizers among Veeam usersHyper-IP
31 WAN optimizers: Tips & tricks Veeam Backup & Replication leaves little room for bandwidth reduction by WAN optimizersBuilt-in WAN optimizations in v6:Multiple TCP streams to maximize throughputNetwork traffic compressionConsider using on unreliable networks, but don’t expect them to add you extra bandwidthUpdate to Backup & Replication 6.1 Patch 1 before evaluatingVeeam Backup & Replication not using all available bandwidth?Increase the amount of TCP streams (default is 5)DownloadStreamsNumber (DWORD)
32 Caching WAN accelerators: Tips & tricks Multiple TCP streams can cause issues!Disable multiple streams in B&R traffic throttling settingsDisable network traffic compression in VeeamLow (dedupe-friendly) compression level might be a better optionIf required, have network admins configure bypass on Veeam backup proxy servers to avoid polluting caches
33 Veeam Backup & Replication 6.5 We never stand still
34 What’s coming in 6.5 Veeam Explorer for Microsoft Exchange Veeam Explorer for SAN SnapshotsVMware vSphere 5.1 supportWindows Server 2012 support
35 Veeam Explorer for Microsoft Exchange Visibility into Exchange VM backupsImmediate: No need to provision storage, restore the VM or restore the mailbox storeAgentlessRequires no special backups or metadata collection—even works with existing Veeam backups (and SAN snapshots)Free!Included in all versions of Veeam Backup & Replication 6.5, including Free EditionEliminates need for expensive standalone tools licensed per-mailboxCurrently available in “exclusive beta”Just restored a 145 GB #MSExchange Public Folder database in 2 min. Then restored a single item all under 10 min. SWEET!The new #veeam explorer for exchange looks veeamy. That's right, I just made up a new word (it means awesomesauce)Even the *beta* Exchange Explorer works a treat. Saved literally, hours of work.. and saved my bacon. #recommendReally excited with the Explorer beta for #Microsoft Exchange VM backups - we have lots of interested customers ready for this!
36 Veeam Explorer for Microsoft Exchange (continued) CapabilitiesBrowse: familiar Explorer-type interfaceSearch: familiar Outlook-like Find, including Advanced FindExport: export to PST file, MSG file or attachmentUses casesE-discoveryItem-level restore: export and send to affected userMailbox archiveSupports Exchange Server 2010
37 Veeam Explorer for SAN Snapshots Veeam restores from SAN snapshotsSupports tiered data protection strategyPerform all restores through familiar, easy-to-use Veeam interfaceSupports HP StoreVirtual VSA and HP LeftHandSAN snapshots + Veeam restore = Best RPOs and RTOs for operational recovery
38 Veeam Explorer for SAN Snapshots (continued) Fast: recover entire VM or individual items in < 2 minutesFully automated: clone & promote snapshot, present to vSphere, clean upRestores directly from VM files on the SAN snapshot: no staging or intermediate restores requiredFlexibleSpecific VMIndividual guest files: Windows, Linux, et alIndividual Microsoft Exchange itemsFreeWorry-free: automated process eliminates human errors and protects integrity of SAN snapshots and production LUNSAgent-free: no agents to deploy on hosts or VMsLiterally free: included in all editions of Veeam Backup & Replication 6.5, including Free Edition