1. Best Practices for Backing Up Your System
Luca Ravazzolo
Technology Architect

2. Types of backups Cold file-level backup Caché online backup
Caché shutdown
Server-level copy to disk/tape
Caché restarted
Caché online backup
Caché’s backup tool copies data blocks from CACHE.DAT files to disk file or tape.
Full or various incremental backups

3. Types of backups SAN or disk array backup
Backup I/O stays within the SAN or the array
Block level copy from device to device (disk, tape, virtual tape)
All vendors have some type of software to control backups.
To backup a consistent image, a point-in-time snapshot or clone is made of the source device.

4. Types of backups: others
CDP: Continuous Data Protection (Near-CDP)
Use of separate appliance to journal changes out-of- band allowing for recovery to any point-in-time,
Depending on space available can restore to most any point-in-time.
SAN-based Replication
Provides a disk-to-disk copy within the SAN, perhaps over long distances,
Destination can be archived to tape.

5. Advantages and Challenges

6. Caché online backup Advantages: Challenges:
Caché stays up, users continue to work
Simple to implement, may not need 3rd party software
Challenges:
Only backs up the CACHE.DAT data – must also backup journals, other files.
Restores typically take multiple steps
Create a Caché instance
Restore “*<date>.cbk” files from storage
Apply most recent full backup, then cumulative & incrementals
Apply journal files

7. Disk array/SAN-based snapshot
Advantages:
Point-in-time copy of all data (Caché and otherwise)
Requires no downtime (when using Caché write daemon freeze and thaw)
Challenges:
Requires snap/clone technology
Requires additional software to coordinate
There are two main types of storage snapshot, called the
copy-on-write (or low-capacity) snapshot and the
split-mirror snapshot.
Utilities are available that can automatically generate either type.
-A copy-on-write snapshot utility creates a snapshot of changes to stored data every time new data is entered or existing data is updated. This allows rapid recovery of data in case of a disk write error, corrupted file, or program malfunction. However, all previous snapshots must be available if complete archiving or recovery of all the data on a network or storage medium is needed.
-A split-mirror snapshot utility references all the data on a set of mirrored drives. Every time the utility is run, a snapshot is created of the entire volume, not only of the new or updated data. This makes it possible to access data offline, and simplifies the process of recovering, duplicating, or archiving all the data on a drive. However, this is a slower process, and it requires more storage space for each snapshot. --
from

8. CDP or replication Advantages Challenges
CDP allows restore to nearly any point-in-time
Replication allows geographically separated backups
Challenges
Non-Caché technologies require coordination with Caché, i.e.
May end up with Caché in a crash-consistent state and require recovery before use
Requires appliances and software
CDP = Continuous Data Protection

9. Coordinating with Caché
External Backup
Coordinating with Caché

10. Freeze the write daemon(s)
For a consistent database image on your backup media (i.e. a CACHE.DAT without integrity errors) the write daemon’s cycle must be complete.
Use the Backup.General.ExternalFreeze() method
Keeps write daemon from writing
Waits for current write daemon cycle (if active) to finish
Switches journal file
Logs information to the cconsole.log file.

11. Freezing the write daemon
ExternalFreeze command:
OS command returns a code:
5 – successful
3 - failure
While frozen, all updates are made as usual to database cache
Processes continue to run normally UNLESS:
Available buffers in the database cache falls too low.
The ExternalFreeze lasts longer than the default limit (600 seconds)
#csession cache –U%SYS “##class(Backup.General).ExternalFreeze()”
#echo $?
%SYS>SET rc=##class(Backup.General).ExternalFreeze()

12. Thaw the write daemon
Use Backup.General.ExternalThaw to allow write daemon(s) to resume writing.
Thaw command:
OS-level command returns one of these codes:
5 – success
3 - failure
#csession cache –U%SYS “##class(Backup.General).ExternalThaw()”
#echo $?
%SYS>SET rc=##class(Backup.General).ExternalThaw()

13. Another useful method
Use Backup.General.ExternalSetHistory to log successful backups in the Backup History
log is name of an externally created backup log
desc is free text
%SYS>SET log=“/var/logs/backup.log”,desc=“Full Backup”
%SYS>S rc=##class(Backup.General).ExternalSetHistory(log,desc)

14. Who runs the freeze/thaw?
The operating system user that executes the freeze/thaw command must have access to Caché.
In normal install, the “backup” user must be a Caché user.
%Service_Terminal must allow OS-level authentication.
Caché “backup” user needs RW on the %DB_CACHESYS resource as well as use of %Admin_Operate and %Service_Terminal

15. Case Study: External Backup
Using snapshots, a de-duplication appliance and replication for an external backup of Caché

16. External backup 1: Caché & snaps
Backup software initiates the backup process from media server
MEDIA
Invoke script on server running Caché to FREEZE write daemon
Backup software initiates clone or snapshot of all Caché arrays
MEDIA
Invoke script on server running Caché to THAW write daemon

17. External backup 2: Mount & copy
Backup software mounts snapshot on the media server
MEDIA
Backup software does file level copy from snapshot to disk-based backup appliance.
MEDIA
Backup releases the snapshot via command-line interface call to disk controller
MEDIA

18. Ext Backup 3: Replicate, verify & archive
Backup software initiates a backup copy to a secondary data center
MEDIA
In secondary data center, replicated backup is restored, mounted in a Caché instance and an integrity check is run to verify structural integrity.
Depending on space and policy backup is kept online and/or archived to tape for long term storage.

19. Timings and best practices
Backup software initiates the backup process from media server
MEDIA
Backup software:
Must be able to call freeze/thaw script on Caché server
Must be able to initiate the snapshot
Most commercial backup software will work well including EMC Networker, Symantec NetBackup, IBM Tivoli (TSM), etc.

20. Timings and best practices
Sample scripts available from the WRC
Time to freeze and return depends on
Database activity
Current write daemon phase (i.e. is it writing to disk?)
Invoke script on server running Caché to FREEZE write daemon
04/02-02:30:00 (1098) 0 ExternalFreeze: Suspending system
04/02-02:30:00 (1098) 0 ExternalFreeze: Description: Backup Performed by TSM at: :30:00
04/02-02:30:01 (1098) 0 ExernalFreeze: Start a journal restore for this backup
with journal file: /jrn/
04/02-02:30:02 (1098) 0 ExernalFreeze: System suspended

21. Timings and best practices
Creating the clone or snap - this period is when write daemon(s) are frozen.
Timing is based on array controller activity
If greater than a few minutes there is a risk of running into freeze timeout.
50 seconds frozen with
IBM DS5300 using FlashCopy on
a few TB of data with active systems
Backup software initiates clone or snapshot of all Caché arrays
MEDIA
04/02-02:30:02 (1098) 0 ExernalFreeze: System suspended
04/02-02:30:52 (9109) 0 ExternalThaw: Resuming system

22. Timings and best practices
Thawing the write daemon takes seconds at most.
Best practice is to be sure to thaw the database on any error along the way.
Perhaps have an independent job to check database status and thaw if frozen---so a failed backup will never leave Caché frozen.
Invoke script on server running Caché to THAW write daemon

23. External backup 2: Mount & copy
Backup software mounts snapshot on the media server
MEDIA
Backup software does file level copy from snapshot to disk-based backup appliance.
MEDIA
Use of a de-duplication appliance as the file- level backup target speeds backup and saves space.
Timings vary a lot here---disk used, dedupe rate etc.

24. Ext Backup 3: Replicate, verify & archive
Backup software initiates a backup copy to a secondary data center
MEDIA
SAN level replication or replication via de- duplication appliance.
Timings vary a lot here based on bandwidth and de-dupe rate if applicable.

25. Ext Backup 3: Replicate, verify & archive
Integrity checks vary in timing
Another option is to have media server in primary data center run the check.
In secondary data center, replicated backup is restored, mounted in a Caché instance and an integrity check is run to verify structural integrity.
Depending on space and policy backup is kept online and/or archived to tape for long term storage.

26. Final points
Considering cost and effort, Caché online backup works well for small to medium size databases (~ 100s of GB total) with generous RTOs
Use InterSystems Mirroring in conjunction with your backup mechanism.
Perhaps there will be no need to restore a backup
If needed, the mirror destination will have CACHE.DAT files and journal files.

27. Final points Backup should have minimal impact on live database
Using SAN/disk controller based backups offloads the work to other appliances/servers
SAN/disk-based backups meet the fastest RTOs.
Restore from backup RPOs are as good as the most recently available journal file.

28. Best Practices for Backing Up Your System
Luca Ravazzolo
Technology Architect