1. LTFS EE PGA2 Enhancements

2. Agenda
LTFS EE Overview
PGA1 Enhancements
PGA2 Enhancements
Demo

3. What is LTFS File format File system Index Partition Data Partition
BOT
EOT
Index Partition
Data Partition
Guard Wraps
LTFS Index XML
File
1) Open File Format for data written to tape
Developed and disclosed by IBM
Describes the format of data and meta data stored on tape
Meta data is based on XML schema
Applicable to LTO5, LTO6 and TS1140
Requires tape partitioning
Approved by LTO Consortium
2) LTFS software
Externalizes the LTO5 / LTO6 / TS1140 tape as file system
Enables standard applications to write/read LTFS tapes
Supports update, edit, and delete of files on LTFS tape
Available on Linux, Mac OS X and Windows
Makes tape look and work like any removable media (e.g., USB drive, removable disk)

4. Creating Operational Storage Solutions
Goals:
A single tiered storage system
A single namespace for all your file data
Policy based placement of data
Scalable, low cost
Open, non-proprietary, cross platform interchange
Archive
Users & Applications
NFS, CIFS, FTP, …
GPFS file system
LTFS EE
Summary: Non-proprietary Policy based, low cost scalable storage.
TS3500 4

5. Use case: Long Term Data Retention (Archiving)
Large amount of data, larger files
Infrequently accessed
Longer retention periods
LAN
Common name space
SAN
Define the difference between backup and archive.
Backup – A secondary copy of production data used for restore or disaster recovery in the case of data loss or data corruption. Backup data is kept for a relatively short period of time.
Archive – A primary copy of less frequently accessed information that has been moved off primary production disk to lower cost storage. Archive data is kept for a long period of time.
A key difference between backup and archive is how the data is retained. Backup is ultimately overwritten as time goes on, whereas archived data is usually a permanent record, stored without deletion for an extended period of time.

6. Tape / Disk Cost Comparison
A 12 Year TCO Archiving Study on Back-end storage costs including hardware, maintenance, floor space and energy found disk storage is 15 times the cost of tape
The cost of energy alone for the average disk-based solution exceeds the entire TCO for the average tape-based solution
15X
238X
1Clipper Notes report “In Search of the Long-Term Archiving Solution - Tape Delivers Significant TCO Advantage over Disk”,The Clipper Group, Dec.23, This was a general TCO study and did not specifically focus on LTFS

7. GPFS- LTFS Tiered Operational Storage
Single file system view
G:/
File Namespace
C:/user defined namespace
Subdir 1
One.txt
Video1.mpg
Move files older than X
Move files named *.mpg
Never move files named *.db2
Move files opened for modification
Database.db2
Dept0A
Policy Based Tier Migration
Memo.txt
Movie1.mpg
Manuf
Inv02.txt
Repair.mpg
Sales.db2
LTFS Pool 1
LTFS Pool 2
LTFS
LTFS
LTFS
File Storage Tiers
One.txt
Video1.mpg
G:/
Subdir 1
Memo.txt
Movie1.mpg
One.txt
Inv02.txt
Repair.mpg
Video1.mpg
Memo.txt
Database.db2
Dept0A
Memo.txt
Movie1.mpg
Manuf
Inv02.txt
Repair.mpg
Sales.db2

8. LTFS EE Export / Import User / Application Global Namespace GPFS
Disk Tier
Tape Tier
File
Migration
Cartridge
Export
Tape
File
Recall
Cartridge
Import

9. LTFS EE Multi-Site User / Application NFS or CIFS GPFS AFM or rsync
GPFS / LTFS EE cluster
GPFS / LTFS EE cluster
Policy Based
Migration
Policy Based
Migration
Can duplex single site tapes via LTFS EE or GPFS Policy. If multi-site electronic replication is desired, two implementations are possible. GPFS AFM can be used to replicate between the sites. Or a simple script using rsync can be used to do the replication.
TS3500
TS3500

10. LTFS EE Rapid Application Solution (Bundle)

11. Agenda
LTFS EE Overview
PGA1 Enhancements
PGA2 Enhancements
Demo

12. LTFS EE PGA1 enhancements
Improved migration performance
Premigration
Tape optimized recalls
Additional OS support
Improved data resiliency
New tape storage pool management
LTFS EE PGA1 GA-ed December 20, 2013

13. Improved migration performance

14. Migration performance improvement

15. Premigration
Migration
Pre-PGA1
Migration
PGA1 or
Premigration

16. Agenda
LTFS EE Overview
PGA1 Enhancements
PGA2 Enhancements
Demo

17. LTFS EE PGA2 enhancements
New OS Support
Redhat Enterprise Linux (RHEL) Version 6.5
SUSE Linux Enterprise Server 11 SP3
New Tape Library Support
TS4500
TS3310
Bulk File Recall
Read Starts Recall (RSR) - Early trigger of recalling migrated file
Preserving File System Objects on Tape
Rebuild GPFS from tape
LTFS EE PGA2 announced June 10, 2014

18. Additional Operating System Support
CIFS
NFS
Windows
Linux
Etc.
GPFS Node 1
TS3500 Tape Library
GPFS Node 2
Global name space
GPFS file systems (user data and metadata)
LTFS EE
RHEL 6.2
RHEL 6.4
RHEL 6.5 New
SLES 11 SP1
SLES 11 SP2
SLES 11 SP3 New

19. New Tape Library Support
TS4500
TS3310
TS3200 / TS3100 / TS2900 available via RPQ.

20. Licensed Slot Capacity
TS4500 HD2 Base Frame
Up to 3.3X the slot density of TS3500
Up to 2.2 PB native capacity in a single frame
Two I/O Stations
Grow left or right of base frame
TS3500 Tape Library
TS4500 Tape Library
Model Lx3
Model Lx5
Up to 12 Drives
16 or 32 I/O
36 I/O - LTO
32 I/O
Licensed Slot Capacity
Entry
Interm
Full
Base HD
LTO 64
129
100
200
400
730
3592 58
117
550
In the TS3500 family, only S frames could be HD. With the TS4500, all frames are HD.

21. TS4500 HD2 Drive Expansion frames
Up to 2.4X slot density
Up to 1.3X drive density
New HD Capacity on Demand option
2 I/O Stations optional
No capacity reduction for tape drives
TS3500 Expansion
TS4500 Expansion
Model Dx3
Model Dx5
Up to 12 Drives
Up to 16 Drives
Licensed Slot Capacity
Base
HD w/ I/O
HD w/o I/O
LTO
500
882
970
3592
660
740

22. TS4500 tape drive support LTO-5, LTO-6, and TS1140
Up to 16 drives in each frame
Support for drive migration from TS3500 frames

23. New option for customer-usable rack space
10U of rack space above each TS3500 or TS4500 library frame
Opportunity for “zero footprint” rack-mount components such as Fibre Channel switches, Ethernet switches, IBM FlashSystems, and/or tape-attached servers

24. TS4500 control - Integrated management
Reporting
Tape System Reporter (integrated, improved)
Other
LDAP, DNS, NTP, SSL
(integrated, improved)
Management
GUI/CLI
(improved)
Monitoring
(SMTP), syslog, SNMP, mobile
(integrated, improved)
Admin and Service LANs
Support
Tape System Service Console (TSSC) capabilities
(integrated)
Integrated
Management
Console (IMC)

25. Transparent and selective bulk recall
New ltfsee recall command
Supports GPFS policy recall command
Supports recall list file (1 full path filename per line)
Supports stdin input
ls <file spec> | ltfsee recall
find . -type f | ltfsee recall
Takes advantage of the “Tape Optimized Recall” feature
Supports GPFS policy recall command (ie. mmapplypolicy)
Supports recall list file (1 full path filename per line) (eg. ltfsee recall ./recall_list.txt)
Supports stdin input
ls <file spec> | ltfsee recall
find . -type f | ltfsee recall (sends ltfsee recall all the file names in the current directory)
Takes advantage of the “Tape Optimized Recall” feature

26. Pre-PGA2 PGA2 Read Starts Recall (RSR) and Preview Size (PS) Stub
(on disk)
Remaining file
(on tape)
Remaining file
(on disk/tape)
Trigger recalls
Blocked
PGA2
Stub
(on disk)
Remaining file
(on disk/tape)
Remaining file
(on tape)
For example, say you have a video surveillance application, and the video is parsed into 1 hour files. You could keep the first few (5) minutes on disk, and while that was being read, concurrently the remaining 55 minutes would be recalled. So by the time the 5 minutes was up, the recall would have completed, and the video read would continue uninterrupted.s
Early trigger of recall
of migrated file called
Enhanced Normal
Recall
RSR
enabled
Blocked PS
(triggers
recalls) PS
(triggers
recalls)

27. Preserving file system objects on tape
Use GPFS policy with list rules: internal ltfseesave command
Saves symlinks, empty file (file size 0), empty directories (directories with no files or subdirectories) to tape
Same as migrate/premigrate and allows up to 3 pools

28. Rebuild GPFS from tape
For DR scenarios (GPFS file system is lost), rebuilds a GPFS file system into a specific directory from the specified tapes
If multiple versions/generations of a file are found, the latest version/generation is selected
Includes entire file system structure: regular files, regular directories, symlinks, empty files, empty directories
Similar to the ltfsee import command, but can be used even if the tape has not be exported.
The rebuild command will takes tapes that is either not-exported or already exported as we do not know when the GPFS file system is "gone". You can actually use rebuild at any time even if the GPFS file system is not "gone". It basically allows you to import tapes into the GPFS file system.

29. References
LTFS EE InfoCenter:
GPFS InfoCenter:
LTFS EE Redbooks:
LTFS EE Installation Demo:

30. Questions?

31. Agenda
LTFS EE Overview
PGA1 Enhancements
PGA2 Enhancements
Demo

32. LTFS EE – Demo CIFS NFS TS4500 Tape Library Global name space
Windows
Linux
Etc.
CIFS
NFS
Global name space
GPFS file system
LTFS EE
I am using PuTTy for command line access to the RHEL server.
I am using a vnc client as a GUI into the RHEL server.
vncserver :1 –geometry 1200x650 seems to work well.
TS3500 GUI:
Log libs / Cart summary
EE CLI Commands:
status, drives, pools
df –h ll
ltfsee info files file1.jpg
ltfsee info files file2.jpg
du –ahc
vncviewer: Look at a jpg on a mounted tape. Then look at a jpg on an unmounted tape.
mmapplypolicy gpfs1 -P /root/Jeff/downloads/testfiles/gpfs_migration_policy1 –L 3 -I test
ltfsee info jobs
Optional: tail -f /var/log/ltfsee.log (shows the ltfsee log in real time)
Detailed demo kept in C:\$User\Presentations\Tape\LTFS\LTFS EE\Demos\LTFS EE demo script.doc
TS4500 Tape Library

33. Disclaimers and Trademarks
The performance data contained herein was obtained in a controlled environment based on the use of specific data. Actual results that may be obtained in other operating environments may vary significantly. These values do not constitute a guarantee of performance.
Product data is accurate as of initial publication and is subject to change without notice.
No part of this presentation may be reproduced or transmitted in any form without written permission from IBM Corporation.
References in this document to IBM products, programs, or services do not imply that IBM intends to make these available in all countries in which IBM operates. Any reference to an IBM program product in this document is not intended to state or imply that only IBM's program product may be used. Any functionally equivalent program may be used instead.
The information provided in this document has not been submitted to any formal IBM test and is distributed "As Is" basis without any warranty either express or implied. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer's ability to evaluate and integrate them into their operating environment.
While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk.
The following terms are trademarks or registered trademarks of the IBM Corporation in either the United States, other countries or both:
IBM, S/390, ES/3090, ES/9000, AS/400, RS/6000, MVS/ESA, OS/390, VM/ESA, VSE, TPF, OS/2, OS/400, AIX, DFSMS/MVS, DFSMS/VM, ADSTAR Distributed Storage Manager, DFSMSdfp, DFSMSdss, DFSMShsm, DFSMSrmm, FICON, ESCON, Magstar, Seascape, TotalStorage, Tivoli, AIX, OS/400, TSM, BRMS
Other company, product, and service names mentioned may be trademarks or registered trademarks of their respective companies.
Linear Tape-Open, LTO, LTO Logo, Ultrium and Ultrium Logo are registered trademarks of HP, IBM and Quantum in the US and other countries.