1. ASM-based storage to scale out the Database Services for Physics
Database Service Meeting - April 11th, 2006
Luca Canali, CERN

2. Outline Storage for Oracle 10g A solution based on cost/performance
Some performance measurements and benchmarks
ASM at CERN, lessons learned
DB Service Meeting, 11-Apr-2006
Luca Canali

3. Architectural Goal
High-end performance, scalability and HA at low cost
RAC
Conventional -> Scale UP
Grid-like -> Scale OUT
ASM
DB Service Meeting, 11-Apr-2006
Luca Canali

4. Storage Solutions for RAC
RAC implements ‘shared everything’ clustering
Common storage solutions for RAC
High-end SAN (ex: EMC2 symmetrix, Hithachi)
Specialized NAS (ex: Netapps filer)
SAN + low-cost storage + ASM
Infortrend storage arrays with FC controllers and SATA disks
QLogic HBAs and FC Switches
Less common solutions
Directly attached storage
Solid state disks
DB Service Meeting, 11-Apr-2006
Luca Canali

5. Automatic Storage Manager
ASM is a volume manager and cluster filesystem for Oracle DB files
Raw IO, Direct IO, Asynch IO
Implements S.A.M.E. (stripe and mirror everything)
‘the software glue’ to scale out for performance and HA
Online storage reconfiguration (ex: in case of disk failure)
Ex: ASM ‘filesystems’ -> disk groups:
DiskGrp1
DiskGrp2
Mirroring
Striping
DB Service Meeting, 11-Apr-2006
Luca Canali

6. Performance, Capacity and Cost
ASM proven to ‘scale out’ low-cost storage:
I/Os per second:
Tests at CERN showed good scalability up to max tested (64 HDs)
~ 100 IOPS per disk (SATA disks, small random IO)
Sequential throughput:
Scales out, but limited by fabric to 2Gbps (per HBA)
Tests on a 4 node RAC at CERN -> ~800MB/s for seq. read
High capacity: leverages SATA disks (typical DB size 5-10 TB)
Comparison with the top performers: Solid State Disks (SSD)
SSD has highest performance: ~100K IOPS, latency < 1 ms
BUT cost/capacity (SSD vs. SATA) > 1000, while cost/IOPS ~ 1
DB Service Meeting, 11-Apr-2006
Luca Canali

7. Orion 10.2, Sequential IO (RO)
Bottleneck: Disk controller = 2 Gb
(from other tests HBA = Gb)
DB Service Meeting, 11-Apr-2006
Luca Canali

8. Sequential IO measured with SQL
4 Disk arrays = 4 x 2 Gb
(measured with parallel query)
DB Service Meeting, 11-Apr-2006
Luca Canali

9. Orion 10.2, Small Random IO (RO)
DB Service Meeting, 11-Apr-2006
Luca Canali

10. Small Random IO, with SQL
8675 IOPS
135 IOPS (uniformly) per disk
Extra tuning: equivalent to using only the external edges of the disks
DB Service Meeting, 11-Apr-2006
Luca Canali

11. Lessons Learned Performance Administration:
ASM and ASMLib scale out, tested with 64 HDs.
IO is spread uniformly across disks and mirror pairs for HA and performance (no data yet on write activity, but no issues expected)
Administration:
We experienced a few stability issues with 10.1, but all fixed in No pending issues with ASM.
‘ASM DBA’ need storage admin and sysadmin skills. Ex: we configure ourselves HBAs multipathing, LUN mapping and SAN switch zoning.
We don’t use arrays’ RAID, but deploy a custom ASM config using JBOD (HA and performance in return for the added complexity)
Single disk failure rate, so far as expected: MTBF = 60 years.
DB Service Meeting, 11-Apr-2006
Luca Canali

12. Conclusions
Storage for RAC on Linux at PSS successfully consolidated using Infortrend disk arrays and Oracle ASM.
Scalability and performance tests are positive.
Extra effort for administration
Specialized operation required mainly during installation and replacement of failed disks
DBAs occasionally need to wear the storage admin hat
More info on wiki:
DB Service Meeting, 11-Apr-2006
Luca Canali