1. PAE SuperCluster Peformance
June 12, 2012

2. SuperCluster/Exadata Dashboard
Focus Area
Previous Status
(05/23)
Current Status (06/12)
Comments
Virtualization (scaling-within-a-box) G
OLTP: Consistent performance across zones achieved by manually re-targeting ixgbe interrupt with SR-IOV VF. Currently testing on zones w/ SR-IOV VF IB support verifying that interrupt re-targeting is working as expected.
DSS: CPU-intensive queries scaling close to linear from 1 to 8 zones (1 zone/core) on single-socket in split PCI-E LDOM. IO-intensive query doesn’t scale due to IO bottleneck.
Exadata Expansion Rack Evaluation
Avg IB latency of cells in Expansion Rack same as in SSC rack
rds-stress peaks at 1.08M IOPS at 8 cells. Degrades to 0.9M at 21 cells.
Orion 8k peak of 850K IOPS, avg 815K 6-8 cells. Throughput degrades slowly to 21 cells at 790K IOPS.
Orion 8K IOPS: 24 cells scaling from 1 to 4 T4-4’s non-partitioned: 2.45M IOPS
1M Throughput: 24 cells: 1xT4-4=10.8GB/s, 4xT4-4=24GB/s
IOPS issues due to CR , Throughput issues under investigation
2-socket 11gR2 LDOM vs T4-2
11GR2 LDom has 5% better remote latency than T4-2 (lmbench)
iGenRAM & iGenCPU tests result in equivalent performance
Storage Cells vs Cached Storage Array (log file sync)
Flexcube log file sync issue due to I/O saturation of Storage Cells.
Igen oltp log file sync improved from 6-9ms to 2ms w/ disabling nxge driver
Storage Cell Flashlog Disk First investigation
Niwot Controller Cache is faster than Aura flash for log writes: 97% of log writes satisfied by disk. If controller cache turned off 100% log writes satisfied by flash.
With heavy concurrent read workload disk writes redo 100MB/s faster than flash.
Exadata Flashcache with Aura2 R
On hold: Exadata Storage Server software to support Aura2 not be available.
RAC Scalability
On Hold: Resources redirected to Storage Cells vs. Cached Storage Array project.

3. SuperCluster/Exadata – Highlights
Performance issues reported by Field/non-DPA Orgs
T4-2 vs 2-Socket 11gR2 LDOM CPU/Memory Performance
Issue: T4-2 has better CPU and memory performance compared to 2-socket 11gR2 LDOM CPU/Memory
Resolution: PAE repeated tests using iGenCPU and iGenRAM resulted in equivalent performance
Exadata Storage Cell vs Cached Storage Array (log file sync issues)
Issue: iGenOLTP on SSC had 35% better Response Time with Redo Logs on ZFSSA 7320 than on Exadata Storage Cells with FlashLog
Resolution: Disabling nxge driver reduced Log File Sync from 6-9ms to 2ms. Response Time with Storage Cells is now 1.6x better than ZFSSA
Issue: General Observation that with Exadata FlashLog only small percentage of redo writes are satisfied by flash.
Resolution: Purpose of FlashLog is to eliminate outliers when controller cache is congested. Confirmed that Niwot Controller Cache on Exadata Storage is faster than Aura flash for log writes: 97% of log writes satisfied by disk. If controller cache turned off 100% log writes satisfied by flash. FlashLog behavior is correct.
SuperCluster/Exadata – Highlights
Confidential

4. SuperCluster/Exadata – Lowlights
Performance issues reported by Field/non-DPA Orgs
Issue: Flexcube team reported that performance for given batch step was 2.2x faster on T4-2 w/ ST6180 than on SSC 11gR2 Ldom w /3xCells due to redo log performance.
Review of data showed I/O saturated on the storage cells. Also test conditions were not exactly the same.
Highlights that SSC/Exadata performance work has concentrated on read performance only. Will expand scope to include write-intensive workloads.
Evaluate Exadata Expansion Rack
Target Goals for Orion I/O tests on Full Rack SSC connect to Full Exadata Expansion Rack (4xT4-4 and 24xExadata Storage Cells)
3M IOPS (random 8K reads)
48 GB/s (random 1M reads)
Actual Results
2.45M IOPS (random 8k reads)
24GB/s (random 1M reads)
Performance shortfall due mainly to 56 event queues/socket limitation which results in cpus sharing interrupts (CR )
Exadata FlashCache with Aura2
Exadata Storage Server software to support Aura2 is still not available for testing.
SuperCluster/Exadata – Lowlights
Confidential

5. SuperCluster/EECS Performance Dashboard
Focus Area
Previous Status
(02/12)
Current Status (03/25)
Comments
MWM-E
Met and exceeded December RR goals G
Self-Tuning/worker thread optimization now works generally well based on MWM-E
Lock-Less Resource Manager (LLRM) does not show overhead. LLRM is recent EECS optimization
Improved OTD performance by 20%. Can reach IR 5900 using OTD or IR 6000 without OTD
Virtualization (scaling-within-a-box)
N/A
WIP
Characterize & Analyze VM, both Zones and SR-IOV LDOMs, scaling based on current SSC split PCIe LDoms configurations
Current low-level SR-IOV IB preliminary performance status :
Single Connection latency over VF to external client is 5-10% worse than baremetal to external client
Request/response profile performance – IPoIB over VF performance is 30% less than as expected. Under investigation
OTD
Observe throughput drop with higher connections (IPoIB to WLS). Under Investigation
Differentiator exploration – OTD + ILB:
Solaris ILB is Layer 4 kernel Load Balancer. When conducting layer 4 LB, ILB consumes ¼ CPU of OTD for delivering the same throughput
Found out ILB performance under IB UD mode is much better than under CM mode due to lack of soft-interrupts feature under IB CM mode. Prototyping in progress
Infinibus/Coherence
Proof of concept status:
Unit tests passed, both C and Java portion, on Solaris x86 platform
SPARC porting in progress
TBD: likely need OFUV library optimization – from gcc to Sun Studio

6. OTD vs OTD + ILB Performance With Layer 4 LB Rules
OTD Only
With ILB (+OTD)

7. OTD Differentiator Next Step OTD + ILB + OpenFlow
OS Kernel (ILB)
Hardware
To-Do:SSL offloading and acceleration
Back-end servers
Clients