Presentation on theme: "WellPoint The Fastest Essbase, Hyperion Planning, and OBIEE Performance Imaginable. Jonathon Eastman WellPoint, Inc."— Presentation transcript:
WellPoint The Fastest Essbase, Hyperion Planning, and OBIEE Performance Imaginable. Jonathon Eastman WellPoint, Inc.
Introduction. Why Oracle Exalytics. Business Case. Extreme Performance Infrastructure. Lessons Learned… including performance. Questions??? AGENDA
Health Benefits Company 37 Million Members Q $71 Billion Revenue 10K 2013 ≈ 50K Associates and Contractors. About WellPoint
Jonathon Eastman, Director EPM/BI. Center of Excellence – EPM. ●Cross functional team of CPAs to technical experts. ●We manage system tools and processes critical for the forecasting and budgeting, internal management reporting, and external GAAP and Statutory Reporting functions. We report to CAO even though we perform many functions traditionally considered “IT”. About Me And Our Department
Uber reliant on Essbase named Essbase users. 40 Reporting Cubes, 500+ Production Cubes. Reporting Tools. ●Financial Reporting/Workspace ●Excel Add-In/Smart View. ●Interactive Reporting (Dashboards). ●OBIEE with mobility. Hyperion Planning. 300 users. EPM OBIEE WellPoint
interRel Consulting. WellPoint Consultants last 7 years. We won the Oracle Innovation award OpenWorld 2013 (EPM/BI). WellPoint’s Partner. Consulting TrainingSupport Infra structure Press
Why Oracle Exalytics – Part 1. Over 1000 pages. Expected acquisitions and new business requirements. More data, metadata, and processes.
Aging and complex infrastructure. 13 AIX servers and 5 windows servers. OS Patching and upgrades costly and time consuming. Why Oracle Exalytics – Part 2.
Why Oracle Exalytics – Part 3. Tested, Certified, Packaged, Deployed, Upgraded, Managed, and Supported Together. AIX Last Tested, Last Released. Performance!!!!
Align with an initiative. ●Amerigroup acquisition for initial 3 Exalytics boxes. ●ACA Exchanges for 4 th Exalytics box. ●Easier to find budget. Replace aging infrastructure. ●Easy way to get concrete numbers. ●Priced internally against WellPoint Standard Offerings. ●We did not shop numbers against outside vendors. WellPoint’s Business Case – Part 1.
Roughly 25% cost savings. Can’t release hard numbers. Includes new software ●Converted to BI Foundation Suite ●Received Mobility and others. ●Majority of our cost. Better Servers. ●More CPU (80 cores to 120 cores). ●More RAM (640 GB to 3 TB). WellPoint’s Business Case – Part 2.
Savings were enough to justify. Normalize server CPU/RAM specs. Take out the software portion. Apply performance ratio to the cost. Administrative savings. ●Server consolidation (patching, upgrades, monitoring, etc.). Subjective savings ●Better lifecycles due to performance improvements. WellPoint’s Business Case – Part 3.
Extreme Performance Infrastructure.
WellPoint’s New Infrastructure – Part 1 F5 – to load balance for to EPM web applications (Round Robin). 3 Exalytics X2-3 boxes (No Flash Storage). ZFS storage device. ●EPM web application share (RM1, LCM). ●Essbase App Directory
WellPoint’s New Infrastructure – Part 2 Added 4 th Exalytics Server. Oracle VM. DEV, TEST, and PROD Fully utilize box.
Budget correctly. Transceivers do not come with the Exalytics boxes. ●10G networking (We did not use Infiniband). ●Required on the switch side and on the server side. Fiber Channel cards and transceivers do not come with the ZFS appliance. Virtualization and ASR (automated service request) require commodity OEL box (linux). Load balancing/Fail over potential costs. Lessons Learned – Procurement Part 1.
Lessons Learned – Procurement Part 2. Who does What?
Lessons Learned – Procurement (RACI) ActivityWLP ITWLP FDSMWLP IBMOracle Governance RICC User Provisioning RI Backup & Restore AAR Job Scheduling AAR EPM & BI Suite Application Admin & Support IR EPM & BI Suite Application Installation IR Exalytics Installation and configuration. AAAR Monitoring AC R General Administration IA R Proactive OS & Firmware Patching AA R 24hr Hardware Failure Resolution AA R
Power, Networking, Disaster Recovery. Final Roles and Responsibilities. Hardware failure. Information Technology – Governance, User Security. EPM CEE – Everything EPM, Associate Support Exalytics OS Patching, admin support.
Not Like Apple computer all ready to go. Now ships with Oracle Linux (5.5) pre-installed. Oracle Virtual Linux now option as fixed price install. Done on-site. 1 week to install OBIEE/Time Ten on three Exalytics boxes. Need to fill out Pre-Installation Questionnaire. Host Names, static ip addresses, DNS Servers, gateways. Logistics and Facility access. Database configuration. Create SS Schemas, Temporary dba access to run OBIEE RCU. Port Configuration. Oracle ACS performs installation. OBIEE, TimesTen, Essbase, but not all of EPM. (Like Planning) Lessons Learned – Installation.
Exalytics comes with HugePages memory set really high to support TimesTen Oracle In- Memory Database. Essbase/EPM does not use and it can potentially degrade performance. Lessons Learned – Configuration Part 1
Serious Error~( )~Unable to Create Request Server Thread JVM hs_err_pid.log Check pid_max for linux OS. ●cat /proc/sys/kernel/pid_max ●Output: kernel.pid_max = Change two ways. ●sysctl -w kernel.pid_max= ●echo > /proc/sys/kernel/pid_max Lessons Learned – Configuration Part 2
500 Gigabyte linux tmpfs for TimesTen. ●This in combination with Hugepages greatly reduces available memory. Load balancing requires a shared drives for LCM and RM1 directories. ●Oracle Clustered file systems supported. ●SAMBA/NFS not supported. If using a ZFS, need technical expertise configuring Multipath. Oracle VM – Guest Size Limitation (Patch Soon). Lessons Learned – Configuration Part 3
Lessons Learned – Performance!!! Benchmark Hardware IBM Power 595 (9119-FHA) 32 Core 64-Bit Power6 Processers 5.0 GHZ. 4 cores dedicated to Virtual IO. Split into 4 separate LPARS 96 Gigabytes RAM 2 Terabytes EMC Storage AIX 6.1
Saw over 2-3X improvement across the board. Serial calculations represented roughly 200,000 lines of calculation script code. Export = 2 level 0. Lessons Learned – BSO Performance.
Lessons Learned – BSO Performance Optimizing data cache and index caches improved performance. As today, must empirically optimize cache settings on a per application basis. Performance could get worse if cache is set too high.
Buffer loads saw a 2X increase. Already loading in memory. Commits and aggs performed 3-4X better Setting the Pending Cache Size Limit improved performance. Set too high and performance went down. Lessons Learned – ASO Performance
MDX Query times had the biggest variability improvement – from 2-8x better 95% of our queries are fast, so we looked for large dynamic Hierarchies with lots of formulas for test. 2-4x performance improvement report batch bursting in workspace. Lessons Learned – Query Performance.
Local storage cannot handle multiple simultaneous batch processes (non-flash). WellPoint originally looked at volatile RAMDISK before san was supported. ZFS performance equaled RAMDISK. Running 4 calc dims at once with 200,000 lines of calc script code. (Screenshot to follow). Lessons Learned – Storage Performance.
Lessons Learned – ZFS Performance IOPS
Midday Direct Admin and Headcount Batch Job to create two ASO databases. 5X Faster. Midday P&L, Balance Sheet, Cash Flow, statistical metrics. Many bso calculations exported into an aso reporting cube. 2X Faster. Nightly Batch process for all our “GAAP” databases. Cost Allocations, Direct Admin, Headcount, Enrollment, Quality Improvement, MLR Rebates, MLR Segments, P&L. 2.5X Faster. Lessons Learned – Batch Performance.
Performance hit over our x2-3 servers connected to ZFS. ●XEN overhead. ●Exalytics X3-4 come with 22 93G flash drives. ●LVM (Logical Volume Manager) Oracle ACS coming by end of year to tune linux and zfs. Contact me if you curious about the Oracle’s hypervisor’s impact on performance. Lessons Learned – Virtualization Performance.
Can you realize even better gains? Hardware CPU – Number of cores, Processor Speed, Cache, etc…. Hard Disk IO – Stand alone disk, performance RAID, Storage Area Networks (SAN), RAMDISK. Cube Design – Index, Structure, and Cache optimization. Script optimization – Calculating the minimum data necessary, utilizing CPU efficiently, etc… Load Optimization – Buffers, parallel loads, etc…. Batch process optimization – Take advantage of parallelism. Lessons Learned – Performance Variability.
Great Performance compared to high-end hardware 2-3x with cost savings. You can see even better gains if using lessor hardware. Exalytics has some configurations slanted towards OBIEE that may need to be altered if you focus more on EPM. You still need to optimize you application caches, outlines, and data loads. Takeaways.