How we made our Agile Development Cycles More "Agile" and Less "Fragile"  LeanLogistics Introduction  Development Cycle  Business Problem  Technical Solution Cody Pack Oracle DBA Manager Michael George System Administrator

LeanLogistics  Founded as a SaaS software company from day one  Industry leader in SaaS transportation technology development  LeanTMS ® single instance, private cloud, serving customers in over 100 countries  Managed Transportation Services – 100 in-house users of our app  40 developers  Java application on top of Oracle database

Our Technology

Our Customers

Development cycle  Three major releases per year  Twelve minor releases  Two-week sprints Sprint Kickoff Daily Standup Frequent Demos Sprint Review Team makes a commitment to complete a set of work What did I do yesterday? What will I complete today? What obstacles are in my way? Ensure our solution will solve the problem we first identified Project stakeholder receives a working demo and have an opportunity to provide feedback

Business Problem Production Cluster Production Array Non Production Cluster Non Production Array RMAN DWTXDWTX  Each major release involves 28 database clones, which is 84 clones per year.  Depending on the needs of the release cycle, sometimes three clones a week are needed.  RMAN clones reached over 30 hours in duration.  Schedule didn’t allow for failures of the clone process. The entire Development and Release Management teams were severely impacted if failure occurred.

Hardware Limitations Encountered  Production and our primary testing environment shared the same array, utilizing separate RAID groups.  Each clone writes about 6TB of data to the SAN-based storage. Not a problem for the fiber network, but it was for the aging EMC Clariion.  Excessive contention writing to too few drives. Purchasing more drives for the RAID groups helped. A data utilization rate of 50% hurt.  Excessive contention on the controllers. Rebalancing access across the controllers alleviated the issue.  Excessive contention on the bus to the disk drives caused the array’s only write-cache to fill and cause write-through. Deactivating cache for the test LUNs allowed production use of cache, impacted testing performance.  The bag of tricks was getting empty, a new storage array for the testing environment was needed!

New Array Technologies  Thin-Provisioning  Solid-state storage  Data Compression  Data De-duplication  Snapshots and thin-clones

Considered Options All flash, data compression, data deduplication, thin-provisioning 11TB (actual) system: ~$250,000 Hybrid-storage (flash and spinning disk), data compression, thin-provisioning 45TB (actual) system: ~$170,000

Our Selection  Nimble Storage  No data de-duplication, but has more base storage  We have lots of stale data, putting it all on flash storage is expensive  Met our needs for ~65% of the cost CS500

Results  About a 60% reduction in storage consumption due to compression  Combined all down-level environments’ databases to leverage zero-copy cloning  80% of our reads are at 1.1ms latency, 0.5ms for writes.  We were satisfied enough that we bought a second for production storage.  StorageTek arrays consumed 2/3 of a rack, a RAMSAN - 6U, and the Clariion an entire rack. Two Nimble CS500’s replace them all and consume 6U of rack space.  We have also moved the Vmware storage for our production and primary testing environments to data stores on the Nimble for improved performance during release rebuilds. Before After Rack Usage

Solution Production Cluster Production Array Internal Active DG Nimble CS500 DWTXDWTX  Migrated internal use Active Dataguard to Nimble CS500 Array. Active Dataguard

Non Production Cluster DWTX Non Production Cluster DWTX Solution Internal Active DG Nimble CS500 DWTX  Migrated internal use Active Dataguard to Nimble CS500 Array.  Attached all Non Production clusters to the new array.

Non Production Cluster DWTX Non Production Cluster DWTX Solution Internal Active DG DWTX  Migrated internal use Active Dataguard to Nimble CS500 Array.  Attached all Non Production clusters to the new array.  Leveraged Array level Snap Clone technology. Array Level Snap

Solution Internal Active DG DWTX  Migrated internal use Active Dataguard to Nimble CS500 Array.  Attached all Non Production clusters to the new array.  Leveraged Array level Snap Clone technology.  Thin provisioned non production databases off the snap. Array Level Snap Non Production Cluster DWTX Non Production Cluster DWTX

Benefits Internal Active DG DWTX  Clone duration decreased from 30 hours to 30 minutes.  Provided the DBA team with greater flexibility.  Clone failures no longer a business risk to Development and Release Management teams  Testing Accuracy greatly increased by high performance Array.  107TB allocated to the databases, 96TB “in use” by all the database volumes  Between data compression and the thin-clones, 22.5TB of the available 45TB is in use, 3.25TB of which are the snapshots, the basis of the thin-clones. Array Level Snap Non Production Cluster DWTX Non Production Cluster DWTX

Questions?