1. Optum DTR Process Overview

2. Agenda UHG Application Tuning
DTR: The Foundation of UHG Tuning Efforts
Key Participants
DTR Lifecycle
UHG Results
The Secret Sauce
Areas of Ongoing Focus
Questions/Discussion

3. UHG Application Tuning Program
Business demand and transaction volumes grew with no end in sight
Two Choices:
Throw ever-increasing amounts of hardware at the problem, often to no avail
Tune high-value, high ROI consumers
In 2004 UHG chose the latter, and from this decision the DTR program/partnership was developed.

4. DTR: The Foundation of UHG Tuning Efforts
In response to corporate mandate to control CPU consumption, a process was developed by which tuning actions could be created, reviewed, tested and implemented. The Detailed Tuning Recommendation (DTR) program was the output from this exercise.
A DTR is a tuning recommendation created for application jobs, transactions, SQL, or system resources for the purposes of:
Reducing Business Chargeback
Improving Response Time
Improving Cycle Time (Batch Throughput)
Reducing Abends/Contentions
Decreasing system overhead and/or peak hour demand
Summarized: Improve efficiency of UHG assets
Standardized format makes for easier processing
DTRs are intentionally detailed for the purpose of providing clarity for the recommended change so as to minimize the amount of discovery work needed by the application team
The process also includes a reporting database (compiling DTR savings & costs) containing all DTRs for current year as they progress through the DTR life cycle.
Every DTR is assigned a unique serial number so its progress can be tracked.

5. Key Participants APM Team (Tuner): Identifies the tuning opportunity
Writes the DTR (Detailed Tuning Recommendation)
Supports the Application team throughout development lifecycle
Application Team:
Validate the recommendation
If funding is needed, estimate the work and gain approvals
Complete any additional analysis needed to confirm scope/impacted components
Complete necessary build/development activity
Complete necessary testing
Schedule and execute implementation activities
DTR Administrator:
Maintains inventory of all DTRs
Tracks DTRs throughout lifecycle
Measures and publishes tuning results
Creates scorecards for aggregated results
Other Contributors:
System teams (CICS, DBA, Capacity Management, etc.): Identify hotspots or areas of interest
Business teams: Communicate SLA issues and/or areas of growth
Problem Management: Issues that require performance attention

6. DTR Process Workflow Opportunities are identified in several ways:
Identify Opportunity and Create Tuning Recommendation
Application Implementation
Measurement
Publish Results
Opportunities are identified in several ways:
Application team has a performance issue where tuning expertise is needed
Problem identified as a result of a Production Incident
Instrumentation/exception reporting identifies a situation where tuning could address consumption or performance
Create Tuning Recommendation:
APM team creates Detailed Tuning Request (DTR) that outlines suggested changes that will address opportunity

7. DTR Process Workflow Application Implementation:
Identify Opportunity and Create Tuning Recommendation
Application Implementation
Measurement
Publish Results
Application Implementation:
DTR information is forwarded to the Application Service Manager
Application Team:
Assesses DTR for effort and timing
Request funding if needed (ROI is a significant factor in decision to fund)
Keep DTR Administrator updated on progress
Schedule and implement DTR
Confirm implementation

8. DTR Process Workflow Measurement:
Identify Opportunity and Create Tuning Recommendation
Application Implementation
Measurement
Publish Results
Measurement:
DTR Administrator gather performance metrics across a standard measuring period following implementation (usually two weeks).

9. DTR Process Workflow Publish Results:
Identify Opportunity and Create Tuning Recommendation
Application Implementation
Publish Results
Measurement
Publish Results:
DTR Administrator team notifies to interested stakeholders with the results of the DTR and updates annualized totals

10. UHG Tuning Results – Executive View
Chargeback savings roll-up with breakout by:
DTR Phase
Area/type of savings
Tuning results reflect:
Traditional success in tuning DB2 usage
Active work to manage DB size through purging, archiving, etc.
Distributed-related chargeback for storage and memory

11. UHG Tuning Results – Application View
Ability to report results by Application

12. UHG Tuning Results – Inception to Date
Since inception of DTR program in 2004, over 1900 DTRs have been implemented with cumulative compounded CPU savings of +56 million CPU hours.
We would require double the current amount of mainframe capacity without the DTR program.

13. The Secret Sauce Enterprise Support for program:
Proven results
Partnership with Application, Systems and Business teams to identify and implement changes
Funding: Common pool of funding established to remove “no money to tune” barrier from the application teams while maintaining centralized control and visibility into where tuning efforts are directed.
Business willingly funds because they receive 17:1 ROI – for $1 spent on tuning their chargeback bill is reduced by $17 (average ROI )
Lightweight Process
APM vs. Application roles: The APM team makes a recommendation; the Application team maintains ultimate ownership of their code
Defensible measurements with supporting documentation
The APM Team:
Singularly focused on Application performance
Consolidated into one team, shared learning and information
Highly skilled former developers with deep system and operations knowledge
Flexible – not dedicated to a single app; can “run to the fire”

14. Areas of Ongoing Focus Aging DTRs: Improved Predictability of Outcomes
As inventory ages, recommendations run risk of becoming obsolete
Improving internal processes to keep inventory fresh and relevant
Improved Predictability of Outcomes
Ensuring All Tuning is Captured via DTR
“If no DTR then the savings did not happen”
Non-Chargeback Metrics
Most current aggregated reporting is mainframe and chargeback-based (CPU or Storage)
Opportunities exist to improve how we produce and report aggregated results for:
Abend/contention reduction
Response time improvements
Distributed environmental measurements
Improved/Common Access to DTR Information
Enhanced partnership with Application teams:
Performance assessment/tuning earlier in SDLC
Business Logic

15. Questions/Discussion