2. Optimizing Oracle Exadata with Oracle Support Services [CON7054]
A Client View from KPN
Eric Zonneveld
Architect Engineered Systems
Oracle Advanced Customer Support
Jan van Dijken
Service Delivery Engineer
October 02, 2014

4. Program Agenda 1
KPN Overview Platinum Services Advanced Monitoring and Resolution Advanced Customer Support Lessons Learned 2 3 4 5

5. “We have multiple suppliers who deliver support and updates, but that makes it difficult for us to meet our service levels. We believe Oracle Platinum Services can improve the way we support our environment so that we are able to meet our service levels and, at the same time, lower our costs.”
Play video
Michel Bosman
Lead Architect, KPN

6. KPN Overview
Infrastructure, project and organization

7. KPN Overview Project 2010-2011 2012-2013 2014 Project startup (Mar)
Ordering (Nov)
POC Systems delivered X2-2 quarter (Dec)
Phase I: Initiation and testing (Jan)
Phase II: POC and pilots (Feb)
Phase III: EXA demo (Mar)
Phase IV: DTAP systems delivered X2-2 full (Jun)
First critical application chain live (Nov)
Platinum Services (Aug)
DTA systems delivered X3-8 full (Jan)
Oracle ACS and AM&R contract for service delivery (IM, PM, CM) (Jan)
EXATEAM created (Jan)
Migration DTA -> X3 (Jun) Separate production and non-production at rack level
2014
Delivery second production P2 based on 12c (Jan)
Migrate workload P1 to P2 (Apr)
Exalogic (May)
Exalytic (Aug)

8. KPN Overview
Infrastructure
Haarlem
Aalsmeer
40 km
Dataguard

9. KPN Overview Total Hardware 24 compute nodes 62 storage nodes
560 cores
10,8 TB memory
55,8 TB flash
1,824 TB storage
16 IB switches
6 CISCO switches
4 KVM
Per Site
Storage bandwidth 50GB/s X2 93GB/s X3 per rack
1,5M IOPS (read) 1M IOPS (write) per full rack

10. KPN Overview Interesting landscape … Interesting objectives …
Challenges
Interesting landscape …
Huge Oracle database landscape (>2,000), lots of Oracle middleware and applications
All hardware and OS platforms in 27 datacenters
All versions (e.g. Oracle database 7 and up)
Interesting objectives …
Data center migration and application / chain consolidation
Company strategy is to have vendor / partner take more responsibility
Decrease spend

11. KPN Overview Which leads to … While at the same time …
Challenges
Which leads to …
Increasing workloads because of consolidation
Increasing demands for performance, stability and availability
While at the same time …
Aggressive outsource strategy
Continuous re-organization
Challenging indeed

12. KPN Overview “Old fashioned” support did not work “New way” of working
How to engage
“Old fashioned” support did not work
Tiered model
Offshore
“New way” of working
DBA centric, end-2-end responsible
1 team onshore
Strong combination between Oracle (run) and KPN (change)

13. KPN Overview Oracle Platinum Services
Organization
Oracle Platinum Services
Change management (Patching)
Oracle Advanced Support Cloud Services
Incident management, problem management
Oracle Onsite Support
Innovation, life cycle management (infra upgrade)
KPN ITS Platform Business Applications
Change management, business consulting, backup & recovery

14. KPN Overview Scalability Performance Benefits of introducing Exadata
Servicing 594 databases (incl. 132 standby databases)
220TB storage used (net)
58 applications
Performance
Performance gains ranging from 2x to 100x
Special hardware and backup infra no longer needed
Application scaling up 10x volume without a problem

15. KPN Overview Availability Faster life cycle management
Benefits of introducing Exadata
Availability
Availability increase (no unplanned downtime)
Real Application Clusters
Active dataguard
Maximum availability architecture
Faster life cycle management
Most up-to-date and secure platform
Introduction of 12c at least a year before other platforms

16. KPN Overview But what about support? Benefits of introducing Exadata
Incident Management SLA’s (notification, diagnostic, resolution)
MTTR
Service request handling
Problem management
Business KPI’s (anything)

17. Platinum Services
Make the invisible visible

18. Platinum Services Part of hardware support “Free” but with rules
What is it?
Part of hardware support
“Free” but with rules
Limited scope for incident management
Oracle Advanced Support Gateway server
OEM12c, customized with metrics, thresholds and escalation rules
Patching
Change management (in ACS portal)

19. Platinum Services Visibility of incidents (not events!)
What changed?
Visibility of incidents (not events!)
Visibility of problems
Visibility of (wrong) configuration
RDS in Oracle executable
Visibility of Oracle best practices not being applied
Huge backlog
Moved from >120 open SR’s after engagement to average 15 SR’s currently

20. Oracle Monitoring
Advanced Support Gateway
Platinum aids company DBA’s with the incident management process

21. Oracle Advanced Support Cloud Services
Advanced Monitoring and Resolution

22. Oracle Advanced Support Cloud Services
What is it?
Extends Platinum Services, adds problem management
Broader scope
More events
Diagnostics
RCA’s
Resolution
Integration with client IM/PM processes (ITIL-based)
Service Now

23. Oracle Advanced Support Cloud Services
What changed?
More responsibility delegated to Oracle (IM and PM process)
Shifts focus from KPN DBA’s from running the infrastructure (run) to aid applications and business processes, introduce new functionality (change)
Advanced Support portal

24. Oracle Advanced Support Cloud Services
What changed?
Increased service level objectives, e.g.
Resolution
Severity 1: 80% < 1 day, MTTR average < 1 day
Severity 2: 80% < 3 days, MTTR average < 3 days
Availability reporting
Of (by the client) nominated critical database services and related applications, >99.99%
Incident summary reporting, quarterly
Production / non-Production, by date and severity, by category (Exadata software, database, compute node, storage, switch, other)

25. Oracle Onsite Support
Customer Integration

26. Oracle Onsite Support
What is it?
Acts as the “glue” between Oracle Platinum, AM&R and the client
Life cycle management (upgrades)
Innovation
New functionality, new releases
Best practices, Standardization
Expert troubleshooting
On site, close contact, customer intimacy

27. Lessons Learned
2 years of engagement, what worked and what didn’t?

28. Lessons Learned Oracle should be more visible in what they do
Room for improvement
Oracle should be more visible in what they do
Frequent face-to-face reporting in a format that appeals to the customer
Client does not want to know the differences within Oracle (Platinum, AM&R and ACS), should be one team and one voice
Incident to problem promotion can be improved
Portal could be improved
Google search and user friendliness
CI tagging

29. Lessons Learned Continuous learning within AM&R / Platinum
Room for improvement
Continuous learning within AM&R / Platinum
Enhancement of KA’s (knowledge articles) must become second nature
Change management
Integration of Oracle changes into the CM system and processes of the client
Work on “awareness” of customers of patch frequency, hardware maintenance etc.
Communication between support engineers and customer
Do not assume customers read all mail or plough through the portal every day

30. Lessons Learned
What worked?
Insight into Exadata significantly improved (exposure of problems)
Co-operation between KPN and Oracle
Service delivery management (TAM)
Co-engineering between Oracle and KPN
Local ACS resources
Complex problem solving

31. Oracle Services
Advanced Customer Support
and Support + + =