Aaron Rimel Team Lead October 2015 Upgrading Oracle Enterprise Manager: Why and How

© 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner. 1 Innovating at the intersection of Health Care and IT World’s largest standalone health care IT company $2.9 billion in revenues 21,000 associates+ worldwide 14,000 client facilities in 30+ countries 4,900 physician practices 590 ambulatory facilities 3,500 extended care facilities 150 employer sites 1,790 retail pharmacies R&D commitment and clinical culture $3.8 billion of cumulative R&D 3,000+ person software development org 1,800 clinicians More than 140 health IT patents #4 Top 100 Healthiest Workplaces in America #22 Most innovative companies 2014

© 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner. 2 CernerWorks 52,403 Systems 1,034 Network Circuits 90 PB Disk Storage CernerWorks 52,403 Systems 1,034 Network Circuits 90 PB Disk Storage Managed services 386 contracts 248,576 Contracted Users / 247,856 Peak Users 245,851 Beds = 27 % of total beds in United States 36,971 Facilities Managed OMS 33 Clients DBA 4 Clients Disaster Recovery 33 Clients Remote Hosted 356 Clients ITWorks 15 Clients Cloud Cerner Wellness 359 Databases Patient Portal 943 (290 Acute, 684 PWx) Healthe Record RHIO / HIE 13 Sites Healthe Transaction/Exchange 3 TPA Clients Skybox 180 Clients SaaS PowerWorks 2,718 Clients 20,215 Concurrent Users CommunityWorks 72 Clients

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 3 CERNER Oracle Management Server version Three EM installations, Production/Sandbox/Development Repository version: – 3.5tb in size EM Agents: 1500 Agents, 97% RHEL, 3% other Targets 25k Tableau, Vertica AWR Warehouse

© 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner. 4 OEM Monitoring at Cerner 60M Raw Metrics Daily 540 Tickets 33,000 Incidents 18,000 Incidents 2,300 Alarms

© 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner. 5 Corrective Action statistics

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 6 EM – Way to go ! We love the ease of installation. =) Gold agent image, ora check plug-in Single point solution for Agent Lifecycle Flash is gone… (almost)

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 7 EM : Upgrade Vs Install ? 13.1 Upgrade or Fresh Install? Large repository, extended upgrade time Creating a new multi EM installation Repository clean-up, upgraded from 10.1.x,10.2.x,11.1,12.1.x Methodical upgrade, allows for a client focued upgrade, making changes to client systems when it is right for them.

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 8 How we Manage Agents with Same Configuration We were way behind the Oracle Patch recommendations. Prior to Golden Agent Images, upgrading/patching agents would have taken a FTE. We almost never patched agents. EM now has a patch/psu rotation. We can now plan and execute our EM agent upgrades staying current with PSU’s or applying a one-off to fix an immediate problem. Upgrading agents prior to GAI, took nearly a year to accomplish due to the vast number of agents that exist. The implementation of Gold Agent Images has altered our deployment schedule to be in-line with PSU.

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 9 Solution: Gold Agent Lifecycle Feature Requirements (One time activity) Software Library Configured Source Agent to Create Gold Image – Fresh deploy or use existing agent Step1: Apply patches, configuration changes to the Source Agent Step2:Test the Source Agent – Check if agent is up, uploading Step3: Create Gold Image Step4: Use Gold Image to deploy, update, upgrade Agents (Recurring) Step5: Apply any new patches, Config changes to Source Agent Step6: Create another version of Gold Image

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 10 Solution: Gold Agent Lifecycle Feature We have updated /etc/sudeors to allow for our OS user to run commands specific to the agent, ie root.sh, allroot.sh to make the upgrade even easier by removing the need for root access to the machines. Activity with Agent Gold ImagePassword RequiredRoot.sh execution Fresh Deploy Password required and supports SSH keys, PBRUN, Sudo, all EM supported authentications Yes Update (Patching) agent with newer version of Gold Image Password requiredRoot.sh depends on what is the type of update a)If the patch need sbin changes then it requires root.sh execution. b)For config property changes, root.sh is not required Upgrade agent using Gold imageNo password required, uses existing OMS-Agent communication

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 11 oem ALL=(root,oracle) NOPASSWD:/bin/su - oracle*, /usr/bin/id, /u01/oracle/product/oemagent/agent13c/*/agentdeployroot.sh, /u01/oracle/product/oemagent/agent13c/*/*/agentdeployroot.sh, /u01/oracle/product/oemagent/agent13c/*/sbin/nmosudo *, /u01/oracle/product/oemagent/agent13c/sbin/nmosudo *, /u01/oracle/product/oemagent/agent13c/*/root.sh

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 12 Base Directory Structure of Fresh Agent Install Using a Gold Image |_____core |_____ |_____plugins |_____agent_inst |_____sbin |_____plugins.txt |_____plugins.txt.status |_____agentimage.properties Base Directory Structure of Upgraded Agent Using a Gold Image |_____GoldImage_ |_____core |_____ |_____sbin |_____plugins |_____agent_inst Base Directory Structure Agent is upgraded or updated using a gold image multiple times |_____GoldImage_ |_____core |_____ |_____sbin |_____plugins |_____GoldImage_.. |_____agent_inst Changes to Agent Directory Structure Tip: Sudoers file should have $oracle_home set and agent user profile with right oracle home. Don't hard core the agent base dir/sbin in the sudoers file.

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 13 Why Out of Place Agent Updates ? Simple rollback process - If Gold Agent Update fails then your old Agent will come up automatically Simplified old Agent home cleanup Reduce downtime Multiple Agents with multiple patches in a single downtime While updating old Agent is brought down later in the process

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 14 Solution: Gold Agent Lifecycle Feature Dashboard, real time tracking, maintenance via clicks Evaluated as compliance against the latest ‘image’ Agents yet to be updated with Gold Image

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 15 Solution: Gold Agent Lifecycle Feature Drifters – Detects manual Patching and drifts from Current Gold Image Chart showing subscriber’s versions Displays all artifacts included in the version of Gold Image

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 16 Solution: Gold Agent Lifecycle Feature 1.Gold Image can be Imported/Exported from a Test to Prod EM 2.Agents can be updated with Gold Image using Push or Pull 3.Define Batch size, success criteria and frequency which will reduce workload on OMS and help you plan your updates 4.Updates can be planned during maintenance and comfortable time periods 5.Get notification for Success/Failed/Status for each Agent update in a Batch 6.Ability reclaim space by cleaning out old Agent Oracle Homes 7.Run just Prerequisites before actual update operation

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 17 Solution: Gold Agent Lifecycle Feature Export Gold Images across Test /Dev/Prod EM environments 1 2 3

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 18 Solution: Gold Agent Lifecycle Feature Single view for all activities

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 19 Solution: Gold Agent “EMCLI” scripting option Used EMCLI Gold Agent commands for updating 2500 Agents! Worked like a charm Repeatable and reliable process Use cases executed: Upgraded Agents to Updated all Agents to Again updated Agents from to

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner. 20 Gold Agent Lifecycle Feature Simplified Agent Lifecycle Operations at scale Flexible Deploy/Update/Upgrade Agents at Scale and Ease with Gold Images End users can update during their comfortable time periods Reduced Downtime Patch and Test Source Agent Replacing updated software for changes reduces or removes downtime Automation at Scale Create Gold Image & Subscribe Agents Make changes at mass scale