1. Execution
Monitoring
Notification
Aychin Gasimov
DOAG 2016

2. Pros Cons Enterprise Manager Cloud Control Wide range built-in metrics
Reach functionality
Includes job system
Incident management and reporting
Not just monitoring but also management
Pros
Configuration Complexity
BUGs and week support
Cons

3. Database can send incidents and problems saved into ADR to Enterprise Manager
Incident Rule Set
Check
Apply
threshold
Generate event
New Event
Incident occures
Store in ADR
Send to EM
Diagnostic Incident

4. Define metrics
Generate Event
Process Event

5. Design flow
Admin Groups
Templates
Event Rules
Jobs
Credentials

6. Define hierarchy and create Admin Groups
Templates
Event Rules
Jobs
Credentials
Define hierarchy and create Admin Groups
Groups targets by their type and properties
Targets become the member automatically
Automates target management on subgroups level
Each entity is separate admin group
ADMGRP
Production
Database Instance
Cluster DB
Host
ASM
Cluster ASM
Mission Critical
Development
Staging
Test
Lifecycle Status
Target type

7. Create Monitoring Templates and Template Collections
Admin Groups
Templates
Event Rules
Jobs
Credentials
Create Monitoring Templates and Template Collections
One template per target type and lifecycle status
Centralized metric management
Starting from define Adaptive Thresholds
Group Monitoring Templates to Template Collections
Associate Template Collections with Admin Groups
Template
Collection for Prod
ADMGRP
Production
Database Instance
Cluster DB
Host
ASM
Cluster ASM
Mission Critical
Development
Staging
Test
Lifecycle Status
Target type

8. Create Event Management Rulesets
Admin Groups
Templates
Event Rules
Jobs
Credentials
Create Event Management Rulesets
Associate with Admin Groups
Define
Target availability
Metric alert
Metric Evaluation Error
Define rule for notifying about Problems (from ADR)
SMS notification for Mission Critical and Production targets
Create incidents for specific conditions
System Response time
Development
Mission Critical
Create Incident
Send
Capacity
Performance
Send SMS
and
New event
Lifecycle
Action

9. Create Jobs Use Jobs to execute OS scripts Create Jobs in Job Library
Admin Groups
Templates
Event Rules
Jobs
Credentials
Create Jobs
Use Jobs to execute OS scripts
Create Jobs in Job Library
Define them as “OS Command” and “Command Type” as “Single Operation”
Define notification
In Job definition (notify only owner)
In Incident Rules
Submit from library to activate as required
Associate with Admin Groups
Associate with Admin Group

10. Central location for scripts and losg
Admin Groups
Templates
Event Rules
Jobs
Credentials
Create Job to sync central folder over enterprise
Maintain one place to manage all scripts
Synchronize central location to local hosts
Benefits:
Ease of maintenance
High availability
High redundancy
Apply the Job to the highest level of the Admin Groups
Execution logs can be synced back to central folder
Management host
/home/oracle/scripts
/home/oracle/logs
Management host
/home/oracle/scripts
Central location for scripts and losg
/home/oracle/logs
Management host
rsync -a --delete DEST
/home/oracle/scripts
/home/oracle/logs

11. Define credentials for the targets
Admin Groups
Templates
Event Rules
Jobs
Credentials
Define credentials for the targets
Create Global Named credentials for MC and Production databases
Use same global credentials for Primary and Standby databases as default ones
Define “Database Host Credentials” for Database Targets, they used by Jobs
If some target types mostly use same passwords, then define default credentials for them

12. Adding new target to EM Cloud Control
Save target
Set credentials
Set Lifecycle Status
Lifecycle Status = Production
Contact = BOTH
Database Instance
Template Collection
Cluster DB template
Host template
DI template
Prod-Grp
Jobs on Prod-Grp
Incident Ruleset for
Prod-DI-Grp

13. General recommendations
Use corrective actions, specify them in templates
To fix issues
To gather additional information
Wait some time after configuration to filter out useful metrics and adjust thresholds
Check already generated incidents and add them to your rules if required
Backup important components periodically and before major changes. Use emcli utility with next options:
export_admin_group
export_incident_rule_set
export_jobs
export_metric_extension
export_template
Backup OMS configuration (emctl exportconfig)

14. Availability problems
Monitoring should be uninterruptable
Scripts should be able to run without interruption
EM 13c provides new Always On Monitoring (AOM)
Configure Agent Out-of-Bound notification
Job executions can be offloaded to local crontabs
AOM
Target Availability events
AOM
Scripts running over Job system
Metric Alerts
Custom Solution

15. Custom solution for Jobs availability
Custom script resync_em_to_cron.sh
Extracts data from EM repository
Converts it to crontab format
Appends to crontab
Create EM Job which calls the above script with “add” option
Create EM Job which calls the scripts with del option
EM Jobs
DEPLOY_JOBS_TO_CRONTAB
UNDEPLOY_CRONTAB_ENTRIES
EM Repository DB
… add %TargetName%
… del
resync_em_to_cron.sh
hosta
resync_em_to_cron.sh
hostc
resync_em_to_cron.sh
resync_em_to_cron.sh
hostb
hostd

16. https://aychin.wordpress.com/downloads
Questions
Questions