Combining Metrics and Logs for Holistic System/Application Analysis Sharath Kumar M N Elasticsearch Solutions Architect, Peoplesoft Oracle May 10, 2017 Confidential – Oracle Internal/Restricted/Highly Restricted

Metrics :- A metric is a quantifiable measure that is used to track and assess the status of a specific process Confidential – Oracle Internal/Restricted/Highly Restricted

Logs:- record of incidents or observations Confidential – Oracle Internal/Restricted/Highly Restricted

Need For Holistic Analysis   Confidential – Oracle Internal/Restricted/Highly Restricted

Metrics 10.121.123.104 - - [01/Nov/2016:21:01:00 +0100] "apache" cpu 30 10.121.123.104 - - [01/Nov/2016:21:01:05 +0100] " apache" cpu 35 10.121.123.104 - - [01/Nov/2016:21:01:15 +0100] "apache" cpu 32 Logs 10.121.123.104 - - [01/Nov/2016:21:01:04 +0100] "GET /cluster HTTP/1.1" 200 1272 10.121.123.104 - - [01/Nov/2016:21:01:17 +0100] "GET /cpc/auth.do?loginsetup=true&targetPage=%2Fcpc%2F HTTP/1.1" 302 466 10.121.123.104 - - [01/Nov/2016:21:01:18 +0100] "GET /cpc?loginsetup=true&targetPage=%252Fcpc%252F HTTP/1.1" 302 - Confidential – Oracle Internal/Restricted/Highly Restricted Confidential – Oracle Internal/Restricted/Highly Restricted Confidential – Oracle Internal/Restricted/Highly Restricted 6 6

Application Metrics + Logs Confidential – Oracle Internal/Restricted/Highly Restricted

Architecture Collect/Ship Data Datastore, Search, analytics engine Visualize Edge Nodes Queue Parse/ Enrich Data Confidential – Oracle Internal/Restricted/Highly Restricted

Filebeat Lightweight Shipper for Logs Tail files Ensure At-Least-Once Delivery Extensions: Modules Filebeat comes with internal modules (Apache, Nginx, System, and MySQL)  Filebeat consists of two main components: prospectors and harvesters.They work together to tail files and send event data to the output that you specify. A harvester is responsible for reading the content of a single file. The harvester reads each file, line by line, and sends the content to the output. A prospector is responsible for managing the harvesters and finding all sources to read find all files on the drive that match the defined glob paths and starts a harvester for each file Filebeat keeps the state of each file and frequently flushes the state to disk in the registry file. Confidential – Oracle Internal/Restricted/Highly Restricted

Config file (.yml) Confidential – Oracle Internal/Restricted/Highly Restricted

Metricbeat Lightweight Shipper for Metrics Extensions: Modules and Metricsets Come with many modules (Apache, Nginx, System, Redis, MySQL,PostgreSQL, MongoDB, Kafka ….)  Confidential – Oracle Internal/Restricted/Highly Restricted

Config file (.yml) Confidential – Oracle Internal/Restricted/Highly Restricted

Event Structure Confidential – Oracle Internal/Restricted/Highly Restricted

Logstash Data collection engine with real- time pipelining capabilities  Unify data from disparate sources and normalize the data into destinations of choice Parse and Enrich Logs Confidential – Oracle Internal/Restricted/Highly Restricted

Logstash Pipeline Confidential – Oracle Internal/Restricted/Highly Restricted

Parse Logs 192.164.4.12 - frank [10/Oct/2016:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 { "clientip": "192.164.4.12", "userId": "frank", "timestamp":"10/Oct/2016:13:55:3 6 -0700", "verb": "GET", "request": "/apache_pb.gif", "httpversion": "HTTP/1.0", "response": 200 "bytes" :2336 } filter { grok{ match=>{ "message"=>"%{IPORHOST:clientip} %{USER:userId} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion}))" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" } Confidential – Oracle Internal/Restricted/Highly Restricted

Enrich Logs 192.164.4.12 - frank [10/Oct/2016:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 { . "geoip": { "continent_name": "North America", "country_iso_code": "US", "region_name": "California", "city_name": "Mountain View", "location": { "lat": 37.386, "lon": -122.0838 } } filter { geoip { source => "clientip" } Confidential – Oracle Internal/Restricted/Highly Restricted

Elasticsearch Distributed, full text search and analytics engine Based on Lucene Its RESTful Highly available Very Fast Connectors for Big Data – “Elasticsearch-Hadoop” Confidential – Oracle Internal/Restricted/Highly Restricted

Kibana Analytics and visualization platform designed to work with Elasticsearch  Browser-based interface for near real time analytics Option of visualizing data in time series. Perform advanced data analysis and visualize your data in a variety of charts, tables, and maps Confidential – Oracle Internal/Restricted/Highly Restricted

Lets See it in Action!! Confidential – Oracle Internal/Restricted/Highly Restricted

Demo – Attachment Processing Confidential – Oracle Internal/Restricted/Highly Restricted

Demo – Attachment Processing Confidential – Oracle Internal/Restricted/Highly Restricted

Demo – Attachment Processing Confidential – Oracle Internal/Restricted/Highly Restricted

The Best Part is …. Confidential – Oracle Internal/Restricted/Highly Restricted

Open Source Powerful Simple Confidential – Oracle Internal/Restricted/Highly Restricted Confidential – Oracle Internal/Restricted/Highly Restricted 25

Benefits Holistic Analysis / Unified Analytics Better Troubleshooting / RCA Deeper Insights into System and Applications “Data” is Power – Intrinsic Values Confidential – Oracle Internal/Restricted/Highly Restricted

Sample Use Cases DevOps , IoT, ITOps , Application Monitoring & Analytics Confidential – Oracle Internal/Restricted/Highly Restricted

Q&A Confidential – Oracle Internal/Restricted/Highly Restricted