1. The ELK stack - get to know logs
Igor Rudyk
DevOps / System Integrator

2. Agenda Introduction. What is ELK, and why do we need it? The ELK stack
Logstash
ElasticSearch
Kibana
Architecture
Demo

3. Can you check the errors from yesterday between 9:09 and 9:27

4. So what’s a log

5. Log = timestamp + data

6. Real Real Lifecycle of a log
Transmit
Analyze
Record
Store
Delete

7. Default problem Tools? Multiple log time formats
Apr 28 20:21:59
[27/Apr/2015:07:05: ]
:27:32
Mon, 27-Apr-15 06:27:02 UTC
:07:
Starts not with timestamp or without timestamp
Error messages with really unhelpful info
No rotation
No scaling
Tools?
grep
awk / sed / cut
less / tail
vi / vim
regular expression
...

8. Logging Solutions Solutions Collections Transport Parsing Storage
Analysis
Alerting
Visualizer
Commercial
Logstash
Logstash shipper or logstash-forwarder (Lumberjack)
RPM installation
Logstash shipper or logstash-forwarder (Lumberjack, encrypted transport is the default)
Output plugins
Central server-master with a hot-standby in case of failure
Codecs plugins
Grok debugger
ElasticSearch,
MongoDB,
AWS S3
and much more
Kibana,
graylog2
Riemann NO
fluentd
Input plugins
Install from source or via gem
Load-balance between multiple hosts or have a master with a hot-standby in case of failure
Plugins
Doesn’t provide any storage tier itself but allows you to easily configure where your logs should be collected
splunk
Splunk Universal Forwarder
SSL security
Splunk
YES
Graylog2
ElasticSearch
loggly
Hosted

9. What is ELK, and why do we need it?
ELK is a stack of programs that help dealing with logs.
Includes:
Aggregation of logs
Search capabilities
Aggregation of statistics
Visualizations

10. Logstash
Filters
Outputs
Unstructured
Documents

11. Inputs
Logs:
Lumberjack - resilient, compressed, secure (logstash-forwarder)
Remote syslog
Files
Devices:
Event log, Collectd
Netflow, WMI
Event Queue:
Redis, RabbitMQ
Kafka, ZeroMQ
Streaming APIs:
Twitter
(IMAP)
Amazon S3, ganglia, sqs, varnishlog, etc
Full list

12. Filters Why Do I like Logstash?
grep
date
json
grok
Full list
Why Do I like Logstash?
It uses Grok filter for parsing standard and non standard logs:
Log Line:
27/10/14 07:39:28 [localhost-startStop-1] [] INFO com.vidmind.config.LoggingPropertyPlaceholderConfigurer - streams.limit.general = 0
Pattern:
%{DATESTAMP} %{SYSLOG5424SD} ?? %{WORD:ErrorLevel} %{JAVACLASS}

13. Outputs Storage: ElasticSearch MongoDB Event Queue: S3 Redis, RabbitMQ
Graphite
File
...
Notification:
Zabbix
Nagios
Riemann
PagerDuty
Event Queue:
Redis, RabbitMQ
Kafka, ZeroMQ
tcp/udp
SaaS:
AWS CloudWatch
Hipchat
Jira
Full list

14. Logstash - Forwarder (Shipper) Configuration file{
"network": {
"servers": [[logstash_indexers]]
"timeout": 15,
"ssl ca": "logstash-forwarder.crt" },
"files": [
"paths": [
"/usr/share/tomcat7/logs/*.json.log" ],
"fields": { "type": "tomcat", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]" }
"/usr/share/tomcat7/logs/*.activities.log"
"fields": { "type": "activities", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]" } } ]

15. Logstash-Indexer Configuration file
input {
lumberjack {
codec => json{}
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" }
output {
elasticsearch {
host => " "
protocol => "http"
cluster => "[[elasticsearch_cluster_name]]"
manage_template => false
index => "logstash-%{system}-%{type}-%{+YYYY.MM.dd}" }

16. ElasticSearch Configuration file (yaml-based configuration)
cluster.name: [[elasticsearch_cluster_name]]
node.name: "[[node_name]]"
node.master: false / true
node.data: false / true
index.number_of_replicas: 1
#Security
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [[elasticsearch_servers]]
action.disable_close_all_indices: true
action.disable_delete_all_indices: true
action.disable_shutdown: true
script.disable_dynamic: true

17. The ELK stack General Architecture
Logstash
ElasticSearch
Kibana

18. The ELK stack Our Scaled Architecture
Logstash-Indexer
ElasticSearch
Kibana
Logstash-Forwarder
Logstash-Indexer
ElasticSearch
Kibana
Logstash-Indexer
ElasticSearch
Kibana

19. DEMO