1. Storage Centralized Logging (Log Aggregator)by
RCI Storage Team

2. Agenda RCI Storage PoC target A bit of history Why Centralized Logging
Basic Workflow
What options do we have?
Why Graylog2?
Graylog2 vs. Others
Architecture
Installation
Where to?

3. RCI Storage PoC Target Log aggregator for RCI storage
FLOSS (with support)
KISS philosophy:
Easy to use
Easy to manage
Reliability, Availability & Scalability
Low (HW & SW) requirements

4. A bit of history
1980s:
Syslog was developed by Eric Allman as part of the Sendmail project
Syslog originally functioned as a de facto standard
The Internet Engineering Task Force documented the status quo in RFC 3164
It was standardized by RFC 5424
Various companies have attempted to claim patents for syslog implementations

5. A bit of history
1998:
Balázs Scheidler ported the existing nsyslogd code to Linux … later became syslog-ng
2004:
Rainer Gerhards decided to write a new strong syslog daemon to compete with syslog-ng
2010:
systemd is born
logging is based on systemd-journald:
a daemon responsible for event logging
append-only binary files serving as its logfiles
sysadmin may choose whether to log system events with systemd- journald, syslog-ng or rsyslog

6. Why Centralized Logging
Easy & fast access to all logs
History
Reduce risk of log loss
Correlation
System tuning
Discover HW and/or SW problems
Improve design
Security (and other) audits

7. Basic Workflow Collect Applications create logs in different ways
Multiple log files
Transport
Scribe, nsq, kafka: more app oriented
Flume, Heka, Logstash, Chukwa, fluentd: more log oriented
Store
For how long?
How much volume?
Access?
Analyze
Batch, real Time?
Frontend, CLI, …?
Alerts?

8. What options do we have? logentries.com www.logtrust.com rigabyte.com
sematext.com/logsene
moonlit-software.com
logentries.com
stackify.com
github.com/facebookarchive/scribe
github.com/mozilla-services/heka
flume.apache.org

9. Graylog vs others SPLUNK: ELK: Graylog: $$$$$ Closed source
Huge amount of apps
ELK:
Complex
Great for huge systems
Graylog:
Doesn't correlate
Weak analysis
Lightweight

10. Why Graylog2?
Supports *syslog, Windows logs, CISCO, Apache, APC MGE, Audit Daemon, Bind9, PHP, Ruby, Juniper, MongoDB, MySQL, Nginx, Firewalls, PAM, crond, Postfix, Puppet, Snort, Squid, SSH, sudo, systemd, Heroku (app logging)
GELF logging format
Overcomes syslog limitations:
Limited to length of 1024 bytes
No data types in structured syslog: you don’t know what is a number and what is a string
Strict RFCs but many syslog dialects -> can't parse all of them
No compression
Libaries and appenders for many programming languages and logging frameworks so it is easy to implement:
GELF is a great choice for logging from within applications

11. Why Graylog2? Logs are stored in MongoDB & elasticsearch
Search engine (possibility of saving searches)
Streams: let you "aggregate" in real time messages
Establish alerts
Dashboards to build pre-defined views of data
Marketplace:
Plugins
Content packs
GELF library

12. Architecture

13. Installation Install & configure mongoDB & Elasticsearch
Install & configure Graylog
Get your hands dirty:
Create a new input
Add servers
vi /etc/rsyslog.d/90-graylog2.conf
Add these 2 lines:
$template GRAYLOGRFC5424,"<%PRI%>%PROTOCOL-VERSION% %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
Restart rsyslogd

14. Searching Simple search: OR search: AND search: NOT search
ssh
OR search:
ssh telnet
AND search:
ssh AND telnet
NOT search
ssh NOT telnet
EXACT search:
"ssh telnet"
Source search:
source:rcdt01
source:rcdt0?
source:rcdt*
source|message|full_message
Fuzzy searches:
source:rcdt0~
Range queries:
[] -> inclusive
{} -> exclusive
http_response_code:[500 TO 504]
http_response_code:{400 TO 404}
bytes:{0 TO 64]
http_response_code:[0 TO 64}
http_response_code:>400
http_response_code:<400
http_response_code:>=400
http_response_code:<=400
http_response_code:(>=400 AND <500)

15. Where to? Scalability: Hadoop? "Just add nodes" "Just add Hadoop" ;)
Hive?
MapReduce?
Impala?
Hbase?
Import?
/Export?
elasticsearch
mongoDB
Hadoop
Graylog2

16. MHO