Elasticsearch – An Open Source Log Analysis Tool Rob Appleyard and James Adams, STFC Application-Level Logging for a Large Tier 1 Storage System

Introduction First, a little about what we do –RAL = UK’s LHC Tier 1 site –CASTOR for LHC storage CERN Advanced Storage manager Disk & Tape My responsibility Domain-specific solution developed at CERN for WLCG

CASTOR Logs CASTOR is a complex system… …and produces a lot of logging information –From the application daemons, not the system daemons.

CASTOR Logs

~2GB/day from the node I showed (highest volume) ~30GB/day collected overall ~200 source nodes ~70,000,000 log events/day

Where does it all come from? CASTOR logs each interaction between the various components –…in great detail. The window to the right shows 10 lines of logging from one daemon Time period is ~0.07s

One Log Message… T11:02: :00 lcgcstg01 stagerd[22773]: LVL=Info TID=22822 MSG="Request moved to Wait" REQID=45bea7cd-acb1-4d1f-a66f-45aa41663c3a NSHOSTNAME=cexperimentlsf.ads.rl.ac.uk NSFILEID= SUBREQID=0fd d07-ff31-e053-05b6f6821b16 Type="StagePutDoneRequest“ Filename="/castor/ads.rl.ac.uk/prod/experiment/prodInput/proddata/data/datast ore/ff/ff/datafile.data" Username=”experiment001" Groupname=”experiment" SvcClass=”experimentInput"

One Log Message… T11:02: :00 lcgcstg01 stagerd[22773]: LVL=Info TID=22822 MSG="Request moved to Wait" REQID=45bea7cd-acb1-4d1f-a66f-45aa41663c3a NSHOSTNAME=cexperimentlsf.ads.rl.ac.uk NSFILEID= SUBREQID=0fd d07-ff31-e053-05b6f6821b16 Type="StagePutDoneRequest“ Filename="/castor/ads.rl.ac.uk/prod/experiment/prodInput/proddata/data/datast ore/ff/ff/datafile.data" Username=”experiment001" Groupname=”experiment" SvcClass=”experimentProdInput"

What’s wrong with the old way? Most CASTOR logs are like this The files are big, but they’re easy to parse… –So why not just use normal UNIX commands? grep, awk, sed, etc… –With modern hardware, a 5 million-line logfile can be grepped in reasonable (<1 minute) time periods

What’s wrong with the old way? Our system is distributed! –Multiple management nodes –Multiple storage nodes Grepping on one node? OK (more-or-less) Grepping for the same string across a 200+ node system? No.

The First Solution - DLF DLF = ‘Distributed Logging Facility’ CERN-developed monitoring system for CASTOR Store all the log information in a big Oracle DB Source: CASTOR end-to-end monitoring, by T Rekatsinas et al, URL: _219_4_ pdf

Searching DLF DLF offers a CASTOR- customised search function –Which is pretty neat! –The problem is… –…that searches take… –…a very… –…very… –…long… –…time.

Running DLF Scalability was a killer. –By 2013, simple queries were taking >1 hour. –Fundamental architecture couldn’t cope.

The Hunt for Better CERN’s solution used the Hadoop stack and an Apollo message broker… …and a lot of bespoke Python

The Hunt for Better This didn’t work for our use case –We tried adapting it… –But we just ended up spending ages hacking the Python.

Plan B Our problems are not unique. –There are some really nifty off-the-shelf solutions to these issues… –Let’s see if they scale! Spoiler: They do.

The ELK Stack ELK stack= –Elasticsearch –Logstash –Kibana 3 separate pieces of software –But they are designed to fit together URL for (recently renamed) developer:

Logstash Sequence: –Data arrives in format A… –…process B occurs… –…data out in format C. In our case this is: –CASTOR nodes send log messages in –JSON-ise –Send to Elasticsearch Screengrab from Logstash documentation

Elasticsearch Distributed RESTful search and analytics engine Built on Apache Lucene –Apache 2 license Behind the scenes: shard-based storage –Admin defines no. of primary/replica shards –Users don’t need to think about this

Kibana Web-based data visualisation system –Lucene query syntax –Lots of pretty graphs –Heavily integrated with Elasticsearch ES indices are per-day

Cool Kibana Plots (1)

Cool Kibana Plots (2)

Sysadmin Use Cases Common questions: –“What happened to this user’s request?” –“When did we first see an ORA-5555 error message?” –“Tell me everything you have from the past 5 days about the file with ID= ” We get the answers fast and in a useful format.

Sysadmin Use Cases

Challenges Encountered (1) CASTOR’s logging conventions are sometimes messy… 168 lines of code in Logstash to sort out changing field names, case variations, typos, etc… –3 different field names used for a file’s ID –‘CASTOR_Pivilege’ Lucene query syntax needs to be learned –Handling of quotation marks is odd –Simple sample query: castor_MSG: “Marking transfer as scheduled”

Challenges Encountered (2) Load on hardware is non-trivial –Currently running on 10 (obsolete) batch nodes… Raw messages not stored Tuning proved difficult –None of the published HOWTOs deal with working at this scale –We are very happy to discuss our experiences and offer advice

Other uses Application log search is just our use case Others: –Syslog search –Logging from Condor batch farm –Open big data analysis service for other uses

Conclusion Elasticsearch fits our requirements very well. –Powerful –Cheap to run –Quick querying, even at high scale If you need to manage logs from distributed sources, you should try this!

Any Questions?