Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan

Hier uw naam Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan

Agenda About VanMeijel Why we need insight in Metacom (our ERP-system) Requirements for monitoring tool Developing the tool Final solution Demo ! Please feel free to ask questions at any time

About VanMeijel Founded in 1987 Based in The Netherlands, Emmeloord Satellite office in Belgium, Sint-Niklaas Around 110 employees Software is targeted at construction companies 300+ customers, users

Metacom – Our ERP system Contains modules for - Estimation - Reporting - Planning - Invoicing - Cost Control - Procurement - Relation Management - Resource Planning - and many more

Metacom – Our ERP system Build in OpenEdge, currently version Millions of code lines - Around 20 – 25 years of age - Lots of different techniques (DB-access on client, client-server, procedural, OO) - Deployed via Round Table, recently moved to Subversion / Env / Jenkins

About VanMeijel Why we need insight in Metacom Requirements for monitoring tool Developing the tool Final solution Demo Please feel free to ask questions at any time

Application usage is changing - People are working mobile - People want / expect Apps - Customers don’t want big server parks: web interface Changing world Integrations Maintenance Different license models Move to webinterface Why we need insight in Metacom

Customers are integrating 3th party applications One webservice often replaces many “real” users Integrations weaken our license model Changing world Integrations Maintenance Different license models Move to webinterface Why we need insight in Metacom

What methods are called How often are they called How many business objects are being created / edited per method How many systems are connecting to Metacom Changing world Integrations Maintenance Different license models Move to webinterface Why we need insight in Metacom

System is complex Around 3000 – 4000 windows/dialogs Contains lots and lots of settings What is the performance of our webservice methods Changing world Integrations Maintenance Different license models Move to webinterface Why we need insight in Metacom

In the future we need to move to “pay for usage” How many business documents are created - For example how many invoices Are business documents updated frequently or not? How many reports are being generated Changing world Maintenance Integrations Different license models Move to webinterface Why we need insight in Metacom

How many users are active? On what time of the day / week? How about evenings, weekends? Changing world Maintenance Integrations Different license models Move to webinterface Why we need insight in Metacom

What functions / modules are used What is not used anymore Top 100 of most used functions Changing world Maintenance Integrations Different license models Move to webinterface Why we need insight in Metacom

About VanMeijel Why we need insight in Metacom Requirements for monitoring tool Developing the tool Final solution Demo Please feel free to ask questions at any time

Requirements monitoring tool Web based Real time Easy to analyze Flexible, schemaless database Light-weight; JSON

About VanMeijel Why we need insight in Metacom Requirements for monitoring tool Developing the tool Final solution Demo Please feel free to ask questions at any time

Events are send to our AppServer API (written in OpenEdge) - Client - Triggers - Business Logic - Webservices AppServer API sends events to webserver Webserver stores events and hosts a viewer Architecture First try, january 2014 Different license models Developing the tool

NodeJS as webserver MongoDB as database Visualization was a problem Architecture First try, january 2014 Second try, march 2014 Developing the tool

We replaced MongoDB with ElasticSearch Visualization is super easy with Kibana Migration from MongoDB to ElasticSearch was difficult Architecture First try, january 2014 Second try, march 2014 Developing the tool

About VanMeijel Why we need insight in Metacom Requirements for monitoring tool Developing the tool Final solution Demo Please feel free to ask questions at any time

Clients push events to AppServer AppServer start HTTP-session to REST-service in cloud Fires JSON event, simplified example: { “customerid”:”vma”, “datetime”:” T14:36:54”, “appid”: “mtc”, “appversion”: “ ” “type”:”ui”, “function”: “wcustomers” } OpenEdge events NodeJS Webserver ElasticSearch database Kibana for visualisation Final solution

OpenEdge events NodeJS Webserver ElasticSearch database Kibana for visualisation Final solution var restify = require('restify'); var elasticsearch = require('elasticsearch'); var server = restify.createServer( { name : "stats" }); server.listen(‘1234’,’ ’, function() { console.log('%s listening at %s ', server.name, server.url);}); server.use(restify.queryParser()); server.use(restify.bodyParser()); server.use(restify.CORS()); var client = new elasticsearch.Client({ host: 'localhost:9200' }); server.post({path : '/stats', version: '0.0.1'}, postNewMtcStat); function postNewMtcStat(req, res, next){ var body = req.body; res.setHeader('Access-Control-Allow-Origin','*'); client.create({index: 'usage', type: 'usage', body: req.body }, function(err, success){ if(success){ res.send(201, 'saved'); return next(); }else{ return next(err); } }); }

OpenSource, Apache License Based on Lucene Distributed database, very scalable Near real-time search Document oriented, schema changes not a problem! OpenEdge events NodeJS Webserver ElasticSearch database Kibana for visualisation Final solution

OpenEdge events NodeJS Webserver ElasticSearch database Kibana for visualisation Final solution Cool Flexible Fast (but not on this laptop ) Very easy to use

OpenEdge events NodeJS Webserver ElasticSearch database Kibana for visualisation Final solution

DEMO

QUESTIONS

Thank you for your