Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 (Ab-) using Round Robin Databases with Perl Mike Schilli, Yahoo! 06/18/2008.

Similar presentations


Presentation on theme: "1 (Ab-) using Round Robin Databases with Perl Mike Schilli, Yahoo! 06/18/2008."— Presentation transcript:

1 1 (Ab-) using Round Robin Databases with Perl Mike Schilli, Yahoo! 06/18/2008

2 2 Overview What are Round Robin Databases? RRD and Perl – which module to use? Practical Examples

3 3 Round Robin Databases Archives of fixed size for unlimited data Overwrite old spots if full

4 4 Round Robin Archives start

5 5 Round Robin Archives Overwrite if full

6 6 Round Robin Archives RRDs with multiple archives with different granularity

7 7 Data Flow with rrdtool Data Source Archive

8 8 Data Source Conversions

9 9 Steps and Heartbeat

10 10 Steps and Heartbeat

11 11 Steps and Heartbeat

12 12 Generous Heartbeat

13 13 Generous Heartbeat

14 14 Steps and Heartbeat Long story short: Heartbeat > Step if you’re roughly feeding in “step” intervals. Example: Step=300 (5 min) Heartbeat=600

15 15 Feeding PDPs to the Archive

16 16 Feeding PDPs to the Archive Consolidation Functions (MAX, AVERAGE, etc.) xfiles factor (allowed percentage of missing values)

17 17

18 18 Simple Example Web Orders: 0: : : : : :00150

19 19 Simple Example

20 20 Simple Example rrdtool 'create' 'myrrdfile.rrd' '--start' ' ' '--step' '3600' 'DS:orders:GAUGE:7200:U:U' 'RRA:MAX:0.5:1:100' rrdtool 'update' 'myrrdfile.rrd' ' :100' rrdtool 'update' 'myrrdfile.rrd' ' :110' rrdtool 'update' 'myrrdfile.rrd' ' :120' rrdtool 'update' 'myrrdfile.rrd' ' :130' rrdtool 'update' 'myrrdfile.rrd' ' :140' rrdtool 'update' 'myrrdfile.rrd' ' :150' rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per Hour' '--end' ' ' '--start' ' ' 'DEF:draw1=myrrdfile.rrd:orders:MAX' 'AREA:draw1#0000FF:Web Orders

21 21 RRDTool Arcane syntax Perl API RRDs.pm Pretty close to RRDTool syntax Calls rrdtool under the hood

22 22 To the Rescue: RRDTool::OO CPAN Module RRDTool::OO Takes the drudgery out of the RRD syntax Does what you want by default Translates between Perl and RRD (watch it with debug on) Named parameters instead of fixed order

23 23 Simple Example #!/usr/local/bin/perl -w use strict; use RRDTool::OO; use DateTime; my $rrd = RRDTool::OO->new( file => "myrrdfile.rrd" );

24 24 Simple Example my $start_dt = DateTime->today( time_zone => "local" ); my $dt = $start_dt->clone();

25 25 Simple Example $rrd->create( step => 3600, start => $start_dt->epoch(), data_source => { name => "orders", type => "GAUGE", }, archive => { rows => 100 } );

26 26 Simple Example for my $hour (0..5) { $dt->add(hours => 1); $rrd->update( time => $dt->epoch(), value => $hour*10, ); }

27 27 Simple Example $rrd->graph( image => "mygraph.png", vertical_label => 'Orders per Hour', start => $start_dt->epoch(), end => $dt->epoch(), draw => { type => "area", color => '0000FF', legend => "Web Orders", } );

28 28 Simple Example

29 29 RRDTool Commands rrdtool 'create' 'myrrdfile.rrd' '--start' ' ' '--step' '3600' 'DS:orders:GAUGE:7200:U:U' 'RRA:MAX:0.5:1:100' rrdtool 'update' 'myrrdfile.rrd' ' :100' rrdtool 'update' 'myrrdfile.rrd' ' :110' rrdtool 'update' 'myrrdfile.rrd' ' :120' rrdtool 'update' 'myrrdfile.rrd' ' :130' rrdtool 'update' 'myrrdfile.rrd' ' :140' rrdtool 'update' 'myrrdfile.rrd' ' :150' rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per Hour' '--end' ' ' '--start' ' ' 'DEF:draw1=myrrdfile.rrd:orders:MAX' 'AREA:draw1#0000FF:Web Orders

30 30 Where RRDtool shines Graph time-based data Don’t worry about scaling or axis text Stacked graphs

31 31 Where RRDtool sucks Can’t add data for past events Can’t add data twice at the same timestamp

32 32 Pitfalls Make sure that the first update happens after RRD “start” time Updates need to happen at increasing timestamps On large step sizes (1 year), limit the number of rows to fit within < 2030.

33 33 Debugging with Log4perl use Log::Log4perl qw(:easy) Log::Log4perl-> easy_init( $DEBUG );

34 34 Log4perl Output rrdtool 'create' 'myrrdfile.rrd' '--start' ' ' '--step' '3600' 'DS:orders:GAUGE:7200:U:U' 'RRA:MAX:0.5:1:100' rrdtool 'update' 'myrrdfile.rrd' ' :100' rrdtool 'update' 'myrrdfile.rrd' ' :110' rrdtool 'update' 'myrrdfile.rrd' ' :120' rrdtool 'update' 'myrrdfile.rrd' ' :130' rrdtool 'update' 'myrrdfile.rrd' ' :140' rrdtool 'update' 'myrrdfile.rrd' ' :150' rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per Hour' '--end' ' ' '--start' ' ' 'DEF:draw1=myrrdfile.rrd:orders:MAX' 'AREA:draw1#0000FF:Web Orders

35 35 Example: Stock Portfolio

36 36 Example: Measure Power Usage Mastech MAS-345 Serial Interface (RS-232)

37 37 Example: Measure Power Usage

38 38 Example: Measure Power Usage

39 39 Aberrant Behaviour

40 40 The End Thanks!

41 41 References RRDTool::OO on CPAN RRD::Simple on CPAN Portfolio Rendering article: RRDTool article: magazine.com/issue/44/Perl_RDDtool.pdfhttp://www.linux- magazine.com/issue/44/Perl_RDDtool.pdf RRDTool article (in German): magazin.de/Artikel/ausgabe/2004/06/perl/perl.html

42 42 Bonus Slides

43 43 Aberrant Behaviour Holt Winter Algorithm Data Smoothing/Prediction

44 44 Aberrant Behaviour Holt Winter Algorithm RRAs: HWPREDICT SEASONAL DEVSEASONAL DEVPREDICT FAILURES RRA:HWPREDICT:rows:alpha:beta:sea sonal_period

45 45 Aberrant Behaviour alpha (baseline): : recent observations carry greater weight 0: past observations carry greater weight beta (slope): 0..1 gamma (seasonal): 0..1 failures defaults to 7

46 46 Aberrant Behaviour Example: alpha = 0.50 beta = 0.50 gamma = 0.01 seasonal_period = 2 threshold = 7 window_length = 9


Download ppt "1 (Ab-) using Round Robin Databases with Perl Mike Schilli, Yahoo! 06/18/2008."

Similar presentations


Ads by Google