Presentation on theme: "MIRnet Administrative Data Analysis System (MADAS) Greg Cole, Natasha Bulashova Friends & Partners NCSA."— Presentation transcript:
MIRnet Administrative Data Analysis System (MADAS) Greg Cole, Natasha Bulashova Friends & Partners NCSA
Description System converts netflow data into structured data stored in a series of relational database tables System provides means of browsing summary statistics in graphic and table format A work in progress since 1998; first version in summer of 1999, second in fall of 2000 (for HPIIS review), third in February FOR MORE INFO...
Primary Domains table *************************** 1. row *************************** domainid: 715 domainname: anl.gov latitude: longitude: domainlabel: Argonne Natl Lab createtime: modifytime: origin: US shortlabel: Argonne Natl Lab location: pdomainid: 715 rdomainid: 715 loccity: Chicago locstate: IL loccountry: United States orgclass: US Government,US Govt DOE worldclass: North America regionclass: USA Great Lakes *************************** 2. row *************************** domainid: 948 domainname: doe.gov latitude: longitude: domainlabel: US Department of Energy createtime: modifytime: origin: US shortlabel: US-DOE location: Washington, DC pdomainid: 948 rdomainid: 948 loccity: Washington locstate: DC loccountry: United States orgclass: US Government,US Govt DOE worldclass: North America regionclass: USA Atlantic Central Heart and soul of MADAS system Adding new intelligence to this database enables entirely new classes of analysis Currently maintains 11,771 domain records (January 10, 2001)
Other Primary Tables IP Today (last 24 hours of ipheaders records) Country Codes Parent domains Color mappings | code | country | worldclass | | ?? | Unknown | Unclassified | | AC | Ascension Island | Other | | AD | Andorra | Europe | | AE | United Arab Emirates | Middle East | | AF | Afghanistan(Islamic St.) | Middle East | | AG | Antigua and Barbuda | North America | | AI | Anguilla | Other | | AL | Albania | Europe | | AM | Armenia | Middle East | | AN | Netherland Antilles | Other | | parentid | parentname | | 1308 | ac.jp | | 3 | ac.ru | | 959 | ac.uk | | 986 | edu.tw | | 6 | free.net | | 735 | nasa.gov | | 41 | nlanr.net | | 4762 | ircache.net | | 100 | ras.ru | | code | value | | ?? | pink | | CA | lblue | | CH | purple | | DE | lbrown | | DK | green | | EE | dgray | | FI | white | | FR | cyan | | IL | gold | | IT | lred | | JP | dpink | | NL | lpurple | | NO | gray | | Other | lyellow | | PL | orange | | RU | blue | | SE | lgray | | TW | yellow | | UK | marine | | US | lgreen |
Capabilities With these tables (updated every 10 minutes), we can provide all sorts of live (and historical) traffic analysis between world regions, countries, country regions, cities, institutions, organizations, network protocols by year, month, day, hour, minute,.. But..
Need to use Indexed Summary Tables Database mirsum 8 tables updated live every 10 minutes 2 Heap (RAM-based) tables used for most live queries Pre-query optimizer selects best tables for current query Domain_date_proto Domain_date_proto_mm Domain_date Domain_date_mm Country_date_proto Country_date_proto_mm Country_date Country_date_mm Heap_domain_date_proto Heap_domain_date_proto_mm
A word about technologies No proprietary software Mysql for database PHP for query interface Web/CGI for stats interface Perl for code/CGI base –DBI for interaction with Mysql –GD::Graph graphics libraries
Perl Code (object-oriented) Analysis that in original MADAS system took lines of perl code, now looks like: #### 2 ########## # chart showing total volume with breakdown by top countries my $self = MADAS::Country->new( database => "mirsum", table => "domain_date", variable => "origin_dest", imagemapcgi => "/cgi-bin/madas/printtable.pl", imagemap => 0, percent => 1, graphtype => "bars", title1 => "Total MIRnet Traffic Flow by Destination Country", rh_input => \%in); $self->set_title2("Period: ". $self->get_timebegin. " - ". $self->get_timeend. " "); $self->doit();
World Regions (by country)
Countries (by domain)
US Regions Russian Regions
US Government DOENASA DOD
Advantages Higher-level analysis of network usage (not just for engineers) System encourages exploration Better understanding of users and their applications Immediate feedback on traffic problems/issues
Future Plans Evaluate shared use of Domains and DNSdata tables (perhaps via LDAP) Standard monthly and quarterly reports of traffic utilization Monster query Project level accounting/analysis more...
Future Plans (continued) Create always-running server to maintain data, provide instant stats, manage web site/interface Provide statistical analysis routines Create database to maintain all global settings Port-level analysis (looking for napster, etc.) more...
Future Plans (continued) Explore integration/sharing with HPIIS projects (others?) Develop data maintenance applications for Domains database Develop world-map graphics applications more...
Future Plans (continued) Develop partnerships analyses (looking at domain-domain and machine-machine partnerships) Add additional organizational classes (i.e., US Govt DOE, University) Add state-level analyses Clean-up/refine Domains database more...
Future Plans (continued) Add science classifiers and project identifiers to regular traffic flows Integrate this with database describing high performance network science applications Integrate back-end reporting with front-end reservation system
Future plans (continued) Authentication system for machine- level inquiry/analysis Device independent display of usage (for text-only, , WAP devices) Handle IP address cache expiration problem Etc....