1. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (1) 7. Monitoring & Analysis Tools

2. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (2) 7. Monitoring & Analysis Tools  Tremendous variety of tools are available for monitoring many aspects of networks  Simple commands usually included in operating systems  Free (open-source) applications  Commercial packages and systems  Category  Passive monitoring tools  Traffic Flow Analysis –NetFlow (C), cflowd (F), FlowScan (F), Sniffer Pro (C), argus (F), i-Flow (C)  Network Utilization –MRTG (F), RMON (C)  Visualization –RRD (F)  Active monitoring tools  Network Performance –ping (S), traceroute (S), Network Vantage (C), NetPerf (F), etc.

3. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (3) 7. Monitoring & Analysis Tools - NetFlow Network Data Analyzer: Data Presentation Flow Control and Configuration Partner Applications NetFlow Data Export: Data Switching Data Aggregation Data Export NetFlow FlowCollector: Data Collection Data Filtering Data Aggregation Data Storage RMON Probe Accounting/Billing Network Planning RMON Application Image From NetFlow PPT by Michael Lin, Cisco Systems  Cisco IOS NetFlow Infrastructure

4. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (4) 7. Monitoring & Analysis Tools - NetFlow  NetFlow FlowCollector  provides fast and scalable data collection from multiple NetFlow Export-enabled devices  performs data volume reduction through selective filtering and aggregation  stores flow information in flat files on disk for post-processing by consumer applications. NetFlow Consumer Applications NetFlow Consumer Applications NetFlow FlowCollector NetFlow Enabled Devices Image From NetFlow PPT by Michael Lin, Cisco Systems

5. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (5) 7. Monitoring & Analysis Tools - NetFlow  Network Data Analyzer  Receives flow data from NetFlow FlowCollector(s)  Performs time-based analysis and data sorting  Configures FlowExports and FlowCollectors  Produces histograms, bar charts, and pie charts NetFlow FlowCollectors NetFlow FlowCollectors NetFlow FlowAnalyzer NetFlow FlowAnalyzer Image From NetFlow PPT by Michael Lin, Cisco Systems

6. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (6) 7. Monitoring & Analysis Tools - cflowd  Freely available NetFlow analysis tool from CAIDA  Functionality  Input  NetFlow export data from Cisco routers  Collect  collect: Information of flow obtained from NetFlow  store: uses arts++ file format (binary file format specification for storing network data)  Analyze  predetermined statistics text format, using ARTS utility (e.g., xartsprotos)  query and visualize using java front-end

7. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (7) 7. Monitoring & Analysis Tools – cflowd Source: http://www.caida..org

8. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (8) 7. Monitoring & Analysis Tools – FlowScan  Traffic Reporting & Visualization Tool  developed by Dave Plonka(U. Wisconsin)  analyzes and reports on flow data exported by routers  produces graph images which provide a continuous, near real-time view of the network traffic across a network's border  freely available  FlowScan binds together (1) a flow collection engine (a patched version of cflowd) (2) a high performance database (Round Robin Database - RRD) (3) a visualization tool (RRDtool)

9. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (9) 7. Monitoring & Analysis Tools – FlowScan loads and executes report modules of the administrator’s choosing Source: “FlowScan”, Dave Plonka

10. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (10) 7. Monitoring & Analysis Tools – arts++  ART is a binary file format specification for storing network data  ART was initially developed at ANS (American Nuclear Society) by David Bolen (1992)  ARTS was licensed to CAIDA (1998)  ARTS data objects are generally composed of three parts: a header, a list of attributes and a data section  CAIDA has developed a C++ class library for ARTS called arts++  arts++ Functionality  efficient data archival  aggregation in the time domain (AS, net, port, protocol, interface..)  version-specific formats  support for iostreams and UNIX file descriptors

11. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (11) 7. Monitoring & Analysis Tools – ARGUS  Audit Record Generation and Utilization System  A powerful flow-based, passive monitoring tool for IP networks  Provides tools for various analysis of network activity  Probe system: argus  Collector/Analysis tools: ra, racount, ragator, ramon, rasort, raxml  Developed originally by CMU in 1993, now coordinated by QoSient LLC as open source project  Current Release Version : 2.0.5  Current Developing Version: 2.0.6  http://www.qosient.com/argus  Fixed model Real-Time Flow Monitor after IETF RTFM

12. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (12) Argus Architecture Source: QoSient LLC

13. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (13) Argus Data Model  Argus flow modeled after IPPM Framework  Type-P and Type-P1-P2 flows  Bidirectional flow model <- RTFM  Packets of Type-P  Defined in RFC 2330 from IETF IPPM WG  To remove the ambiguity in the definition of Network Performance Metrics  The generic notion where in some contexts P will be explicitly defined(Type-P), partially defined(Type-P1- P2), or left generic  Example:  IP-connectivity  IP-Type-P-Connectivity, IP-Port-HTTP-Connectivity

14. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (14) Argus Flows  An Argus Flow is simply a set of datagrams that share a common set of datagram attributes.  Destination Address  Network Addresses  Addresses, Protocol, NSAPs, TTL, Session IDs, Application data, etc.  Supports 13 simultaneous flow models, enabling Layer 2, 3, 4, and 5 based flow tracking and reporting

15. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (15) Argus Flow Models  Layer 5  RTP and RTCP (Type-P)  8-tuple: SrcIPAddr,DstIPAddr,L4Protocol,SrcPort,DstPort, rh_ver, rh_seq, rh_ssrc  Layer 4  TCP and UDP (Type-P)  5-tuple: SrcIPAddr, DstIPAddr, L4Protocol, SrcPort, DstPort  ESP (Type-P)  4-tuple: SrcIPAddr, DstIPAddr, L4Protocol, SPI  ICMP ECHO (Type-P1-P2)  7-tuple: SrcIPAddr, DstIPAddr, L4P, type, code,id, seq where the type is either ECHO REQUEST or REPLY.  ICMP INFO TYPE (Type-P1-P2)  5-tuple: SrcIPAddr, DstIPAddr, L4P, type, code where the type is either REQUEST or REPLY.  ICMP UNREACHABLE/REDIRECT (Type-P1-P2)  Mapped to any supported Argus flow type.  6-tuple: SrcIPAddr, DstIPAddr, L4P, type, code, object  IGMP (Type-P)  4-tuple: SrcIPAddr, DstIPAddr, L4P, type

16. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (16) Argus Flow Models  Layer 3  IPv4 (Type-P)  3-tuple: SrcIPAddr, DstIPAddr, L4Protocol  Fragments (Type-P1-P2)  Mapped to any supported Argus flow type.  Fragments (Type-P)  4-tuple: SrcIPAddr, DstIPAddr, L4Protocol, ip_id  Layer 2  LLC SNAP Encapsulation (Type-P)  5-tuple: SrcMACAddr, DstMACAddr, L3Proto, SrcSAP, DstSAP  ARP (Type-P1-P2)  3-tuple: ARP_SPA, ARP_TPA, Eaddr  where the EAddr value is either the SrcMacAddr of the REQUEST or the dstMACAddr of the REPLY.  All other traffic: (Type-P)  3-tuple: SrcMACAddr, DstMACAddr, L3Protocol

17. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (17) Argus Flow Record Format  Common Type Length Value (TLV) Structure  Common 16 byte header struct ArgusRecord { unsigned char type, cause; unsigned short length; unsigned int status; unsigned int argusid; unsigned int seqNumber; union { struct ArgusMarStruct mar; struct ArgusFarStruct far; } ar_union; }; 081624 31 TypeCauselength status Argus ID Sequence Number Management Audit Record(MAR) / Flow Activity Record (FAR) A Start MAR must be the first record in an ArgusRecord Stream A Stop MAR should be the last record

18. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (18) Argus Record Format  Type  Type of Argus Record: MAR or FAR  Length  Length of entire argus record  Status  Connectivity status, transition status  Argus ID  A unique identifier for the source argus  Sequence Number  Management Audit Record (MAR)  Provides information about argus itself  Start MAR --- Status MAR ---- Stop MAR  Flow Activity Record (FAR)  Provides information about network transaction flows that argus track  The FAR are generated either because of state or because of time  Start FAR: transaction started, Stop FAR: transaction stopped  Status FAR –Default time out : every 60 seconds

19. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (19) Argus Flow Record -MAR struct ArgusRecord { unsigned char type, cause; unsigned short length; unsigned int status; unsigned int argusid; unsigned int seqNumber; union { struct ArgusMarStruct mar; struct ArgusFarStruct far; } ar_union; }; struct ArgusRecord { unsigned char type, cause; unsigned short length; unsigned int status; unsigned int argusid; unsigned int seqNumber; union { struct ArgusMarStruct mar; struct ArgusFarStruct far; } ar_union; }; struct ArgusMarStruct { struct timeval startime, now; unsigned char major_version, minor_version; unsigned char interfaceType, interfaceStatus; unsigned short reportInterval, argusMrInterval; unsigned int argusid, localnet, netmask, nextMrSequenceNum; unsigned long long pktsRcvd, bytesRcvd; unsigned int pktsDrop, flows, flowsClosed; unsigned int actIPcons, cloIPcons; unsigned int actICMPcons, cloICMPcons; unsigned int actIGMPcons, cloIGMPcons; unsigned int actFRAGcons, cloFRAGcons; unsigned int actSECcons, cloSECcons; int record_len; };

20. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (20) Argus Flow Record - FAR struct ArgusFarStruct { unsigned char type, length; unsigned short status; unsigned int ArgusTransRefNum; struct ArgusTimeDesc time; struct ArgusFlow flow; struct ArgusAttributes attr; struct ArgusMeter src, dst; }; struct ArgusFarStruct { unsigned char type, length; unsigned short status; unsigned int ArgusTransRefNum; struct ArgusTimeDesc time; struct ArgusFlow flow; struct ArgusAttributes attr; struct ArgusMeter src, dst; }; struct ArgusTimeDesc { struct timeval start; struct timeval last; }; struct ArgusTimeDesc { struct timeval start; struct timeval last; }; struct ArgusFlow { union { struct ArgusIPFlow ip; struct ArgusICMPFlow icmp; struct ArgusMACFlow mac; struct ArgusArpFlow arp; struct ArgusRarpFlow rarp; struct ArgusESPFlow esp; } flow_union; }; struct ArgusFlow { union { struct ArgusIPFlow ip; struct ArgusICMPFlow icmp; struct ArgusMACFlow mac; struct ArgusArpFlow arp; struct ArgusRarpFlow rarp; struct ArgusESPFlow esp; } flow_union; }; struct ArgusIPAttributes { unsigned short soptions, doptions; unsigned char sttl, dttl; unsigned char stos, dtos; }; struct ArgusIPAttributes { unsigned short soptions, doptions; unsigned char sttl, dttl; unsigned char stos, dtos; }; struct ArgusARPAttributes { unsigned char response[8]; }; struct ArgusARPAttributes { unsigned char response[8]; }; struct ArgusAttributes { union { struct ArgusIPAttributes ip; struct ArgusARPAttributes arp; } attr_union; }; struct ArgusAttributes { union { struct ArgusIPAttributes ip; struct ArgusARPAttributes arp; } attr_union; }; struct ArgusMeter { unsigned int count, bytes, appbytes; }; struct ArgusMeter { unsigned int count, bytes, appbytes; };

21. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (21) Argus Flow Record – FAR - Argus Flow struct ArgusIPFlow { unsigned int ip_src, ip_dst; unsigned char ip_p, tp_p; unsigned short sport, dport; unsigned short ip_id; }; struct ArgusIPFlow { unsigned int ip_src, ip_dst; unsigned char ip_p, tp_p; unsigned short sport, dport; unsigned short ip_id; }; struct ArgusICMPFlow { unsigned int ip_src, ip_dst; unsigned char ip_p, tp_p; unsigned char type, code; unsigned short id, ip_id; }; struct ArgusICMPFlow { unsigned int ip_src, ip_dst; unsigned char ip_p, tp_p; unsigned char type, code; unsigned short id, ip_id; }; struct ArgusMACFlow { struct ether_header ehdr; unsigned char dsap, ssap; }; struct ArgusMACFlow { struct ether_header ehdr; unsigned char dsap, ssap; }; struct ArgusArpFlow { unsigned int arp_spa; unsigned int arp_tpa; unsigned char etheraddr[6]; unsigned short pad; }; struct ArgusArpFlow { unsigned int arp_spa; unsigned int arp_tpa; unsigned char etheraddr[6]; unsigned short pad; }; struct ArgusRarpFlow { unsigned int arp_tpa; unsigned char srceaddr[6]; unsigned char tareaddr[6]; }; struct ArgusRarpFlow { unsigned int arp_tpa; unsigned char srceaddr[6]; unsigned char tareaddr[6]; }; struct ArgusESPFlow { unsigned int ip_src, ip_dst; unsigned char ip_p, tp_p; unsigned short pad; unsigned int spi; }; struct ArgusESPFlow { unsigned int ip_src, ip_dst; unsigned char ip_p, tp_p; unsigned short pad; unsigned int spi; }; 7 4 3 6 3 8 ip arp rarp esp icmp mac

22. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (22) Argus Flow - Canonical Record struct ArgusCanonicalRecord { struct ArgusRecordHeader ahdr; struct ArgusFarStruct far; struct ArgusMacStruct mac; union { struct ArgusTCPObject tcp; struct ArgusESPStruct esp; struct ArgusICMPObject icmp; struct ArgusIGMPObject igmp; struct ArgusDHCPObject dhcp; struct ArgusRTPObject rtp; struct ArgusRTCPObject rtcp; struct ArgusARPObject arp; struct ArgusAHObject ah; struct ArgusFRAGObject frag; } acr_union; struct ArgusAGRStruct agr; struct ArgusTimeStruct time; struct ArgusVlanStruct vlan; struct ArgusMplsStruct mpls; }; struct ArgusCanonicalRecord { struct ArgusRecordHeader ahdr; struct ArgusFarStruct far; struct ArgusMacStruct mac; union { struct ArgusTCPObject tcp; struct ArgusESPStruct esp; struct ArgusICMPObject icmp; struct ArgusIGMPObject igmp; struct ArgusDHCPObject dhcp; struct ArgusRTPObject rtp; struct ArgusRTCPObject rtcp; struct ArgusARPObject arp; struct ArgusAHObject ah; struct ArgusFRAGObject frag; } acr_union; struct ArgusAGRStruct agr; struct ArgusTimeStruct time; struct ArgusVlanStruct vlan; struct ArgusMplsStruct mpls; }; struct ArgusMacStruct { unsigned char type, length; unsigned short status; union { struct ArgusETHERObject ether; } phys_union; }; struct ArgusMacStruct { unsigned char type, length; unsigned short status; union { struct ArgusETHERObject ether; } phys_union; }; struct ArgusAGRStruct { unsigned char type, length; u_short status; unsigned int count; struct timeval laststartime, lasttime; struct ArgusTimeObject act, idle; }; struct ArgusAGRStruct { unsigned char type, length; u_short status; unsigned int count; struct timeval laststartime, lasttime; struct ArgusTimeObject act, idle; }; struct ArgusTimeStruct { unsigned char type, length; u_short status; struct ArgusTimeEntity src, dst; }; struct ArgusTimeStruct { unsigned char type, length; u_short status; struct ArgusTimeEntity src, dst; }; struct ArgusMplsStruct { unsigned char type, length; unsigned short status; unsigned int slabel; unsigned int dlabel; }; struct ArgusMplsStruct { unsigned char type, length; unsigned short status; unsigned int slabel; unsigned int dlabel; }; struct ArgusETHERObject { unsigned char ethersrc[6]; unsigned char etherdst[6]; }; struct ArgusETHERObject { unsigned char ethersrc[6]; unsigned char etherdst[6]; }; struct ArgusTimeEntity { struct ArgusTimeObject act, idle; }; struct ArgusTimeEntity { struct ArgusTimeObject act, idle; }; struct ArgusTimeObject { int n; unsigned int min; unsigned int mean; unsigned int stdev; unsigned int max; }; struct ArgusTimeObject { int n; unsigned int min; unsigned int mean; unsigned int stdev; unsigned int max; }; struct ArgusRecordHeader { unsigned char type, cause; unsigned short length; unsigned int status; unsigned int argusid; unsigned int seqNumber; }; struct ArgusRecordHeader { unsigned char type, cause; unsigned short length; unsigned int status; unsigned int argusid; unsigned int seqNumber; }; struct ArgusVlanStruct { unsigned char type, length; unsigned short status; unsigned short sid, did; }; struct ArgusVlanStruct { unsigned char type, length; unsigned short status; unsigned short sid, did; };

23. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (23) Argus Flow Record – acr union struct ArgusTCPObject { unsigned char type, length; unsigned short status; unsigned int state; unsigned int options; unsigned int synAckuSecs, ackDatauSecs; struct ArgusTCPObjectMetrics src, dst; }; struct ArgusTCPObject { unsigned char type, length; unsigned short status; unsigned int state; unsigned int options; unsigned int synAckuSecs, ackDatauSecs; struct ArgusTCPObjectMetrics src, dst; }; struct ArgusTCPObjectMetrics { unsigned int seqbase, ackbytes; unsigned int bytes, rpkts; unsigned short win; unsigned char flags, pad; }; struct ArgusTCPObjectMetrics { unsigned int seqbase, ackbytes; unsigned int bytes, rpkts; unsigned short win; unsigned char flags, pad; }; struct ArgusESPStruct { unsigned char type, length; u_short status; struct ArgusESPObject src, dst; }; struct ArgusESPStruct { unsigned char type, length; u_short status; struct ArgusESPObject src, dst; }; struct ArgusICMPObject { unsigned char type, length; unsigned short status; unsigned char icmp_type, icmp_code; unsigned short iseq; unsigned int osrcaddr, odstaddr; unsigned int isrcaddr, idstaddr; unsigned int igwaddr; }; struct ArgusICMPObject { unsigned char type, length; unsigned short status; unsigned char icmp_type, icmp_code; unsigned short iseq; unsigned int osrcaddr, odstaddr; unsigned int isrcaddr, idstaddr; unsigned int igwaddr; }; struct ArgusESPObject { unsigned int spi, lastseq, lostseq; }; struct ArgusESPObject { unsigned int spi, lastseq, lostseq; }; struct ArgusDHCPObject { unsigned int respaddr; }; struct ArgusDHCPObject { unsigned int respaddr; }; struct ArgusIGMPObject { unsigned char igmp_type, pad; unsigned int igmp_group; }; struct ArgusIGMPObject { unsigned char igmp_type, pad; unsigned int igmp_group; }; struct ArgusARPObject { unsigned char respaddr[6]; unsigned short pad; }; struct ArgusARPObject { unsigned char respaddr[6]; unsigned short pad; }; struct ArgusAHObject { unsigned int src_spi, dst_spi; unsigned int src_replay, dst_replay; }; struct ArgusAHObject { unsigned int src_spi, dst_spi; unsigned int src_replay, dst_replay; }; tcp esp icmp igmp dhcp rtp struct ArgusRTPObject { unsigned char type, length; unsigned short status; struct rtphdr src, dst; unsigned short sdrop, ddrop; unsigned short ssdev, dsdev; }; struct ArgusRTPObject { unsigned char type, length; unsigned short status; struct rtphdr src, dst; unsigned short sdrop, ddrop; unsigned short ssdev, dsdev; }; struct ArgusRTCPObject { unsigned char type, length; unsigned short status; struct rtcphdr src, dst; unsigned short src_pkt_drop, dst_pkt_drop; }; struct ArgusRTCPObject { unsigned char type, length; unsigned short status; struct rtcphdr src, dst; unsigned short src_pkt_drop, dst_pkt_drop; }; rctp struct ArgusFRAGObject { int fragnum, frag_id; unsigned short status, totlen, currlen, axfraglen; }; struct ArgusFRAGObject { int fragnum, frag_id; unsigned short status, totlen, currlen, axfraglen; }; arp ah frag

24. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (24)  Record generator (server) supports multiple access methods.  Local storage  Near-real time record access  Collector (client) initiated associations  TCP based control exchange  Proprietary protocol for capability negotiation  TCP or UDP based data transfer  SASL (Simple Authentication and Security Layer, RFC 2222) mediated security Argus Transport Model

25. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (25) Access Methods  Local Storage  Information Base for Transport Reliability  Enable retransmission capability  Support guaranteed delivery  Provide bulk transfer capability  Near-Real Time Access  Push based record transfer  Integrated management capabilities  Keep Alive/Heartbeat  Probe status and state reporting

26. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (26) Argus Record Stream  Collection of Management and Flow Activity Records  Management records convey Argus status/state (MAR)  Flow Activity Records (FAR) convey monitored flow state  Argus Stream/Files have same structure Start MAR argus Record (required) FAR Argus Record (optional) …. Status MAR Argus Record (optional) FAR Argus Record (optional)... Stop MAR Argus Record (required) Start MAR argus Record (required) FAR Argus Record (optional) …. Status MAR Argus Record (optional) FAR Argus Record (optional)... Stop MAR Argus Record (required)

27. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (27) Argus Practical Experiences  Data Model Supports a lot of applications  Security Assurance  Detect Service Failure  Detect DoS attack  Detect Network Configuration Problem (Policy enforcement Validation )  Accounting/Billing  Bidirectional Flow Model  Performance Monitoring in Passive mode (IPPM Metrics)  Connectivity and reachability : unidirectional and bidirectional  Packet Loss: TCP state machine, sequence number tracking logic  Round-Trip Delay: -R option, TCP handshake establishment round trip delay metrics are provided by default  Packet Jitter and Jitter variance  Traffic Management  Operations Management

28. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (28) NG-MON  Next Generation Network Traffic MONitoring and Analysis System  Developed at DPNM Lab, POSTECH  Targeting 10 Gbps or higher networks  To support various analysis applications  Multimedia streaming & conferencing, P2P, game traffic analysis  Network security attack detection and analysis  SLA monitoring  Usage-based billing, Customer relationship management

29. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (29) NG-MON - Requirements  Distributed, load-balancing architecture for scalability  subdivide monitoring system into several functional components  efficient load sharing between phases and within each phase  pipelined and parallel architecture  Lossless packet capture  Flow-based analysis  aggregate packet information into flows for efficient processing  Considerations for small storage requirements  Support for various applications

30. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (30) NG-MON - Design  NG-MON is composed of 5 phases  Packet Capture  Flow Generation  Flow Store  Traffic Analysis  Presentation & Reporting Packet Capturer Flow Generator Flow Store Traffic Analyzer Presenter Web Server Network Device User Interface Web browser raw packet packet header information flow information stored flows analyzed data

31. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (31) NG-MON - Packet Capture Probe #1Probe #2Probe #3 Network Link Splitting Device divided raw packet pkt header messages  Distribution of raw packets  by using splitting function provided by an optical splitter  by using mirroring function provided in network devices  Probe  captures all packets coming into probe  export buffer-queues: one to one with flow generators  fills buffer-queues with packet header’s 5-tuple based hashing  collect the scattered packets in the same flow into the same buffer-queue

32. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (32) NG-MON - Flow Generation  Distribution of packet header information  5-tuple based hashing in the probe  Packet header messages of potentially the same flow get delivered to the same flow generator  Flow generator receives packet header messages and generates flows and exports flow messages to flow store Flow Generator #1 Flow Generator #2 Flow Generator #3 Flow Generator #4 pkt header messages flow messages

33. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (33) NG-MON - Flow Store  Separation of write operations from read operations  the destination address of flow message is assigned to the flow store according to the time  While one or more flow stores are inserting flow data, the other flow stores are queried by the traffic analyzers  Flow store provides traffic information to support various analysis applications  provides an analysis API to analyzers t 1 t 2t 3 Database Query / Response Flow Store #1 Flow Store #2 Flow Store #3 Traffic Analyzer #1 Traffic Analyzer #2 flow messages Write operations Read operations

34. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (34) NG-MON - Traffic Analysis & Presentation  Analyzer extracts information from Flow Stores and can perform application specific analysis  Separate analyzer is needed for each application Flow Store #1 Flow Store #2 Flow Store #3 Web Server Presenter Traffic Throughput Analyzer Usage-based billing application DDoS or DoS Attack Analyzer Other applications

35. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (35) NG-MON - Implementation Phase Packet Capture Flow Generator Flow StoreAnalyzerPresenter Development Tool pcap library C language MySQL C language MySQL PHP jpgraph library Hardware System  Xeon 2.4 GHz 2 CPUs  1 Gbytes memory  2-1000 Mbps NICs  80 GB hard disk  Pentium-III 800 GHz CPU  256 Mbytes memory  2-100 Mbps NICs  20GB hard disk OSRedhat Linux 7.2

36. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (36) NG-MON - Deployment at POSTECH http://ngmon.postech.ac.kr Packet Capture Flow Generator Flow Store AnalyzerPresenter 141.223.182.40 EnterFLEX at Computer Center 141.223.182.40 EnterFLEX at Computer Center Flow Store 141.223.182.[31,32,33,34] POSTECH Computer Center 141.223.182.[31,32,33,34] POSTECH Computer Center 141.223.182.38 EnterFLEX at Computer Center 141.223.182.38 EnterFLEX at Computer Center 141.223.182.37 EnterFLEX at Computer Center 141.223.182.37 EnterFLEX at Computer Center 141.223.182.36 EnterFLEX at Computer Center 141.223.182.36 EnterFLEX at Computer Center INTERNET 1Gbps Optical link Router NetOptics 1Gbps Optical Splitter Packet Capture Flow Generator Packet Capture Flow Generator Packet Capture Flow Generator POSTECH Gigabit Campus Network

37. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (37) NG-MON - Host Data Sent Minute View

38. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (38) NG-MON - Host Data Exchanged Minute View

39. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (39) NG-MON - Detailed Host Data Received Minute View

40. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (40) NG-MON – Network Security Analysis Minute View

41. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (41) NG-MON – Detailed Security Analysis Minute View

42. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (42) NG-MON - Application Protocol Minute View

43. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (43) NG-MON - Application Protocol Minute View

44. POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (44) Flow-based Passive Monitoring Tools Summary Input Monitoring Output A/PFunctionsScope Hardware Agilent AdvisorL, W A, PP, U, R, LRGUI InMon sFlowL,WPFlow generation RsFlow SnifferProL, W, GA, PP, URGUI Software EtherealL, WPPRGUI cflowdNetFlowPFlow table generation RTable ArgusL, WPP, U, R, LRCLI NG-MONL, W, GPP,URGUI Input: L – LAN, W – WAN, G - Giga Measurement: A – Active, P – Passive, P – Protocol distribution, U – Utilization, R – RTT, L – Packet Loss Scope: R – Real time, O - Offline