1. CN8861 Network Management Lecture-6 15 June 2014 Govi Ravindran Dept. of Electrical & Computer Engineering Ryerson University

2. Section 1 Distributed Management

3. RFC References – Distributed Management  RFC 3165, “Definitions of Managed Objects for the Delegation of Management Scripts”  RFC 3231, “Definitions of Managed Objects for Scheduling Management Operations”  RFC 2982, “Distributed Management Expression MIB”

4. Overview  Distributed Management System –Definition, Characteristics, and Technologies  Distributed Management MIBs –Script MIB –Schedule MIB –Expression MIB

5. Distributed Management Systems  Complexity of Management Tasks –In a centralized system, the central manager carries out the entire management task. –In a weakly distributed system, complex tasks are executed at the top level manager whereas simple management tasks are delegated to mid-level managers. –Strongly distributed systems decentralize management processing to each and every network element; management tasks are no longer confined to managers.

6. Distributed Management Systems  Computational load on the top-level manager is reduced by delegating functions elsewhere. Resource requirements at the top-level manager is reduced.  The communication load on the network is reduced. By locating mid-level managers as close to the devices as possible, only processed data moves between managers. With dynamic delegation, management tasks can be assigned and changed dynamically.

7. Distributed Management MIBs  IETF Distributed Management (DISMAN) Working Group had proposed standard MIBs to realize –Distributed Managers and/or –Self-Managing devices  The DISMAN MIBs that are of interest include: –Schedule MIB –Script MIB –Expression MIB

8. IETF Distributed Management MIBs mib-2 Schedule MIB (schedMIB) Script MIB (scriptMIB) {mib-2 63} {mib-2 64} Event MIB (dismanEventMIB) {mib-2 88} Expression MIB (dismanExpressionMIB) {mib-2 90} {mgmt 1} mgmt {internet 1} internet {.iso.org.dod 1}

9. Script MIB  Script MIB defines managed objects for delegation of management functions.  Management functionality is delegated as scripts which are transferred to the location where they are executed.  The Scripts can be started remotely, arguments can be passed to them, and results can be returned back to the initiator.

10. Script MIB - Capabilities  Script MIB provides following capabilities: –Transfer management scripts to a distributed manager. –Transfer arguments for management scripts. –Initiate, suspend, resume or terminate management scripts. –Monitor and control running management scripts. –Transfer the results produced by the management scripts.

11. Script MIB – Script Transfer  There are two different ways to transfer management scripts to a distributed manager: –Push Model: The high-level manager pushes the script to the distributed manager. –Pull Model: The manager tells the distributed manager the location of the script and the distributed manager retrieves the script itself.  The Script MIB also supports management scripts that are part of the Script MIB implementation.  Scripts are stored in an non-volatile storage. This allows a distributed manager to restart scripts.  Every script is identified by an administratively assigned name.

12. Script MIB – Framework

13. Script MIB – Script Transfer  The ‘push model’ is realized by a table which allows a manager to write scripts by sending a sequence of SNMP set requests.  The ‘pull model’ is realized by the use of URLs that point to the script source. –The manager writes the URL by an SNMP set request. –The distributed manager is then responsible for retrieving the script using the protocol specified in the URL (such as FTP or HTTP).

14. Script MIB – Script Execution  The Script MIB allows execution of management scripts –Arguments can be passed to a script. –Scripts can return a single result value. –Scripts can also export complex results. For instance, a URL can be returned that points to a file containing script output.  When runtime errors terminate active scripts, an exit code and an error message is left in the MIB.  Running scripts have associated status object which allows script execution to be suspended, resumed, or aborted.  A history of finished scripts is kept in the MIB.

15. Distributed Management Script MIB smTraps scriptMIB {.iso.org.dod.internet.mgmt.mib-2 64} smObjects smConformance {scriptMIB 2} {scriptMIB 3} smExtsnTable smScriptException smNotifications {scriptMIB 1} smLangTable smScriptObjects smRunObjects smScriptAbort smScriptResult smScriptTable smCodeTable smLaunchTable smRunTable

16. Script MIB – Language Group  Language Group –The smLanguageGroup is used to provide information about the languages and the language extensions supported by a Script MIB implementation. This group includes two tables. The smLangTable lists all languages supported by a Script MIB implementation and the smExtsnTable lists the extensions that are available for a given language.

17. Script MIB – Script Group  Script Group –The ‘smScriptGroup’ consists of a single table called ‘smScriptTable’ which lists all scripts known to a Script MIB implementation. –The source of a script is defined by the ‘smScriptSource’ object. –‘smScriptSource’ may contain a URL pointing to a remote location. –If the ‘smScriptSource’ is null, the script source is read from the smCodeTable or from local storage. –The ‘smScriptStorageType’ object is used to distinguish between scripts read from local storage and scripts read from the smCodeTable.

18. Script MIB – Script Group  Scripts are automatically loaded once the ‘smScriptAdminStatus’ object is set to ‘enabled’. –Loading a script includes retrieving the script, compiling (if required), and making the code available to the runtime system.  The ‘smScriptOperStatus’ object is used to indicate the status of the loading process. –‘smScriptOperStatus’ object will start in the state ‘retrieving’, switch to the state ‘compiling’ and finally reach the state ‘enabled’. –Errors during the retrieval or compilation phase will result in an error state such as `compilationFailed'.

19. Script MIB – Script Group smScriptObjects {scriptMIB.smObjects 3} smScriptTable {smScriptObjects 1} smCodeTable smScriptOwner smScriptName smScriptDescr smScriptLanguage smScriptSource smScriptAdminStatus smCodeIndex smCodeText smCodeRowStatus {smScriptObjects 2} smScriptOperStatus smScriptStorageType smScriptRowStatus smScriptError smScriptLastChange

20. Script MIB – Code Group  The smCodeGroup consists of a single table, the ‘smCodeTable’ –‘smCodeTable’ provides the ability to transfer and modify scripts via SNMP set requests. –A script can be fragmented over multiple rows of the smCodeTable in order to handle SNMP message size limitations.

21. Script MIB – Launch Group  The smLaunchGroup contains a single table, the smLaunchTable.  The smLaunchTable allows the following operations: –associate a script with an owner used during script execution –provide arguments and parameters for script invocation –invoke scripts with a single set operation

22. Script MIB – Launch Group smRunObjects {scriptMIB.smObjects 4} smLaunchTable {smRunObjects 1} smLaunchOwner smLaunchName smLaunchScriptOwner smLaunchScriptName smLaunchArgument smLaunchMaxRunning smLaunchMaxCompleted smLaunchLifeTime smLaunchExpireTime smLaunchStart smLaunchControl smLaunchAdminStatus smLaunchOperStatus smLaunchRunIndexNext smLaunchStorageType smLaunchRowStatus smLaunchError smLaunchLastChange smLaunchRowExpireTime

23. Script MIB – Run Group  Every row in the smRunTable contains the argument passed during script invocation, the result, the script exit code, run state, and start and end time-stamps.  There are three writable objects in the smRunTable: –The smRunLifeTime object defines the maximum time a running script may run before it is terminated by the Script MIB implementation. –The smRunExpireTime object defines the time that a completed script can stay in the smRunTable before it is aged out. –The smRunControl object allows running scripts to be suspended, resumed, or aborted.

24. Script MIB smRunObjects {scriptMIB.smObjects 4} smRunTable {smRunObjects 2} smRunIndex smRunArgument smRunStartTime smRunEndTime smRunLifeTime smRunExpireTime smRunExitCode smRunResult smRunControl smRunState smRunError smRunResultTime smRunError

25. IETF Script MIB

27. Script MIB – Application  Software update: –Updating and configuring software on a managed node –Following scripts can be used to accomplish the above:  a script checking the version currently installed,  a script to stop and uninstall the old software release,  an install script for the new release, and a  a configure script for the new release

28. Schedule MIB  Schedule MIB supports scheduling of actions periodically or at specified dates and times.  The actions can be used to trigger management functions in a distributed manager.  Schedules can be enabled or disabled by modifying a control object.

29. Schedule MIB – Schedules  Periodic schedules are based on fixed time periods between the initiation of scheduled actions.  Calendar schedules trigger scheduled action at specified days of the week and/or days of the month.  One-shot Schedules are similar to calendar schedules. The difference is a one-shot schedule will automatically disable itself once it has been invoked.

30. Schedule MIB – Actions  Scheduled actions are invoked by SNMP set operations on local MIB variables.  Scheduled actions in the Schedule MIB are further restricted to objects of type INTEGER. –Simple actions such as setting a status MIB object (e.g. ifAdminStatus). –Complex actions such as triggering a management script.

31. Distributed Management Schedule MIB schedTraps schedMIB {.iso.org.dod.internet.mgmt.mib-2 63} schedObjects schedConformance {schedMIB 2} {schedMIB 3} schedTable schedActionFailure schedOwner schedName schedDescr schedInterval schedWeekday schedMonth schedDay schedHour schedMinute schedValue schedType schedAdminStatus schedOperStatus schedFailures schedLastFailure schedLastFailed schedNotifications {schedMIB 1} schedLocalTime schedVariable schedContextName schedStorageType schedRowStatus schedTriggers

32. Expression MIB  The Expression MIB is a powerful way to monitor large, complex systems.  Expression MIB is a way to create new, customized MIB objects for monitoring. –Expression MIB supports externally defined expressions of existing MIB objects. –In the Expression MIB the results of an evaluated expression are MIB objects that are used like any other MIB objects. –Without these capabilities monitoring would be limited to the objects in predefined MIBs.  The MIB is usually appropriate in a relatively powerful, resource-rich managed system and not necessarily in a severely limited environment.

33. Expression MIB - Sampling  The MIB supports three types of object sampling for the MIB objects that make up the expression: absolute, delta, and changed. –Absolute samples are simply the value of the MIB object at the time it is sampled. –Delta sampling requires the implementation to maintain the value of the last sample. –Changed sampling is a boolean indicating whether or not the object changed value since the last sample.

34. Expression MIB - Evaluation  If there are no delta or change values in an expression, the evaluation occurs on demand, i.e. when a requester attempts to read the value of the expression. In this case the requester gets a freshly calculated value.  For expressions with delta or change values, evaluation goes on continuously, every sample period. In this case requesters get the value as of the last sample period.

35. Simple Expression MIB  No wildcards –Such an implementation would allow expressions made up of individual MIB objects but would not be suitable for expressions applied across large tables as each instance in the table would require a separate expression definition. –Furthermore it is suitable for tables with arbitrary, dynamic instances, as expressions definitions could not predict what instance values to use. –Significantly reduces retrieving and processing complexity  No deltas –Leaving out delta processing significantly reduces state that must be kept and processing burden. –Unfortunately it makes expressions on counters unusable, as counters have meaning only as deltas. –An implementation without deltas might be useful for a resource limited, self-managing system that has no need for expressions or events on counters.

36. Expression MIB – Structure  Resource Section  Has objects to manage resource usage by wildcarded delta expressions, a potential major consumer of CPU and memory.  Definition Section  Contains the tables that define expressions.  The expression table contains parameters that apply to the entire expression, such as the data type of the result, and the sampling interval if it contains delta or change values.  The object table contains the parameters that apply to the individual objects that go into the expression, including the object identifier, sample type, discontinuity indicator, and such.  Value Section  Contains the values of evaluated expressions.  For a given expression only one of the columns is instantiated, depending on the result data type for the expression.

37. Expression MIB - Structure dismanExpressionMIB {.iso.org.dod.internet.mgmt.mib-2 90} dismanExpressionMIBObjects dismanExpressionMIBConformance {dismanExpressionMIB 2} expValue {dismanExpressionMIB 1} expResource expDefine expValueTable expExpressionTable expObjectTable expErrorTable

38. Section 2 Service Level Monitoring

39. Defining Delay, Jitter, & Packet loss  Multi-media applications are sensitive to the Delay, Jitter, and Packet Loss characteristics of data networks.  Delay is the time taken for packets to travel across a network. One-way delay calculations require clock synchronization across nodes, whereas measuring round-trip delay is easier.  Jitter is the variation in delay. A jitter buffer temporarily stores arriving packets in order to minimize delay variations.  Packet loss describes the % of packets that did not reach their intended destination. Reasons for packet loss include link failure, network congestion, and Random Early Discard (RED) among others.

40. Cisco IP SLA  Cisco IP SLA is an IOS feature that enables measurement of jitter, latency, or packet loss between network end- points.  IP Services can be simulated by specifying various packet sizes, ports, class of service, packet spacing, and frequencies.  Jitter/Latency/Packet-loss measurements per class of service are made to validate service differentiation for data, voice, and video.  Cisco IP SLA also helps to establish an network performance baseline and allow the user to understand trends and anomalies from the baseline.

41. SAA and RTTMON  Cisco IP SLA is supported by the Service Assurance Agent (SAA) and the Round-Trip Time Monitoring (RTTMON) MIB. –The SAA and the RTTMON MIB are Cisco IOS software features available in versions 12.0 (5)T and higher.  The features enable you to test and collect delay, jitter, and packet loss statistics on the data network. –Cisco IOS routers are used as probes to simulate end points. –The probes can be configured to monitor the network for delay and jitter and alert network management stations when thresholds are exceeded.

42. SLAs for IP Networks Measure Either PE–CE or PE-PE Links Enterprise Site 2 Enterprise Site 1 Measure Either CE–PE or CE–CE Links P Router SP Converged IP Network

43. UDP Jitter Operation Source IP Core Responder Send train of packets with constant Interval Receive train of packets at Interval impacted by Network Add a receive time stamp, and calculate delta, the processing time.  Per-direction inter-packet delay (Jitter)  Per-direction packet loss  Average Round Trip Delay

44. Responder Source Router Responder Target Router T1 T4 T3 T2  = T3 - T2  Responder factors out destination processing time making results highly accurate  Responder allows for one-way measurements for latency, jitter, and packet loss.

45. UDP Jitter Operation Time Frequency = IP SLA UDP Jitter test packet – Operation 1 destination1 = IP SLA UDP Jitter test packet – Operation 2 destination2 Interval Number of Packets

46. IP SLA Example IP Class of Service One-way Latency Packet LossJitter Priority Voice Traffic < 80 ms < 5%< 35 ms Real-Time Traffic - Video < 80 ms< 3% Priority Data Traffic < 100 ms< 2% Best Effort Traffic No target

47. Polling the RTTMON MIB  The source SLA probe collects and places data in the RTTMON SNMP MIB tables. rttMonStatsTable rttMonLatestJitterOper  ‘show rtr collection−stats’

48. Polling the RTTMON MIB saarouter1# show rtr collection−stats Collected Statistics Entry Number: 100 Target Address: 172.18.179.10, Port Number: 14384 Start Time: 13:06:04.000 09:25:00 Tue Mar 21 2000 RTT Values: NumOfRTT: 600 RTTSum: 873 RTTSum2: 1431 Packet Loss Values: PacketLossSD: 0 PacketLossDS: 0 PacketOutOfSequence: 0 PacketMIA: 0 PacketLateArrival: 0 InternalError: 0 Busies: 0 Jitter Values: MinOfPositivesSD: 1 MaxOfPositivesSD: 1 NumOfPositivesSD: 23 SumOfPositivesSD: 23 Sum2PositivesSD: 23 MinOfNegativesSD: 1 MaxOfNegativesSD: 1 NumOfNegativesSD: 1SumOfNegativesSD: 1 Sum2NegativesSD: 1 MinOfPositivesDS: 1 MaxOfPositivesDS: 1 NumOfPositivesDS: 7 SumOfPositivesDS: 7 Sum2PositivesDS: 7 MinOfNegativesDS: 1 MaxOfNegativesDS: 1 NumOfNegativesDS: 18 SumOfNegativesDS: 18 Sum2NegativesDS: 18

49. Simulating a voice call  Simulating a G.711 voice call: –RTP/UDP port 14384 –64 Kb/s –172 byte packets (160 byte payload + 12 byte RTP header)  The jitter probe operation includes: –Send the request to RTP/UDP port number 14384. –Send 172 byte packets –Send 3000 packets for each frequency cycle. –Send every packet 20 milliseconds apart for a duration of 60 seconds and sleep 10 seconds before starting the next frequency cycle. –((3000 datagrams * 160 bytes per datagram)/ 60sec) * 8 = 64 kb/s

50. Section 3 Agent EXtentisibilty (AgentX)

51. Why AgentX?  There is a real need to dynamically extend the set of managed objects within a node.  The SNMP framework does not describe how a set of managed objects supported by an agent may be changed dynamically

52. Agent EXtentisibilty (AgentX)  Facilitates extension of SNMP Agents with dynamic addition of new MIB module implementations.  Separates SNMP protocol engines from MIB Instrumentation.  Maintains transparency to Management Stations.  Proposed IETF Standard –RFC 2741 (AgentX Protocol) –RFC 2742 (AgentX MIB)

53. AgentX Framework  In the AgentX framework, the SNMP agent is defined to consist of: –a master agent, which sends and receives SNMP protocol messages but has little or no direct access to management information. –one or more subagents, which are shielded from the SNMP protocol messages, but have access to management information. –a protocol (agentX) that operates between the master agent and subagents, permitting subagents to connect to the master agent

54. AgentX Design Features  Master agents are concerned with SNMP protocol operations and the translations to and from AgentX protocol operations  Subagents are concerned with management instrumentation  Subagents are not aware of any other existing subagents.  A single subagent is "authoritative" for a particular region of the MIB –"region" may extend from an entire MIB down to a single object- instance

55. AgentX - Basic Structure Master Agent Protocol Operations MIB AgentX Sub-agent SNMP

56. Master/Sub Agent Functions  Master agent –sends and accepts SNMP protocol messages. –sends and receives AgentX protocol messages to access management information from sub-agents. –provides instrumentation for certain MIB objects. –forwards notifications on behalf of subagents.  A sub-agent –initiates AgentX session with the master agent –registers MIB regions with the master agent –provides instrumentation for managed objects –performs management operations on variables –initiates notifications

57. AgentX – Usage Scenarios  Subagents implement separate MIB modules –for example, subagent `A' implements "mib-2", subagent `B' implements "host-resources-mib". This will be the most common subagent configuration.  Subagents implement rows in a "simple table". –A simple table is one in which row creation is not specified, and for which the MIB does not define an object that counts entries in the table. This is the most commonly defined type of MIB table  Subagents share MIBs along non-row partitions. –Subagents register "chunks" of the MIB that represent multiple rows, due to the nature of the MIB's index structure. Examples include registering tcpConnEntry.a.b.c.d, where a.b.c.d represents an IP address on a subagent’s interface

58. AgentX – OID Registration  MIB Registration –Example: REGISTER (mib-2)  RANGE Registration –Example: REGISTER (mib-2.interfaces, mib-2.tcp)

59. AgentX – VarBind Splitting MIB ABC A BC Sub-agent Master Agent SNMP MANAGER

60. Section 4 HP Openview NNM: Scalability and Distribution

61. NNM Scalability and Distribution  Scalability is referred to –the configuration of NNM to perform well for a wide range of network sizes and degrees of complexity.  Distribution of NNM is –Refers to the distribution of network management functions to multiple, distributed systems.  Objective is to optimize: –System resources at the management station. –Reduce the amount of management traffic over the network.

62. NNM Scalability/Distribution Features  Distributed Discovery and Monitoring  Management Consoles  Support for viewing large map (the ‘panner’)  On-demand Sub-maps  Filters  Distributed Threshold Monitoring  Event Forwarding

63. Distributed Discovery and Monitoring  Distributed discovery and monitoring is the ability to move much of the network discovery, topology monitoring, and status polling from the central management station to one or more distributed stations.

64. Distributed Discovery & Monitoring  Consists of Management and Collection stations  Management station makes available network management functionality to users, either directly or via one or more management consoles.  Collection stations function as data collection points. A collection station typically performs: –network discovery, –topology checks, –IP status monitoring, –data collection, –event forwarding

65. Distributed Discovery & Monitoring Management Station A Collection Station A Collection Station B Collection Station n Changes in status and topology are relayed from the collection stations to the management station. Discovery and status polling occur at the local level. Managed Devices

66. NNM Filters  A filter is essentially a way to remove unneeded data from further processing.  Filters result in less data to process resulting in improved NNM performance and usability for the administrators. –for instance, to an administrator whose only concern is the routers and hubs, a filter can be applied to the map, so that only routers and hubs appear.  Filters can be applied at different points in the system: –Discovery filtering at Management or Collection station. –Topology filtering at the Collection station. –Map filtering at the Management station. –Failover filtering at the Management station.

67. NNM Filters  A data-stream filter acts as a screen through which objects flow. –contains criteria that permit some objects to pass the screen while others are removed at the point of the filter. –once an object has passed a data-stream filter, later changes to the filter have no effect on it.  A set-defining filter is applied to a pool of candidate objects to determine which objects belong in the final set based on some criteria contained in the filter. –If a set-defining filter is changed, all candidate objects are re- evaluated to determine which ones belong in the new set.

68. NNM Filters Filter TypePurpose DiscoveryLimit scope of objects added to database. TopologyLimit information forwarded from collection station to management station. MapShow only items of interest to operator on map. PersistenceDisable on-demand sub-maps for third party applications. FailoverLimit nodes polled by management station when collection station fails.

69. Distributed Discovery & Monitoring NNM Adv Edition A NNM Station C Dual-role station Domain A Devices NNM Adv Edition B Domain B Devices LAN data flow WAN data flow Filter in data path NNM Station D Collection station NNM Station E Collection station NNM Station F Dual-role station NFS Objects discovered & monitored by Station C Objects discovered & monitored by Station D Objects discovered & monitored by Station E Objects discovered & monitored by Station F

70. Section 5 RMON

71. RMON  SNMP Remote Network Monitoring (RMON) was created to enable efficient management of remote networks using dedicated devices.  RMON defines a MIB module with object definitions that permit dedicated network management capabilities. –RFC 1271, "Remote Network Monitoring Management Information Base," published in 1991. –RFC 2819, published in May 2000, updates RMON to use the SMIv2 specification.

72. RMON  Remote network monitoring devices, often called remote probes or monitors are stand-alone devices and devote significant internal resources for the sole purpose of managing a network. –An organization may employ many of these devices, one per network segment, to manage its internet.  RMON MIB objects are not intended for processing by a management application and for direct manipulation by humans.  Most RMON objects are suitable for the management of any type of network –Objects specific to managing Ethernet networks are defined in the ‘etherStatsTable’ and ‘etherHistoryTable’. –The RMON-2 MIB objects enable analysis up to the application layer.

73. RMON System

74. RMON Benefit  Offline Operation: –There are times when a management station is not in contact with its remote monitoring devices. by design or during network failures –RMON MIB allows a probe to be configured to collect statistics continuously or run diagnostics, even when communication with the management station may not be possible or efficient. The probe may notify the management station when an exceptional condition occurs. Fault, performance, and configuration information may be continuously accumulated and forwarded to the management station efficiently.

75. RMON MIB Structure  An RMON MIB consists of a several functional groups. –Functional groups have control and data tables. –The control data specify how and when to collect the statistical data.

76. RMON MIB Structure  All groups in RMON MIB are optional. Implementations of this MIB must also implement the system group of MIB-II and the IF-MIB. The objects are arranged in following groups: –ethernet statistics –ethernet history –alarm –host –hostTopN –matrix –filter –packet capture –event

77. RMON MIB Structure

78.  The Ethernet Statistics Group –The ethernet statistics group contains statistics for each monitored Ethernet interface on this device.  The Alarm Group –The alarm group periodically compares statistical samples to configured thresholds. If the monitored variable crosses a threshold, an event is generated. –This group requires the implementation of the event group.  The Host Group –The host group contains statistics associated with each host discovered on the network. This group discovers hosts on the network by keeping a list of source and destination MAC addresses seen in packets received from the network.