21 st Century Network Management Dale Blakemore and Bruce Williams Information Services

Network Communications Services 2 Agenda Overview of the Griffith University NOC operation Griffith University Port Database (GUPD)‏ Plone and other tools (mrtg with rrdtool, smokeping, layer 2 trace, attack search)‏ Future vision and roadmap Questions

Network Communications Services 3 We had a problem… Like most people we had a number of disparate network management systems that together gave us a view of services and systems. These systems all worked differently and only provided limited use in troubleshooting real network problems. In addition to this other systems such as building management and control systems (eg swipe card, aircon, lighting) and video systems were moving onto the IP network and we were then told that these are critical systems and if it they have problems bad things will happen.

Network Communications Services 4 Administrator The old way... previous monitoring procedures Login

Network Communications Services 5 So we created the NOC It was decided to create our own custom interface to these disparate systems that would provide us with near real time information for the whole network that we could see, understand and most importantly ACT ON! The Griffith NOC incorporates a collection of commercial network applications and custom in house written tools that combine to provide a snapshot of high level information about the Universities network and services.

Network Communications Services 6 The new way… Login Central location for network monitoring NOC Website

Network Communications Services 7 Network and Service Monitoring The Network Operations Center facilitates the Griffith University Network Services team in providing a highly reliable and robust network infrastructure. Some of the services monitored by the NOC Include: Router/switch interface bandwidth and CPU utilization Radius authentication services Light weight access point client associations DNS and DHCP services Building management services (Air Conditioning etc.)‏

Network Communications Services 8 NOC Advantages The Griffith NOC provides the following advantages Threshold breach alerting Real-time fault notification Reduced time in identifying faults and causes The ability to become proactive Remote network monitoring

Network Communications Services 9 SNMP and Syslog NOC Server SNMP Polling Switch FirewallRouter SNMP Traps and Syslog

Network Communications Services 10 Outage and threshold alerting NOC Server IM Notification NotificationSMS Notifications

Network Communications Services 11 Front end user interface The front end has been written in a combination of PHP/Perl/AJAX and Javascript and is designed to be used from any standard compliant browser such as Firefox and projected onto a large display screen. The following slides provide a description of each applet and the final product interface.

Network Communications Services 12 Last Alert This applet will display the last event received form Intermapper or from a broadcast Jabber message sent to the NCS team. This example shows the Griffith web server has returned from an outage with status OK.

Network Communications Services 13 NCS Staff Status This applet displays the Instant messaging status of NCS staff, the status message below the name will be updated as the client status changes

Network Communications Services 14 Outages This applet will list all network devices that are currently unable to be contacted by the network management systems. In this example we have an access point in the n16 building down.

Network Communications Services 15 Campus Status This applet displays the top level status of all Griffith campuses. NCS staff are able to drill down into this view for more detailed information.

Network Communications Services 16 Top Talkers This applet gathers the top 10 interfaces on the network by frames per second. The output is logarithmically scaled to prevent the bar graph exceeding the table boundaries

Network Communications Services 17 Graphs The main display will rotate several graphs with information on bandwidth, possible network attacks and TCP session counters from key points on the network.

Network Communications Services 18

Network Communications Services 19

Network Communications Services 20 Griffith University Port Database (GUPD)‏ Griffith University has developed a web based database to document switch port to wall outlet mapping. Currently the database holds over data port records for Griffith campuses with voice records to be added in the coming months. Documenting to this level has drastically reduced requests for port activations with support staff able to determine the VLAN and connection state of any outlet within the university.

Network Communications Services 21

Network Communications Services 22 Plone Content Management System Griffith network team uses the Plone CMS to augment the NOC by providing access to contracts, procedures, policies, templates, and documentation.

Network Communications Services 23 Handy tools The NOC provides a launch point to some open source and in house tools written to improve productivity and automate some of the more tedious tasks.

Network Communications Services 24 Handy tools continued… Attack search queries a database of radius start records for username to IP address binding Subnet audit uses flood ping and SNMP requests to gather information on port usage, DNS registrations and switch software versions Smoke ping written by Tobias Oetiker provided ping graphs of critical or troubled links Layer two trace performs an automated CDP trace from the NOC box to any IP address on the Griffith network Jabbercast sends a XMPP protocol message from a web page to all NCS team members AP Graphs grabs information on wireless radio associations and graphs the usage (even LWAPP’s!)‏

Network Communications Services 25 Future vision and roadmap Incorporate the cabling routes and pit information into a searchable map view perhaps using google earth maps which are GPS accurate (already have the above info in cad format which took two years to complete)‏ Include a services view of information. Eg peoplesoft, blackboard and other key applications can see their infrastructure, performance details, SLA statistics, etc)‏ Update GUPD to provide more realtime port information and the ability to change vlans, shut ports, etc.

Network Communications Services 26 Questions? Our guiding principles for improving network management are to automate where ever possible, document thoroughly, to gain improved efficiency and to build repeatable processes.