1. 5/15/2015How to Build a NOC Quilt NOC Workshop Panel Discussion: Indiana University/Global NOC WiscNet Pacific Northwest GigaPop October 3, 2007 How to Build a NOC

2. 9/11/2007How to Build a NOC Customers and Expectations Who are your customers and what are their expectations/SLA’s? Campus, University System, StateNet, GigaPoP, RON, National Backbone, International Connections 24x7, Business Hours, Best Effort Problem Resolution, Triage, Problem Identification, Service Desk

3. 9/11/2007How to Build a NOC Supported Services In addition to networking, what other services does your organization support? Computer Operations Support Center Security Response Grid Computing Consulting

4. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  How large and complex is your network?  Large and Simple, or Small and Complex  Types of Networking Supported  Optical DWDM  Layer 2 – Switch Network  Layer 3 – Routed Network  How many hats being worn? Incestuous operational and support relationships?

5. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  What level troubleshooting and/or monitoring will your NOC do?  Troubleshooting is based on delegation of responsibility within a NOC organization and/or other related NOC’s  Monitoring based on SLA’s and health of the network & services provided

6. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  How will you communicate outages and planned work to customers?  Phone  Email / Listserv  Web page announcements  RSS

7. 9/11/2007How to Build a NOC Staffing Staffing requirements due to SLA's or after hour service response policies 24x7, Business Hours, Best Effort

8. 9/11/2007How to Build a NOC Staffing Service Hours - Hours of coverage? Not all NOCs need to be 7x24x365, but what about holidays? Weekends? On-call?

9. 9/11/2007How to Build a NOC Staffing What level of staff needs to be present, and when? Tier One: Service Desk (Call Center), Customer Service, Problem Assessment, Network Knowledgeable Tier Two: Engineering, Problem Resolution, Perform Maintenance Tier Three: Advanced Engineering, Complex Problem Resolution, Escalation Point, Network Planning

10. 9/11/2007How to Build a NOC Staffing Means of responding to issues when NOC is not staffed 24x7? Other group within organization answering phone, email, and watches monitoring, contacting the on-call Monitoring sends message directly to the on-call pager Out-Source after-hours

11. 9/11/2007How to Build a NOC Organizational Structure What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs?

12. 9/11/2007How to Build a NOC Organizational Structure Escalation practices and policies When to move a ticket to an escalation group or person within an organization When to inform key personnel within organization or network supported about outages/problems

13. 9/11/2007How to Build a NOC Organizational Structure Writing/updating procedures, training manuals, etc. Who is charged with this? When is it accomplished? NOC personnel in conjunction with their other responsibilities Dedicated resources

14. 9/11/2007How to Build a NOC NOC Location What is your facility like? Does your facility have any unique or particular advantages? How do you want to arrange your staff? Separate offices? "War room"?

15. 9/11/2007How to Build a NOC NOC Funding How is your organization funded University funds State appropriations GigaPoP / RON revenue Contracts, grants

16. 9/11/2007How to Build a NOC Tools How will you track trouble tickets? Enterprise wide systems shared used on university or state wide level Proprietary system supported by the NOC and/or other related support groups Commercial application or Homegrown

17. 9/11/2007How to Build a NOC Tools How will you track customer information? (Database needs, CRM?) Ticketing system Database Web or Wiki information repository

18. 9/11/2007How to Build a NOC Tools How will you monitor and troubleshoot? Tools, specifically. Network monitoring system like Nagios, WhatsUp Gold, HP OpenView Weather Maps MRTG

19. 9/11/2007How to Build a NOC Tools Are you writing any of your own tools? Who will maintain your applications?

20. 9/11/2007How to Build a NOC Reporting What are the key metrics for a NOC? How will you measure these? Uptime availability Nodes monitored Trouble tickets Phone calls Emails

21. 9/11/2007How to Build a NOC NOC Evolution What factors have determined operational changes for your organization - new services, expanded hours, increased number of customers, new equipment types, deeper skill level

22. 5/15/2015How to Build a NOC October 3, 2007 Building a NOC Indiana University/Global NOC Case Study Steve Peck

23. 9/11/2007How to Build a NOC Customers and Expectations Who are your customers and what are their expectations/SLA’s Indiana University Indiana GigaPoP & IP Grid I-Light (state of Indiana Higher Ed) Internet2 National LambdaRail CIC OmniPoP IU/Global NOC

24. 9/11/2007How to Build a NOC Customers and Expectations Who are your customers and what are their expectations/SLA’s TransPAC2 AMPATH MAN LAN HOPI Connecticut Education Network (pending) OneNet (consulting) IU/Global NOC

25. 9/11/2007How to Build a NOC Supported Services In addition to networking, what other services does your organization support? REN-ISAC (Security service) Open Science Grid (Grid monitoring service) IU/Global NOC

26. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  How large and complex is your network? YES!!!  Types of Networking Supported  Optical DWDM  Layer 2 – Switch Network  Layer 3 – Routed Network  How many hats being worn? Yes!  Incestuous operational and support relationships? Yes! IU/Global NOC

27. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  What level troubleshooting and/or monitoring will your NOC do?  Within the division of work between our Service Desk and Network Engineering groups, our NOC is able to perform all levels of troubleshooting and monitoring. Ranges from simple layer 3 connections to complex DWDM systems. IU/Global NOC

28. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  How will you communicate outages and planned work to customers?  Email / Listserv  Web page announcements  RSS Feeds  Web and iCalendar based Maintenance/Outage Calendars  Phone (in limited circumstances) IU/Global NOC

29. 9/11/2007How to Build a NOC Staffing Service Hours - Hours of coverage? Not all NOCs need to be 7x24x365, but what about holidays? Weekends? On-call? Service Desk: 24x7x365 Engineering: Business Hours & On-Call Systems Engineering: Business Hours & On-Call IU/Global NOC

30. 9/11/2007How to Build a NOC Staffing Staffing requirements due to SLA's or after hour service response policies Service Desk: 24x7x365 Engineering: Business Hours & On-Call Systems Engineering: Business Hours & On-Call IU/Global NOC

31. 9/11/2007How to Build a NOC Staffing What level of staff needs to be present, and when? Service Desk: 24x7x365 Engineering: Business Hours & On-Call Systems Engineering: Business Hours & On-Call IU/Global NOC

32. 9/11/2007How to Build a NOC Staffing Means of responding to issues when NOC is not staffed 24x7? We have on-call rotation for Engineering and System Engineering groups, as well as Service Desk Supervisors. IU/Global NOC

33. 9/11/2007How to Build a NOC Organizational Structure What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs? Service Desk 2 Shift Supervisors (Day & Night shifts) 5 Senior Technicians (at least one on every shift) 13 Technicians (including hourlys) 5 Off Front-Line support personnel IU/Global NOC

34. 9/11/2007How to Build a NOC Organizational Structure What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs? Network Engineering 17 Network Engineers –Network Engineering Team –Network Planning Team Systems Engineering 7 Systems Engineers (+1 open position) –Application Developers –System Administrators IU/Global NOC

35. 9/11/2007How to Build a NOC Organizational Structure Escalation practices and policies Service Desk has 15 minutes to assess problem or outage before escalating to Engineering. Standard “escalation” processes for outages and problems (immediate, 1 hour, 4 hours, 12 hours, etc.) IU/Global NOC

36. 9/11/2007How to Build a NOC Organizational Structure Writing/updating procedures, training manuals, etc. NOC personnel in conjunction with their other responsibilities (Service Desk & Engineering) Recently have hired dedicated resources to focus on internal documentation environment IU/Global NOC

37. 9/11/2007How to Build a NOC NOC Location What is your facility like? State of the art Does your facility have any unique or particular advantages? Showpiece for tours, on the edge of downtown Indianapolis, close to State Capitol How do you want to arrange your staff? Separate offices? "War room"? War room (for most part), plus offices for appropriate staff IU/Global NOC

38. 9/11/2007How to Build a NOC NOC Funding How is your organization funded Contracts, grants University funds State appropriations GigaPoP revenue IU/Global NOC

39. 9/11/2007How to Build a NOC Tools How will you track trouble tickets? Footprints ticketing system (manufactured by Numara) for all GRNOC networks & projects Peregrine for all IU campus related tickets IU/Global NOC

40. 9/11/2007How to Build a NOC Tools How will you track customer information? (Database needs, CRM?) Ticketing system Great Database (developed in-house) Web or Wiki information repository IU/Global NOC

41. 9/11/2007How to Build a NOC Tools How will you monitor and troubleshoot? Tools, specifically. Monitoring: AlertMon homegrown web based alert interface links Nagios to Footprints ticketing system Visualization: Weather Maps, Utilization Graphs (SNAPP) Management: GRNOC Database and linked systems (RADIUS, DNS, etc.) Other special-purpose tools (examples: Spanning Tree state map, Juniper Firewall Filter Grapher, Syslog Analysis Scripts, prefix list diff checker, etc.) IU/Global NOC

42. 9/11/2007How to Build a NOC Tools Are you writing any of your own tools? Yes. Large deployment of custom developed and open source software with a sprinkle of commercial software. Who will maintain your applications? Systems Engineering Team: Software Developers and System Administrators. IU/Global NOC

43. 9/11/2007How to Build a NOC Tools: Monitoring AlertMon: “Big Board” front-end to monitoring system Nagios: open source network monitoring system with custom-developed Plug-ins and monitoring agents. Monitor variety of services: BGP session status, Interface up/down, IS-IS Adjacency, Router/Switch CPU load, MSDP status, etc. IU/Global NOC

44. 9/11/2007How to Build a NOC Tools: Monitoring Nagios Monitoring System IU/Global NOC

45. 9/11/2007How to Build a NOC Tools: Weathermaps “Mini Maps” embedded in NOC web sites IU/Global NOC

46. 9/11/2007How to Build a NOC Tools: Weathermaps IU/Global NOC

47. 9/11/2007How to Build a NOC Tools: Management Network Management System including: –Contact Management –Device Management Inventory RADIUS-based authentication DNS record generation Configuration archiving Tied to utilization measurement system –Circuit Management (Layer 0, Layer 1, Layer 2) –IP Address Management –Services Management GRNOC Database IU/Global NOC

48. 9/11/2007How to Build a NOC Tools: GRNOC DB IU/Global NOC

49. 9/11/2007How to Build a NOC Tools: GRNOC DB IU/Global NOC

50. 9/11/2007How to Build a NOC Reporting What are the key metrics for a NOC? How will you measure these? Uptime availability Services monitored Trouble tickets Phone calls Emails IU/Global NOC

51. 9/11/2007How to Build a NOC NOC Evolution What factors have determined operational changes for your organization - new services, expanded hours, increased number of customers, new equipment types, deeper skill level Yes! Optical networking and dynamic circuit switching has made the most difference IU/Global NOC

52. 5/15/2015How to Build a NOC October 3, 2007 Building a NOC WiscNet Case Study Kevin Schmidtke

53. 9/11/2007How to Build a NOC Supported Services In addition to networking, what other services does your organization support? –Service & System support, providing support for all servers, mainframes and applications serving UW- Madison as well as some UW- System wide applications. –All NOC staff are cross trained to provide both network support as well as server and application support WiscNet

54. 9/11/2007How to Build a NOC NOC Funding How is your organization funded –Budget is completely revenue based –Service support (monitoring, troubleshooting and incident management) and server hosting fees account for 64% of total budget/revenue. –Network support accounts for 36% of budget/revenue WiscNet (44%), UW Madison campus (41%) and BOREAS Net (15%) WiscNet

55. 9/11/2007How to Build a NOC Customers and Expectations Who are your customers and what are their expectations/SLA’s –UW Madison Campus –WiscNet –BOREAS Net No defined SLA’s for any of customers Customers expect maximum up-time, efficient incident management, effective change management at a reasonable price. WiscNet

56. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  How large and complex is your network? –UW Madison Campus Cisco based network reaching 80 campus buildings Managing/monitoring over 1,140 3750 Cisco switches Managing/monitoring over 40,000 ports across campus Managing/monitoring 1,930 wireless access points across campus WiscNet

57. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  How large and complex is your network? –WiscNet State-wide backbone connecting 500 + members 20+ local telcos provide circuits to members throughout the state. Members include all UW campuses and institutions across the state, all state technical colleges, all private colleges in the state, the majority of K-12 school districts and some libraries across the state. WiscNet

58. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  How large and complex is your network? –BOREAS Net Regional Optical Network WiscNet is the operator A collaboration of four major research institutions in the upper Midwest: Iowa State University, the University of Iowa, the University of Minnesota, and the University of Wisconsin-Madison. WiscNet

59. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  What level troubleshooting and/or monitoring will your NOC do?  Node up/node down  Network Closet environmental alarms  UPS alarms  Backbone error monitoring  Access Switch port errors  Bandwidth threshold monitoring WiscNet

60. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  How will you communicate outages and planned work to customers? –Outage pages for incidents/unplanned outages –E-mail communications for changes/planned outages In house tools used to determine sites affected and build custom e-mail list. WiscNet

61. 9/11/2007How to Build a NOC Staffing Service Hours –NOC level 1 staffed 24X7X365 including all holidays –Level 2 engineering on-call during off-shift and holidays –Remote ‘hand and eyes’ on call for UW Campus and BOREAS Net WiscNet

62. 9/11/2007How to Build a NOC Staffing What level of staff needs to be present, and when? –A minimum of two NOC level 1 support personnel staffed at all times except holidays; only one NOC level 1 support personnel staffed on holidays. –Level 2 network support (Operational Engineers) and Network engineers available during the day. WiscNet

63. 9/11/2007How to Build a NOC Staffing After hour service response policies –Varies for each of the networks depending on on-call or best effort support model –Immediate phone support expected for on-call staff –If on-call staff presence is necessary, one hour on-site response is expected WiscNet

64. 9/11/2007How to Build a NOC Organizational Structure Escalation practices and policies –Escalation policies and procedures defined in an online knowledge database –Policies and procedures differ from network- to-network –On-call and best effort support models in place WiscNet

65. 9/11/2007How to Build a NOC Organizational Structure What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs? –UW Campus NOC Level 1 on site at all times Operation Engineers/Level 2 on-site during the day, on call after hours Network Engineers on-site during the day, best effort after hours WiscNet

66. 9/11/2007How to Build a NOC Organizational Structure What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs? –WiscNet NOC Level 1 on-site at all times Network Engineers on site during the day, best effort after hours Tech Support (service support) on site during the day, best effort after hours Telco providers available at all times Remote tech contact is best effort WiscNet

67. 9/11/2007How to Build a NOC Organizational Structure What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs? –BOREAS Net NOC Level 1 on-site at all times Network Engineers on site during the day, best effort after hours Level 3 Communications available at all times WiscNet

68. 9/11/2007How to Build a NOC Organizational Structure Writing/updating procedures, training manuals, etc. –Shared responsibility between NOC staff, Tech Support and Network Engineering –All support/training documents stored in an online knowledge database –Ability to share documents between instances of the knowledge database –A defined lifecycle for each article; draft, team review, production, update/review, etc. WiscNet

69. 9/11/2007How to Build a NOC NOC Location What is your facility like? –Large control center facility. Network support and service support duties performed in same area –War room and the primary workstation in control center available to manage major incidents. Primary workstation in control center is preferred. –Network engineers and tech support staff in adjacent campus buildings. WiscNet

70. 9/11/2007How to Build a NOC Tools How do you track customer information? (Database needs, CRM?) –All customer information (support, configuration, contact, etc.) is maintained in database tools developed in-house. –Different networks use different tools –Customer information can be searched by site id, circuit id, vlan, IP address, etc. WiscNet

71. 9/11/2007How to Build a NOC Tools How will you monitor and troubleshoot? Tools, specifically. –UW Madison campus – HP Network Node Manager, syslogs and custom monitoring tools. –WiscNet – HP Network Node Manager, syslogs and custom monitoring tool. –BOREAS Net – Infinera Digital Network Administrator WiscNet

72. 9/11/2007How to Build a NOC Tools Are you writing any of your own tools? –None of our support tools are written at the NOC. The NOC identifies the need for new tools or modifications to existing tools and the network engineers do the development. Who will maintain your applications? –The tools are maintained by the Network Engineering staff WiscNet

73. 9/11/2007How to Build a NOC Tools How do you track trouble tickets? –Incidents for all supported networks are tracked through Clarify –Each network has a unique product name –The subject line includes the site name and/or device name –Weekly reports are generated for each of the networks and distributed WiscNet

74. 9/11/2007How to Build a NOC Reporting What are the key metrics for a NOC? –Incident management Detection Troubleshooting Communication Escalation How will you measure these? –Post incident review –Customer feedback –Problem management WiscNet

75. 9/11/2007How to Build a NOC NOC Evolution What factors have determined operational changes for your organization? –New technologies & network configurations –Changes in support models dictated by campus –Newly configured networks –Incident tracking trends and analysis. WiscNet

76. 5/15/2015How to Build a NOC October 3, 2007 Building a NOC Pacific Northwest GigaPop Case Study Linda Hornung

77. 9/11/2007How to Build a NOC Customers and Expectations Our customers are in the Pacific Northwest and Pacific Rim. We track customer information in a database designed and maintained locally. Time zones have driven our need for 7x24. We cover 15 time zones which also makes scheduling planned maintenance difficult. PNWGP

78. 9/11/2007How to Build a NOC Customers and Expectations Escalation policies also drive our need for on-call schedules and on-site personnel. Our customers expect 48 hours notice prior to work, unless it is an emergency. Outages are communicated via an application we have built locally. We also want to know when our customers have planned work. PNWGP

79. 9/11/2007How to Build a NOC Customers and Expectations Expectations are included in the contract and located on the PNWGP web page. We prefer to have the NOC as the primary customer contact point for our organization in order to maintain quality of service when a ticket moves between groups. PNWGP

80. 9/11/2007How to Build a NOC Supported Services In addition to network monitoring for PNWGP, our NOC monitors: UW campus connectivity, including two hospitals (both layer 2 and layer 3, and wired as well as wireless, many UPSs), Approximately 500 sites throughout WA state for the K-20 network, Pacific Wave, and Transit Rail. PNWGP

81. 9/11/2007How to Build a NOC Supported Services We do not do any host, server, or customer application monitoring. We do not do desktop support. We do not reset passwords or arrange services such as email and web. We do not monitor data center security alarms such as for fire or flood. PNWGP

82. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  PNWGP: approx. 20 customers, 5 node sites, MPLS, BGP, VPN  Campus: 115,000 connected devices, ~100 remote locations, ~7,000 wireless APs, ~4,000 layer 2 switches, ~150 routers. PNWGP

83. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  Over 450 Washington state K-20 sites monitored.  Transit Rail, approx. 5 node sites,BGP and ISIS status. PNWGP

84. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  PNWGP troubleshoots with the customer on routing problems, latency, and loss of connectivity.  IS-IS and IGP status is monitored.  We manage DWDM, Sonet, and MPLS circuits.  Complexity is increased with escalation paths being different depending on what isn’t working. PNWGP

85. 9/11/2007How to Build a NOC Monitoring and Troubleshooting  Outages and planned events are sent via email announcement in a standard format.  We include the date/time of the work or when the outage began.  If the customer’s connectivity is entirely down, we also call or page them.  Updates are sent at predefined intervals for large events, or when we have a change in status. PNWGP

86. 9/11/2007How to Build a NOC Staffing We are 7x24 with full-time staff. Weekends we only have one person covering each day, so vacations and sick time are problems. Holidays are covered by on-site and on- call staff. On-call consists of a 7-day period and rotates among all NOC staff on a regular schedule. PNWGP

87. 9/11/2007How to Build a NOC Staffing Tier 1 are student staff, also called Network Analysts. Tier 2 are full-time staff, most are titled Network Specialists. Tier 3 are full-time staff, titled Network Engineers. PNWGP

88. 9/11/2007How to Build a NOC Staffing We advertise to all customers an on-site 7x24 staff for immediate response to outages. Our SLAs indicate when a problem should be escalated beyond the oncall staff to a manager, director, and higher at any time of the day or week. PNWGP

89. 9/11/2007How to Build a NOC Staffing Day shift M-F requires that we have engineers (Tier 3) staff in the building. Off-hours we have Tier 2 staff always present in the building. They will escalate as necessary, either due to outage severity or complexity, or at a customer’s request. PNWGP

90. 9/11/2007How to Build a NOC Staffing To maximize staffing efficiencies, 2 nd and 3 rd shift personnel report to Computer Operations managers, rather than the NOC manager. These staff provide all of the same support and have the same training/access as the daytime Tier 2 staff in the NOC. PNWGP

91. 9/11/2007How to Build a NOC Staffing We have on-call lists with a primary and secondary person for backup when NOC staff is not on-site. A separate call list exists for escalation to managers. Engineering and other service groups are also available on-call 7x24x365. PNWGP

92. 9/11/2007How to Build a NOC Organizational Structure PNWGP has recently adopted a tiered approach for staffing. NOC Network Engineers are available after 6pm M-F and weekends only via on- call. One manager, reporting to the Director of Operations. PNWGP

93. 9/11/2007How to Build a NOC Organizational Structure Our escalation practices and policies are based on length or severity of outage. At predetermined intervals additional management levels are notified of severe outages in order to help with escalation at other organizations (telcos), or to keep peers updated at the affected sites. PNWGP

94. 9/11/2007How to Build a NOC Organizational Structure Many people in the NOC work to provide and update training materials; SMEs write procedures. We use a wiki to maintain our documentation. We were able to have a tech writing expert give us training on writing effective procedural documents and re-organize our wiki. PNWGP

95. 9/11/2007How to Build a NOC NOC Location Currently, our NOC has some single- occupant offices and some multiple (2-3). We are all on the same floor. We find that being located near the Network Engineering team is quite helpful for urgent escalations. Next year we will be moving into a new location and we do not yet have details on what that will look like. PNWGP

96. 9/11/2007How to Build a NOC NOC Location Consider lighting and noise control with shared offices. How many monitors will each person need? Will you use a large central monitor for some things? Provide an impromptu meeting space for collaboration on big events. PNWGP

97. 9/11/2007How to Build a NOC NOC Location Conference bridges greatly enhance collaboration across geographic distances whether working on outages or events. PNWGP

98. 9/11/2007How to Build a NOC NOC Funding UW funding comes primarily from the state tax payers of Washington,and UW Medical Centers. Funding also comes from the PNWGP and customers and WA state K-20 network. If you recharge you will need a business and billing model. Or will you use time and materials? PNWGP

99. 9/11/2007How to Build a NOC Tools A very useful tool is live chat or IM for coordinating efforts no matter where your office is. Our customer information is tracked in a home-grown database which has grown and morphed over a dozen years. New needs such as SLAs and layer 1 info now require significant investment in upgrades. PNWGP

100. 9/11/2007How to Build a NOC Tools Our monitoring system – surprise – is also “homegrown”. We monitor interface state and IP reachability; performance and protocol state connectivity will soon be integrated into our “event system” (NMS) Automated tools can page the appropriate group to notify them of outages or threshold conditions. PNWGP

101. 9/11/2007How to Build a NOC Tools We have a separate Tools team (with 10 staff members) who design, write, implement, and maintain tools. This allows us to have full-featured and robust tools. One trade-off is fewer “one-off” tools for specific or isolated issues. PNWGP

102. 9/11/2007How to Build a NOC Tools PNWGP uses Request Tracker, RT, an open-source application to track trouble tickets. Weekly reports are generated for our Directors by sector, severity, and type. Monthly reports are generated by sector for billing purposes. PNWGP

103. 9/11/2007How to Build a NOC Reporting Key metrics we track include: 1.Ticket numbers by sector for billing 2.Phone call volumes 3.Duration of outages 4.Root Cause Analysis for high-impact events Outage time is measured by duration of the customer impact. After Action Review and Follow-up is conducted for serious events. PNWGP

104. 9/11/2007How to Build a NOC Reporting Monthly report is emailed to the customer for traffic sent to/from their site. Our internal reporting includes “operational impacts” to groups under our main organization. How do you measure your NOC’s success? Response times? Reduced calls? PNWGP

105. 9/11/2007How to Build a NOC NOC Evolution Factors that have determined operational changes for our organization have been increased size, complexity and number of networks monitored; Need to respond to outages 24 hours/day with on-site personnel rather than paging; Skill and responsibility levels have increased significantly, and continue to do so. PNWGP

106. 9/11/2007How to Build a NOC Panel Contact Information Linda Hornung Pacific Northwest GigaPop lhornung@cac.washington.edu Steve Peck Indiana University/Global NOC speck@iupui.edu Kevin Schmidtke WiscNet kevin.schmidtke@doit.wisc.edu