5/15/2015How to Build a NOC Quilt NOC Workshop Panel Discussion: Indiana University/Global NOC WiscNet Pacific Northwest GigaPop October 3, 2007 How to.

Slides:



Advertisements
Similar presentations
Steve Lewis J.D. Edwards & Company
Advertisements

Top-Down Network Design Chapter Nine Developing Network Management Strategies Copyright 2010 Cisco Press & Priscilla Oppenheimer.
Common NOC Practices 4/05/2007 The Quilt NOC Common Practice Panel April 4, 2007.
Case Franchise Support & Training. Case Support Who? Who do I call for support? What? What types of questions can be answered by the corporate support.
MUNIS Platform Migration Project WELCOME. Agenda Introductions Tyler Cloud Overview Munis New Features Questions.
MONITORING THE CONNECTICUT EDUCATION NETWORK Aliza Bailey 10/20/2010.
CIT 470: Advanced Network and System Administration
End to End Uptime Monitoring
Position Opening Page 1 Job Classification: Network Server Administrator Opening Date: June 27, 2014 Location: Dobson Office Closing Date: July 7, 2014Grade:
Dave Jent, PI Luke Fowler, Co-PI Ron Johnson, Co-PI
Documenting the Existing Network - Starting Points IACT 418 IACT 918 Corporate Network Planning.
Introduction <Header Title> Last saved: YYYY-MM-DD
Network Operations Centers Training Programs
1 25\10\2010 Unit-V Connecting LANs Unit – 5 Connecting DevicesConnecting Devices Backbone NetworksBackbone Networks Virtual LANsVirtual LANs.
© 2014 Fair Isaac Corporation. Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac.
Chapter 13 Network Management and Operations. Agenda Objectives Scope Functions Software Security Physical Facility Staffing.
Beyond HIPAA, Protecting Data Key Points from the HIPAA Security Rule.
Common Solutions Group Workshop: Help Desk Survey Results.
Microsoft Premier Support for Office 365 Service Introduction
MANAGED SERVICES OPERATIONS. Increasing IP Infrastructure Complexity Requires Greater Need for Services Data Center B2B Links Branch Offices Distribution.
1 Customer Network Operations Center. 2 Agenda Overview Benefits Value Additional Services Questions Monitoring Options.
1 Network Quarantine At Cornell University Steve Schuster Director, Information Security Office.
EC4019PA Intrusion & Access Control Technology (IACT) Chapter 4- CAMS Prepared by Sandy Tay.
8/20/2015How to Build a NOC Pacific NorthWest GigaPop November 29, 2007 How to Build a NOC.
You will be responsible for professional and technical work related to network and telecommunications solutions (Local Area Network/ Wireless, Wide Area.
Net Optics Confidential and Proprietary Net Optics appTap Intelligent Access and Monitoring Architecture Solutions.
Term 2, 2011 Week 3. CONTENTS The physical design of a network Network diagrams People who develop and support networks Developing a network Supporting.
Network Administration. What is a Systems Administrator?  Person responsible for:  Setting up servers  Configuring the environment for web and other.
Current Job Components Information Technology Department Network Systems Administration Telecommunications Database Design and Administration.
© 2006 Avaya Inc. All rights reserved. Avaya Services Michael Dundon Business Development Manager.
Common Devices Used In Computer Networks
Cisco Threaded Case Study
Objectives Overview Identify the qualities of valuable information Describe various information systems used in an enterprise Identify the components of.
SCSC 311 Information Systems: hardware and software.
So You Have Assigned the SLA, Now What? Dave McGaugh Network Architecture, PNWGP Quilt Fiber Workshop - 13/Oct/2006 St Louis, MO.
Steve Peck Service Desk Manager Global Research NOC At Indiana University.
Technical Policy and Standards Andy Gorton – Senior Architect: Institutional Networks.
Existing Solution 1 Proactive – SLR 6 hour schedule activities for managed services Stop light report is posted and reviews any open tickets for.
Unit 8a Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users Component 8 Installation and Maintenance of Health IT.
workshop eugene, oregon What is network management? System & Service monitoring  Reachability, availability Resource measurement/monitoring.
Computer Emergency Notification System (CENS)
Delivering Circuit Services to Researchers: The HOPI Testbed Rick Summerhill Director, Network Research, Architecture, and Technologies, Internet2 Joint.
Session 7 - Maintenance - contract and day-to-day Maintenance Support Presenter  Grenville Powell (Managing Director - of Shokaz Integrated Computing.
Business Data Communications, Fourth Edition Chapter 11: Network Management.
Based on work by DoIT Network Services, UW-Madison The Network and the Role of Tools January 6, 2006 Ron Kraemer, Deputy CIO.
Oncology Patient Enrollment Network OPEN Support Services Lucille Patrichuk OPEN Implementation Manager OPEN Conference September 18, 2008.
BNL Tier 1 Service Planning & Monitoring Bruce G. Gibbard GDB 5-6 August 2006.
Connect. Communicate. Collaborate perfSONAR MDM Service for LHC OPN Loukik Kudarimoti DANTE.
ORGANIZING IT SERVICES AND PERSONNEL (PART 1) Lecture 7.
IT Priorities Minimize CAPEX Maximize employee productivity Grow the business Add new compute resources real- time to support growth Meet compliance requirements.
NETWORKING FUNDAMENTALS. Network+ Guide to Networks, 4e2.
Adoption and Use of Electronic Medical Records (in Federally Qualified Health Centers) and Supporting an ASP Community Care Network of Virginia, Inc.
How to Build a NOC. Identify Customers –Who are your customers? Understand Customer Expectations –What are your user expectations? –SLA’s? Support Service.
Component 4: Introduction to Information and Computer Science Unit 7: Networks & Networking Lecture 1 This material was developed by Oregon Health & Science.
Spring 2004 Internet2 Member Meeting NLR Service Center Update Dave Jent Indiana University.
1 Network Quarantine At Cornell University Steve Schuster Director, Information Security Office.
Staff Assessment Technology Services Department Palmyra Area School District.
Installation and Maintenance of Health IT Systems Unit 8a Troubleshooting; Maintenance and Upgrades; and Interaction with Vendors, Developers, and Users.
A local area network (LAN) is a group of computers and associated devices that share a common communications line or wireless link. Typically, connected.
Information Technology Division Customer Service Support Center.
© 2014 Level 3 Communications, LLC. All Rights Reserved. Proprietary and Confidential. Simple, End-to-End Performance Management Application Performance.
Taking your Business Technology Further. First Communications: At A Glance Technology Provider since 1998, serving thousands of Businesses throughout.
OPEN SOURCE NETWORK MANAGEMENT TOOLS
Fourth Dimension Technologies
SupportU 24x7: Implementing and Maintaining a Co-Managed Help Desk
SCSU Technology Update FY05
Based on work by DoIT Network Services, UW-Madison
Relate to Clients on a business level
Presentation transcript:

5/15/2015How to Build a NOC Quilt NOC Workshop Panel Discussion: Indiana University/Global NOC WiscNet Pacific Northwest GigaPop October 3, 2007 How to Build a NOC

9/11/2007How to Build a NOC Customers and Expectations Who are your customers and what are their expectations/SLA’s? Campus, University System, StateNet, GigaPoP, RON, National Backbone, International Connections 24x7, Business Hours, Best Effort Problem Resolution, Triage, Problem Identification, Service Desk

9/11/2007How to Build a NOC Supported Services In addition to networking, what other services does your organization support? Computer Operations Support Center Security Response Grid Computing Consulting

9/11/2007How to Build a NOC Monitoring and Troubleshooting  How large and complex is your network?  Large and Simple, or Small and Complex  Types of Networking Supported  Optical DWDM  Layer 2 – Switch Network  Layer 3 – Routed Network  How many hats being worn? Incestuous operational and support relationships?

9/11/2007How to Build a NOC Monitoring and Troubleshooting  What level troubleshooting and/or monitoring will your NOC do?  Troubleshooting is based on delegation of responsibility within a NOC organization and/or other related NOC’s  Monitoring based on SLA’s and health of the network & services provided

9/11/2007How to Build a NOC Monitoring and Troubleshooting  How will you communicate outages and planned work to customers?  Phone  / Listserv  Web page announcements  RSS

9/11/2007How to Build a NOC Staffing Staffing requirements due to SLA's or after hour service response policies 24x7, Business Hours, Best Effort

9/11/2007How to Build a NOC Staffing Service Hours - Hours of coverage? Not all NOCs need to be 7x24x365, but what about holidays? Weekends? On-call?

9/11/2007How to Build a NOC Staffing What level of staff needs to be present, and when? Tier One: Service Desk (Call Center), Customer Service, Problem Assessment, Network Knowledgeable Tier Two: Engineering, Problem Resolution, Perform Maintenance Tier Three: Advanced Engineering, Complex Problem Resolution, Escalation Point, Network Planning

9/11/2007How to Build a NOC Staffing Means of responding to issues when NOC is not staffed 24x7? Other group within organization answering phone, , and watches monitoring, contacting the on-call Monitoring sends message directly to the on-call pager Out-Source after-hours

9/11/2007How to Build a NOC Organizational Structure What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs?

9/11/2007How to Build a NOC Organizational Structure Escalation practices and policies When to move a ticket to an escalation group or person within an organization When to inform key personnel within organization or network supported about outages/problems

9/11/2007How to Build a NOC Organizational Structure Writing/updating procedures, training manuals, etc. Who is charged with this? When is it accomplished? NOC personnel in conjunction with their other responsibilities Dedicated resources

9/11/2007How to Build a NOC NOC Location What is your facility like? Does your facility have any unique or particular advantages? How do you want to arrange your staff? Separate offices? "War room"?

9/11/2007How to Build a NOC NOC Funding How is your organization funded University funds State appropriations GigaPoP / RON revenue Contracts, grants

9/11/2007How to Build a NOC Tools How will you track trouble tickets? Enterprise wide systems shared used on university or state wide level Proprietary system supported by the NOC and/or other related support groups Commercial application or Homegrown

9/11/2007How to Build a NOC Tools How will you track customer information? (Database needs, CRM?) Ticketing system Database Web or Wiki information repository

9/11/2007How to Build a NOC Tools How will you monitor and troubleshoot? Tools, specifically. Network monitoring system like Nagios, WhatsUp Gold, HP OpenView Weather Maps MRTG

9/11/2007How to Build a NOC Tools Are you writing any of your own tools? Who will maintain your applications?

9/11/2007How to Build a NOC Reporting What are the key metrics for a NOC? How will you measure these? Uptime availability Nodes monitored Trouble tickets Phone calls s

9/11/2007How to Build a NOC NOC Evolution What factors have determined operational changes for your organization - new services, expanded hours, increased number of customers, new equipment types, deeper skill level

5/15/2015How to Build a NOC October 3, 2007 Building a NOC Indiana University/Global NOC Case Study Steve Peck

9/11/2007How to Build a NOC Customers and Expectations Who are your customers and what are their expectations/SLA’s Indiana University Indiana GigaPoP & IP Grid I-Light (state of Indiana Higher Ed) Internet2 National LambdaRail CIC OmniPoP IU/Global NOC

9/11/2007How to Build a NOC Customers and Expectations Who are your customers and what are their expectations/SLA’s TransPAC2 AMPATH MAN LAN HOPI Connecticut Education Network (pending) OneNet (consulting) IU/Global NOC

9/11/2007How to Build a NOC Supported Services In addition to networking, what other services does your organization support? REN-ISAC (Security service) Open Science Grid (Grid monitoring service) IU/Global NOC

9/11/2007How to Build a NOC Monitoring and Troubleshooting  How large and complex is your network? YES!!!  Types of Networking Supported  Optical DWDM  Layer 2 – Switch Network  Layer 3 – Routed Network  How many hats being worn? Yes!  Incestuous operational and support relationships? Yes! IU/Global NOC

9/11/2007How to Build a NOC Monitoring and Troubleshooting  What level troubleshooting and/or monitoring will your NOC do?  Within the division of work between our Service Desk and Network Engineering groups, our NOC is able to perform all levels of troubleshooting and monitoring. Ranges from simple layer 3 connections to complex DWDM systems. IU/Global NOC

9/11/2007How to Build a NOC Monitoring and Troubleshooting  How will you communicate outages and planned work to customers?  / Listserv  Web page announcements  RSS Feeds  Web and iCalendar based Maintenance/Outage Calendars  Phone (in limited circumstances) IU/Global NOC

9/11/2007How to Build a NOC Staffing Service Hours - Hours of coverage? Not all NOCs need to be 7x24x365, but what about holidays? Weekends? On-call? Service Desk: 24x7x365 Engineering: Business Hours & On-Call Systems Engineering: Business Hours & On-Call IU/Global NOC

9/11/2007How to Build a NOC Staffing Staffing requirements due to SLA's or after hour service response policies Service Desk: 24x7x365 Engineering: Business Hours & On-Call Systems Engineering: Business Hours & On-Call IU/Global NOC

9/11/2007How to Build a NOC Staffing What level of staff needs to be present, and when? Service Desk: 24x7x365 Engineering: Business Hours & On-Call Systems Engineering: Business Hours & On-Call IU/Global NOC

9/11/2007How to Build a NOC Staffing Means of responding to issues when NOC is not staffed 24x7? We have on-call rotation for Engineering and System Engineering groups, as well as Service Desk Supervisors. IU/Global NOC

9/11/2007How to Build a NOC Organizational Structure What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs? Service Desk 2 Shift Supervisors (Day & Night shifts) 5 Senior Technicians (at least one on every shift) 13 Technicians (including hourlys) 5 Off Front-Line support personnel IU/Global NOC

9/11/2007How to Build a NOC Organizational Structure What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs? Network Engineering 17 Network Engineers –Network Engineering Team –Network Planning Team Systems Engineering 7 Systems Engineers (+1 open position) –Application Developers –System Administrators IU/Global NOC

9/11/2007How to Build a NOC Organizational Structure Escalation practices and policies Service Desk has 15 minutes to assess problem or outage before escalating to Engineering. Standard “escalation” processes for outages and problems (immediate, 1 hour, 4 hours, 12 hours, etc.) IU/Global NOC

9/11/2007How to Build a NOC Organizational Structure Writing/updating procedures, training manuals, etc. NOC personnel in conjunction with their other responsibilities (Service Desk & Engineering) Recently have hired dedicated resources to focus on internal documentation environment IU/Global NOC

9/11/2007How to Build a NOC NOC Location What is your facility like? State of the art Does your facility have any unique or particular advantages? Showpiece for tours, on the edge of downtown Indianapolis, close to State Capitol How do you want to arrange your staff? Separate offices? "War room"? War room (for most part), plus offices for appropriate staff IU/Global NOC

9/11/2007How to Build a NOC NOC Funding How is your organization funded Contracts, grants University funds State appropriations GigaPoP revenue IU/Global NOC

9/11/2007How to Build a NOC Tools How will you track trouble tickets? Footprints ticketing system (manufactured by Numara) for all GRNOC networks & projects Peregrine for all IU campus related tickets IU/Global NOC

9/11/2007How to Build a NOC Tools How will you track customer information? (Database needs, CRM?) Ticketing system Great Database (developed in-house) Web or Wiki information repository IU/Global NOC

9/11/2007How to Build a NOC Tools How will you monitor and troubleshoot? Tools, specifically. Monitoring: AlertMon homegrown web based alert interface links Nagios to Footprints ticketing system Visualization: Weather Maps, Utilization Graphs (SNAPP) Management: GRNOC Database and linked systems (RADIUS, DNS, etc.) Other special-purpose tools (examples: Spanning Tree state map, Juniper Firewall Filter Grapher, Syslog Analysis Scripts, prefix list diff checker, etc.) IU/Global NOC

9/11/2007How to Build a NOC Tools Are you writing any of your own tools? Yes. Large deployment of custom developed and open source software with a sprinkle of commercial software. Who will maintain your applications? Systems Engineering Team: Software Developers and System Administrators. IU/Global NOC

9/11/2007How to Build a NOC Tools: Monitoring AlertMon: “Big Board” front-end to monitoring system Nagios: open source network monitoring system with custom-developed Plug-ins and monitoring agents. Monitor variety of services: BGP session status, Interface up/down, IS-IS Adjacency, Router/Switch CPU load, MSDP status, etc. IU/Global NOC

9/11/2007How to Build a NOC Tools: Monitoring Nagios Monitoring System IU/Global NOC

9/11/2007How to Build a NOC Tools: Weathermaps “Mini Maps” embedded in NOC web sites IU/Global NOC

9/11/2007How to Build a NOC Tools: Weathermaps IU/Global NOC

9/11/2007How to Build a NOC Tools: Management Network Management System including: –Contact Management –Device Management Inventory RADIUS-based authentication DNS record generation Configuration archiving Tied to utilization measurement system –Circuit Management (Layer 0, Layer 1, Layer 2) –IP Address Management –Services Management GRNOC Database IU/Global NOC

9/11/2007How to Build a NOC Tools: GRNOC DB IU/Global NOC

9/11/2007How to Build a NOC Tools: GRNOC DB IU/Global NOC

9/11/2007How to Build a NOC Reporting What are the key metrics for a NOC? How will you measure these? Uptime availability Services monitored Trouble tickets Phone calls s IU/Global NOC

9/11/2007How to Build a NOC NOC Evolution What factors have determined operational changes for your organization - new services, expanded hours, increased number of customers, new equipment types, deeper skill level Yes! Optical networking and dynamic circuit switching has made the most difference IU/Global NOC

5/15/2015How to Build a NOC October 3, 2007 Building a NOC WiscNet Case Study Kevin Schmidtke

9/11/2007How to Build a NOC Supported Services In addition to networking, what other services does your organization support? –Service & System support, providing support for all servers, mainframes and applications serving UW- Madison as well as some UW- System wide applications. –All NOC staff are cross trained to provide both network support as well as server and application support WiscNet

9/11/2007How to Build a NOC NOC Funding How is your organization funded –Budget is completely revenue based –Service support (monitoring, troubleshooting and incident management) and server hosting fees account for 64% of total budget/revenue. –Network support accounts for 36% of budget/revenue WiscNet (44%), UW Madison campus (41%) and BOREAS Net (15%) WiscNet

9/11/2007How to Build a NOC Customers and Expectations Who are your customers and what are their expectations/SLA’s –UW Madison Campus –WiscNet –BOREAS Net No defined SLA’s for any of customers Customers expect maximum up-time, efficient incident management, effective change management at a reasonable price. WiscNet

9/11/2007How to Build a NOC Monitoring and Troubleshooting  How large and complex is your network? –UW Madison Campus Cisco based network reaching 80 campus buildings Managing/monitoring over 1, Cisco switches Managing/monitoring over 40,000 ports across campus Managing/monitoring 1,930 wireless access points across campus WiscNet

9/11/2007How to Build a NOC Monitoring and Troubleshooting  How large and complex is your network? –WiscNet State-wide backbone connecting members 20+ local telcos provide circuits to members throughout the state. Members include all UW campuses and institutions across the state, all state technical colleges, all private colleges in the state, the majority of K-12 school districts and some libraries across the state. WiscNet

9/11/2007How to Build a NOC Monitoring and Troubleshooting  How large and complex is your network? –BOREAS Net Regional Optical Network WiscNet is the operator A collaboration of four major research institutions in the upper Midwest: Iowa State University, the University of Iowa, the University of Minnesota, and the University of Wisconsin-Madison. WiscNet

9/11/2007How to Build a NOC Monitoring and Troubleshooting  What level troubleshooting and/or monitoring will your NOC do?  Node up/node down  Network Closet environmental alarms  UPS alarms  Backbone error monitoring  Access Switch port errors  Bandwidth threshold monitoring WiscNet

9/11/2007How to Build a NOC Monitoring and Troubleshooting  How will you communicate outages and planned work to customers? –Outage pages for incidents/unplanned outages – communications for changes/planned outages In house tools used to determine sites affected and build custom list. WiscNet

9/11/2007How to Build a NOC Staffing Service Hours –NOC level 1 staffed 24X7X365 including all holidays –Level 2 engineering on-call during off-shift and holidays –Remote ‘hand and eyes’ on call for UW Campus and BOREAS Net WiscNet

9/11/2007How to Build a NOC Staffing What level of staff needs to be present, and when? –A minimum of two NOC level 1 support personnel staffed at all times except holidays; only one NOC level 1 support personnel staffed on holidays. –Level 2 network support (Operational Engineers) and Network engineers available during the day. WiscNet

9/11/2007How to Build a NOC Staffing After hour service response policies –Varies for each of the networks depending on on-call or best effort support model –Immediate phone support expected for on-call staff –If on-call staff presence is necessary, one hour on-site response is expected WiscNet

9/11/2007How to Build a NOC Organizational Structure Escalation practices and policies –Escalation policies and procedures defined in an online knowledge database –Policies and procedures differ from network- to-network –On-call and best effort support models in place WiscNet

9/11/2007How to Build a NOC Organizational Structure What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs? –UW Campus NOC Level 1 on site at all times Operation Engineers/Level 2 on-site during the day, on call after hours Network Engineers on-site during the day, best effort after hours WiscNet

9/11/2007How to Build a NOC Organizational Structure What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs? –WiscNet NOC Level 1 on-site at all times Network Engineers on site during the day, best effort after hours Tech Support (service support) on site during the day, best effort after hours Telco providers available at all times Remote tech contact is best effort WiscNet

9/11/2007How to Build a NOC Organizational Structure What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs? –BOREAS Net NOC Level 1 on-site at all times Network Engineers on site during the day, best effort after hours Level 3 Communications available at all times WiscNet

9/11/2007How to Build a NOC Organizational Structure Writing/updating procedures, training manuals, etc. –Shared responsibility between NOC staff, Tech Support and Network Engineering –All support/training documents stored in an online knowledge database –Ability to share documents between instances of the knowledge database –A defined lifecycle for each article; draft, team review, production, update/review, etc. WiscNet

9/11/2007How to Build a NOC NOC Location What is your facility like? –Large control center facility. Network support and service support duties performed in same area –War room and the primary workstation in control center available to manage major incidents. Primary workstation in control center is preferred. –Network engineers and tech support staff in adjacent campus buildings. WiscNet

9/11/2007How to Build a NOC Tools How do you track customer information? (Database needs, CRM?) –All customer information (support, configuration, contact, etc.) is maintained in database tools developed in-house. –Different networks use different tools –Customer information can be searched by site id, circuit id, vlan, IP address, etc. WiscNet

9/11/2007How to Build a NOC Tools How will you monitor and troubleshoot? Tools, specifically. –UW Madison campus – HP Network Node Manager, syslogs and custom monitoring tools. –WiscNet – HP Network Node Manager, syslogs and custom monitoring tool. –BOREAS Net – Infinera Digital Network Administrator WiscNet

9/11/2007How to Build a NOC Tools Are you writing any of your own tools? –None of our support tools are written at the NOC. The NOC identifies the need for new tools or modifications to existing tools and the network engineers do the development. Who will maintain your applications? –The tools are maintained by the Network Engineering staff WiscNet

9/11/2007How to Build a NOC Tools How do you track trouble tickets? –Incidents for all supported networks are tracked through Clarify –Each network has a unique product name –The subject line includes the site name and/or device name –Weekly reports are generated for each of the networks and distributed WiscNet

9/11/2007How to Build a NOC Reporting What are the key metrics for a NOC? –Incident management Detection Troubleshooting Communication Escalation How will you measure these? –Post incident review –Customer feedback –Problem management WiscNet

9/11/2007How to Build a NOC NOC Evolution What factors have determined operational changes for your organization? –New technologies & network configurations –Changes in support models dictated by campus –Newly configured networks –Incident tracking trends and analysis. WiscNet

5/15/2015How to Build a NOC October 3, 2007 Building a NOC Pacific Northwest GigaPop Case Study Linda Hornung

9/11/2007How to Build a NOC Customers and Expectations Our customers are in the Pacific Northwest and Pacific Rim. We track customer information in a database designed and maintained locally. Time zones have driven our need for 7x24. We cover 15 time zones which also makes scheduling planned maintenance difficult. PNWGP

9/11/2007How to Build a NOC Customers and Expectations Escalation policies also drive our need for on-call schedules and on-site personnel. Our customers expect 48 hours notice prior to work, unless it is an emergency. Outages are communicated via an application we have built locally. We also want to know when our customers have planned work. PNWGP

9/11/2007How to Build a NOC Customers and Expectations Expectations are included in the contract and located on the PNWGP web page. We prefer to have the NOC as the primary customer contact point for our organization in order to maintain quality of service when a ticket moves between groups. PNWGP

9/11/2007How to Build a NOC Supported Services In addition to network monitoring for PNWGP, our NOC monitors: UW campus connectivity, including two hospitals (both layer 2 and layer 3, and wired as well as wireless, many UPSs), Approximately 500 sites throughout WA state for the K-20 network, Pacific Wave, and Transit Rail. PNWGP

9/11/2007How to Build a NOC Supported Services We do not do any host, server, or customer application monitoring. We do not do desktop support. We do not reset passwords or arrange services such as and web. We do not monitor data center security alarms such as for fire or flood. PNWGP

9/11/2007How to Build a NOC Monitoring and Troubleshooting  PNWGP: approx. 20 customers, 5 node sites, MPLS, BGP, VPN  Campus: 115,000 connected devices, ~100 remote locations, ~7,000 wireless APs, ~4,000 layer 2 switches, ~150 routers. PNWGP

9/11/2007How to Build a NOC Monitoring and Troubleshooting  Over 450 Washington state K-20 sites monitored.  Transit Rail, approx. 5 node sites,BGP and ISIS status. PNWGP

9/11/2007How to Build a NOC Monitoring and Troubleshooting  PNWGP troubleshoots with the customer on routing problems, latency, and loss of connectivity.  IS-IS and IGP status is monitored.  We manage DWDM, Sonet, and MPLS circuits.  Complexity is increased with escalation paths being different depending on what isn’t working. PNWGP

9/11/2007How to Build a NOC Monitoring and Troubleshooting  Outages and planned events are sent via announcement in a standard format.  We include the date/time of the work or when the outage began.  If the customer’s connectivity is entirely down, we also call or page them.  Updates are sent at predefined intervals for large events, or when we have a change in status. PNWGP

9/11/2007How to Build a NOC Staffing We are 7x24 with full-time staff. Weekends we only have one person covering each day, so vacations and sick time are problems. Holidays are covered by on-site and on- call staff. On-call consists of a 7-day period and rotates among all NOC staff on a regular schedule. PNWGP

9/11/2007How to Build a NOC Staffing Tier 1 are student staff, also called Network Analysts. Tier 2 are full-time staff, most are titled Network Specialists. Tier 3 are full-time staff, titled Network Engineers. PNWGP

9/11/2007How to Build a NOC Staffing We advertise to all customers an on-site 7x24 staff for immediate response to outages. Our SLAs indicate when a problem should be escalated beyond the oncall staff to a manager, director, and higher at any time of the day or week. PNWGP

9/11/2007How to Build a NOC Staffing Day shift M-F requires that we have engineers (Tier 3) staff in the building. Off-hours we have Tier 2 staff always present in the building. They will escalate as necessary, either due to outage severity or complexity, or at a customer’s request. PNWGP

9/11/2007How to Build a NOC Staffing To maximize staffing efficiencies, 2 nd and 3 rd shift personnel report to Computer Operations managers, rather than the NOC manager. These staff provide all of the same support and have the same training/access as the daytime Tier 2 staff in the NOC. PNWGP

9/11/2007How to Build a NOC Staffing We have on-call lists with a primary and secondary person for backup when NOC staff is not on-site. A separate call list exists for escalation to managers. Engineering and other service groups are also available on-call 7x24x365. PNWGP

9/11/2007How to Build a NOC Organizational Structure PNWGP has recently adopted a tiered approach for staffing. NOC Network Engineers are available after 6pm M-F and weekends only via on- call. One manager, reporting to the Director of Operations. PNWGP

9/11/2007How to Build a NOC Organizational Structure Our escalation practices and policies are based on length or severity of outage. At predetermined intervals additional management levels are notified of severe outages in order to help with escalation at other organizations (telcos), or to keep peers updated at the affected sites. PNWGP

9/11/2007How to Build a NOC Organizational Structure Many people in the NOC work to provide and update training materials; SMEs write procedures. We use a wiki to maintain our documentation. We were able to have a tech writing expert give us training on writing effective procedural documents and re-organize our wiki. PNWGP

9/11/2007How to Build a NOC NOC Location Currently, our NOC has some single- occupant offices and some multiple (2-3). We are all on the same floor. We find that being located near the Network Engineering team is quite helpful for urgent escalations. Next year we will be moving into a new location and we do not yet have details on what that will look like. PNWGP

9/11/2007How to Build a NOC NOC Location Consider lighting and noise control with shared offices. How many monitors will each person need? Will you use a large central monitor for some things? Provide an impromptu meeting space for collaboration on big events. PNWGP

9/11/2007How to Build a NOC NOC Location Conference bridges greatly enhance collaboration across geographic distances whether working on outages or events. PNWGP

9/11/2007How to Build a NOC NOC Funding UW funding comes primarily from the state tax payers of Washington,and UW Medical Centers. Funding also comes from the PNWGP and customers and WA state K-20 network. If you recharge you will need a business and billing model. Or will you use time and materials? PNWGP

9/11/2007How to Build a NOC Tools A very useful tool is live chat or IM for coordinating efforts no matter where your office is. Our customer information is tracked in a home-grown database which has grown and morphed over a dozen years. New needs such as SLAs and layer 1 info now require significant investment in upgrades. PNWGP

9/11/2007How to Build a NOC Tools Our monitoring system – surprise – is also “homegrown”. We monitor interface state and IP reachability; performance and protocol state connectivity will soon be integrated into our “event system” (NMS) Automated tools can page the appropriate group to notify them of outages or threshold conditions. PNWGP

9/11/2007How to Build a NOC Tools We have a separate Tools team (with 10 staff members) who design, write, implement, and maintain tools. This allows us to have full-featured and robust tools. One trade-off is fewer “one-off” tools for specific or isolated issues. PNWGP

9/11/2007How to Build a NOC Tools PNWGP uses Request Tracker, RT, an open-source application to track trouble tickets. Weekly reports are generated for our Directors by sector, severity, and type. Monthly reports are generated by sector for billing purposes. PNWGP

9/11/2007How to Build a NOC Reporting Key metrics we track include: 1.Ticket numbers by sector for billing 2.Phone call volumes 3.Duration of outages 4.Root Cause Analysis for high-impact events Outage time is measured by duration of the customer impact. After Action Review and Follow-up is conducted for serious events. PNWGP

9/11/2007How to Build a NOC Reporting Monthly report is ed to the customer for traffic sent to/from their site. Our internal reporting includes “operational impacts” to groups under our main organization. How do you measure your NOC’s success? Response times? Reduced calls? PNWGP

9/11/2007How to Build a NOC NOC Evolution Factors that have determined operational changes for our organization have been increased size, complexity and number of networks monitored; Need to respond to outages 24 hours/day with on-site personnel rather than paging; Skill and responsibility levels have increased significantly, and continue to do so. PNWGP

9/11/2007How to Build a NOC Panel Contact Information Linda Hornung Pacific Northwest GigaPop Steve Peck Indiana University/Global NOC Kevin Schmidtke WiscNet