1. Network Operations Centers Training Programs
Quilt NOC Workshop Panel Discussion: CENIC Indiana University/Global NOC WiscNet Pacific Northwest GigaPop October 3, 2007
Network Operations Centers Training Programs

2. Network Operations Centers Training Programs
NOC Training Programs
WiscNet
Kika Barr
October 3, 2007
Network Operations Centers Training Programs

3. Network Operations Centers Training Programs
Who We Support
53 Higher Education Institutions
Over 300 k12s
12 CESAS – Cooperative Education Service Agency
18 Technical Colleges
13Library Systems
36 Government Institutions
1 museum
I hospital
BOREAS-Net
9/11/2007
Network Operations Centers Training Programs
WiscNet

4. Network Operations Centers Training Programs
Staff Workload
tickets per month
9/11/2007
Network Operations Centers Training Programs
WiscNet

5. Network Operations Centers Training Programs
Our Staff
Size
2 full time tech support specialists
1 ¾ time tech support specialist
1 Network Architect
Mix
The Tech Support Specialist are strong generalists
Yet, the staff tend to divide itself on the level of the expertise by service (Jack of all trades, master of some)
9/11/2007
Network Operations Centers Training Programs
WiscNet

6. Recent Training Content
Training offered to staff in the last 6 months
Seminars about Security
Fiber training
Fiber Optics training
Firewall training
Cisco related training
Juniper training
9/11/2007
Network Operations Centers Training Programs
WiscNet

7. Network Operations Centers Training Programs
Training Resources
Recent training was designed and provided by;
Vendors
Cisco
Packeteer
Engineers on staff
9/11/2007
Network Operations Centers Training Programs
WiscNet

8. Network Operations Centers Training Programs
Training Budget
We allocate training budget from the organization budget
Try to stay local to save on expenses
Take advantage of vendor training opportunities
9/11/2007
Network Operations Centers Training Programs
WiscNet

9. Network Operations Centers Training Programs
Training Logistics
Methodology for scheduling training
When training is out of the office, we normally just send 1 tech support member
When training is in house, our NOC is made aware that staff is on training. Tech Support specialist have their laptops on with IM so NOC can reach us if there is an emergency
Voice mail to Tech Support goes to mail list
Where do you train?
In house using conference rooms. No training rooms available
travel
9/11/2007
Network Operations Centers Training Programs
WiscNet

10. Network Operations Centers Training Programs
Lessons Learned
If you don’t use it, you lose it
Consistency and relevance
Delicate balance between overdoing it and not doing it enough
9/11/2007
Network Operations Centers Training Programs
WiscNet

11. Training Program Example
WiscNet Support Staff if Tier 2
Training as needed
Have been discussing with engineers about internal training and a more structured planned
Competencies:
Troubleshooting
Ticketing
Tools
Customer Services
Service Expertise
Optical networks around the corner
9/11/2007
Network Operations Centers Training Programs
WiscNet

12. Demonstrated Competencies
Excellent phone skills
Excellent understanding of WiscNet services
Technical proficiency in several areas of technology – router configuration, DNS, troubleshooting, firewall configuration, WAN infrastructure, and Optical networks
9/11/2007
Network Operations Centers Training Programs
WiscNet

13. Keeping the Program Current
Challenge for us because of time constraints
Hoping to work with WiscNet engineers to design a more predictable training train
9/11/2007
Network Operations Centers Training Programs
WiscNet

14. Network Operations Centers Training Programs
NOC Training Programs
WiscNet
Kevin Schmidtke
October 3, 2007
Network Operations Centers Training Programs

15. Network Operations Centers Training Programs
Who We Support
Number and type of sectors your organization supports
UW Madison Campus – network backbone, wireless
WiscNet – statewide educational network
BOREAS Net – regional optical network, research focused
9/11/2007
Network Operations Centers Training Programs
WiscNet

16. Network Operations Centers Training Programs
Staff Workload
# of incidents opened per month
300 Clarify cases opened on average
WiscNet cases – 68%
UW Madison cases – 29%
BOREAS Net cases – 2%
9/11/2007
Network Operations Centers Training Programs
WiscNet

17. Network Operations Centers Training Programs
Our Staff
Size of Staff
10 staff
7 Professional Senior
3 Professional Junior
1 Process & Procedure coordinator
Staff Mix
All Junior staff scheduled with a Senior staff member. Seniors perform in a lead capacity.
9/11/2007
Network Operations Centers Training Programs
WiscNet

18. Network Operations Centers Training Programs
Content
Examples of training offered to your staff in the last 6 months
Infinera DNA application training
Infinera hardware configuration training
Modified WiscNet process and procedures
9/11/2007
Network Operations Centers Training Programs
WiscNet

19. Network Operations Centers Training Programs
Training Resources
Who designed the content and provided the training for your staff in the last 6 months
Most training was developed in-house
Most training was delivered by subject matter experts on staff
9/11/2007
Network Operations Centers Training Programs
WiscNet

20. Network Operations Centers Training Programs
Funding Sources
Two Sources
New network support contracts are expected to deliver training to all staff as part of the service rollout and acceptance
Training budget allocation for each staff. Traditionally this is used for specific professional development
9/11/2007
Network Operations Centers Training Programs
WiscNet

21. Network Operations Centers Training Programs
Training Logistics
Scheduling training
Training is typically presented in a minimum of two sessions to ensure all staff can attend. Rarely do we have an all-staff training session
Training facility
Several training rooms are available in data center complex
9/11/2007
Network Operations Centers Training Programs
WiscNet

22. Training Program Example
Knowledge Database Repository
All formal training results in an article or a series of articles
Many departments have instances of the knowledge database and articles can be shared from one instance to another
Only the originating instance administrator can modify the content of the article
9/11/2007
Network Operations Centers Training Programs
WiscNet

23. Training Program Example
Knowledge Database instance administrator
Creates all new knowledge database articles or recruits a subject matter expert for content
Reviews all articles originating from another department
Is responsible for making all NOC staff aware of new process or procedure
Ensures all staff review and accept responsibility for content of articles
9/11/2007
Network Operations Centers Training Programs
WiscNet

24. Training Program Example
Knowledge Database Lifecycle
All articles start out in draft form
All articles are set to team review
All staff members ‘sign off’ on content of the article
All articles are reviewed at a minimum of 6 months for accuracy and relevance
Articles are reviewed by Instance Administrator and/or subject matter expert
9/11/2007
Network Operations Centers Training Programs
WiscNet

25. Network Operations Centers Training Programs
Lessons Learned
Transition from hardcopy to online knowledge database required trust in materials.
All articles are keyword searchable. Review failed searches to determine better ways to define articles
Keep a copy of knowledge database on stand alone server on separate UPS
9/11/2007
Network Operations Centers Training Programs
WiscNet

26. Pacific Northwest GigaPop Linda Hornung
NOC Training Programs
Pacific Northwest GigaPop
Linda Hornung
October 3, 2007
Network Operations Centers Training Programs

27. Network Operations Centers Training Programs
Who We Support
PNWGP
University of Washington Campus
University of Washington Medical Centers
Washington State K-20 network
TransitRail (4/2007)
9/11/2007
Network Operations Centers Training Programs
PNWGP

28. Network Operations Centers Training Programs
Staff Workload
Our NOC opens tickets per month across the various customer bases we handle
9/11/2007
Network Operations Centers Training Programs
PNWGP

29. Network Operations Centers Training Programs
Our Staff
Full time staff in the NOC and Computer Operations, who provide 2nd and 3rd shift coverage.
Four tier 3
Seven tier 2
Five tier 2 for off-shifts.
New NOC students – 6 students work 4pm-8pm M-F, and 9am-6pm Sat/Sun. Considered tier 1.
9/11/2007
Network Operations Centers Training Programs
PNWGP

30. Recent Training Content
Several of our staff have been to Cisco router classes and Cisco ONS classes.
Several others attended Juniper ONS classes provided on the UW campus.
Professional development courses such as leadership, communication, project management, team building.
Many presentations by Network Engineering and Network Architecture teams on new services and network design that have recently been implemented.
9/11/2007
Network Operations Centers Training Programs
PNWGP

31. Network Operations Centers Training Programs
Training Budget
No defined budget, but this past year has seen a tremendous increase in training opportunities!
We have been able to have training from outside companies, our vendors, the UW Professional Development program, and many in-house experts.
9/11/2007
Network Operations Centers Training Programs
PNWGP

32. Network Operations Centers Training Programs
Training Logistics
Because of the different shifts and the need to provide continuous coverage, we will always offer at least two different training sessions for NOC and other interested staff.
Usually train in local conference rooms.
Vendor classes have been held in larger facilities on campus. Always in well-connected facilities. 
9/11/2007
Network Operations Centers Training Programs
PNWGP

33. Network Operations Centers Training Programs
Lessons Learned
The staff that usually does our NOC training must have taken a Train the Trainer course.
Have defined training goals.
Use mixed media – don’t just talk.
Engage the audience through occasional questions or scenarios.
Provide take-away reference material.
9/11/2007
Network Operations Centers Training Programs
PNWGP

34. Training Program Example
Tier 1 staff: the focus is on escalation within the NOC and being able to gather appropriate initial information.
Competencies:
Troubleshooting
Ticketing
Tools
9/11/2007
Network Operations Centers Training Programs
PNWGP

35. Training Topics for Tier 1 Staff
Architecture overview
Services overview
Customer information
Monitoring
Opening trouble tickets
Handling and issuing outages
Escalation
9/11/2007
Network Operations Centers Training Programs
PNWGP

36. Demonstrated Competencies
Basic phone skills (answer, hold, transfer)
Searching tickets and for previous information
Using tools to find out information on equipment location, customer contacts, circuit ID, etc.
9/11/2007
Network Operations Centers Training Programs
PNWGP

37. Keeping the Program Current
Continuous project to keep the content updated and accurate
Reliance on an internal wiki for procedure and information
9/11/2007
Network Operations Centers Training Programs
PNWGP

38. Network Operations Centers Training Programs
NOC Training Programs
CENIC
Cindy Abercrombie
October 3, 2007
Network Operations Centers Training Programs

39. Network Operations Centers Training Programs
Who We Support
CalREN Community (-approx site count)
California Community Colleges - 112
University of California - 11
California State University - 25
California K-12 System - 74
California Private Institutions - 9
Associates and sites outside California - 11
NLR
Layer 1 Services
Support equipment / deploy services across 250 sites
Pwave, CIIX, TransitRail, AARNet
Support equipment and services
9/11/2007
Network Operations Centers Training Programs
CENIC

40. Network Operations Centers Training Programs
Staff Workload
CalREN Monthly Ticket Stats
Avg 670 tickets opened
Community requests for support and outages
Deployments, upgrades, maintenance
Avg 375 support tickets resolved
NLR Monthly Ticket Stats
Avg 105 tickets resolved
Deployments, outages, maintenance
9/11/2007
Network Operations Centers Training Programs
CENIC

41. Network Operations Centers Training Programs
Our Staff
15 NOC Engineers
1/3 of our team have > 5yrs. Network support experience
2/3 of our team have 2-3 years combination LAN/WAN and end-user technical support, with 1-2 years large-scale network support
Other Network Operations Support Resources
Our senior engineering team works with us daily on all types of request
Our Sys Admin team responds to NOC needs quickly and provides support to our community too
9/11/2007
Network Operations Centers Training Programs
CENIC

42. Network Operations Centers Training Programs
Content
Examples of Training Provided
Cisco MSTP Test and Turn-up
Configuring BGP on Cisco routers
Cisco CRS-1 Essentials
DNS in Practice
9/11/2007
Network Operations Centers Training Programs
CENIC

43. Network Operations Centers Training Programs
Training Resources
Cisco
Global Knowledge
CENIC Senior Engineers
Outside Specialty Organization
9/11/2007
Network Operations Centers Training Programs
CENIC

44. Training Funding & Logistics
Methodology for scheduling training
2 training sessions on same topic, sometimes weeks apart or months apart
Reduce upgrade, maintenance and deployment activities
Reduce NOC coverage and include senior engineering resources
Purchase bulk training hours and use Cisco Learning Credits
9/11/2007
Network Operations Centers Training Programs
CENIC

45. Network Operations Centers Training Programs
Lessons Learned
Discussing training opportunities during recruiting process
Coordinate training with at least two people at a time, the more the better
Purchasing bulk training hours is more cost effective than a couple of courses at a time
If course feedback for a particular trainer is good, request the same trainer for future courses
Make training and professional development a regular part of communications and schedules
9/11/2007
Network Operations Centers Training Programs
CENIC

46. Network Operations Centers Training Programs
Day1 Training Program
Start of a Shift
Tools
Process
Escalations
Schedule Maintenance
Outages
Announcements
Contact Information
Where
Updates and requests for info
9/11/2007
Network Operations Centers Training Programs
CENIC

47. Non-Technical Training Topics
High visibility sites
Education acronyms
Our role in other network support organizations
Preparing turn-over work for the next shift
Standard signatures and greetings
How we do lunch
Program your cell phone
How to report you are delayed or absent
9/11/2007
Network Operations Centers Training Programs
CENIC

48. Demonstrated Competencies
Using tools to find out information on equipment location, customer contacts, circuit ID
Searching tickets and for previous information
Document and articulate the known facts of the current issue
Know how to use our vendors’ processes
9/11/2007
Network Operations Centers Training Programs
CENIC

49. Keeping Network Operations
Everyone owns network operations
Sharing is caring = keep wiki current, send
Ask questions to understand b/c there’s too much to memorize
9/11/2007
Network Operations Centers Training Programs
CENIC

50. Network Operations Centers Training Programs
NOC Training Programs
IU/Global NOC
Steve Peck
October 3, 2007
Network Operations Centers Training Programs

51. Network Operations Centers Training Programs
Who We Support
17; Campuses, University System, StateNet, GigaPoP, Regional Aggregation point, National Backbones, International Connections (total includes separate networks supported by the national backbones)
9/11/2007
Network Operations Centers Training Programs
IU/Global NOC

52. Network Operations Centers Training Programs
Staff Workload
906 incidents a month
9/11/2007
Network Operations Centers Training Programs
IU/Global NOC

53. Network Operations Centers Training Programs
Our Staff
Size of Staff
25 Service Desk
17 Network Engineering
7 Systems Engineering
Staff Mix - Service Desk
Shift Supervisors (Day & Night shifts)
Senior Technicians (at least one on every shift)
Technicians (including hourlys)
Off Front-Line support personnel (5 people)
9/11/2007
Network Operations Centers Training Programs
IU/Global NOC

54. Network Operations Centers Training Programs
Our Staff
Staff Mix
Network Engineering
Network Engineering Team
- Day-to-day problem assessment / resolution, After Hours on-call rotation.
Network Planning Team
- Longer term network planning tasks.
Systems Engineering
Software Developers
System Administrators
9/11/2007
Network Operations Centers Training Programs
IU/Global NOC

55. Network Operations Centers Training Programs
Content
Service Desk
New Internet2 Network
Connecticut Education Network
Automated reporting tool
IU Core network transition to GRNOC Eng
Indiana GigaPoP changes
CIC OmniPoP follow up
NLR L2 VLAN / Circuit determination
New database circuit schema
9/11/2007
Network Operations Centers Training Programs
IU/Global NOC

56. Network Operations Centers Training Programs
Content
Engineering
Internal training: Ex: Ciena CoreDirector training from resident in-group expert.
Training for new staff as necessary: Ex: commercial Cisco BGP training for a new Network Engineer.
Systems Engineering
One-on-one training of new staff
Ad-hoc training from in-group experts on various technologies and systems.
9/11/2007
Network Operations Centers Training Programs
IU/Global NOC

57. Network Operations Centers Training Programs
Training Resources
Service Desk
Senior staff picks topics and focus, utilize internal expertise to provide and design the content
Engineering
Internal formal and informal training sessions by in-house experts.
Commercial training for individual engineers as needed.
9/11/2007
Network Operations Centers Training Programs
IU/Global NOC

58. Network Operations Centers Training Programs
Training Resources
Methodology for scheduling training
Training Facilities
Service Desk
Definitely a challenge! Once a month from 4:00 pm to 7:00 pm; two techs man the SD; use conference room; meetings are video recorded;
9/11/2007
Network Operations Centers Training Programs
IU/Global NOC

59. Network Operations Centers Training Programs
Lessons Learned
If you have access to excellent internal training resources within your organization, you *HAVE* to utilize them!
Must have a clear picture of staff responsibilities to ensure that appropriate training takes place in a prioritized manner.
9/11/2007
Network Operations Centers Training Programs
IU/Global NOC

60. Training Program Example
From I-Light Service Desk Training Session
There are three main components to what
we call I-Light:
Layer 1 fiber optic network
Layer 2/3 network on the fiber
Copper based circuit network
9/11/2007
Network Operations Centers Training Programs
IU/Global NOC

61. Training Program Example
From I-Light Service Desk Training Session
The I-Light NOC will be monitoring and supporting TDM border routers at universities, and taking calls from their primary technical contacts. However, calls will sometimes be from end users.
The dmarc for all of I-Light is the customer facing port on the last piece of equipment I-Light owns. TDM customers are used to IHETS helping troubleshoot beyond that dmarc.
9/11/2007
Network Operations Centers Training Programs
IU/Global NOC

62. Training Program Example
From I-Light Service Desk Training Session
Most of the universities have Cisco 7206 VXRs, Cisco 2620 XM routers, or possibly 1700 series routers. These routers use several different versions of IOS. Some have QOS, some don't. However, all commands should be uniform.
9/11/2007
Network Operations Centers Training Programs
IU/Global NOC

63. Panel Contact Information
Linda Hornung
PNWGP
Steve Peck
Indiana University/Global NOC
Kevin Schmidtke
WiscNet
Cindy Abercromie
CENIC
Kika Barr
WiscNet
9/11/2007
Network Operations Centers Training Programs