1. “Internet2 Presentation”
Brian Stengel, Director of Operations, Kinber
Peter J. Heverin, Project Manager, Kinber
Mike Carey, Network Engineer, Kinber
Jon Paul Herron, Director of Engineering, GlobalNOC
Luke Fowler, Senior Manager, Systems Engineering, GlobalNOC
Marianne Chitwood, Director of Operations, GlobalNOC
September 20th, 2012

2. What is PennREN Capital Budget - $128,958,031
a high-speed, state-wide, research & education serving healthcare, K-20 and the public good
Capital Budget - $128,958,031
Federal Stimulus Funds - $99,660,678
Matching Funds - $29,297,353
Outside Plant Infrastructure Constructed for PennREN
48 Stands of NZD Fiber optic Cable
1700± Route Miles
Outsourced Fiber Maintenance
13 Optical Regeneration Service Nodes
56 Service Distribution Access Nodes
Peter talk about PennREN and KINBER, grant process, history

3. Project Route Total PennREN Fiber route is estimated at 1,613 miles.
The route consists of the following;
1,086 miles of new aerial construction.
486 miles of aerial overlash.
14 miles of new underground construction.
27 miles of leased underground conduit.
Network Backbone Engineering is 95% complete overall.
Peter talk about outside plant 3

4. Project Route
Peter talk about route 4

5. PennREN Locations Service Nodes Amp Nodes Access Nodes
Lehigh University
401 N. Broad
Penn State Hershey Medical Center
Indiana University of Pennsylvania
Allegheny Center Mall
Slippery Rock University of Pennsylvania
Penn State Erie (Behrend)
Clarion University of Pennsylvania
Penn State Dubois
Penn State – State College
Windstream
Bucknell University
University of Scranton
Amp Nodes
East Stroudsburg University of Pa
Kutztown University of Pa
West Chester University of Pa
Millersville University of Pa
Shippensburg University of Pa
Pitt Johnstown
Community College of Beaver Co
Allegheny College
Pitt Titusville
Penn State Hazelton
Access Nodes
46 Locations on PennREN fiber
Peter talk about the locations

6. Project Teams Fiber OSP Building entrances Cabinets, Installation
Splicing
Testing
Last mile solutions
Hardware procurement
Design
Integration
Project Management
Professional Services
Plan/Design
Optical engineering
Field services
Professional services
Staging
Shipping
Configuration
Testing
Turn-up
Field Services
Plan/Design
Consulting
Engineering
Support
Training
Service Desk, Tier 1, Tier 2
Systems and Network Tools
Network Engineering
Net, Perf, Change Management
Operations support
Peter talk about the RFP for OSP and for network equipment
Earthlink sourcing for OOB MPLS Network
Last mile solutions

7. Node Installation Status
Peter talk about the status of equipment installations and transition to Mike
Segment in Production
Equipment Installed
Under Construction

8. Co-Location at Member Sites
Construction Diagrams
Mike to start here talk about meeting with members to negotiate space, power, etc… for putting in equipment. Talk about the greenfield opportunity to design from the ground up….

9. Optical Core/DWDM System
Talk about the selection of the core DWDM system – ADVA equipment

10. Packet Core – MPLS
Talk about the selection of the MX480

11. Plan – Build – Integrate
Talk about our approach of taking the Torrey Point design, ADVA/Juniper BOM and working with the various teams to take design to plan. Then moving from plan to build and then integration. Talk about using GRNOC systems as the unit of record and that our Google sites and collaboration tools free for us and not very structured… so we needed GRNOC structured systems for final unit or record.
Talk about coordination with teams and how GRNOC provides support in PM, NE, and SD

12. Peering Points (Initial)
Talk about how we will deliver external network connectivity and how GRNOC is helping us with this
Internet
Internet
R&E
R&E

13. Network Management
PS1
Performance Measurement Servers at every Service Node
PS2
PS3
In-band management
Deploying performance measurement servers at each service node
3 servers per site:
Active throughput measurement (1G and 10G)
Active latency measurement
local data collection / ad hoc performance measurement
Specific performance measurement tools include:
BWCTL
OWAMP
MaDDash
GlobalNOC tool-set updated to support all PennREN devices, including:
SNAPP – High-resolution SNMP-based network utilization data
LLAMA – DWDM layer performance measurement data
Central storage of other passive data like syslog, configuration, flow, etc. for regular and ad-hoc processing & analysis
Out-of-band access
External Networks DB
External Networks

14. Operations - Support GRNOC provides to KINBER:
Service Desk – 24x7x365 call center support, ticket management, technical support coordination, and workflow support
Network Engineering – Expert network engineers work with the Service Desk to ensure fast problem resolution, provisioning, and strategic engineering and planning
Software and Systems – Provides support through a fully integrated system of network management, measurement, and visualization tools
The GlobalNOC at Indiana University provides carrier-grade operations, tools, and network expertise while placing a singular focus on the unique requirements of the research and education (R&E) Community
GRNOC supports 20+ R&E networks across the country
Talk about the GRNOC people as being their best asset and the tremendous job they are doing for us… allowing us to keep our costs low and exploiting the experience of GRNOC to bring this network up and into production.

15. Operations - Maintenance
Warranty
Hardware replacement (pre-ship)
Software maintenance
Fiber Maintenance
Emergency Restoral
Routine Maintenance
OSP Records
Warranty
Hardware replacement (pre-ship)
Software maintenance
Host IT/Site teams provide remote eyes and hands support upon request
Talk about how GRNOC will help us manage maintenance on the network, change management, incidents, etc…
KINBER Network Engineers

16. Foundation for Services
Multi-Degree
ROADMs
MPLS PE Switches
3rd Party Providers
West DWDM Ring
East DWDM Ring
Our foundation for services
ASBR Router
External Networks
External Networks

17. On-Net: Delivery Access Nodes Access Node Access Node Access Node
Talk about how our standards will apply to access nodes once we start installing equipment
Access Nodes
Internet
Internet
R&E
R&E

18. Off-Net: Delivery First/Last Mile NNI Commercial Co-Lo Member Off-Net
Carrier CX
First/Last Mile
Commercial Co-Lo
Private
Member Off-Net
NNI
Talk about how GRNOC will help us
Internet
Internet
R&E
R&E

19. EPC – Ethernet Port Connection Service
PennREN
R&E networks
VPLS (pt – pt, multipoint)
Customer Router
KMEX
Service Node: PE Switch
Commodity Internet
Customer Subscribes to 1/10GE Ethernet Port(s)

20. Services Services available to a customer with an EPC Member B
Member A
KINBER Member Exchange
R&E Transit
VPLS – Pt-Pt, Multipoint
Commodity Internet
Member C
Member A

21. Services KINBER Member Exchange R&E Transit Commodity Internet
Community-wide, distributed service for members to exchange traffic across a common network. Similar to an Internet Exchange
Member-to-Member peering, ad-hoc R&E activities
Best-effort traffic exchange within the community
R&E Transit
Transit service to major R&E networks such as Internet 2, ESNet, NLR…
Provided by KINBER affiliates
Virtual private networks with committed bandwidth can be established using VPLS instances in the PennREN network
VPWS – Virtual Private Wire Service – Point to Point
VPLS – Virtual Private LAN Service - Multipoint
VPLS – Virtual Private LAN Service (Point-to-Point Multi-Point)
Commodity Internet
Access to commodity Internet service is available over the PennREN network

22. Optical – Wave/Lambda Services
Optical Waves 10G can be provisioned across the network
Member A
Member A
You may want to discuss plans for GMPLS and how we can use NM and SM in coordination with GRNOC.
Member B
Member A
Member B

23. Performance Measurement
PerfSONAR Measurement Archives allow exchange of data with other network operators
Regularly scheduled testing across the backbone
User-initiated testing for applications such as problem diagnosis
Multiple routing tables allow us to support both 1G and 10G testpoints on a single host

24. Performance Measurement
Deploying performance measurement servers at each service node
3 servers per site:
Active throughput measurement (1G and 10G)
Active latency measurement
local data collection / ad hoc performance measurement
Specific performance measurement tools include:
BWCTL
OWAMP
MaDDash 24

25. Network Measurement
GlobalNOC tool-set updated to support all PennREN devices, including:
SNAPP – High-resolution SNMP-based network utilization data
LLAMA – DWDM layer performance measurement data
Central storage of other passive data like syslog, configuration, flow, etc. for regular and ad-hoc processing & analysis.

26. PennREN Service Desk GlobalNOC Specialized Support Technician
Footprints PennREN project created – integrated to TickMon, Operations Calendars, Trouble Tickets
Telephone number for PennREN customers integrated into shared GlobalNOC phone queue
established to receive customer inquiries and or communications from vendors….this is monitored 24x7x365

27. PennREN Service Desk Network and Member impact guidelines defined
Web form for customers/vendors to submit trouble tickets into Footprints
Change management process/form created
Internal documentation developed/published for staff training and reference

28. PennREN Service Desk Pro-active network monitoring
Support for scheduled maintenances and changes
Vendor coordination
Customer install process
Reporting
Security
Tools and Communications

29. Weighing network needs
Early Service Definition can guide the network plan in the right way.
Projects are often looked at from the perspective of competing demands of cost, time, and quality. Networks have something similar, with Cost, Time, Capabilities, and Reach:
Cost & Time: pretty obvious
Reach: How BIG is the network? What all needs to be connected?
Capability: Speed, Service Level expectations, AND some idea of the kinds of services & features it will need to have day 1, and day 1826
For PennREN, Cost and Time were mostly dictated by the award. Reach was also given high priority by the award. Capability and service were the area of most planning. Given the short time, the set cost, and the scope of the network, what services and capabilities were possible, and how do they maximize this as much as possible?

30. A new network is more than a construction project
It’s a system:
Goals
Infrastructure
People
Services
Operations
Business
Make decisions and plans early (also, there will be more decisions than you think, so watch for bottlenecks)
The work on all of these areas starts right away
We’ve found that new network builds often seem to focus so heavily on the build that all the other things get postponed until later. But really, a new network will be a larger system, with the infrastructure but also the services, operations, and business models that all need to be pointing the same direction, toward the network goals we mentioned before.
In PennREN, everyone involved has been working hard to make sure that happens throughout the project.

31. Service Definition
Unrealistic to imagine services will be completely defined from the beginning of the design
But, enough understanding is needed to guide the design and build plan
Early “anchor” users help a lot!
In PennREN’s case, it was enough to know that “off-net” connections for commodity and Internet2 access were needed, and that there was a need for multipoint, point-to-point L2 services
This helps to guide not just the network plan for what to buy, what protocols are needed, etc, but also the Operations plan. This includes what will need to be monitored, who the providers will be, who the users will be, how to define services in the database, and more.

32. Communications Communication needs: to be high-bandwidth Early is good
Changing staff can be disruptive, do it carefully.
Informal is good
Multi-channel is good
reliable technologies are good
Face to face is good
Build-out like PennREN is big, and involves lots of organizations: fiber building group, administrative group, NOC, integrator, etc
Everyone knows communication is important to do something like this. We found that this communication was most effective as the relationships grew, and the people understood the total context of the project and how best to work together. Getting to that point fast is important. Face to face helps.
Changing people involved is disruptive during projects like this. New people have to come up to speed technically, but also have to assimilate themselves into how the group works.
Meetings and formal tracking are important. But equally important are the informal communications at technical levels. Using chatrooms was immensely helpful for technical staff from all the organizations to get the real details done.

33. Documentation Documentation: Have a place for documents
Keep the place for documents clean
Keep the purpose of each document clear/distinct

34. Operations Operations Preparedness:
Start early, there are a LOT of things to think about, especially:
Turn-up/acceptance process
Expectations for facilities
How and where to keep network data
We find that it makes things much easier to be thinking about operating the network throughout the build. For example:
building a network produces a LOT of data. Think about how you might need to use that data for troubleshooting, maintenance, or upgrades, and put it somewhere where it’ll be easy and fast to find when you need it.
the number of small, but long-lived decisions is more than you think. How are you identifying circuits? How are you defining individual services? What documentation of each site is needed? How will you keep track of asset changes due to maintenance, RMAs? What arrangements will you have with facilities for power, remote hands, etc?

35. Build-outs make for lots of information
Need to be ready to put it somewhere where it’ll be usable later
Data entry is cheap
Entering/documenting is best when the information and its context are fresh

36. The lowly management network
Sometimes doesn’t get enough attention
It can get crazy complicated or crazy expensive
NOTE: in an SDN world, this becomes even MORE important!
Maybe more on specific technical recommendations?