1. Network Operator Perspective MPLS: 12 Years After Tom Bechly
IETF 74, San Francisco IAB Plenary March 18, 2009

2. MPLS: A Successful Protocol
MPLS has been and is a successful protocol
From perspective of RFC 5218 (What Makes for a Successful Protocol?), MPLS was used for its intended purpose and at intended scale
Goal was to switch packets to support rapidly expanding global networks
MPLS is “wildly successful” (RFC 5218) in that its use has exceeded its original design goal thru development of numerous extensions
From service provider perspective MPLS was successful in supporting growth, reducing cost, and providing basis for new services
Original goal of bringing Layer 2 switching speed to Layer 3 was accomplished, but somewhat discounted over time due to hardware evolution
L2 was hardware switched and L3 was process switched
MPLS was easily leveraged for traffic engineering, VPNs, and layer 2 transport.
For the service provider, MPLS has become one the most reached for and extended tools in the tool chest (150+ RFCs)

3. MPLS
Customer Edge (CE) Router
Provider Edge (PE) Router/Switch
Provider Core Router/Switch
MPLS CORE
CE Router
PE Router
P Router
Enables network edge routers to apply simple MPLS labels to packets or frames
Forwards packets by swapping labels with minimal lookup
Integrates Layer 2 switching and Layer 3 routing
This diagram introduces how MPLS manages network traffic.
MPLS is a high-performance method of forwarding packets (frames) through a network. It enables routers at the edge of a network to apply simple labels to packets (frames). ATM switches or existing routers in the network core can switch packets according to the labels with minimal lookup.
MPLS integrates the performance and traffic management capabilities of Data Link Layer 2 (frame relay and ATM) with the scalability and flexibility of Network Layer 3 (IP) routing.
MPLS offers additional benefits when applied to ATM networks. MPLS integrates IP routing with ATM switching to offer scalable IP-over-ATM networks.

4. MPLS/RSVP-TE Benefits
MPLS with RSVP-TE provides overall path control in network
Use with constraint based routing
Control over latency and delay variation
Bridges gap between ability to deploy capacity versus current demand in existing network
Use of MPLS allowed gathering measurement statistics on LSPs
Probably more important than actual path control
Provides ability to accurately measure traffic between router pairs
Traffic volumes, latency, and delay variation
Measure traffic between hubs, metros, and regions
Measure asymmetry of flows, over time
A time series depiction can be built to trend traffic for efficient investment and to provide required service
MPLS became an enabler for the development of additional services
L2 VPNs and L3 VPNs

5. Verizon Public IP
AS 701 was initially implemented as an overlay over a dedicated frame relay network
Path control was effected thru manipulating path of frame relay PVCs
As capacity requirements increased, the network was migrated to an overlay over ATM
The cost of this became untenable, as capacity requirements continued to increase
MPLS with RSVP-TE deployed in EMEA (AS 702) in 1999
First deployment of RSVP-TE in production network
Deployed in US (AS 701) in 2000
Deployed for traffic engineering to provide control over path selection that was not available thru L3 protocols
Shortest path algorithm did not always provide optimal route
MPLS technology has enabled the Verizon Public IP network to grow to be one of the largest in the world

6. Verizon IP Network 410 unique switch/router hubs (PoPs)
Our global IP network spans more than 446,000 miles across 150 countries on six continents and is backed by experienced sales and service reps around the globe.
Verizon continues to invest in our facilities-based strategy to deliver the performance and reach to meet our customer’s needs from Small Business to Enterprise to Government to our Wholesale market segments of ISPs and leading content providers.
410 unique switch/router hubs (PoPs)
Six continents, 150+ countries

7. Verizon Layer3 VPN Services: VBNS+ and Private IP
vBNS (very-high-performance Backbone Network Service) was established in 1995
Cooperative research and development agreement between Verizon (formerly MCI) and National Science Foundation (follow on to NSFnet)
Evolved to a commercial product: vBNS+ for gov and edu market
MPLS routing/switching implemented in network in 1999
Initially MPLS was implemented for traffic engineering
L3VPN (RFC 2547) was implemented in 2001
There are approximately 40 nodes in 19 US cities, full mesh of TE LSPs
Verizon PIP (Private IP) was established in 1999
Layer 3 VPN (RFC 4364), wide area network for business customers
Quality of Service, strong SLAs, etc.
Large global network
There are approximately 625 nodes across 162 cities in 59 countries
Uses LDP for label distribution, with partial mesh of LSPs
Designed and supported by Verizon’s next generation engineering organization, it has been in existence for more than ten years; MPLS has been a principle switching technology within the vBNS+ platform for the past five years. The vBNS+ has a proven and history of extraordinary performance for numerous government clients who rely upon the network for vital/critical applications.
Verizon recognizes that when customers entrust their critical operations to a network service, they are quite literally putting their ability to operate in the hands of the network provider.
We take this responsibility seriously and are proud to report that during its nine years in operation, the vBNS+ network has provided an exceptionally high level of availability to customers large and small.” This is despite the evolution, expansion and upgrade of the entire service platform and all its elements, including the replacement and/or upgrade of every switch, router, and trunk which makes up the network. For example, in 1999, Verizon (formerly MCI) completed the transition from ATM to MPLS as the core switching service in the network without adversely impacting our customers’ service.
In addition to the technical evolution of the physical components of the network, Verizon has continually led the way in delivering new and advanced services to our vBNS+ customers. In all cases, the advanced networking features and high performance of the vBNS+ network allow clients to enable new applications and next generation capabilities that go beyond traditional wide area networking.

8. Private IP Global Reach
This is the complete list of countries where Private IP is currently available.
vBNS+ International coverage is supported by provisioning on Private IP routers with 6 gateways deployed back in the US.
MP10163v5.03

9. Verizon Layer 2 Services: MAE® Services and Converged Packet Architecture (CPA)
MAE® Services established 1992 as metro Internet Exchange point
Evolved into MPLS based national service for extended peering and L2 VPNs (VPWS), implemented in 2002
Service interworking (ATM, Frame Relay, and Ethernet), based on draft Martini pseudowires and draft Shah ARP Mediation
Implemented across public internet within full mesh of GRE tunnels
ISIS, RSVP-TE signaled LSPs, and LDP signaled pseudowires
CPA supports Ethernet access and Ethernet services
L2 VPNs: both EVPL (PWE3) and VPLS (RFC 4762)
Quality of Service, strong SLAs, etc.
Large global network
There are approximately 115 nodes across 27 countries
RSVP-TE used to signal LSPs
Full mesh for EVPL and VPLS
Currently 10,000+ LSPs

10. Lessons Learned
Implementation defects significantly impact early perception of technology
For AS 701, there was internal resistance to moving from ATM underlay network to MPLS
When defects in the MPLS implementation on vendor equipment were encountered these initially viewed by some as defects in the technology

11. Lessons not Learned (VPLS)
RFC 4762: Virtual Private LAN Service (VPLS) Using Label Distribution Protocol (LDP) Signaling
Hierarchy is managed thru HVPLS, specified within RFC
RFC 4761: Virtual Private LAN Service (VPLS) Using BGP for Auto-Discovery and Signaling
Hierarchy is managed thru route reflectors and multi-segment pseudowires
Both approaches are currently in production in different service provider networks
Some vendors have implemented both standards
This increases to overall cost and complexity of technology and network development across the community
Resolution and mitigation of differences is far more economic during protocol development than once into implementation
Gateway function has high development and operational cost
The added costs and complexity are continuously accretive

12. Lessons not Learned (RFC5085 – PW VCCV)
Pseudowire Virtual Circuit Connectivity Verification (VCCV) – RFC 5085
Three modes of operation: (Type 1: PWE3 Control Word Bit, Type 2: MPLS Router Alert Label, Type 3: MPLS PW Label with TTL == 1
Mode is negotiated, so all three are optional
Vendors, to this point, have not implemented all modes nor the same modes
This leads to interoperability issues in mixed vendor networks
Delays significantly availability of feature
Adds to development and integration costs
VCCV Mode
Vendors Y
Vendors X
Control Word*
Yes No
Router Alert Label
TTL Expiry*

13. Continuing Challenges
Latency sensitive customers
These are typically financial customers that are sensitive to a 2ms increase or change in latency
Require traffic to be on path with deterministic low latency
Due to network event traffic may be rerouted, via Fast Reroute and the re-signaled LSP
Paths are recalculated periodically to ensure low latency path
Once optimal path is available, traffic is re-routed (make before break) to this path
As this path could be significantly shorter (2 – 10ms), there will be out of order packets that may impact some hosts
Nodes in network within the core, may carry a high number of LSPs
Latency sensitive customers are requesting notification on any maintenance that will impact LSPs carrying their traffic

14. MPLS Going Forward MPLS has been an extremely successful protocol
It has been widely deployed and extended
MPLS based networks and facilities to continue to grow and expand
This growth is continuing and will continue for some time