1. 1 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 10 Gigabit Networking at TelstraClear Jörg Micheel

2. 2 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 Outline Some Metro Ethernet equipment in use at TelstraClear Test equipment, testing process Project: University of Auckland 10 Gigabit Campus Ring Measurement results and performance figures Outlook: REANNZ Advanced Network implementation

3. 3 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 Extreme Networks Black Diamond – metro core switch Black Diamond 10808 (BD10K) 22 rack mount units 1280 Gbps capacity Up to 48 10-Gigabit ports Up to 480 10/100/1000 ports Powerful VLAN, Virtual router Layer2 and Layer3 capabilities Proprietary EAPS link-protection protocol provides continuity in case of fiber cut L2/L3 Quality-of-Service L2/L3 hardware filtering and priority Jumbo frames at 9212 High availability, hardware redundancy

4. 4 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 Extreme Networks Summit X450 – edge switch X450-24t with 24 ports 10/100/1000 copper, four combined SFP GigE ports X450-24x with 24 ports 1-GigE SFP, four combined 10/100/1000 ports Optional dual 10-Gigabit Ethernet uplinks 1 RU form factor 160 Gigabits-per-second capacity 65 million packets-per-second forwarding performance Coming: X450a with stacking capability Other features similar to Black Diamond series

5. 5 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 Network test equipment – EXFO PacketBlazer 8510 – wire speed full duplex 10/100/1000 8510G – wire speed full duplex 10G LAN/WAN PHY BERT testing at Layer 1 Frame testing at Layer 2 RFC 2544 Performance test suite for network devices Throughput, back-to-back packets, frame loss and latency, jitter Detailed QoS analysis Can extend concept of device testing to (parts of) a network

6. 6 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 University of Auckland 10 Gigabit campus ring (1) Requirements: Resilient infrastructure to interconnect eight University of Auckland campus sites Provide multiple 1-Gigabit services at Layer 2 between campuses Solution: Dual-implementation with Summit X450 at six sites within Auckland metropolitan domain Dark fiber between the sites Supports multiple unconstraint 1-Gigabit VLAN services with QoS up to the maximum 10 Gigabit backbone capacity Resiliency via Ethernet Advanced Protection Switching (EAPS) SLA: 99.99% end-to-end throughput SLA: 500 milliseconds maximum round-trip-time SLA: 250 milliseconds maximum packet delay variation

7. 7 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 University of Auckland 10 Gigabit campus ring (2) ITSS Symonds St Newton Data Medicine Epsom Tamaki

8. 8 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 University of Auckland – test setup ITSS Symonds St Newton Data Medicine Epsom Tamaki 12 VLANS

9. 9 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 University of Auckland – throughput results 10 Gigabit backbone performance: full line rate for any packet sizes between 128 and 9212 bytes Any single 1-Gigabit loop VLAN: loss free at any packet size 10 loop 1-Gigabit VLAN performance (saturated core, with all services active):

10. 10 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 University of Auckland – delay and jitter results Fiber run length 44.7 km – 218.7 microseconds (RI=1.467) OTD(min) = 258.4 microseconds Switching, buffering accounts for 39.6 microseconds (15% of total) Per switch latency 3.3 microseconds, inclusive serialization RTTD(64) 517.0 microseconds at 100% of 1-Gigabit load PDV(64) 1.2 microseconds  Meets SLA requirements comfortably, after adjustment for additional 10 km fiber spur

11. 11 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 University of Auckland – latency as a factor of packet size

12. 12 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 University of Auckland – EAPS protection switching Measured the impact of a ring break upon site-to-site traffic Worst case scenario – re-routing once around the entire ring Measurements conducted at single site Metric – number of packets lost Worst case timing was measured at 217 milliseconds Typical timing is around 100-150 milliseconds  Exceeds SLA expectations for sub-second failover

13. 13 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 University of Auckland – test summary Installation meets / exceeds original design parameters Network is capable of delivering unconstraint full 1-Gigabit services across the metro domain Fiber spur by far dominates latency (85%) Switching latency is negligible, a few microseconds per packet at most Serialization delay can become the dominating factor with large packet sizes and many hops  Consider the QoS impact when designing applications for either bulk data transfer or sensitive audio/video conferencing applications

14. 14 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 REANNZ Advanced Network 10 Gigabit backbone overlay on TelstraClear DWDM Gigabit access in all major cities Layer 2 network with QoS Routers in Auckland, Wellington, Christchurch International access: 155 Mbits/sec to Sydney, 622 Mbits/sec to California Rollout in 2006 and 2007

15. 15 NZNOG 2006 – Victoria University Wellington, March 23 rd, 2006 Acknowledgements and references University of Auckland ITSS department – Robert Beattie, Manager REANNZ – Charles Jarvie, CEO A cast of dozens of hands at TelstraClear and JazzTech Thank you!