1. Hawaii IPv6 Deployment Experiences Webcast for IPv6Hawaii.org (IEEE GLOBECOM 2009 IPv6 FORUM, November 30, 2009) U. Hawaii Chief Internet Engineer President, IPv6 Forum Hawaii alan.whinery@ipv6hawaii.org Alan Whinery

2. 2 Acknowledgements Antonio Querubin –Systems Engineer, Lavanet Bill Becker –Network Administrator, Honolulu Community College –Pacific Center for Advanced Technology Training Latif Ladid –President, Global IPv6 Forum David Lassner –U. Hawaii VP for Info Tech UH ITS Network Team

3. 3 IPv6 is not a “project” We don't have an “IPv6 person”, or an “IPv6 team”, or an “IPv6 initiative”. It is our policy to deploy IPv6 where we deploy IPv4 As upgrades or maintenance or changes are scheduled, IPv6 is on the to-do list.

4. 4 IPv6 is not a “project” Start now Don't forklift Consider IPv6 in the course of your design and purchasing decisions Work toward including IPv6 in what you do.

5. 5 Most things support IPv6 Now Clients –Windows (XP,Vista,7) –Mac OS X Router/Switch –Cisco, Juniper, Brocade (Foundry) Server –Linux, Solaris, Win2003/2008, MacOS –Apache, BIND, Postfix, Sendmail

6. 6 UH System Network 18 Campuses and learning centers on 6 islands ~50 separate research/outreach facilities ISP for –State Government –Bishop Museum Internet2 connector for –Dept. Of Education –NOAA NWS and NMFS

7. 7

8. 8

9. 9

10. 10 U. H. IPv6 History January 2004: “turned on” peerings with 3 RENs IPv6 Demos to Korea/Japan in 2004 ITS network engineers and Honolulu Community College v6 enabled ever since Addressing moved from “provider” to self in December 2007 Turned on State-wide OSPFv3 and IPv6 addressing in 2008

11. 11 U.H. IPv6 History Initial campus deployment (2004) was through Cisco 7500 running an experimental IOS load –(Juniper supported in main release from 2003) Distribution to user vlans was through a trunk into (inter-)campus VLANs Although operational DNS servers running BIND were capable early, we waited for Infoblox appliance to do IPv6 records –For user interface

12. 12 U.H. IPv6 History Support for Cisco Catalyst 6500 was late in the game Our production, mission-critical IPv4 multicast infrastructure had us running non- standard IOS in various places Cisco 3550's are still not capable, but we anticipate IPv6 and other factors will drive replacement of those devices

13. 13 Watershed Moments Acquired address block 12/2007 (2008) Long-awaited Cisco IOS Catalyst 6500 and 3750, permitting us to “go native” on most core infrastructure In June 2009, we added IPv6 to our TWTC peering, which made “commodity” IPv6 viable –Prior paths to commercial sites were circuitous –Netflix “Instant” over IPv6 works well

14. 14 Current UH IPv6 Deployment Status DNS Stores and answers forward and reverse IPv6 records –Forward currently served in IPv4 packets –Reverses currently served in IPv4 and IPv6 State-wide, 99% of facility gateway routers have IPv6 3 campuses have deployed IPv6 in some form –Manoa, Honolulu CC and Maui CC UH Manoa has native v6 turned on to all user segments

15. 15 v6 Peerings UH –TW Telecom –Pacific Northwest GigaPoP –DREN (Hawaii Intranet Consortium) –Hawaii Internet eXchange –AARNET (Seattle, Sydney, [LA]) Lavanet –Hurricane Electric –Sprint –AT&T –Verizon Business –TowardEX –Hawaii Internet eXchange

16. 16 Lava.net Status Multiple v6 peers network core (unicast and multicast) public servers DNS hosting Lavanet web site mail (SMTP/IMAP/POP) NTP most office workstations DSL or frame-relay (on request)

17. 17 UH Short Term Goals DNS –workflow DNS Address management –full services in IPv4 packets –full services in IPv6 packets Dual stack all IP routing –Currently @ ~ 99% Dual-stack all public-facing services –Currently @ ~ 5%

18. 18 UH Short Term Goals Provide v6/SLAAC to dual stack clients Provide Tunnel endpoints for 6to4, Teredo, ISATAP –Currently have in-state 6to4 with Lavanet –Teredo is harder to affect –ISATAP is a “good idea”

19. UH Medium Term Goals Develop single stack v6 capability DHCP6 Translation Proxying Move email infrastructure to IPv6

20. 20 Mail

21. 21 Mail Careful planning will be necessary Training the email sysadmins will be necessary When email is not flowing, a crisis is occurring

22. 22 The Big Island Router Memory Thing ($$$) Problems on the south REN path from Mauna Kea Observatories on the Big Island of Hawaii –Cisco 3750's with BGP tables of ~12,000 prefixes –Adding IPv6 overwhelmed the fixed-config memory –We were required to re-design and buy an extra Juniper M7i to support v6 there This would have occurred regardless of IPv6 –This is an expense to expedite, not simply to deploy

23. 23 Cost IPv6 is not value-added software –Cisco now has “feature parity” –Juniper has stopped charging for it Most of our costs, Lavanet's costs are in staff time and training. Lavanet has participated in Opensource projects and contributed IPv6 code Cost can be controlled if you simply place IPv6 on your requirements list, start requiring it, and don't panic The Big Island router memory re-design is so far the highest-cost IPv6 deployment measure (by far).

24. 24 List of Problems: Native IPv6 Deployment To User Networks Honest: not a single one.

25. 25 UH Client OS Distribution Volume of HTTP GETs categorized by User-Agent

26. 26 Out-Of-Box V6 Readiness

27. 27 Tunneled Forms of IPv6 Teredo (Significant incidental traffic) –Included with Windows XP, Vista, 7 –Used from behind NAT device (real/virtual) 6to4 (some incidental traffic) –Included with Windows/Mac OS –tunnels via well-known address (i.e. 192.88.99.1) ISATAP (?? traffic) –Included in Windows, some Cisco IOS –Tunnels through guessable domain name

28. 28 Tunneled v6 In The Wild Sources of incidental 6to4, Teredo seem to be applications which require IPv6, e.g. P2P clients –Teredo can be used as an indicator of NAT –There may be more insidious things present Setting up local tunneling services can mitigate cost and issues for tunneled clients Native IPv6 deployment should stop 6to4, but Teredo will persist from behind NAT Un-managed tunnels can represent increased attack surface and firewall by-pass.

29. 29 UH Teredo Traffic All clients use one of three Teredo servers: –207.46.48.150 (Microsoft Asia) –213.199.162.214 (Microsoft Europe) –65.55.158.80 (Microsoft USA) NAT causes Teredo traffic Virtual machine NATs cause Teredo traffic Exceedingly complicated Presumably initiated by an application install

30. 30 SNMP Cisco devised interim MIBs –Which persist in our current IOS Cisco's MIB support notes are fiction CISCO-IETF-IP-MIB

31. 31 Graphing v4/v6 The old MRTG model of graphing interface Octet-counts doesn't do per protocol accounting Various non-optimal things can be done –ACLs feeding counters, etc The following graphs were by using 8 “bpf” counters fed by individual filter expressions –No packet was examined –Not a scalable approach Data represents 1 day on our TWTC v6/v4 peering

32. 32

33. 33

34. 34

35. 35 Performance High-throughput transfers can exhibit throughput differences between v4 and v6 Usually because v6 is being processor- switched (configuration?) v4 and v6 paths to a resource often differ. We have been happy if it worked at all –Now we need it to perform well The time for optimization is at hand.

36. 36 Comparing v6/4 paths (UH)

37. 37 Comparing v6/4 paths (LavaNet)

38. 38 Every Firewall, ACL,etc Web server access controls acls firewall setups PHP code to return restrict content based on IP address MaxMind GeoIP –is v6 capable

39. 39 IPv6 Deployment Scenarios Laissez-faire (ignoring your destiny) –Your resources will be unreachable via IPv6 –external IPv6 resources will either be unreachable or tunneled per client You probably have significant tunneled traffic now –Your existing IPv6 traffic will be high latency, poor performance Possibly without the end-users' knowledge, they will simply blame you for bad performance Or they will blame IPv6 and turn it off –Requires IPv4 addresses “I don’t believe the v6 transition is occuring” –Means “I choose denial instead of participation”

40. 40 IPv6 Deployment Scenarios IPv6 Only ☺ - Client support sketchy - Translation necessary to reach IPv4 Internet + Some value in enabling v6-only servers Dual-stack ☺☺☺ + Client support good + No translation necessary + Serves potential v6-only groups - Requires IPv4 addresses

41. 41 Stateless Auto-configuration (SLAAC) Many operating systems have IPv6 turned on by default With SLAAC, if your router interface is using v6, then you are too. You may use v6 without realizing it Your machine determines your IPv6 address, and adds it to the prefix advertised by the router Some OS build the RH 64 bits using the MAC address Others will make up random (currently only Vista and W7) –complicates address accounting/management

42. 42 Getting a DNS Server address Stateless auto-configuration gets you an address and gateway But no DNS server Of course, if you have DNS through IPv4, you will learn v6 addresses through that DNS server Currently, the only way for a v6-only host to auto-learn the name server address is DHCPv6 Attachments to SLAAC are proposed –RFC 5006 (IPv6 Router Advertisement Option for DNS)

43. 43 IPv6: Apple OSX 10.4+ On by default Missing DHCP6 Can't specify v6 address for networked printer, because the preferences pane for printer set-up considers a colon ‘:’ as preceding a port number (? 10.6) –Printer can, however, be specified by name

44. 44 Apple OS X Applications Firefox – once required v6 “turn on” –This seems to have changed Safari – does browse IPv6 ping – works with separate “ping6” traceroute – works with separate “traceroute6” SSH client – works telnet – works to router: fe80::209:7bff:fedc:400%en0 email – no server to test to yet

45. 45 IPv6: Windows XP (SP2+) You can add it to an interface with the interfaces “Properties” pane, just like IP(v4) or IPX/SPX or NetBIOS Once added, there is no GUI config, although some things can be accomplished with the command line Will not do DNS queries in IPv6 packets Will receive IPv6 info from DNS in IPv4 packets Is Ultimately doomed.

46. 46 Windows XP Applications Firefox – will browse IPv6 IE7 – will browse IPv6 ping – works –Tries first address as returned by DNS tracert – works –Tries first address as returned by DNS Telnet – doesn’t appear to work Thunderbird – no server to test to yet

47. 47 IPv6: Windows Vista and 7 On by default Does DHCP6 There have been some problems –Passing of ICMP6 messages to applications

48. 48 Windows Vista Applications Firefox – will browse IPv6 IE7 – will browse IPv6 ping – works –Tries first address as returned by DNS tracert – works –Tries first address as returned by DNS Telnet – doesn’t appear to work Thunderbird – no server to test to yet

49. 49 IPv6: Ubuntu 8 On by default Does DHCP6, if you install it Since Linux (and BSD OS) are typically used for reference implementations, support is pretty good

50. 50 Ubuntu Linux Applications Firefox – will browse IPv6 ping – works as “ping6” traceroute – works as “traceroute6” Telnet – doesn’t appear to work Linux is a kernel. –Linux distributions are operating systems. They differ as to what apps they provide for various roles. –“Distributions” means, Red Hat, Ubuntu, Suse, Debian, Slackware, etc.

51. 51 Steps To Dual-stack IPv6/(4) Deployment Get addresses Configure routers Configure DNS Configure public-facing services (web/mail/etc) Configure clients –Probably only necessary to the extent that you have Windows XP

52. 52 Steps to single-stack IPv6 Deployment Get addresses Configure routers Configure DNS (in v6 only) Configure public-facing services (web/mail/etc) Provide gateway to v4 Configure clients –Need DNS server entry –Manual or DHCP

53. 53 IVI V6 to V4 gateway Implementation of Internet Draft From CERNet and 清華大學 (Beijing) License unclear Involves patches to out-dated kernel (2.6.18) –Which doesn’t compile under current libc/gcc I have seen it work well, in February 2009, at Joint Techs, Texas A&M

54. 54 Trying Out Your IPv6 It’s hard to know whether you are using it. –ShowIP add-on for Firefox helps –But it isn’t perfect When the OS provide resolution and connectivity –The applications still may Or may not

55. 55 Perl Programming Can't simply handle addresses as integers anymore, without a 128-bit data type. –Math::BigInt –Net::IPv6Addr (eats RFC 1884 addr format)

56. 56 pcap filters expressions tcpdump udp and 'udp[16:4] = 0x20010000' –capture packets from Teredo clients tcpdump net 20C0:FFEE::/32 –capture for the specified 6 net tcpdump ip6 –capture all ipv6 traffic tcpdump proto ipv6 –capture ipv6 encapsulated in IPv4 OR IPv6 Used to specify packet captures: with tcpdump, wireshark, ngrep, etc

57. 57 Dirty Tricks: OK! You can direct the DNS AAAA record for an existing IPv4 services to a separate device –Use Apache as a transparent proxy to make it look like the content has a v6 address –This is GREATLY simplified if the content has an alternate name –Some scenarios/services can simply use a different host

58. 58 Dirty Tricks: OK! Nothing says that the interface or device that offers services via IPv6 is required to be the same as the one that offers those services over IPv4

59. 59 Multicast For IPv6 Multicast, the capabilities of Layer 2 switches become critical –Currently our v6 Multicast is only experimental –We do not have plans to migrate our multicast applications

60. 60 What can I reach with IPv6? More and more. See http://ipv6hawaii.org “Things You Can Reach With IPv6”

61. 61 Hawaii IPv6 Forum –http://ipv6hawaii.org Returning To Work On Monday