Remote Network Labs: An On-Demand Network Cloud for Configuration Testing Huan Liu, Dan Orban Accenture Technology Labs

Configuration is hard A few reasons –Many SW images for each box IOS: 5+ trains, 8 packages for routers, 5 packages for switches –Primitive CLI interface –Commands may behave differently a design may work on paper, but not in practice –Local configuration at a switch/router could have global impact Sample manifestations –More outages are caused by operator errors rather than equipment failures 1,2 –3 in 4 new BGP prefix advertisements are results of misconfiguration ( prefixes/day) 3 –Average 7.17 to 9.63 errors per firewall config 4 1.Configuration management delivers business resiliency, The Yankee Group, Nov D. Oppenheimer, A. Ganapathi, and D. Patterson, Why internet services fail and what can be done about these, in Proc. USENIX USITS, Oct R. Mahajan, D. Wetherall and T. Anderson, Understanding BGP Misconfiguration, SIGCOMM A. Wool, A Quantitative Study of Firewall Configuration Errors, Computer,

Current solutions and their problems Solution 1: Simulator (e.g., RouterSim, OPNET) –Cannot capture all aspects of real hardware (e.g., many IOS images) –Support limited # of commands –Have simulation models for a limited number of routers Solution 2: Emulator (Dynamips) –Limited set of interfaces are simulated –Support a limited set of Cisco routers Solution 3: Build a smaller scale network in Lab to mimic the production network, and test in lab before roll out –Costly investment (routers are not cheap) –Poorly utilized (only during testing), yet necessary in case configuration changes again –May not be available when you need it (spare parts problem) –Time consuming to wire up –Hard to make sure the test network is the same as designed on paper. 3

What your lab looks like?

Our solution: Remote Network Labs Setup a virtual lab, then test –Equipment could be anywhere (no longer needs to be co-located) –Design from anywhere (no longer needs to be physically on site) –Quickly and easily reconfigurable Advantages –Enable efficient sharing of equipment between projects No more procurement delay Essential zero cost to each project No need for physical lab space –Enable design from a Web browser Everything digitized, including the wiring –Save/restore legacy environment –Fully automate lab set up No physical work to mount equipment and wire Reduce travel (go Green) 5

Distributed architecture Internet San Jose Chicago Client site Web browsers 6

How it works Internet Interface SW Interface SW When press Deploy button, send instruction to build tunnel Front endBack end 7

Making a router/firewall/server part of the lab inventory Internet Interface SW 8

User defines mapping 9 Copyright © 2007 Accenture All Rights Reserved.

Key differences from prior work 10 Copyright © 2007 Accenture All Rights Reserved. Real routerDistributedWire EmulabNNL2 tunnel PlanetLabNYL3 ONLNNL2 tunnel VININYL3 tunnel WAILYNReal wire RNLYYL2 + L3 tunnel

Use case 1 (layer 2 config) Configuring failover using Cisco Catalyst 6500 switches with a Firewall Services Module (FWSM) Can also capture transient behavior 11 Copyright © 2007 Accenture All Rights Reserved.

Use case 2 (test automation) Full automation (from setup to teardown) –Implementing web services API –Implementing library for traffic generation/capture, CLI parsing. Add delay/jitter Inject/observe anywhere 12 Copyright © 2007 Accenture All Rights Reserved. Inject Observe Add delay/jitter WAN

Challenges/drawbacks Additional delay WAN bandwidth is not free L2 interface diversity Performance testing (see paper for more details) 13 Copyright © 2007 Accenture All Rights Reserved.