Addressing a Globally Distributed Network The Future of Testing at Akamai Polish Network Operators Group 2013 Thomas Higdon – Akamai Technologies

©2013 AKAMAI | FASTER FORWARD TM A little about me Software engineer at Akamai in the platform infrastructure team We develop the software that goes on every Akamai server out there (well over 100,000).

©2013 AKAMAI | FASTER FORWARD TM Some statistics Accelerating Daily Traffic of: 10+ Tbps 25+ million hits per second 1.5+ trillion deliveries/day 50+ petabytes/day 10+ million concurrent streams 15-30%+ of Web Traffic Delivering 250,000+ Domains 5,000+ Customers 9 of the top 10 Banks 9 of the top 10 Social Media Sites All top 30 M&E Companies All top 60 eCommerce Sites A Global Platform: 132,000+ Servers 86 Countries 650+ Cities 1,150+ Networks 2,200+ Locations

©2013 AKAMAI | FASTER FORWARD TM A little about me I’m addressing the problem of how to get new platform software out there faster, and with a smaller likelihood of disruption We’re using…

©2013 AKAMAI | FASTER FORWARD TM

Akamai in a nutshell Akamai is a globally-distributed cloud of physical servers Machines at Akamai are grouped into functional units called networks. Edge networks - CDN services, object/file storage, analytics, etc. Infrastructure – distributed file transfer, messaging, monitoring, reporting, etc.

©2013 AKAMAI | FASTER FORWARD TM Akamai in a nutshell Each network runs a different set of software that depends on the role of the network in the ecosystem. Each also runs a common set of platform software Generally OS software and Akamai-specific File transfer MessagingMonitorin g Network-specific software OS

©2013 AKAMAI | FASTER FORWARD TM Akamai in a nutshell Infra network File transfer MonitoringMessaging Operating system/kernel Edge network Network-specific software

©2013 AKAMAI | FASTER FORWARD TM Akamai in a nutshell Infra network File transfer MonitoringMessaging Operating system/kernel Edge network Network-specific software File transfer MonitoringMessaging Operating system/kernel Network-specific software File transfer MonitoringMessaging Operating system/kernel File transfer MonitoringMessaging Operating system/kernel File transfer MonitoringMessaging Operating system/kernel

©2013 AKAMAI | FASTER FORWARD TM Akamai in a nutshell Infra network File transfer MonitoringMessaging Operating system/kernel Edge network Network-specific software Edge network Network-specific software Edge network Network-specific software

©2013 AKAMAI | FASTER FORWARD TM Akamai in a nutshell State of a network Installed software and configuration Dynamic configuration/metadata Persistent (across installation) filesystem state Applied manual changes Each network tends to have built-up state necessary for proper functionality not well-documented or defined

©2013 AKAMAI | FASTER FORWARD TM An Akamai “instance” A given unique set of Akamai networks that function together Self-contained with limited external dependencies

©2013 AKAMAI | FASTER FORWARD TM Akamai “instances” Production – 10 5 SQA – 10 2 Dev – 10 2

©2013 AKAMAI | FASTER FORWARD TM Commoditizing an Akamai instance Instances are labor-intensive to manage We’d like the ability to create these as a commodity Automatically bootstrapping from scratch is possible, but difficult.

©2013 AKAMAI | FASTER FORWARD TM Commoditizing an Akamai instance Solution: virtualization Represent the state of an instance using virtual machines. Save the virtual machines and clone them. SQA – 10 2 SQA copy– 10 2 SQA – 10 2

©2013 AKAMAI | FASTER FORWARD TM Cloning Akamai instances Static IP configuration Isolation traditional NAT “reverse” NAT SOCKS proxy Akamai “authgate”

©2013 AKAMAI | FASTER FORWARD TM Accessing an instance NAT SOCKS authgate / NAT SOCKS authgate / Client

©2013 AKAMAI | FASTER FORWARD TM Managing the Akamai side Private cloud infrastructure is great for managing VMs and their resources We needed another level of management “Akamai” networks Akamai instances Expose specific Akamai machine types and services

©2013 AKAMAI | FASTER FORWARD TM An Akamai instance service Service-oriented Language/test harness agnostic Share common resources

©2013 AKAMAI | FASTER FORWARD TM An Akamai instance service Instance service Private Cloud Orchestrator 2. “Clone these machines” 3. Creates resources 4. “Here’s your instance!” 5. “Ok, let’s use it.” Gateway User 1. “Give me an instance”

©2013 AKAMAI | FASTER FORWARD TM A well-managed instance Leverage expertise from around the company to create an internal “master” instance Each additional network gets it closer to a “real” instance. Forking?

©2013 AKAMAI | FASTER FORWARD TM Addressing existing instances Can also address existing instances of physical machines Production Lab instances Akamai Production Instance Lab test instance User Instance service

©2013 AKAMAI | FASTER FORWARD TM Testing at Akamai Unit testingDev QA SQA ChecklistingAlerting (?)

©2013 AKAMAI | FASTER FORWARD TM Unify these stages Dev QA SQA Checklisting Use the same code to write tests by using a common interface to refer to an Akamai instance.

©2013 AKAMAI | FASTER FORWARD TM Unify these stages Dev QA SQA Checklisting Use the same code to write tests by using a common interface to refer to an Akamai instance.

©2013 AKAMAI | FASTER FORWARD TM Takeaways We’re virtualizing Akamai and providing this as an internal service We’re simplifying our testing using the power of the cloud.

©2013 AKAMAI | FASTER FORWARD TM Thank you!