Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University

Motivation Downtime and management are expensive for services Downtime can cost millions of dollars per hour [IW00] Management can account for 5080% of IT budgets [Brown05] Operator mistakes are a major source of downtime Cause up to 36% of service failures [Oppenheimer03] Misconfiguration is the most common type of mistake More than 50% of service failures [Oppenheimer03] 24 out of 42 mistakes in a multi-tier service [Nagaraja04]

Understanding Misconfigurations Misconfigurations occur frequently for two main reasons: Servers and services are becoming increasingly complex Operators modify servers configs often, as service evolves via software and hardware upgrades, node additions and removals Example: Apache One config file has 240+ lines relating to performance, structure, modules, and others Config files must change every time a node is added or removed in the 2 nd tier

Our Work Idea: Automatically generate server config files as the system and its workload evolve Goals: Reduce config complexity and operator effort, and generate optimized config files efficiently Contributions: Infrastructure for automatic configuration of services Heuristics for efficient parameter tuning Quantitative evaluation for a realistic service

Outline Motivation Approach and Contributions Automatic Configuration Infrastructure (ACI) Tuning Parameters Evaluation Conclusions

Key Observations Behind ACI Most of each servers config doesnt change Node additions and removals require tedious and mistake-prone reconfigurations Changes to a config parameter affect only a few other parameters

Key Observations Behind ACI Most of each servers config doesnt change Config file templates and generation scripts Node additions and removals require tedious and mistake-prone reconfigurations Changes to a config parameter affect only a few other parameters

Key Observations Behind ACI Most of each servers config doesnt change Config file templates and generation scripts Node additions and removals require tedious and mistake-prone reconfigurations Network of membership daemons Changes to a config parameter affect only a few other parameters

Key Observations Behind ACI Most of each servers config doesnt change Config file templates and generation scripts Node additions and removals require tedious and mistake-prone reconfigurations Network of membership daemons Changes to a config parameter affect only a few other parameters Parameter dependency graph and parameter-tuning heuristics

Global View of ACI (3-tier Service)

Local View of ACI Daemon Scripts + templates Config files Node i Global membership info + tuned parameter values

Outline Motivation Approach and Contributions Automatic Configuration Infrastructure (ACI) Tuning Parameters Evaluation Conclusions

Tuning Parameters Tune parameters for some metric, as service evolves Evolution may cause current config to behave poorly Ex: when throughput is the metric, more memory may allow more threads and higher throughput Problem Hard to know what parameters are affected Hard to select new values for the parameters

Parameter Dependency Graph Idea: Constrain the search space with a parameter dependency graph and value ranges Vertex = parameter, directed edge = dependency After each change, ACI traverses the graph to find the affected parameters and tune them experimentally Ex: if Apache node is upgraded, make Apache parameters sources and tune reachable parameters Overhead of generating graph can be amortized

Generating the Graph 1 st step: find important parameters using CART 2 nd step: find dependencies between important parameters Pair-wise test: parameter A depends on B iff different settings of B lead to different best values for A; check min, mid, max values

Generating the Graph 1 st step: find important parameters using CART 2 nd step: find dependencies between important parameters Pair-wise test: parameter A depends on B iff different settings of B lead to different best values for A; check min, mid, max values A depends on B

Generating the Graph 1 st step: find important parameters using CART 2 nd step: find dependencies between important parameters Pair-wise test: parameter A depends on B iff different settings of B lead to different best values for A; check min, mid, max values A depends on BB doesnt depend on A

Generating the Graph (cont.) An evolution may change the bottleneck tier Dependencies may be affected by provisioning of tiers Ex: if throughput is the metric, different bottleneck tiers may lead to different dependencies Run dependency checking once per tier, each time forcing the tier to become the bottleneck Dependency graph is the union of these results

Tuning = Traversing the Graph In a cycle, tune all member vertices at the same time In an acyclic chain, tune parents first assuming best values for ancestors and current values for descendants A singleton is tuned in isolation Select values using Simplex algorithm and value sets # experiments increases with connectivity, value set size

Summary Find important parameters Find dependencies Tune parameters Normal execution Evolution

Outline Motivation Approach and Contributions Automatic Configuration Infrastructure (ACI) Tuning Parameters Evaluation Conclusions

Evaluation: Performance Tuning 3-tier auction (Apache, Tomcat, MySQL) on 9 nodes; 24 performance parameters Management complexity and mistake-elimination results in the paper Efficiency: Number of throughput-tuning experiments after DB machine upgrade, 2 application server removal

Online Auction Dependencies

Performance Tuning Approaches Exhaustive: O(R N ) runs; R = parameter range and N = # parameters = 24; infeasible Dependency graph + exhaustive: N = 9; infeasible Simplex ACI = Dependency graph + Simplex

Performance Tuning Efficiency ACI needs 2 service changes to amortize the graph overhead EvolutionTuning Approach Throughput (reqs/sec) Number of Experiments DB node upgrade Best prior values343- Simplex search ACI Removal of 2 application servers Best prior values91- Simplex search ACI

Amortizing the Graph Overhead Does dependency graph change when service evolves? DB node upgrade Graph does not change Removal of 2 application server nodes Graph does not change Upgrade MySQL server version Graph is a subset of original one

Conclusions Proposed automatic configuration infrastructure Proposed notion of parameter dependency graph Quantified benefits experimentally for realistic service ACI reduces config complexity, eliminates misconfigs, and produces high-performance services efficiently

Questions?