IBM Almaden Research Center April 23, 2006 End-User Software Engineering for System Administrators Allen Cypher, Eben Haber, Eser Kandogan USER Group, Computer Science

IBM Almaden Research Center EUSE for System Adminstrators © 2006 IBM Corporation2 Overview Field Studies of System Management Autonomic Task Manager for Administrator/A1: System Management Tool Development Studies on Policy-based Autonomic Computing –Field Studies –Experimental Studies Ongoing and Future Work –System Simulation Studies –Activities for IT Management

IBM Almaden Research Center EUSE for System Adminstrators © 2006 IBM Corporation3 System Management Today Complexity Large scale operations, complex layered IT infrastructures, variety of system interactions, many system management tools, various notions of configurations, permissions, etc. Risk Contractual obligations, high availability demands, unacceptable loss of data, dependence of modern life Cost People cost dominating over hardware and software, obstacle to future technology America Online 6 August 1996 outage: 24 hours Maintenance/Human Error Cost: $3 million in rebates NYSE June 8, 2001 >1700 stocks stopped trading for 90 minutes Software Upgrade We upgrade and patch systems all the time. We have about 1,000 production systems that are Sun, HP, Linux, and Dell."

IBM Almaden Research Center EUSE for System Adminstrators © 2006 IBM Corporation4 Field Studies of System Management Web Hosting, Data Management, Operating System, Security, Storage, Data Center Operations 14 Visits, 6 sites Surveys (~ 100 people) Observations (~ 50 days) Video (~ 300 hours) Interviews (~ 30 people) Diary (~ 10 months) Qualitative and quantitative analysis Data Management Poughkeepsie 3 Days Web Hosting Boulder 3 Days + 1 Eve Web Hosting Southbury 1 Week Web Hosting Southbury 1 Week Data Management Charlotte 3 Days Web Hosting Boulder 1 Week Storage Boulder 3 Days Security Urbana 1 Week Operating system Boulder 3 Days Security Urbana 3 Days Storage Greenbelt 3 Days Storage Greenbelt 3 Days Data Center Boulder 3 Days Data Center Boulder 3 Days

IBM Almaden Research Center EUSE for System Adminstrators © 2006 IBM Corporation5 A Case from the Field How can I put a number and date on the same line?

IBM Almaden Research Center EUSE for System Adminstrators © 2006 IBM Corporation6 How can I put a number and date on the same line? Task: Resolve customer issue so that problem does not repeat itself for three consecutive days People: Rob (web admin), Jack (architect), Andy (operating system), Managers, Tech Support, etc. Problem: Intermittent connection spikes between web server and application server that lead to unresponsive application

IBM Almaden Research Center EUSE for System Adminstrators © 2006 IBM Corporation7 The Crit Sit

IBM Almaden Research Center EUSE for System Adminstrators © 2006 IBM Corporation8 Troubleshooting intermittent, multi-system problems is hard. Sysadmins build tools to increase shared situational awareness. Tool building is a collaborative effort too. There is a variety of tool building expertise among admins. Existing scripting languages aren't aimed at non-programmers. Scripting languages dont handle input & output and errors very well. How can I put a number and date on the same line?

IBM Almaden Research Center EUSE for System Adminstrators © 2006 IBM Corporation9 Issues in Tools and Tool Building Diversity: Every shop is different with its own processes and infrastructure. Vendor tools sometimes are just not doing it. Integration: There isnt one tool that does all the things I need to do. Every vendor provides its own management utilities. Transparency: I tend to learn the guts of things in a CLI as close to the heart of the matter as possible, and then translate it into a GUI 24x7 Coverage: We have scripts that monitor our systems and notify us when something goes wrong. We cant be on the job all the time! Trust: I prefer the CLI. These tools seem to be the most truthful and accurate for administration. Efficiency: I need to reset 150 passwords every week! Risk: We cant afford to make mistakes as we move from test to staging to production servers.