CIT 470: Advanced Network and System AdministrationSlide #1 CIT 470: Advanced Network and System Administration Debugging

CIT 470: Advanced Network and System AdministrationSlide #2 Debugging 1.Learn the customer’s problem. 2.Find the problem’s cause and fix it. 3.Have the right tools. Fix things once 1.Fix something once rather than over and over. 2.Avoid the temporary fix trap. 3.Measure twice, cut once, and other advice.

CIT 470: Advanced Network and System AdministrationSlide #3 Learn the Customer’s Problem Understand at a high level what customer is attempting to do and what part is failing. Customer problem reports vary: –My mail program is broken. –I can’t reach the mail server. –My mailbox disappeared! Actual problem might be: –Network problem. –Power failure. –DNS problem.

CIT 470: Advanced Network and System AdministrationSlide #4 Find the Problem and Fix it Approach debugging systematically –Form a hypothesis. –Test hypothesis. –Record results. –Modify hypothesis based on results. Problem is often in the last change. –Last config change, last new hardware, etc. Avoid random changes and workarounds. –Rebooting is not a solution!

CIT 470: Advanced Network and System AdministrationSlide #5 Process of Elimination –Remove parts of system one by one until problem disappears. –Problem must have been in last component. Examples –Remove DIMMs one by one to identify a bad memory unit. –Remove driver or application one by one to identify the source of the conflict.

CIT 470: Advanced Network and System AdministrationSlide #6 Successive Refinement –Add a component at a time, verifying that it works correctly at each step along the way. –Examine output at each step along the way. Examples –traceroute: tests network connectivity one hop at a time until it encounters problem or reaches dest –pipeline: develop a piped set of commands by adding one command a time to the pipeline

CIT 470: Advanced Network and System AdministrationSlide #7 Have the Right Tools Tools to let you see inside devices/systems. network: sniffer, ping, traceroute, telnet/nc network services: netstat, rpcinfo operating system: log files process: system call tracer, e.g. strace performance: top, ps, vmstat, iostat Know how tools draw their conclusions. Tools can make mistakes or mislead you.

CIT 470: Advanced Network and System AdministrationSlide #8 Fix Things Once Fixing something once is faster than fixing it over and over again. Corollaries –Fix the problem permanently. –Don’t reinvent the wheel. –Fix the problem for all hosts at the same time.

CIT 470: Advanced Network and System AdministrationSlide #9 Avoid Temporary Fix Trap Quick fixes aren’t. –A few minutes of your time every day adds up over a month or year. –Temporary fixes accumulate until you spend your entire day doing one quick fix after another. Temporary fixes may be required –Lack of resources (hardware/software) or time. –Must always be followed by permanent fixes. –Add permanent fix to your calendar or request system to ensure that it happens.

CIT 470: Advanced Network and System AdministrationSlide #10 Learning from Carpenters Measure twice, cut once. –Double-check your work before making changes. –ex: Replace UNIX command with echo to be sure you’re modifying the correct files. –ex: Reread configuration file before restarting server. Copy exact –Develop correct solution and test it. –Copy solution exactly to other hosts or sites.

CIT 470: Advanced Network and System AdministrationSlide #11 Automate Automation can fix problems permanently –Log rotation script will ensure you don’t have to manually delete logs to avoid full disks. –Tape jukebox will ensure that you don’t forget to manually swap backup tapes. Avoid using automation for quick fixes –Automation can perform a temporary fix without needing human intervention, e.g. kill runaways. –Problem may grow over time without your awareness, and automation can’t fix buggy software.

CIT 470: Advanced Network and System AdministrationSlide #12 Key Points Learn the customer’s problem. Systematically identify the cause and fix it. –Process of elimination. –Successive refinement. Fix the problem permanently. –Don’t reinvent the wheel. –Test your solution. –Use fix on all of your hosts. Use automation wisely.

CIT 470: Advanced Network and System AdministrationSlide #13 References 1.Mark Burgess, Principles of System and Network Administration, Wiley, Aeleen Frisch, Essential System Administration, 3 rd edition, O’Reilly, Thomas A. Limoncelli and Christine Hogan, The Practice of System and Network Administration, Addison-Wesley, Evi Nemeth et al, UNIX System Administration Handbook, 3 rd edition, Prentice Hall, 2001.