PowerPoint® Slides to Accompany A Gift of Fire: Social, Legal, and Ethical Issues for Computers and the Internet (2nd Edition) by Sara Baase San Diego State University PowerPoint slides created by Sherry Clark Copyright 2003 Prentice Hall

Can We Trust the Computer? A Gift of Fire Can We Trust the Computer? What Can Go Wrong? Case Study: The Therac-25 Increasing Reliability and Safety Perspectives on Failures, Dependence, Risk, and Progress Computer Models

What Can Go Wrong? Facts About Computer Errors Error-free software is not possible. Errors are often caused by more than one factor. Errors can be reduced by following good procedures and professional practices. Q: How do we distinguish between tolerable or unavoidable errors in software versus careless software development?

What Can Go Wrong? The Roles of People in Computer-related Problems: Computer User At home or work, users should understand the limitations of computers and the need for proper training and responsible use. Computer Professional Understanding the source and consequences of computer failures is valuable when buying, developing, or managing a complex system. Educated Member of Society Personal decisions and political, social, and ethical decisions depend on understanding computer risks.

What Can Go Wrong? Categories of Computer Errors and Failures Problems for Individuals: usually in their role as consumers. who are incorrectly identified by inaccurate law enforcement databases. System Failures: affecting large numbers of people and/or costing large amounts of money. Safety-Critical Applications: where people may be injured or killed.

What Can Go Wrong? Problems for Individuals Billing Errors Lack of tests for inconsistencies and inappropriate amounts. Database Accuracy Problems Incorrect information resulting in wrongful treatment or acts. Q: Describe a computer error or failure that has affected you.

What Can Go Wrong? Problems for Individuals (cont’d) Causes: Large population. Human common sense not part of automated processing. Overconfidence in the accuracy of data from a computer. Errors in data entry. Information not updated or corrected. Lack of accountability for errors.

What Can Go Wrong? System Failures Communications: Business: Telephone, online, and broadcast services. Business: Inventory and management software. Financial: Stock exchange, brokerages, banks, etc.. Transportation: Reservations, ticketing, and baggage handling.

What Can Go Wrong? System Failures (cont’d) Causes: Insufficient testing and debugging time. Significant changes in specifications (during and after project begun). Overconfidence in system. Mismanagement of the project. Q: Describe a recent system failure that affected many people or resulted in a great monetary loss.

What Can Go Wrong? Safety-Critical Applications Military Power Plants Aircraft Trains Automated Factories Medicine …others.

What Can Go Wrong? Safety-Critical Applications Causes: Overconfidence. Lack of override features. Insufficient testing. Sheer complexity of system. Mismanagement. Q: What activities do you participate in that are controlled by safety-critical applications?

Case Study: The Therac-25 The Therac-25 was a software-controlled radiation-therapy machine used to treat people with cancer. Overdoses of radiation Normal dosage is 100–200 rads. It is estimated that 13,000 and 25,000 rads were given to six people. Three of the six people died. Q: What determines whether the risks associated with a safety-critical application are acceptable?

Case Study: The Therac-25 Therac-25 Radiation Overdose Multiple Causes: Poor safety design. Insufficient testing and debugging. Software errors. Lack of safety interlocks. Overconfidence. Inadequate reporting and investigation of accidents.

Increasing Reliability and Safety What Goes Wrong? Computer Systems Fail Because: The job they are doing is inherently difficult, and The job is done poorly. Compounding the Reliability Issue: Developers and users exhibit overconfidence in the system. Reused system software may not work in different environments. Q: Identify the elements needed as an incentive to increase reliability and safety.

Increasing Reliability and Safety Professional Techniques Follow good software practices. Exhibit professional responsibility at all levels of development and use. Construct well-designed user interfaces. Take human factors into account. Include built-in redundancy. Incorporate self-checking where appropriate. Follow good testing principals and techniques. Q: What human interface features should be considered for ordinary business applications?

Increasing Reliability and Safety Law and Regulation Criminal and Civil Penalties: to recover loss from faulty or unsafe systems. Liability and Civil Laws: to provide incentives to produce reliable and safe systems. Warranties: to guarantee a certain level of quality. Federal or State Regulations: to protect the public.

Increasing Reliability and Safety Law and Regulation (cont’d) Database Accuracy Enforcement: to protect the public from inaccurate information maintained by private companies and government. Mandatory Licensing of Software Developers: to ensure proper training, competency, and continuing education. Q: How can consumers protect themselves from faulty software?

Perspectives on Failures, Dependence, Risk, and Progress What are acceptable rates of failures? How accurate should software be? Dependence How dependent on computer systems are our ordinary activities? How useful are computer systems to our ordinary activities? Risk and Progress How do new technologies become safer? Can progress in software safety keep up with the pace of change in computer technology?

Computer Models Points to Consider: Models are simplifications of either physical or intangible systems. Those who design and develop models must be honest and accurate with results. Computer professionals and the general public must be able to evaluate the claims of the developers. Q: What problems in your community have been or could be studied with computer models?

Computer Models Evaluating Models Why Models Might Not Be Accurate: Developers have incomplete knowledge of the system being modeled. Data might be incomplete or inaccurate. Power of the computer might be inadequate. Variables are difficult to numerically quantify. Political and economic motivation to distort results. Q: For each item above, give an example of a model or simulation that was inaccurate.

Computer Models Evaluating Models (cont’d) Regarding the Car-Crash Models Described in the Text: How well do the modelers understand the system and/or materials being studied? How accurate and complete are the data? What are the assumptions and simplifications in the model? Do the results or predictions correspond with the real world?

Computer Models Evaluating Models (cont’d) Regarding the Climate Models Discussed in the Text: How well do the modelers understand the system and/or materials being studied? How accurate and complete are the data? What are the assumptions and simplifications in the model? Do the results or predictions correspond with the real world?