2 What is good data?Correctness - data collected according to the exact rules of definition of the metricAccuracy - difference between the data and the actual valuePrecision - number of decimal places needed to express the dataConsistent - no much difference in value when measured using different device or by different person, or repeatedly
3 What is good data?Time-stamp - associate with a particular activity or time period to know when it was collected.Replication - replicate for difference circumstances.
4 How to define the data?Metric terminology must be clear and detailed, understood by all involved.Two kinds of dataRaw dataRefined dataDirect and indirect measurementMost organizations are interested in software quality, cost and schedule.
5 The problem with problems Fig 5.2 (Software Quality Terminology)Faultoccurs when a human error results in a mistake in some software productEg: developer misunderstand the user-interface requirement, and create a design with the wrong understanding, the design fault results in incorrect code.
6 The problem with problems FailureDeparture of a system from its required behaviorCan be discovered during testing and operationCan think of fault and failure as inside and outside views of the system.Faults - problems that developers seeFailures - problems that the users seeNot every fault corresponds to a
7 The problem with problems Reliability of software is defined in terms of failures observed during operation not faults.Terminology is very important here. Refer to page 157 (error, anomalies, defects, bugs, crashes)If investigation of failure reveal the fault, change is made to the product to remove it.
8 The problem with problems Key elements of a problem:Location - where did it occurTiming - when did it occurSymptom - what was observedEnd result - which consequences resultedMechanism - how did it occurCause - why did it occurSeverity - how much was the user affectedCost - how much did it cost
9 The problem with problems These 8 attributes are mutually independent (applies only to initial measurement).Is known as orthogonalityOrthogonality also refers to a classification scheme within a particular category.Non-orthogonality can lead to data loss or corruption, Eg 5.1
10 The problem with problems The 8 attributes should suffice for all types of problems, but are answered differently based on whether it is faults, failures or changes.
11 FailuresFocus on external problems of the system: installation, chain of events, effect on user or other system, cost.Fig 5.3 (failure report), page 160Location - code that uniquely identifies the installation and platform on which the failure was observed.Timing - real time of occurrence, execution time up to the occurrence of failure.
12 Failures Cause Severity type of trigger, type of source Often cross-referenced to fault and change reports.Severityhow serious the failure’s end resultClassification for safety-critical system:CatastrophicCriticalSignificantMinorCan also be measured in terms of cost
13 FailuresCost - how much effort and other resources to diagnose and response to the failureNinth category; Count - number of failures in a stated time interval.Mechanism, cause and cost can only be completed after diagnosis.Data collection form for failure should include at least five categories.
14 Failure ReportLocation: such as installation where failure was observedTiming: CPU time, clock time or some temporal measureSymptom: type of error message or indication of failureEnd result: description of failure, such as “operating system crash”, “ services degraded”, “loss of data”, “wrong output”, “no output”Mechanism: chain of events, including keyboard commands and state data, leading to failureCause: reference to possible fault(s) leading to failureSeverity: reference to a well-defined scale, such as “critical”, “major”, “minor”Cost: cost to fix plus cost of lost potential business
15 Faults Focus on the internals of system Only the developer can see it Locationwhich product(identifier and version) or part of the product contains the fault.Spec, code, database, manuals, plan and procedures, report, standard/policyRequirements, functional, preliminary design, detailed design, product design, interface, database, implementation.
16 Faults Timing - when the fault is created, detected, corrected. Symptom - what is observed during diagnosis (Table 5.2, page 166)End result - actual failure caused by fault, should be cross-referenced to failure report.Causehuman error that led to the faultCommunication, conceptual, clerical
17 Faults Severity - impact on the user Cost - total cost to system providerCount - number of faults found in a product or subsystem or during a given period of operation.
18 Fault Report Location: module or document name Timing: phases of development during which fault was created, detected, and correctedSymptom: type of error message reported or activity which revealed faultEnd result: failure cause by the faultMechanism: how cause was created, detected, correctedCause: type of human error that led to faultSeverity: refer to severity of resulting or potential failureCost: time or effort to locate and correct
19 ChangesOnce a failure is experienced and its cause determined, the problem is fixed through one or more changesProblem is fixed through 1 or more change to any or all of the development productChanges report (Fig 5.5) - report the changes and track the most affected products
20 Changes Cause of change may be Corrective - correcting a faultAdaptive - system changes in some way so the product have to be upgradedPreventive - find faults before they become failurePerfective - redo something to clarify the system structureCount - number of changes made in a given time or given system component.
21 Change Report Location: identifier of document or module changed Timing: when change was madeSymptom: type of changeEnd result: success of change, as evidence by regression or other testingMechanism: how and whom change was performedCause: corrective, adaptive, preventive or perfectiveSeverity: impact on rest of the systemCost: time and effort for change implementation and test
22 How to collect data Requires human observation and reporting Manual data collection – bias, error, omission and delay -> uniform data collection formAutomatic data capture – desirable, essential – recording the execution timeTo ensure data are accurate and complete, planning is essential.
23 How to collect data Planning involves: Decide what to measure based on GQM analysis.Determine the level of granularity (individual modules, subsystem, function etc)Ensure that product is under configuration control (which version)Form design
24 How to collect data Planning involves (continue) Establish procedures for handling the forms, analyzing the data and reporting the results, setting a central collection point
25 How to collect data Keep procedure simple Avoid unnecessary recording Train staff in recording data and procedureProvide results to original providers promptlyValidate all data collected at a central collection point
26 Data collection forms Encourages collecting good, useful data Self-explanatoryShould allow fixed-format data and free-format commentsTable 5.3 (Data collection forms for software reliability evaluation)
27 When to collect dataData collection planning when project planning beginsActual data collection takes place during many phases of development.Can be collected at the beginning of the project to establish initial values and collected again later to reflect activities and resources being studied.
28 When to collect dataData-collection activities should become part of the regular development process.Should be mapped to the process model. (eg: Fig 5.9)
29 When to collect data Eg: Data relating to project personnel (qualifications or experience) can be collected at the start of the project.While other data collection, such as effort, begins at project start and continues through operation and maintenanceCount of the number of specification and design faults can be collected as inspections are performedData about changes made to enhance the product can be collected as enhancements are performed
30 How to store and extract data Raw data should be stored in database set up using DBMS.Can define data structure, insert, modify, delete and extract refined data.Formats, ranges, valid values etc can be check automatically as they are input.
31 How to store and extract data Raw database structureEg: Figure 5.10 (data structure that supports reliability measurement)box - table, arrow denotes a many-to-one mapping, double arrow