1. *Graduate School of Engineering Science, Osaka University
An Experimental Comparison of Checklist-Based Reading and Perspective-Based Reading for UML Design Document Inspection
Giedre Sabaliauskaite*
Fumikazu Matsukawa**
Shinji Kusumoto**
Katsuro Inoue**
*Graduate School of Engineering Science, Osaka University
**Graduate School of Information Science and Technology, Osaka University

2. Contents Background Research Goals Experiment description
Reading techniques
Experimental design
Data analysis
Results
Conclusions and Further Work

3. Reading techniques are used to guide inspectors
Software Inspection
Software inspection is an effective and efficient method to detect defects in early stages of software development life-cycle
Software inspection usually consists of several activities [1]
Planning
Key activity – defects are detected
Reading techniques are used to guide inspectors
Defect detection
Defect collection
Defect correction
[1] O. Laitenberger, J.M. DeBaud, “An encompassing life cycle centric survey of software inspection”, The Journal of Systems and Software, vol. 50, no. 1, 2000, pp

4. Reading techniques
Ad hoc and Checklist-based Reading are the most popular [1]
Ad hoc: no guidance for inspectors
Checklist-based Reading (CBR): Checklist
Scenario-based Reading is a more recent approach: Scenario
Perspective-based Reading (PBR) [2] is one of Scenario-based approaches
Software product is inspected from different perspectives (designer, tester, etc.)
Provides inspectors with different Scenario for each perspective
We analyzed CBR and PBR inspection techniques during experiment
[1] O. Laitenberger, J.M. DeBaud, “An encompassing life cycle centric survey of software inspection”, The Journal of Systems and Software, vol. 50, no. 1, 2000, pp
[2] V.R. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Sorumgard, M.V. Zelkowitz, “The Empirical Investigation of Perspective-Based Reading”, Empirical Software Engineering: An International Journal, vol.1 , no. 2, 1996, pp

5. Related research
Several works are done in the area of Unified Modelling Language (UML) diagram inspection
One of the works is described in [1]
Experimental comparison of CBR and PBR
18 subjects (practitioners) and 2 software systems
UML diagrams inspected – Class, State, Sequence, Collaboration
Results - PBR 3-person teams exhibited higher effectiveness and lower cost per defect than CBR teams
Authors of different studies came to the conclusion that UML inspections need to be further investigated
[1] O. Laitenberger, C. Atkinson, M. Schlich, K. El Emam. An experimental comparison of reading techniques for defect detection in UML design documents. The Journal of Systems and Software, 53: , 2000.

6. Research Goals
Conduct UML diagram inspection experiment in Osaka university
Language used during experiment - Japanese
Compare Reading techniques CBR and PBR with respect to
Defect detection effectiveness
Cost per defect
Time spent on inspection
Compare both individual inspector results and team results
Acquire experience for future inspection
The goals of the research are:
To conduct an experiment in Osaka university to inspect design documents written in the notation of Unified Modelling Language.
To inspect the following UML diagrams:
Use-case
Class
Activity
Sequence
To compare
To analyze
To acquire

7. Structure of CBR and PBR techniques, used during experiment
Diagram-specialized Checklist developed
Included 20 questions
PBR
Three perspectives: User, Designer, Implementer
Three scenarios (one for each perspective) developed
Scenarios consisted of several steps, each of them included
Explanations
Tasks to perform
Questions to answer

8. Assignment of UML diagrams
All inspectors were given system requirement description and Use-Case diagrams
Defects were injected into Activity, Class, Sequence and Component diagrams
Document
Seminar system
Hospital system
CBR
PBR perspective
User
Designer
Implementer
System requirement description 1 
Use-Case diagrams
Activity diagrams 8 7
Class diagrams
Sequence diagrams 12
Component diagrams

9. Experiment design Subjects Objects
59 third year students of the Software Development course, Osaka University
Objects
Seminar information system ( 24 pages )
Hospital information system ( 18 pages )
PBR
CBR
User
Designer
Implementer
Seminar system 7 6 11
Hospital system 10

10. Defects Three types of defects inserted into UML diagrams
Syntactic – missing or unnecessary information
Semantic – misrepresentation of a concept, or unclear representation
Consistency – inconsistency between UML diagrams
In total, 15 defects inserted into the each software system
4 into Activity diagrams
3 into Class diagrams
5 into Sequence diagrams
3 into Component diagrams

11. Experimental Hypotheses
H01: subjects who use PBR technique should spend less time on inspection
H02: subjects who use PBR should have higher cost per defect
H03: individual defect detection effectiveness should be different between CBR and PBR inspectors
H04: team defect detection effectiveness should be different between CBR and PBR inspector teams

12. Data analysis Two types of analysis Individual data analysis
Defect detection effectiveness
Time spent on inspection
Cost per defect
Team data analysis
Subjects assigned into 3-person virtual teams
Effectiveness of CBR and PBR teams compared

13. Percentage of inspectors, who have detected defects relevant to different perspectives
Both CBR and PBR exhibited similar results

14. Defect detection effectiveness (according to defect types)
Syntactic defects: CBR more effective
Semantic and Consistency defects: PBR more effective

15. Defect detection effectiveness (according to UML diagram types)
Class Diagrams: CBR more effective
Activity and Component Diagrams : PBR more effective
Sequence Diagrams: CBR and PBR exhibited similar effectiveness

16. Time spent on inspection
Time spent on inspection, Cost per defect and Defect detection effectiveness
Variables
Reading technique
Mean SD
Time spent on inspection
CBR
62.9
11.7
PBR
51.3
15.1
Cost per defect
6.2
1.6
10.2
3.5
Effectiveness
70.2
11.5
69.1
15.3
PBR: 18% (11 min) less time spent on inspection
CBR: 39% (4 min / defect) lower cost per defect
CBR Effectiveness ≈PBR Effectiveness
Independent samples t-test
Mann-Whitney test
Significant difference in Inspection Time and Cost
No significant difference in Effectiveness

17. Virtual teams Subject assignment into 3-person virtual teams
Any three CBR reviewers are included into team
One reviewer using each of the three PBR perspectives included into team C9 C8 C7 C6 C5 C4 C3 C2 C1
CBR group I5 D5 U5 I4 D4 U4 I3 D3 U3 I6 D6 U6 I2 D2 U2 I1 D1 U1
PBR group
Grouping virtual teams into statistically independent groups
CBR groups: 3 teams included (11 inspectors for Seminar system, 10 subjects for Hospital system)
PBR groups: 6 teams included (7 Users, 6 Designers, 6 Implementers for each systems)

18. Comparisons between CBR and PBR teams
Hospital system
Seminar system
Number of comparisons:
Comparisons C1 C2 C3 C4 C5 C6 C7 C8 C9 U1 D1 I1 U2 D2 I2 … U6 D6 I6
CBR group 1
PBR group 1 C1 C2 C3 C4 C5 C6 C7 C8
C10 U1 D1 I6 U2 D2 I1 … U6 D6 I5
CBR group 2 … …
PBR group 2 …

19. Team defects detection effectiveness
Team data analysis (1/2)
CBR and PBR teams were compared with respect to Team defect detection effectiveness
Number or comparisons, in which either CBR or PBR was more effective, or both techniques showed similar effectiveness
Software system
Team defects detection effectiveness
CBR>PBR
PBR>CBR
CBR=PBR
Seminar
Hospital
8 064
CBR teams more effective than PBR teams

20. Defect detection effectiveness
Team data analysis (2/2)
T-test with significance level of 2.5% was used to evaluate in which comparisons there was a significant difference between CBR and PBR team effectiveness
Two types of comparisons made
All defects included
Syntactic defects omitted
Comparison type
Software
system
Defect detection effectiveness
CBR > PBR
PBR > CBR
CBR = PBR
All defects included
Seminar
Hospital
Syntactic defect omitted
CBR teams more effective than PBR teams

21. Experiment results Individual inspector results
Inspectors who use PBR spend on average 18% less time on inspection
Cost per defect of Inspectors who use PBR is on average 39% higher
There is no statistically significant difference in defect detection effectiveness between individual CBR and PBR inspectors, however on average
PBR is more effective for Semantic and Consistency defects
PBR is more effective for Activity and Component diagrams
CBR is more effective for Syntactic defects
CBR is more effective for Class diagrams
Three-person virtual team results
CBR teams exhibit higher defect detection effectiveness than PBR teams

22. Threats to validity Internal validity External validity
Selection: experiment was a mandatory part of the course
Instrumentation: objects which we used could have influence
External validity
Students were used as subjects
Design documents were similar to those which are used in practice, but the size of systems in industry is usually larger
There were threats to internal and external validity, but they were not considered large in this experiment

23. Conclusions and Further research
UML diagram inspection experiment was conducted
The results of the experiment indicate that
There is no statistically significant difference in effectiveness of CBR and PBR individual inspectors
CBR virtual teams are more effective than PBR virtual teams
Future research will be directed to further investigation of UML diagram inspection
A replication of experiment was conducted in July 2002, during which real team meeting were performed
More detail analysis of both individual and team data
Research of Fault Content Estimation methods

24. Software Engineering Laboratory

25. ADDITIONAL SLIDES: 26-32

26. Experiment Variables and Hypotheses
Independent variables
Reading technique (CBR and PBR)
Dependent variables
Number of defects found
Time spent on inspection
Defect detection effectiveness
Cost per defect
Null Hypotheses
For individual inspectors
H01: PBR Time > CBR Time
H02: PBR Cost < CBR Cost
H03: PBR Effectiveness = CBR Effectiveness
For 3-person virtual teams
H04: PBR Effectiveness = CBR Effectiveness

27. Information systems inspected
Seminar information system
System supports the process of a company which organizes seminars
Includes activities from seminar planning to issuing the qualification certificate
Designs relationships among personnel of seminar company, participants, lecturers, etc.
Hospital information system
System supports the process of a hospital
Includes activities such as patient registration, medical examination, treatment, prescription of the medicines, etc.
Designs the relationships among the personnel of a hospital, patients, pharmacists, etc.

28. Collected data Two types of data collected during experiment
Defect data
Time data
Number of inspectors
Defects
Inspection time (min)
Number of detectable defects
Average number of defects detected
max/min
Average
Seminar information system
PBR User 7
4.43
6 / 3
60.43
90 / 46
PBR Designer 6
5.00
6 / 4
65.50
80 / 51
PBR Implementer 9
6.50
9 / 5
76.67
95 / 40
CBR 11 15
10.55
13 / 8
74.64
90 / 62
Hospital information system
48.29
70 / 25
3.83
5 / 3
59.17
73 / 30
6.33
7 / 5
63.30
77 / 44 10
10.50
12 / 8
70.10
94 / 60

29. CBR Checklist CHECKLIST
Locate the following diagrams: Class Diagrams, Activity Diagrams, Sequence Diagrams and Component Diagrams. Answer the questions related to corresponding diagrams. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have any comments write them in Comment form. …
SEQUENCE DIAGRAMS
10.
Aren’t there any inconsistencies among Sequence Diagram, Class Diagram and Requirements Specification?
 yes  no
11.
Are all the necessary Objects and Messages defined?
12.
Is every Class from Class Diagram included in any of Sequence Diagrams and vice versa?
13.
Is every Use Case from Use-Case Diagram implemented in at least one of Sequence Diagrams?
14.
Are there no redundant elements (Objects of Messages) in Sequence Diagram?

30. PBR Scenario DESIGNER SCENARIO
Assume that you are the Designer of the system. The concern of the designer is to ensure that the designer’s needs are completely satisfied in the following documents: Requirement specification, Class Diagram and Sequence Diagram. During the inspection process you will need to inspect those documents in order to detect defects in Class and Sequence Diagrams from designer’s point of view.
Please perform tasks following Step1 through Step3. For each step you must locate corresponding documents, follow the instructions and answer the given questions. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have some comments write them in Comment form.
Step 3
Locate the Class Diagram and Sequence Diagrams
Make a list of all Objects included in Sequence Diagrams and answer the following questions.
3.1.
Are all the Objects of Sequence Diagrams defined in Class Diagrams?
Are all Classes of Class Diagrams defined in Sequence Diagrams?
3.2.
Are all the Messages between objects Sequence Diagram defined as Methods of the corresponding Class in Class Diagram?
3.3.
Are there no redundant elements in Sequence diagrams?

31. Defect registration form

32. Software development process
The main steps of software development process
Development of Use-case diagrams
Describing system activities in Activity diagrams
Defining static structure of the system in Class diagrams
Modelling dynamic aspects of the system in Sequence diagrams
Detailed description of object states in Statechart diagrams
Development of the Component diagrams
User’s perspective in our experiment covered the second, and partially the third and the fourth steps
Designer’s perspective covered the third and the fourth steps
Implementer’s perspective covered the sixth step, and partially the third and the fourth steps