Download presentation

Presentation is loading. Please wait.

Published byKaylin Welby Modified over 2 years ago

1
Effects of Using Examples on Structural Model Comprehension: A Controlled Experiment Dina Zayan, Michał Antkiewicz, Krzysztof Czarnecki 2014.06.06 http://gsd.uwaterloo.cahttp://necsis.ca

2
Many problems in SE are related to domain knowledge transfer 2

3
For example… Business analysts communicating to other stakeholders who are novices Industry standard which contains domain models applied by novices Specifications written by experts given to suppliers 3

4
Why do we have these problems? 4 Domain Knowledge Complex in nature Difficult to articulate Difficult to communicate

5
Let’s run a mini experiment 5

6
6 In a “second degree price discrimination” strategy, price varies according to the quantity demanded. Larger quantities are available at a lower unit price. Additionally, sellers applying this strategy are not able to differentiate between different types of consumers. Thus, the suppliers will provide incentives for the consumers to differentiate themselves according to preference. This also applies to product quality. This is an Abstraction Airlines often offer multiple classes of seats on flights, such as first class and economy class. This is an Example

7
7 Examples + Abstractions Effective Knowledge Transfer Results in Cognitive Psychology and Software Engineering

8
Modeling Model = Abstractions + Examples 8 Example-Driven

9
Of Course Examples Help! + Capture intuition + Relate to existing knowledge + Show the intended usage of an abstraction 9

10
However… 1.When exactly do examples help? 2.How much can they help? 3.What types of examples help? 4.Which stakeholders benefit the most from examples? 5.Where is a PROOF that they help? 10

11
Provide the Missing Empirical Evidence and Guidelines Objective 11

12
“Analyze the effects of using explicit examples on structural model comprehension in the context of knowledge transfer to model receptors who are novices to the application domain.” 12 Controlled Experiment

13
Two Groups of Participants 13 EDMControl Model Abstractions Experimental Procedures & Requirements List Examples! (13) (12)

14
Instrumentation & Actors Map 14 Experimental Procedures & Requirements List Model Abstractions SME Model Creator Examples!

15
Treatments and Metrics 15 Sobeys Object Diagram EDM Control

16
16

17
17 23 classes, 34 associations, and 8 constraints

18
18 (EDM Only) 4 positive and 2 negative examples

19
Treatments and Metrics 19 Sobeys Object Diagram 1) Diagram Completeness EDM Control

20
Diagram Completeness Mean (Control) = 49.2 % Mean (EDM) = 88.1 % p-value < 0.0002 Maximum score for Control is lower than the minimum score for the EDM + 37% increase 20

21
Treatments and Metrics 21 Sobeys Object Diagram 1) Diagram Completeness 2) Diagram Mistakes EDM Control

22
Diagram Mistakes Mean (Control) = 15.1 mistakes Mean (EDM) = 3.5 mistakes p-value < 0.0053 Maximum number of mistakes for EDM is equal to the minimum number of mistakes for Control - 80% decrease 22

23
Treatments and Metrics 23 Sobeys Object Diagram 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence EDM Control

24
Participants Rated Themselves Difficulty 1 (very easy) to 5 (very difficult) – Control: 3.83 – EDM: 2.46 -27% decrease Confidence 1 (unsure) to 5 (very sure) – Control: 2.33 – EDM: 4.08 + 35% increase 24

25
Treatments and Metrics 25 Study Questions Sobeys Object Diagram 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence EDM Control

26
26

27
27

28
Treatments and Metrics 28 Study Questions Sobeys Object Diagram 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence 1) Task 2 Completeness EDM Control

29
Task 2 Completeness Mean (Control) = 13.6 Mean (EDM) = 21.2 p-value < 0.0012 + 31% increase 29 Max 24

30
Treatments and Metrics 30 Study Questions Sobeys Object Diagram 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence 1) Task 2 Completeness 2) Efficiency EDM Control

31
Task 2 Efficiency Mean (Control) = 0.7 Mean (EDM) = 1.2 p-value = 0.00082 + 71% increase 31

32
Treatments and Metrics 32 Study Questions Sobeys Object Diagram 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence 1) Task 2 Completeness 2) Efficiency 3) Perceived Difficulty 4) Perceived Confidence EDM Control

33
Participants Rated Themselves Difficulty 1 (very easy) to 5 (very difficult) – Control: 3.25 – EDM: 2.08 -23.5% decrease Confidence 1 (unsure) to 5 (very sure) – Control: 3.00 – EDM: 4.15 + 23% increase 33

34
Treatments and Metrics 34 Study Questions Sobeys Object Diagram Participants Questions 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence 1) Task 2 Completeness 2) Efficiency 3) Perceived Difficulty 4) Perceived Confidence 1) Number of domain- related questions EDM Control

35
Number of Domain-Related Questions Control : 10 questions/session EDM: 1 question/session - 90% reduction 35

36
Treatments and Metrics 36 Study Questions Sobeys Object Diagram Participants Questions Feedback 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence 1) Task 2 Completeness 2) Efficiency 3) Perceived Difficulty 4) Perceived Confidence 1) Number of Domain- related Questions 1)UML Class Diagram 2)Examples EDM Control

37
Understanding Without Examples? EDM: “The class diagram was huge, I didn’t know where to start reading to understand, so I looked at the examples one by one while looking at the model abstractions simultaneously.” 37

38
Classes Most Difficult To Comprehend 38 ClassControlEDMMistakes Control/EDM Misses Control/EDM BonusMechanic ✔✔ 5/24/0 Redemption Mechanic ✔✔ 9/03/0 “Partner” Offer ✔ x6/03/0 Bill ✔✔ 9/30/0

39
39

40
How Were the Examples Helpful? Example 2: Regular product vs. product with an offer associated Application of the BonusMechanic class Example 4: Propagation of points across classes and the sequence for accumulating points Account’s “pointsBalance” updated with Bill’s “totalPoints” attribute 40

41
41

42
How Were the Examples Helpful? Example 5: Redemption process Example 6: Violation of the constraint prohibiting accumulation and redemption for the same bill instance 42

43
Are Both Positive and Negative Examples Needed? “Yes” 12/13 EDM participants No comments about the number and content of examples 43

44
Discussion 44

45
Presenting Information in the Form of Explicit Examples Control could not instantiate RedemptionMechanic and RedemptionChannel 45

46
Examples are mostly needed by Novices FixedPriceOff vs. FixedPercentOff – familiar RedemptionMechanic, RedemptionChannel, and BonusMechanic – foreign 46

47
Having a Variety of Examples Useful when 1.Having more than one association linked to a single class 47

48
48

49
Having a Variety of Examples Useful when 1.Having more than one association linked to a single class 2.Complex constraints 49

50
50

51
Having a Variety of Examples Useful when 1.Having more than one association linked to a single class 2.Complex constraints 3.Propagation of attributes through classes 51

52
52

53
Having a Variety of Examples Useful when 1.Having more than one association linked to a single class 2.Complex constraints 3.Propagation of attributes through classes 4.A domain concept that is spread over several classes and associations (Points Accumulation) 53

54
EDM participant who preferred valid examples only answered the related question incorrectly 54

55
Partial Examples 12 participants – partial examples, no complete object diagram One participant – partial examples and complete object diagram 55

56
Conclusions 56

57
Quantitative Analysis 57 Dependent Variable Significant? (p-value) Hyptheses Testing% Improvement due to the use of examples Diagram Completeness Yes (<0.0002)Accept H1+39% Diagram Mistakes Yes (<0.0053)Accept H2-80% Task 2 Completeness Yes (<0.0012)Accept H3+31% Task 2 Efficiency Yes (=0.00082)Accept H4+71%

58
Qualitative Analysis 58 Dependent Variable% Improvement due to the use of examples Perceived difficulty in creating the object diagram. -27% Perceived difficulty in answering Task 2 questions. -23% Confidence in their created object diagrams. +35% Confidence in the answers they provided for Task 2. +23%

59
Limitations & Future Work Partial examples as views on a complete example Variety of examples vs. positive only Cost of creating and maintaining the examples vs. the benefit Empirical studies in an industrial context 59

60
Thank You! Replication package is available for download http://gsd.uwaterloo.cahttp://necsis.caExamples!

61
Example-Driven Modeling Using Unified syntax for abstractions and examples + redefinition Partial examples + automatic completion and constraint checking Specialization + extension 61 http://clafer.org

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google