Presentation is loading. Please wait.

Presentation is loading. Please wait.

Effects of Using Examples on Structural Model Comprehension: A Controlled Experiment Dina Zayan, Michał Antkiewicz, Krzysztof Czarnecki 2014.06.06

Similar presentations


Presentation on theme: "Effects of Using Examples on Structural Model Comprehension: A Controlled Experiment Dina Zayan, Michał Antkiewicz, Krzysztof Czarnecki 2014.06.06"— Presentation transcript:

1 Effects of Using Examples on Structural Model Comprehension: A Controlled Experiment Dina Zayan, Michał Antkiewicz, Krzysztof Czarnecki

2 Many problems in SE are related to domain knowledge transfer 2

3 For example… Business analysts communicating to other stakeholders who are novices Industry standard which contains domain models applied by novices Specifications written by experts given to suppliers 3

4 Why do we have these problems? 4 Domain Knowledge Complex in nature Difficult to articulate Difficult to communicate

5 Let’s run a mini experiment 5

6 6 In a “second degree price discrimination” strategy, price varies according to the quantity demanded. Larger quantities are available at a lower unit price. Additionally, sellers applying this strategy are not able to differentiate between different types of consumers. Thus, the suppliers will provide incentives for the consumers to differentiate themselves according to preference. This also applies to product quality. This is an Abstraction Airlines often offer multiple classes of seats on flights, such as first class and economy class. This is an Example

7 7 Examples + Abstractions Effective Knowledge Transfer Results in Cognitive Psychology and Software Engineering

8 Modeling Model = Abstractions + Examples 8 Example-Driven

9 Of Course Examples Help! + Capture intuition + Relate to existing knowledge + Show the intended usage of an abstraction 9

10 However… 1.When exactly do examples help? 2.How much can they help? 3.What types of examples help? 4.Which stakeholders benefit the most from examples? 5.Where is a PROOF that they help? 10

11 Provide the Missing Empirical Evidence and Guidelines Objective 11

12 “Analyze the effects of using explicit examples on structural model comprehension in the context of knowledge transfer to model receptors who are novices to the application domain.” 12 Controlled Experiment

13 Two Groups of Participants 13 EDMControl Model Abstractions Experimental Procedures & Requirements List Examples! (13) (12)

14 Instrumentation & Actors Map 14 Experimental Procedures & Requirements List Model Abstractions SME Model Creator Examples!

15 Treatments and Metrics 15 Sobeys Object Diagram EDM Control

16 16

17 17 23 classes, 34 associations, and 8 constraints

18 18 (EDM Only) 4 positive and 2 negative examples

19 Treatments and Metrics 19 Sobeys Object Diagram 1) Diagram Completeness EDM Control

20 Diagram Completeness Mean (Control) = 49.2 % Mean (EDM) = 88.1 % p-value < Maximum score for Control is lower than the minimum score for the EDM + 37% increase 20

21 Treatments and Metrics 21 Sobeys Object Diagram 1) Diagram Completeness 2) Diagram Mistakes EDM Control

22 Diagram Mistakes Mean (Control) = 15.1 mistakes Mean (EDM) = 3.5 mistakes p-value < Maximum number of mistakes for EDM is equal to the minimum number of mistakes for Control - 80% decrease 22

23 Treatments and Metrics 23 Sobeys Object Diagram 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence EDM Control

24 Participants Rated Themselves Difficulty 1 (very easy) to 5 (very difficult) – Control: 3.83 – EDM: % decrease Confidence 1 (unsure) to 5 (very sure) – Control: 2.33 – EDM: % increase 24

25 Treatments and Metrics 25 Study Questions Sobeys Object Diagram 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence EDM Control

26 26

27 27

28 Treatments and Metrics 28 Study Questions Sobeys Object Diagram 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence 1) Task 2 Completeness EDM Control

29 Task 2 Completeness Mean (Control) = 13.6 Mean (EDM) = 21.2 p-value < % increase 29 Max 24

30 Treatments and Metrics 30 Study Questions Sobeys Object Diagram 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence 1) Task 2 Completeness 2) Efficiency EDM Control

31 Task 2 Efficiency Mean (Control) = 0.7 Mean (EDM) = 1.2 p-value = % increase 31

32 Treatments and Metrics 32 Study Questions Sobeys Object Diagram 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence 1) Task 2 Completeness 2) Efficiency 3) Perceived Difficulty 4) Perceived Confidence EDM Control

33 Participants Rated Themselves Difficulty 1 (very easy) to 5 (very difficult) – Control: 3.25 – EDM: % decrease Confidence 1 (unsure) to 5 (very sure) – Control: 3.00 – EDM: % increase 33

34 Treatments and Metrics 34 Study Questions Sobeys Object Diagram Participants Questions 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence 1) Task 2 Completeness 2) Efficiency 3) Perceived Difficulty 4) Perceived Confidence 1) Number of domain- related questions EDM Control

35 Number of Domain-Related Questions Control : 10 questions/session EDM: 1 question/session - 90% reduction 35

36 Treatments and Metrics 36 Study Questions Sobeys Object Diagram Participants Questions Feedback 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence 1) Task 2 Completeness 2) Efficiency 3) Perceived Difficulty 4) Perceived Confidence 1) Number of Domain- related Questions 1)UML Class Diagram 2)Examples EDM Control

37 Understanding Without Examples? EDM: “The class diagram was huge, I didn’t know where to start reading to understand, so I looked at the examples one by one while looking at the model abstractions simultaneously.” 37

38 Classes Most Difficult To Comprehend 38 ClassControlEDMMistakes Control/EDM Misses Control/EDM BonusMechanic ✔✔ 5/24/0 Redemption Mechanic ✔✔ 9/03/0 “Partner” Offer ✔ x6/03/0 Bill ✔✔ 9/30/0

39 39

40 How Were the Examples Helpful? Example 2: Regular product vs. product with an offer associated Application of the BonusMechanic class Example 4: Propagation of points across classes and the sequence for accumulating points Account’s “pointsBalance” updated with Bill’s “totalPoints” attribute 40

41 41

42 How Were the Examples Helpful? Example 5: Redemption process Example 6: Violation of the constraint prohibiting accumulation and redemption for the same bill instance 42

43 Are Both Positive and Negative Examples Needed? “Yes” 12/13 EDM participants No comments about the number and content of examples 43

44 Discussion 44

45 Presenting Information in the Form of Explicit Examples Control could not instantiate RedemptionMechanic and RedemptionChannel 45

46 Examples are mostly needed by Novices FixedPriceOff vs. FixedPercentOff – familiar RedemptionMechanic, RedemptionChannel, and BonusMechanic – foreign 46

47 Having a Variety of Examples Useful when 1.Having more than one association linked to a single class 47

48 48

49 Having a Variety of Examples Useful when 1.Having more than one association linked to a single class 2.Complex constraints 49

50 50

51 Having a Variety of Examples Useful when 1.Having more than one association linked to a single class 2.Complex constraints 3.Propagation of attributes through classes 51

52 52

53 Having a Variety of Examples Useful when 1.Having more than one association linked to a single class 2.Complex constraints 3.Propagation of attributes through classes 4.A domain concept that is spread over several classes and associations (Points Accumulation) 53

54 EDM participant who preferred valid examples only answered the related question incorrectly 54

55 Partial Examples 12 participants – partial examples, no complete object diagram One participant – partial examples and complete object diagram 55

56 Conclusions 56

57 Quantitative Analysis 57 Dependent Variable Significant? (p-value) Hyptheses Testing% Improvement due to the use of examples Diagram Completeness Yes (<0.0002)Accept H1+39% Diagram Mistakes Yes (<0.0053)Accept H2-80% Task 2 Completeness Yes (<0.0012)Accept H3+31% Task 2 Efficiency Yes (= )Accept H4+71%

58 Qualitative Analysis 58 Dependent Variable% Improvement due to the use of examples Perceived difficulty in creating the object diagram. -27% Perceived difficulty in answering Task 2 questions. -23% Confidence in their created object diagrams. +35% Confidence in the answers they provided for Task %

59 Limitations & Future Work Partial examples as views on a complete example Variety of examples vs. positive only Cost of creating and maintaining the examples vs. the benefit Empirical studies in an industrial context 59

60 Thank You! Replication package is available for download

61 Example-Driven Modeling Using Unified syntax for abstractions and examples + redefinition Partial examples + automatic completion and constraint checking Specialization + extension 61


Download ppt "Effects of Using Examples on Structural Model Comprehension: A Controlled Experiment Dina Zayan, Michał Antkiewicz, Krzysztof Czarnecki 2014.06.06"

Similar presentations


Ads by Google