1. 1 Sundar Gopalakrishnan, Guttorm Sindre, and John Krogstie: Adapting UML activity diagrams for mobile work process modelling: Experimental comparison of two notation alternatives PoEM 2010 Delft, 9.-10. Sept.

2. 2 Agenda Motivation Background Research method Results Discussion of threats to validity Summary and further work

3. 3 Traditional perspectives to modeling Structural Behavioral Functional Goal and rule-oriented Object-oriented Social communication Actor/role-oriented What about location/place/space... ?

4. 4 Zachman Framework

5. 5 Motivation ’Where’ is increasingly relevant –Outsourcing –Supply Chain Management/Logistics –Virtual organization –Mobile applications and information systems And it is possible to utilize ’where’ to a larger degree (also real time) to know where users, equipment and goods should be, are or where at a certain time –Tracking (RFID, UWB, GPS, GSM, WiFi, Ultrasound…) –Internet of Things (IoT) This paper: –Presents some different notation alternatives based on UML activity diagrams vs. Modeling mobile information systems –Summarizes an analytical evaluation from a previous paper (I-ESA’10) –Makes an experimental comparison of the two most promising ones

6. 6 Possible notations looked at (I-ESA’10) Standard UML, using annotation boxes to indicate context or location Redefining swimlanes to indicate context / location Using colour for context / location These three were compared analytically, using a home care case supported by a mobile IS as an example

7. 7 Alt 1: Using annotation boxes Advantages –Smallest deviation from standard UML AD –Enhanced understandability for those who already know AD Disadvantages –Greatly increases # nodes in diagram (poor expressive economy) –May be confusing if you also need to use notes for something else in addition to context / location

8. 8 Alt 2: Swimlanes for where Advantages –Shifting ”who” to stick figures: AD more uniform with UCD –Swimlanes intuitively indicate location Disadvantages –Many lines from stick figures to activities, poor readability (would be even worse with bigger example) –New usage of swimlanes may confuse those already familiar with AD

9. 9 Alt 3.: using colour Advantages –No increase in # nodes or lines, better readability –With two process design alternatives beside each other, it is easy to spot differences in location Disadvantages –Larger deviation from standard UML –Must add legend –Possible challenge for colour blind users

10. 10 Analytical comparison Analytical comparison (I-ESA’10): Alt 1. (trad. UML with annotations) and Alt 3. (Colours) came out as the two most promising Proposed further work: Make experimental comparison Notation Minimal deviation from standard ExpressivenessIntuitive / Easy to readLess Complexity SimpleLargeSimpleLargeSimpleLargeSimpleLarge Annotated ++++ - - - + Location Swimlanes --++ - -- + Colours --+ ++

11. 11 Experimental design Compare two diagram alternatives, annotation and colour Controlled experiment looking at the participants’ –Performance using the notation –Opinion about the notation Within-subjects design (Latin squares) –Controls better for selection bias –”Doubles” the N Measured variables: –Performance: Understanding: score on 12 True/False questions about a case after reading textual description and seeing diagram Error_detection: score on identifying 5 deliberately seeded errors in a diagram relative to textual description –Opinion: Perceived Ease of Use, Perceived Usefulness, and Intention to Use, measured by answers to a TAM-inspired questionnaire w 14 questions

12. 12 Hypotheses Since the colour notation was best in the analytical comparison, this was hypothesized to have advantages, i.e.: –H1: understanding scores will be better for the colour notation than for the annotated notation –H2: error detection scores will be better for the colour notation than for the annotated notation –H3: participants’ opinion about the colour notation will be more positive than for the annotated notation

13. 13 Experimental tasks 1.Answering a pre-experiment questionnaire investigating relevant competence 2.Reading tutorial about first diagram notation (annotatated or colour depending on group) 3.Reading text and diagram for case (home care or traffic control), answering 12 T/F questions 4.Answering post-task questionnaire giving opinion about the notation 5.Repeating steps 2-4 with the opposite case and notation 6.Repeating 2-6 with the error detection task

14. 14 Latin squares design Group Id(Understanding+ TAM factor) Questionnaire on Error Identification Questionnaire on Group AAnnotated Home Care + Colour Traffic Control Annotated traffic Control + Colour Home Care Group BColour Traffic Control + Annotated Home Care Colour Home Care + Annotated traffic Control Group CAnnotated traffic Control + Colour Home Care Annotated Home Care + Colour Traffic Control Group DColour Home Care + Annotated traffic Control Colour Traffic Control + Annotated Home Care

15. 15 Results 46 students participated, randomly assigned into the four Latin squares groups Clear advantage for colour notation in performance –3 students performed very poorly in error detection (not doing a serious job); might be considered outliers? Slight, but not significant advantage for colour when it comes to opinion Compared variable (N=46) Coloured diagram Annotated diagram Diffe- rence Effect Size Sign.? Y/N (p-value) MeanSDMeanSD Understanding0.9600.0570.9260.0780.03440.51Y (0.01) Error detection4.501.343.931.170.570.45Y (0.001) Errors (w/o outliers, N=43) 4.770.534.161.040.600.77Y (0.001)

16. 16 Conclusion on hypotheses H1: coloured notation would be better than annotated for understanding (answering T/F questions): CONFIRMED H2: coloured notation would be better than annotated for detecting errors: CONFIRMED H3: participants’ opinion about the coloured notation would be more positive: REJECTED

17. 17 Threats to validity Conclusion validity: significant results but small to moderate effects, should have had larger N to make a strong claim about results Construct validity: many other ways of understanding a model than answering T/F questions, and many other work tasks than identifying errors. But at least, this is a relevant task, and the ability to answer questions correctly should indicate to some extent whether a model has been understood

18. 18 Threats to validity, cont. Internal validity: Latin squares design and pre-exp. questionnaire should control very well for any selection bias. Notations were presented in equal detail and style in tutorials, and no preferred or hypothesized outcome was signalled to the students External validity: The biggest challenge: –Students are not practitioners and motivation may be limited in an experiment which has no impact on their job or a delivered product. But the comparative nature of the experiment should mean that performance with both notations are equally hurt by low competence or motivation. –Small experimental tasks are not representative of the more complex tasks in ”real” mobile IS development.

19. 19 Further work Also experiment with different pattern fills instead of colour (e.g. better for colour blind users) More experiments, possibly including practitioners and/or including collaboration among several persons instead of just individuals answering questions Larger industrial case studies, to try out alternative notations with larger and more realistic work tasks