Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 D. (Denis) Ssebuggwawo 1, S.J.B.A. (Stijn) Hoppenbrouwers 1 & H.A. (Erik) Proper 1,2 1 ICIS, Radboud University Nijmegen, The Netherlands 2 Public Research.

Similar presentations


Presentation on theme: "1 D. (Denis) Ssebuggwawo 1, S.J.B.A. (Stijn) Hoppenbrouwers 1 & H.A. (Erik) Proper 1,2 1 ICIS, Radboud University Nijmegen, The Netherlands 2 Public Research."— Presentation transcript:

1 1 D. (Denis) Ssebuggwawo 1, S.J.B.A. (Stijn) Hoppenbrouwers 1 & H.A. (Erik) Proper 1,2 1 ICIS, Radboud University Nijmegen, The Netherlands 2 Public Research Centre -- Henri Tudor, Luxembourg 3 rd Working Conference on The Practice of Enterprise Modeling (PoEM’10) Delft University, The Netherlands 9-10 November, 2010 Assessing Collaborative Modeling Quality Based on Modeling Artifacts

2 Overview Collaborative Modeling Evaluation Hypothesized Model & Alternative Model Empirical Results Conclusion & Future Direction MENU

3 3 Overriding Goals Determine the Efficacy: (Efficiency & Effectiveness ) - evaluate the different constructs (ML, MP, EP, ST) to determine the overall efficiency and effectiveness Efficiency : reduce the effort Effectiveness: improve the quality of the result Determine the Success of collaborative effort : (Success factors) - evaluate the modeling effort to determine (critical) success factorsthat influence the efficiency & effectiveness. Overview

4 4 Modeling Artifacts Anchoring Collaborative modeling Evaluation on modeling artifacts Modeling Language (ML) Modeling Procedure (MP) End-Products (EP) Support Tool or Medium (ST/M) Overview

5 5 The Modeling Artifacts Overview ArtifactExplanation ML Concepts (constructs) in which the modelers express and communicate the solution. MP Processes (methods) for defining the problem and is reaching solution. EP Intermediary and end-products (models ). ST Enabling environment and support tools for the interaction and collaboration, communication, etc.

6 6 Supporting Frameworks SEQUAL (Lindland et al., 1994; Krogstie, et al., 2006) Based on: Semiotic theory Understanding the quality of conceptual models TAM (Davis 1986; Davis et al., 1989) TRA/TPB (Fishbein, 1975; Ajzen, 1991) About: Attitudes, Beliefs, Intentions/Perceptions, Behaviour Explaining & predicting user acceptance of IS/ITs MEM (Moody, 2001; Moody, 2003) Based on : Methodological pragmatism (Theo.Know validation) Evaluating IS design methods & TAM CM Evaluation

7 7 Theory of Reasoned Action (TRA) CM Evaluation PERCEPTIONS INTENTIONS BEHAVIOUR External/ Internal Psychological Behavioural Environment variables variables variables Attitudes toward Act or Behaviour (AB) Subjective Norm (SN) Behavioural Intention (BI) Actual Behaviour (B) Behavioural Beliefs and Outcome Evaluations (  bb i oe i ) Normative Beliefs and Motivation to Comply (  nb j mc j ) Uncontrollable factors Controllable factors Fig. 1. TRA Model

8 8 CM Evaluation Hypothesized Model Interactions … MLMP EP ST ML_1 ML_2 ML_n ST_1ST_2ST_n … EP_1EP_2 EP_n … MP_1MP_2 … MP_n Fig. 2. Hypothesized Model Interactions

9 9 The Constructs Perceived Quality of the Modeling Language (PQML) Perceived Usefulness of the Modeling Procedure (PUMP) Perceived Quality of the End-Products (PQEP) Ease of Use of the Medium or Support Tool (EOUM/ST) CM Evaluation

10 10 CM Evaluation : Original Quality Dimensions ArtifactQuality Dimensions#Sources Modeling Language (ML) construct deficit, construct overload, construct redundancy, construct excess; expressive power, directness, systematicity; syntax, semantic & pragmatic clarity; modeling primitive adequacy 10 Wand and Weber (1993), Lindland et al. (1994), Krogstie et al. (2006), Krogstie et al. (2001), List and Korherr (2006), Nysetvold and Krogstie (2005), Soderstrom et al. (2002), Stirna and Persson (2007) Modeling Procedure (MP) efficiency; effectiveness; ease of application, in-out- description adequacy, process & relation description adequacy, method compatibility, interaction & collaboration adequacy, communication & negotiation adequacy; rule & goal commitment, shared understanding 10 de Brabander and Thiers (1984), Duivenvoorde et al. (2009), Krogstie et al. (2006), Gemino and (2003), Hengst et al. (2006), Reinig (2003), Siau and Wang (2007), Siau and Rossi (1998), Recker (2006), Stirna and Persson (2007), Renger et al. (2008), Becker et al. (2000), Ssebuggwawo et al. (2009) End- Product (EP) correctness, completeness, propriety, clarity, consistency, orthogonality, generality, syntax adherence adequacy, semantics adequacy, pragmatics adequacy; user-comprehensibility; modifiability, re- usability, flexibility; user satisfaction. 15 Lindland et al. (1994), Krogstie et al. (2006), Sedera et al. (2003), Pfeiffer and Niehaves (2005), Paul et al. (2004), Reinig (2003), Rosemann et al. (2001), Stirna and Persson (2007), Schuette and Rotthowe (1998) Medium/ Support Tool (M/ST) tool functionality, performance & reliability; efficiency, effectiveness; satisfaction; synchronicity, negotiation/argumentation adequacy, commenting/proposition adequacy, planning/agenda setting adequacy 9 Dean et al. (1994), Fjermestad and Hiltz (1999), Stirna and Persson (2007), Krogstie et al. (2006), Ssebuggwawo et al. (2009)

11 11 CM Evaluation: Synthesized Quality Dimensions ConstructQuality Dimensions - New Groupings PQML Understandability (ML1), Clarity (ML2), Syntax correctness (ML3), Conceptual minimalism (ML4) PUMP Efficiency (MP 5), Effectiveness (MP6), Satisfaction (MP7), Commitment & Shared Understanding (MP8) PQEP Product Quality (EP9), Understandability (EP10), Modifiability & Maintainability (EP11), Satisfaction (EP12) EOUM /STFunctionality (ST13), Usability (ST14), Satisfaction & Enjoyment (ST15), Collaboration & Communication Facilitation (ST16)

12 Hypothesized (a priori) Model Fig. 3. Hypothesized Model

13 13 Alternative (Competing) Model Fig. 4. Competing Model

14 Modeling Experiment: Collaborative modeling session using COMA tool Evaluation: Using a measurement instrument (Questionnaire) 7-pt Likert Scale Constructs to assess: PQML, PUMP, PQEP and EOUMQuestionnaire Empirical Results Modeling Expt. & Evaluation

15 Goal: Retain factors that account for significant amount variance in the data. Precursor to CFA. Principal Component Analysis (PCA): Data reduction : determing the number of factors Factor rotation : determining the (non-)correlation of factors Common Factor Analysis/Principal Factor Analysis: understanding the relationship btwn: indicators (measured: MLs, MPs, EPs, STs) variables in terms of factor (latent: PQML, PUMP, PQEP, EOUM) variables Empirical Results Validation & Reliability Tests Exploratory Factor Analysis(EFA)

16 EFA Results EFA Results c Empirical Results Validation & Reliability Tests Exploratory Factor Analysis(EFA)

17 A priori hypotheses: Testing: a priori hypotheses/theories Assessing Goodness-of-fit: Assessing the goodness-of-fit based on variance after factor reduction in EFA Assessing validity & reliability : Testing and confirming the validity & reliability of a measurement instrument Empirical Results Validation & Reliability Tests Confirmatory Factor Analysis(CFA) (Structural Equation Modeling (SEM))

18 Fig. 5. Path diagram Model 1 Model 1: Hypothesized CFA Results Model:

19 Fig. 6. Path diagram Model 2 Model 2: Competing CFA Results Model:

20 CFA Results CFA Results c Empirical Results Validation & Reliability Tests Confirmatory Factor Analysis(CFA)

21 Conclusion: Rather than model quality, other artifacts can be used in the evaluation of quality and success of a collaborative modeling effort. Future Direction: Establishing the interdependencies of the artifacts and their impact on the overall quality Measuring the acceptability and adoption of the quality framework in practice Conclusion & Future Direction

22 22 Thank you. Questions ?


Download ppt "1 D. (Denis) Ssebuggwawo 1, S.J.B.A. (Stijn) Hoppenbrouwers 1 & H.A. (Erik) Proper 1,2 1 ICIS, Radboud University Nijmegen, The Netherlands 2 Public Research."

Similar presentations


Ads by Google