1. Ghent, July 6, 2004 1 Construction and Pre-test of a Semantic Expressiveness Measure for Conceptual Models Ann Maes Frederik Gailly Geert Poels Roland Paemeleire (Ghent University)

2. 2 Ghent, July 6, 2004 outline Research objectives and questions Theoretical foundations Research methodology Item generation Item Refinement Reliability and Validity analysis Discussion

3. 3 Ghent, July 6, 2004 Research objectives and questions REA accounting model  one quality: semantic expressiveness SE: “how well a model reflects the underlying reality the model represents”

4. 4 Ghent, July 6, 2004 Alleged benefits of SE Easier integration of representations of accounting and non-accounting phenomena Better user understanding of accounting systems  Empirical test (Dunn & Grabski, 2000)

5. 5 Ghent, July 6, 2004 Empirical test of SE REA <> DCA Measure: a seven-point Likert scale assertion  “the documentation I received provided me with a realistic representation of the accounting information flows of the business” Results:  Users perceive REA-model based accounting system as more semantically expressive  SE associated with higher accuracy in information retrieval tasks

6. 6 Ghent, July 6, 2004 Our research objective Development of better measure for PSE  multi-item measurement instrument  yield a more pure indicant of the conceptual variable  flexible measurement instrument  evaluate the REA model but also other conceptual modeling approaches

7. 7 Ghent, July 6, 2004 Theoretical foundations Conceptual definition of the construct to be measured (  content validity) Dunn & Grabski : SE = property of REA model “how well a model reflects the underlying reality the model represents” McCarthy: SE = property of a schema “the degree to which elements in the final enterprise schema correspond to or capture the meaning of elements in the modelled corporate reality”

8. 8 Ghent, July 6, 2004 Lindland et al. framework for quality in conceptual modeling Language – domain – representation – audience interpretation Semantic quality: correspondence between domain and representation (cf. McCarthy) Language-domain appropriateness: a measure of how the language fits the domain, the extent to which the language makes the kind of statements needed in the domain (cf. Dunn & Grabski)

9. 9 Ghent, July 6, 2004 Wand & Weber research agenda for IS and conceptual modelling research (2002) Grammar quality  script quality Evaluation needs  SE = semantic quality Lastly: semantic quality of a schema is impossible to evaluate empirically as it requires observing and interpreting the problem domain  perceived semantic quality

10. 10 Ghent, July 6, 2004 Perceived semantic expressiveness Definition: “the correspondence between the user understanding of a representation developed according to a language and the user understanding of the reality that needs to be represented”

11. 11 Ghent, July 6, 2004 Research methodology Item Generation Step 1 – Literature Review and Scale Creation Examine literature for existing scales Assess applicability of existing scales and revise if appropriate Develop new items as necessary based on conceptual definition/ theoretical framework Item Refinement Step 2 – Pre-Test Test scale and items using convenience sample Calculate reliability and validity scores and modify scales as necessary Purify measurement instrument Confirmatory Analysis Step 3 – Gather Field Dataset Collect response data from representative random sample Step 4 – Exploratory Factor Analysis Perform factor analysis on initial measurement model Assess reliability and validity of measurement model Remove “problem” indicators, if any from measurement model Step 5 – Confirmatory Factor Analysis (CFA) Examine overall fit and inspect item-level fit for multi-dimensionality Assess modified scales for acceptable reliability

12. 12 Ghent, July 6, 2004 Item generation Item nameQuality propertyDefinition*Item statement correctCorrectness All statements in the representation are correct and relevant to the problem The conceptual schema represents the business process correctly relevantCorrectnessThe conceptual schema shows only relevant entities, relationships, and structural constraints completeCompleteness The representation contains all statements about the domain that are correct and relevant The conceptual schema gives a complete representation of the business process adequateCompletenessEntities, relationships or structural constraints must be added to adequately represent the business process minimalMinimalityThe representation does not contain statements that over constrain the domain None of the entities, relationships, and structural constraints in the conceptual schema can be removed consistentConsistencyThe representation does not contain contradictory statements The conceptual schema contains inconsistencies realisticSingle-item measurement instrument for perceived semantic expressiveness of Dunn and Grabski (2000) The conceptual schema is not a realistic representation of the business process *taken from Lindland et al. (1994)

13. 13 Ghent, July 6, 2004 Item Refinement Pre-test  Experiment with convenience sample  Aim = other quality aspects  Post experiment questionnaire with also two other measures for perception based variables: - perceived ease of use - user information satisfaction

14. 14 Ghent, July 6, 2004 PEOU instrument Item codeItem statement PEOU_1I found the conceptual schema cumbersome (confusing) to use PEOU_2Using the conceptual schema required a lot of mental effort PEOU_3The conceptual schema was clear and understandable to me PEOU_4Overall, I found the conceptual schema easy to use PEOU_5Using the conceptual schema was frustrating * Adapted from (Davis, 1989)

15. 15 Ghent, July 6, 2004 UIS instrument Item code Item questionAnchor at score 1 Anchor at score 7 UIS_1How adequately do you believe the conceptual schema meets the information needs that you were asked to support? adequateinadequate UIS_2How efficient is the conceptual schema for providing the information you needed? efficientinefficient UIS_3How effective is the conceptual schema for providing the information you needed? effectiveineffective UIS_4Overall how satisfied are you with the conceptual schema for providing the information you needed? dissatisfiedsatisfied * Adapted from Seddon and Yip (1992)

16. 16 Ghent, July 6, 2004 Reliability and Validity analysis *** Correlation is significant at the 0,01 level (2-tailed) ** Correlation is significant at the 0,05 level (2-tailed) * Correlation is significant at the 0,1 level (2-tailed)

17. 17 Ghent, July 6, 2004 Reliability analysis on initial PSE instrument Item-Total Correlation Alpha if item deleted Correct0.57280,6553 Relevant0.27730.7120 Realistic0.53970,6499 Complete0.56580.6368 Minimal0.21350.7384 Consistent0.28360.7082 Adequate0.61440.6231 Alll items Alpha = 0,7103

18. 18 Ghent, July 6, 2004 Remaining items… *** Correlation is significant at the 0,01 level (2-tailed) ** Correlation is significant at the 0,05 level (2-tailed) * Correlation is significant at the 0,1 level (2-tailed)

19. 19 Ghent, July 6, 2004 Reliability analysis on refined PSE instrument Item-Total CorrelationAlpha if item deleted Correct0.65010.6682 Realistic0.46480.7284 Complete0.64130.6604 Consistent0.28200.7858 Adequate0.58970.6824 All items Alpha = 0,7535

20. 20 Ghent, July 6, 2004 Discussion “consistent item” Compared to Dunn & Grabski single item measurement Confirmatory analysis  Larger sample  Different set of participants Reliable and valid flexible multi-item measurement instrument for PSE