Presentation is loading. Please wait.

Presentation is loading. Please wait.

Eindhoven Technische Universiteit Measuring User Satisfaction through Experiments B. de Vries.

Similar presentations

Presentation on theme: "Eindhoven Technische Universiteit Measuring User Satisfaction through Experiments B. de Vries."— Presentation transcript:

1 Eindhoven Technische Universiteit Measuring User Satisfaction through Experiments B. de Vries

2 Eindhoven Technische Universiteit Domotica Computable: 5 November 2004 Wildgroei Toekomst droom Losse deelmarkt en

3 Eindhoven Technische Universiteit Innovatie Juiste product ? Juiste doelgroep ? Juiste distributie ? Juiste tijd ? Juiste marketing ? …

4 Eindhoven Technische Universiteit Experiments Observational Experimental

5 Eindhoven Technische Universiteit Characteristics Empirical: Gather evidence through observation and measurement that can be replicated by others Measurement Replicability Objectivity

6 Eindhoven Technische Universiteit Variables Independent: Cause Dependent: Effect

7 Eindhoven Technische Universiteit Scientific research Validity: Are you measuring what you claim to measure ( measuring the right thing) Reliability: The ability to produce the same results under the same condition (Measuring thing right) Error: The difference between our measurements and the value of the construct we are measuring

8 Eindhoven Technische Universiteit Validity Internal validity problems Group threats, regression to the mean, time threats, history, maturation, instrumental change, differential mortality, reactive and experimenter effects External validity problem Over-use pf special participants group, restricted number of participants

9 Eindhoven Technische Universiteit Between groups Treatment (experimental gp.) No Treatment (control gp.) Measurement Random allocation

10 Eindhoven Technische Universiteit Measuring User satisfaction Virtual Reality Bayesian Belief Networks

11 Desk-Cave




15 Eindhoven Technische Universiteit Set-UP 2 synchronized PC’s with dual monitor output 4 LCD Projectors

16 Eindhoven Technische Universiteit Features 1 : 1 Scale 3DS import Immersion

17 Eindhoven Technische Universiteit Bayes Theorem From: Evaluation and Decision (7M834)

18 Eindhoven Technische Universiteit Bayesian Belief Network Norma n Late Martin Late Train Strike

19 Eindhoven Technische Universiteit Node Probability Table Norma n Late Martin Late Train Strike Train Strike Norman late TrueFalse True0.80.1 False0.20.9

20 Eindhoven Technische Universiteit NPT’s Train Strike Martin lateTrueFalse True0.60.5 False0.40.5 Train strike True0.1 False0.9

21 Eindhoven Technische Universiteit Analyzing a BBN p(Norman late) = p(Norman late | train strike) * p(train strike) + p(Norman late | no train strike) * p(no train strike) = (0.8 * 0.1) + (0.1 * 0.9) = 0.17 Marginal probability Conditional probability p(Train|Norman late) = (p(Norman late|train strike) * p(train strike))/ P(Norman late) = (0.8 * 0.1) / 0.17 = 0.47

22 Eindhoven Technische Universiteit Measuring User Satisfaction Using Virtual Reality and Bayesian Belief Networks. 01.11.2004 Maciej A. Orzechowski

23 Eindhoven Technische Universiteit Motivations, aims Current techniques for measuring user preferences (CA, MM, interview) are artificial, lengthy or expensive. For good results we need to get the respondents more involved in the measurement. Can Virtual Reality (VR) improve the quality of measuring preferences: more involved and higher reliability? The aim of this project was to develop and test an interactive VR tool for measuring housing preferences.

24 Eindhoven Technische Universiteit Interactive Virtual Environment (iVE). Modification of a design. Translation of applied modifications into choices. Entering this information into a Bayesian Belief Network. Checking the consistency (if necessary prompting for verification). Learning (updating) the preference network. Solution strategy

25 Eindhoven Technische Universiteit VR System MuseV3 – a Virtual Reality application with functionality of a simple CAD system. Two categories of modifications: Structural modifications (change layout). Textural modifications (change visual impression).

26 Eindhoven Technische Universiteit Structural Modifications Change of internal and external dwelling’s layout. The most important for estimating user preferences. Include following commands: create/resize space; insert openings. Direct impact on overall costs of the dwelling.

27 Eindhoven Technische Universiteit MuseV3 in Desktop CAVE

28 Eindhoven Technische Universiteit Bayesian Belief Network Non-obtrusive interactive method to collect housing preferences. Potential advantages Interaction with the model during the time of preferences estimation. Incremental learning. Possibility to assess:  where the knowledge about preferences is most uncertain.  consistency of measurements.

29 Eindhoven Technische Universiteit Bayesian Belief Network cont. A Bayesian Belief Network (BBN) captures believed relations (which may be uncertain, stochastic, or imprecise) between variables, which are relevant to some problem. Lounge Ext (β1) Garage Ext (β2) Extra Kitchen (β3) 2 Bedrooms (β4) First Floor Ext (β5) Dormer Window (β6) Choice of Lounge Ext Choice of Garage Ext Choice of Extra Kitchen Choice of 2 Bedrooms Choice of First Floor Ext Choice of Dormer Window Price (γ) Family SituationAge

30 Eindhoven Technische Universiteit CPT calculation

31 Eindhoven Technische Universiteit Learning process

32 Eindhoven Technische Universiteit Convergence

33 Eindhoven Technische Universiteit Utility Convergence

34 Eindhoven Technische Universiteit Experiment 1600 letters -> 100 answers -> 64 respondents. Respondents were people searching for a house or who just bought one. 4 kinds of 2 types of tasks (2 traditional, 2 based on MuseV3):  CA: Verbal Description Only (VDO) Multimedia Presentation (MM).  BBN: Preset Options (PO) Free Modification (FM). Each respondent completed both types of tasks.

35 Eindhoven Technische Universiteit Experiment Types – cont. VR ExperimentCA Experiment TypeFree ModificationPreset OptionsMultimedia Presentation Verbal Description Software (Mean of presentation) MuseV3 FMMuseV3 OEMuseV3 SCWeb Pages Collection Method Interaction with 3D environment Questionnaire TaskModification of architectural design Respond to pre designed options Choice from between three design alternatives Interactivity with 3D model Restrained to design constrains Finishes and furniture Walk ThroughN/a Feedback from the system yes none Estimation method Belief Network MNL Model

36 Eindhoven Technische Universiteit Analysis Estimation of separate models for each task. Test for order effect. Comparison of CA and BBN models in terms of:  Internal validity.  Predictive validity. Questionnaire.

37 Eindhoven Technische Universiteit Internal Validity CA vs. BBN Roughly similar between CA and BBN. Estimated utilities are not identical but strongly correlated. The difference in scale suggests that the BBN has a lower error variance. The task order effect suggests VR pre-learning improves the validity.

38 Eindhoven Technische Universiteit External Validity CA vs. BBN Models based on BBN on average predicted correctly 69% of the choices. Models based on CA on average predicted correctly 56% of the choices. High increase in CA model performance when task is preceded by VR task:  for VDO from 56% to 62%.  for MM from 32% to 73%.

39 Eindhoven Technische Universiteit Observed-Predicted

40 Eindhoven Technische Universiteit Conclusions The results support the potential of the suggested approach. The results suggests higher involvement of respondents. This approach is non-obtrusive compared to different preference measurement techniques. The system (tool) can be used to:  To assist individual users in creating their own design.  To derive market potential of housing designs at aggregate level.

41 Eindhoven Technische Universiteit Domotica applications Alarmering: inbraak, zorg, brand Autom. Verlichting Autom. Zonwering …

Download ppt "Eindhoven Technische Universiteit Measuring User Satisfaction through Experiments B. de Vries."

Similar presentations

Ads by Google