Eindhoven Technische Universiteit Characteristics Empirical: Gather evidence through observation and measurement that can be replicated by others Measurement Replicability Objectivity
Eindhoven Technische Universiteit Variables Independent: Cause Dependent: Effect
Eindhoven Technische Universiteit Scientific research Validity: Are you measuring what you claim to measure ( measuring the right thing) Reliability: The ability to produce the same results under the same condition (Measuring thing right) Error: The difference between our measurements and the value of the construct we are measuring
Eindhoven Technische Universiteit Validity Internal validity problems Group threats, regression to the mean, time threats, history, maturation, instrumental change, differential mortality, reactive and experimenter effects External validity problem Over-use pf special participants group, restricted number of participants
Eindhoven Technische Universiteit Between groups Treatment (experimental gp.) No Treatment (control gp.) Measurement Random allocation
Eindhoven Technische Universiteit Measuring User Satisfaction Using Virtual Reality and Bayesian Belief Networks. 01.11.2004 Maciej A. Orzechowski
Eindhoven Technische Universiteit Motivations, aims Current techniques for measuring user preferences (CA, MM, interview) are artificial, lengthy or expensive. For good results we need to get the respondents more involved in the measurement. Can Virtual Reality (VR) improve the quality of measuring preferences: more involved and higher reliability? The aim of this project was to develop and test an interactive VR tool for measuring housing preferences.
Eindhoven Technische Universiteit Interactive Virtual Environment (iVE). Modification of a design. Translation of applied modifications into choices. Entering this information into a Bayesian Belief Network. Checking the consistency (if necessary prompting for verification). Learning (updating) the preference network. Solution strategy
Eindhoven Technische Universiteit VR System MuseV3 – a Virtual Reality application with functionality of a simple CAD system. Two categories of modifications: Structural modifications (change layout). Textural modifications (change visual impression).
Eindhoven Technische Universiteit Structural Modifications Change of internal and external dwelling’s layout. The most important for estimating user preferences. Include following commands: create/resize space; insert openings. Direct impact on overall costs of the dwelling.
Eindhoven Technische Universiteit MuseV3 in Desktop CAVE
Eindhoven Technische Universiteit Bayesian Belief Network Non-obtrusive interactive method to collect housing preferences. Potential advantages Interaction with the model during the time of preferences estimation. Incremental learning. Possibility to assess: where the knowledge about preferences is most uncertain. consistency of measurements.
Eindhoven Technische Universiteit Bayesian Belief Network cont. A Bayesian Belief Network (BBN) captures believed relations (which may be uncertain, stochastic, or imprecise) between variables, which are relevant to some problem. Lounge Ext (β1) Garage Ext (β2) Extra Kitchen (β3) 2 Bedrooms (β4) First Floor Ext (β5) Dormer Window (β6) Choice of Lounge Ext Choice of Garage Ext Choice of Extra Kitchen Choice of 2 Bedrooms Choice of First Floor Ext Choice of Dormer Window Price (γ) Family SituationAge
Eindhoven Technische Universiteit Experiment 1600 letters -> 100 answers -> 64 respondents. Respondents were people searching for a house or who just bought one. 4 kinds of 2 types of tasks (2 traditional, 2 based on MuseV3): CA: Verbal Description Only (VDO) Multimedia Presentation (MM). BBN: Preset Options (PO) Free Modification (FM). Each respondent completed both types of tasks.
Eindhoven Technische Universiteit Experiment Types – cont. VR ExperimentCA Experiment TypeFree ModificationPreset OptionsMultimedia Presentation Verbal Description Software (Mean of presentation) MuseV3 FMMuseV3 OEMuseV3 SCWeb Pages Collection Method Interaction with 3D environment Questionnaire TaskModification of architectural design Respond to pre designed options Choice from between three design alternatives Interactivity with 3D model Restrained to design constrains Finishes and furniture Walk ThroughN/a Feedback from the system yes none Estimation method Belief Network MNL Model
Eindhoven Technische Universiteit Analysis Estimation of separate models for each task. Test for order effect. Comparison of CA and BBN models in terms of: Internal validity. Predictive validity. Questionnaire.
Eindhoven Technische Universiteit Internal Validity CA vs. BBN Roughly similar between CA and BBN. Estimated utilities are not identical but strongly correlated. The difference in scale suggests that the BBN has a lower error variance. The task order effect suggests VR pre-learning improves the validity.
Eindhoven Technische Universiteit External Validity CA vs. BBN Models based on BBN on average predicted correctly 69% of the choices. Models based on CA on average predicted correctly 56% of the choices. High increase in CA model performance when task is preceded by VR task: for VDO from 56% to 62%. for MM from 32% to 73%.
Eindhoven Technische Universiteit Conclusions The results support the potential of the suggested approach. The results suggests higher involvement of respondents. This approach is non-obtrusive compared to different preference measurement techniques. The system (tool) can be used to: To assist individual users in creating their own design. To derive market potential of housing designs at aggregate level.