Presentation on theme: "…continued… Part III. Performing the Research"— Presentation transcript:
1 …continued… Part III. Performing the Research 3 Initial Research4 Research Approaches:4.1 Experience4.2 Modeling and Simulation4.3 Experimental Design4.4 Qualitative Approaches4.5 Quantitative Approaches4.6 Collecting Data from Respondents (surveys, etc…)4.7 Observing System Behavior5 Hypotheses6 Data Collection7 Data Analysis
2 4.1 Experience The main role for experience in research is: effective design of a research project:selecting a do-able and valuable projectselecting and designing appropriate methods and tools:eg: design survey to extract information (everything relevant, and without introducing bias)effective execution of the research:careful execution of experiments/surveys/etc... without introducing errorsnot missing any new insights that may not have been expectedeg: the discovery of penicillinthorough and insightful analysis of results, and drawing of conclusions:through use of appropriate toolspatience, and creativity
3 4.2 Modeling and Simulation What is a model?:a representation of an actual or designed system/object, used to:describe the represented system (for education)gain insights into the represented system (for education, or inference)compare the behavior of alternative versions of the represented system (to optimize design):eg: try different numbers of trucks in an earthmoving system, to minimize costspredict how the system will behave under different scenarios (to plan for future events, or evaluate what happened in the past or present):eg: how will the energy consumption of a building change with different weather conditions
4 What is simulation?:the running of a model that is dynamic (typically that varies with time):involves a class of models that need to be advanced through time in a step-wise manner, either because:there is no known direct solution to the problem,eg: simulation of construction process simulationeg: simulation of building acousticsor, the user needs to interact with the model through time:eg: equipment training simulatorseg: management gaming (to develop individual and group skills in project management)
5 Classifications of models Rosenblueth and Weiner (1945):Material models:transformations of original physical objectsFormal models:logical, symbolic assertions of situations
6 CYCLONE Excavator-Truck Based Earthmoving Model Churchman et al. (1957):iconic models:visual or pictorial representations of aspects of a real systemeg: design drawingseg: simulation model diagramsCYCLONE Excavator-Truck Based Earthmoving ModelDigloadWait fordump truckHaulDumpReturnEXCAVATOR CYCLEDUMP TRUCKS CYCLE1 excavator (1.5 cu yd bucket)5 dump trucks at start(15 cu yd capacity each)SleubackexcavatorGEN10CONRepairp = 0.95p = 0.05
7 analog models:adopt a system with a set of properties that have an appropriate correspondence with the system under investigation:eg: electrical circuit representing heat flow in building (the behavior of the two systems follows the same mathematical principles)eg: hydraulic analog of UK economy (MONIAC)
8 y = m.x + c m * c symbolic models: involves logical or mathematical representationseg: linear regression model*House priceSquare footagey = m.x + ccm
9 Sayre and Crosson (1963): replications: have significant physical similarity to the realityformalisations:symbolic modelseg: linear regression modelsimulations:requires stepwise evaluation to generate resultseg: CYCLONE simulation
10 Fellows and Liu (1997) (synthesis of other classifications): iconicreplicationsanalogssymbolic
11 There are other ways of dividing-up models. A common dichotomy is: The previous divisions classify modeling methods by their means of representing a problem.There are other ways of dividing-up models. A common dichotomy is:Stochastic:this group of models recognize that some aspects of a problem are uncertain, and builds this into the model:eg: how long it takes to perform a construction activityeg: how much concrete will cost per cu ydeg: will the HVAC equipment pass or fail its inspectionmethods that take uncertainty into account include:eg: Monte Carlo samplingeg: PERTeg: Markovian models
12 Deterministic:this group of models ignore uncertaintyeg: only use the expected duration to perform a construction activityeg: use the expected cost rateeg: assume the HVAC equipment will pass inspection
13 The modeling process (from Fellows and Liu (1997), and Mihram (1972): Determine model’s objectives: its purpose(s); who will use it?Study the reality: the system, process, object to be modeled.Synthesize: combine components into model(s).Verify model(s): check model(s) for bugs or discrepancies from what you had intendedValidate model(s): assess its accuracy relative to the system being represented (overall accuracy, consistency across all of the problem)Select most appropriate model(s): such as, that which produces the most accurate results, or that with the most useful scope of application.Apply model(s): for predictions, for making comparisons, for planning, for making inferences, etc...
14 Example 1:Multivariate Linear Regression Modeling (estimating the expected annual energy costs of a house)(see Assignment 2)
15 Example 2:Construction Simulation Modeling (field application of an earthmoving system, comprising 1 excavator and ‘n’ dump-trucks):Objectives: find number of trucks that balances the excavator thereby minimizing construction costsStudy reality: determine the activities, determine the activity durations from historic data, determine the dependences between the activities and the interactions between the equipment, etc…Synthesize: design the simulation diagram (perhaps using CYCLONE or similar), input this to the computer using an appropriate simulation software packageCYCLONE Excavator-Truck Based Earthmoving ModelDigloadWait fordump truckHaulDumpReturnEXCAVATOR CYCLEDUMP TRUCKS CYCLESleubackexcavatorGEN10CONRepair
16 Cumulative Production for 2 truck system simulated actual Verify: run the simulation model, observing the behavior of the simulated system, to look for bugs in the model: eg: input a truck with a 10 cu yd bucket when meant to input 1 cu yd; or designed model so that a truck is filled by just 1 bucket load when it actually takes 10 bucket load cycles…Validate: run the model and compare its performance to available performance data (say for available data on a system with 2 dump-trucks) system comprising or compare to your expectations based on experience, to establish its accuracy, eg: qualitative graph comparison, quantitative statistical (Theil’s test)Cumulative production (cu yds)Time (hrs)Cumulative Productionfor 2 truck systemsimulatedactual
17 Cost versus number-trucks Select most appropriate model: repeat last 3 steps using variations of the model, looking for that which is most accurate, eg: try including breakdown or maintenance of trucks to see if it affects model accuracy, try running as a stochastic versus deterministic modelApply model: perform a sensitivity analysis, running the simulation with 1 truck, then 2 trucks, then 3 trucks, etc, measuring the cost of completing the work for each number of trucksCost to complete work ($)Number dump-trucksCost versus number-truckssensitivity analysis
18 Example 3:Construction Simulation Modeling (field application of an earthmoving system, comprising 1 excavator and ‘n’ dump-trucks):Objectives: find number of trucks that balances the excavator thereby minimizing construction costsStudy reality: determine the activities, determine the activity durations from historic data, determine the dependences between the activities and the interactions between the equipment, etc…Synthesize: design the simulation diagram (perhaps using CYCLONE or similar), input this to the computer using an appropriate simulation software packageCYCLONE Excavator-Truck Based Earthmoving ModelDigloadWait fordump truckHaulDumpReturnEXCAVATOR CYCLEDUMP TRUCKS CYCLESleubackexcavatorGEN10CONRepair
19 4.3 Experimental Design What is an experiment?: A test or study designed to investigate relationships between independent and dependent variables:toss a coin to see if it is biased;independent variables = the force of the flick of the coin; the point of contact on the coin; the time allowed for the coin to spin, etc.. – these are actually all randomizeddependent variable = heads or tailsimmersing a brick in a pool of water to see how much water it absorbs over time:independent variables = the type of brick; air pressure; temperature; time into experiment..dependent variable = quantity of water absorbed
20 Determining occupant satisfaction with automatic light sensors for turning lights on and off: independent variables = the amount of delay after perceiving movement before the lights are turned off; the sensitivity of the sensors to movement; the level of external sources of lighting;..dependent variable = the level of satisfaction of the users (may be measured with interviews post event).
21 Basic approach: Ideally, fix all independent variables except one: eg: fix air pressure and temperature..then vary the remaining independent variable:eg: time (to see how much water is absorbed over time)Dependent Variable(eg: water absorbed)Main Independent Variable(eg: time)Other Independent Variables(eg: temperature, air pressure..)*
22 Main Independent Variable (eg: time) Other Independent Variables Often, cannot fix all independent variables except one (there will be some variance in the other variables):eg: cannot regulate temperature or air pressure perfectlythen the results will include some errors (experimental errors) since these other variables will affect the resultstry to keep them as fixed as possible, then:use repeat experimentation to average out results (randomize errors);or maybe use some other form of curve fitting with smoothingDependent Variable(eg: water absorbed)Main Independent Variable(eg: time)Other Independent Variables(eg: temperature, air pressure..)*
23 Main Independent Variable (eg: time) Other Independent Variables example of repeat experimentation to average out error in results:Dependent Variable(eg: water absorbed)Main Independent Variable(eg: time)Other Independent Variables(eg: temperature, air pressure..)*
24 Main Independent Variable (eg: time) Other Independent Variables example of curve fitting:Dependent Variable(eg: water absorbed)Main Independent Variable(eg: time)Other Independent Variables(eg: temperature, air pressure..)*
25 Designing the set of experiments: Need to identify the problem domain:the range of values to be considered for each independent variableIndependentVariable 1Variable 2Scope of problem(problem domain)
26 Then determine the set of values to be considered within that domain: this may be done on a grid (example with 2 independent variables)IndependentVariable 1Variable ‘n’
27 resolution (fineness) of data samples the question is, how fine should the resolution of the grid be?the finer the resolution, the more information is provided by the results, but the more costly is the set of experimentsone way to determine the resolution is, if building a model of the data, to test the accuracy of the model. If it is not accurate enough, try increasing the resolution of the experimental data. – Can plot sensitivity of accuracy to experiment resolutionerrorresolution (fineness) of data samples*
28 Independent Variable 1 Variable ‘n’ the set of values measured in the experiments may be chosen randomlythe rules for determining the number of experiments is affected, as before, by the cost versus information gleanedin some situations (eg: when we just want to know the average value for a dependent variable) we can use statistics to determine the number of experiments to performthe greater the variance, the more experiments we need – later lectureIndependentVariable 1Variable ‘n’
29 Independent Variable ‘n’ (eg: air pressure) the set of values measured in the experiments may be on a grid that is not constantthis may be because the change in the dependent variable may vary (eg: become less for larger values of time)Independent Variable ‘n’ (eg: air pressure)IndependentVariable 1(eg: time)
30 Sometimes the problem domain is not a regular shape, and may imply some interdependence (correlation) between the independent variables:for example, when one independent variable has a large value, maybe a second independent variable tends to have a larger (or smaller) value;as a practical example, larger houses tend to have higher quality fittings:House SizeHousequality
31 Determining the size of a random sample: Some problems have a finite population to be assessed, for example:What percentage of the population will vote for an Independent candidate?the population of people eligible to vote is in the millions, orWhat percentage of architectural practices use nD CAD tools when designing an airport?the number of architectural practices capable of designing airports is just a handful.Other problems have an infinite population to be assessed:What is the average temperature of a room during a year periodthere are an infinite number of points in time when the temperature could be measured
32 When we sample from a population: For infinite populations, and large finite populations, it is not feasible to measure for every possible situation.Thus, we need to take a statistical sample, and make inferences from those results.Eg: measure the mean number of construction fatalities per year for each of a random selection of construction companiesWhen we sample from a population:the larger the number of samples (the greater the portion of the population we sample) then:the more accurate will be our inferences, butthe more expensive will be the cost of the study (time and $)
33 So, how large should the sample be? (a) we can decide on an acceptable margin of error (E) and level of confidence (often 95%), then calculate the required sample size (n), or(b) we can decide on an acceptable level of confidence and, for a given sample size, calculate the margin of error.Note, the margin of error will be the difference between the mean of the sample and the mean of the total population.Also note, problems with larger variance (or standard deviation) require more samples for a given E and confidence level:Consider the problem of estimating the man-hours required to design a 4,000 sq-ft custom home:If it always took 1,000 hours, then we would only require 1 measurement to establish the man-hours requiredThe more variance between design jobs, the more samples we would need to achieve an accurate estimate of the mean duration
34 Relationship between E and confidence limit: probability densityActual meanEg: 95% confidence limit =(95% of the area under curve)Eg: E = 100 man-hours differenceSample meanGreater confidenceleads to highererror (E)Sample meanActual meanEg: 80% confidence limit =(80% of the area under curve)Eg: E = 73 man-hours differenceprobability densityLower confidenceleads to lowererror (E)
35 Eg: 95% confidence limit = (95% of the area under curve) Relationship between E, confidence limit, and sample size:probability densityActual meanEg: 95% confidence limit =(95% of the area under curve)Eg: E = 100 sq-ft differenceSample meanSmall sample sizeEg: E = 100 man-hours differenceSmaller sample size= less certaintyprobability densityActual meanEg: 95% confidence limit =(95% of the area under curve)Eg: E = 53 man-hours differenceSample meanLarger sample size= more certainty
36 Example: a sample of 31 design firms provided the following data for the man-hours they required to design a 4,000 sq-ft custom home:Sample mean (SM) = 375 man-hours = [( ∑ d ) / n] where:d is the man-hours to design one homen is the number of design firms sampledSample standard deviation (SSD) = 19 = √ [(∑ (d-SM)2) / n ]What is the margin of error (E) for the expected (mean) man-hours if we want to be 90% (p=0.9) confident that we are within the error?the formula to calculate this is: E = z0.9/2∙ (SSD / √n )note: use (n-1) instead of n for small sample sizes less than 30first find z0.9/2 , that is, find z for p = 0.45 (use look-up table such asthus, z = 1.65thus E = 1.65 ∙ (19 / √ 31 ) = 5.6 man-hourswe are 90% confident that the expected value is between man-hours and man-hours
37 What is the margin of error (E) for the expected (mean) man-hours if we want to be 95% (p=0.95) confident that we are within the error?E = z0.95/2∙ (SSD / √n )first find z0.95/2 , that is, find z for p = (use look-up table atthus, z = 1.96thus E = 1.96 ∙ (19 / √ 31 ) = 6.69 man-hourswe are 95% confident that the expected value is between man-hours and man-hours
38 If we want to be 95% (p=0.95) confident that we are within plus or minus 3 man-hours of the actual expected man-hours, how many samples would we need?first, rearrange the formula: n = [( z0.95/2 ∙ SSD ) / E ]2then find z0.95/2 , that is, find z for p = (use look-up table atthus, z = 1.96thus n = [(1.96 ∙ 19) / 3 ] 2 = 154 sampleswe have to perform an additional = 123 samplesAfter the 154 samples have been collected and analyzed, we could recalculate this (with the new SSD) to see if we have enough samplesbut beware that this is a little like “keep collecting data until we get the answer we are looking for” !!!
39 Class example:Estimating average absolute difference between bid price and actual price for a class of construction contracts.
40 What if we want to measure more than one variable? eg: we want to measure:the average time spent designing the house, andthe fee charged for the design work, andthe square footage of the housethen, if the level of confidence and margin of error are the same for all variables, then apply the sample size formula to the variable with the largest sample standard deviation (SSD), since this will be the variable that requires the largest sample size and will thus satisfy all variablesif the level of confidence and/or the margin of error are different for the variables, then calculate the required sample size for each variable and take the largest answer.
41 Up to now, we have been concerned with determining an appropriate sample size when determining the expected (mean, average) value of some continuously valued variable of the population:eg: the mean duration to design houses of a given type?…here, duration is a continuous variableBut how do we determine the sample size for problems where we want to determine the portion of samples in a class?eg: what percentage of urban planners have a doctoral degree?…here, planners either have a PhD or they don’t, it is a categorized variable, not a continuous variable
42 The approach is similar to before, but we have to use a slightly different formula: n = [( zlimit/2))2 ∙ (p ∙ (1-p))] / E 2where:p is the portion of samples in one category, and (1-p) is the portion in the remaining categoriesZ is the confidence limit, andE is the margin of errorthis formula is essentially the same as before with the main exception that it replaces the sample standard deviation (SSD) with (p ∙ (1-p))
43 Eg: we want to know the portion of time a steel-fixer is either cutting reinforcing bars, bending reinforcing bars, or is doing something else:we don’t want to sit there and watch the worker as we have other things to do and, moreover, they may change their behavior;so we make spot observations at randomly selected points in timethe question becomes, how many spot observations should we make?say, for example, we have made 30 spot-observations (at randomly selected times) over a couple of days and we got the following readings:cutting reinforcing steel = 8 observationsbending reinforcing steel = 16 observationsdoing something else = 6 observationsset “p” to the portion for the set of observations closest to a half (as this gives us the largest required sample size)p = 16/30 = (for bending reinforcing steel)lets say we want a confidence limit of 95%, so z0.95/2= 1.96lets say we want a margin of error of E = ±5%thus n = ∙ (0.533 ∙ ( )) / = 382 samples (an additional 352 as we have already completed 30)
44 when all spot observation have been made, we simply multiply the working hours in the day by the portion of spot observations for each task, to get the hours per day spent performing each task.
45 What if we want to know whether or not variance in the means between several groups is significant: eg: we want to know if the type of construction project (commercial, industrial, infrastructure) affects the mean number of fatalities per year (measured, say, as per million man-hours).the expected number of fatalities may vary due to differences in the types of work, but differences in the sampled data may just be due to sampling biaswe can solve this type of problem using ANOVA (ANalysis Of VAriance between groups), the F-test – originally developed by Fischer (hence the F in F-test).
46 What if there are more than one variable: eg: we want to know if the type of construction project (commercial, industrial, infrastructure) affects the mean number of fatalities per year and the mean number of non-fatal accidents:we can solve this type of problem using MANOVA (Multivariate ANalysis Of VAriance between groups)We have just touched the surface of the topic of sampling theory – for a detailed reference, see “Sampling” 2ND Edition by Stephen K Thompson
47 What do we use experimentation for? for theorizing:to provide us with the insight necessary to develop a theory that describes the causal (cause-effect) relationships between the independent and dependent variables;to test the theory we just developed, or indeed any theory of interest (we compare experimental results with those predicted by the theory);Note, experimental theorizing is usually based on some abstraction of the real system under investigation, eg:a physical construction/representation of part of the system under investigation (such as immersing a brick in water as a representation of a brick wall)an empirically derived model that we can experiment with (such as a regression model of house prices)
48 and for empirical modeling (eg: the regression model developed for assignment 1): to build or develop an empirical model that describes the relationships between the independent and dependent variables;to evaluate the performance of alternative models (to select the best); andto validate the model (test its accuracy in all respects)
49 Do not confuse experimentation with ex-post-facto research: Experimentation is something that is planned and conducted with the purpose of understanding relationships between the system variables.it tends to lead to a high degree of internal validity (accuracy within the boundaries of the experiment), but does not extrapolate to the real world as well as ex-post-facto researchEx-post-facto research uses data collected from a system that has not been established specifically for understanding the relationships between its variables, eg:productivity observations for tasks in a given construction projectevaluation of accidents based on data collected by OSHAit usually has a higher external validity than experimentation, but tends to have less internal validityBoth experimentation and ex-post-facto approaches can be used for ‘theorizing’ and ‘empirical modeling’.
50 Replication:A desirable feature of research, in particular experimentation, is replication:this is necessary so that others can validate (or counter) your conclusions, andmore specifically, more replication of an experiment (under identical treatments) facilitates:identification of human error (mistakes made by the experimenter that can be avoided);identification of systematic error (errors that are reproducible and inherent in the experiment – eg: bias when recording room temperature resulting from heating of the air by the measuring equipment); andestimation of experimental error (random effects from unknown factors – eg: error when recording room temperature resulting from random voltage fluctuations in the measuring equipment);Replication of an experiment allows the experimental error to be measured more accurately:often this is measures as Standard Error which is an estimate of the standard deviation of the errors.
51 Replication requires meticulous care in recording all details of the experiment. In any case, it is important to take great care in setting-up and executing an experiment, as well as in recording the resultsso that no unnecessary errors or misconceptions are introduced into the results and conclusions.
52 4.4 Qualitative Approaches Aim of qualitative research:to obtain an in-depth understanding of human behavior and the factors that govern human behavior.it investigates the ‘why’ and ‘how’ of decision making, not just ‘what’, ‘where’, and ‘when’it typically relies on four methods for gathering information:participation in the setting,direct observation,in depth interviews, andanalysis of documents and materials.
53 4.5 Quantitative Approaches Aim of quantitative research:the identification and quantification of phenomena and their relationshipsit attempts to develop mathematical models of certain phenomena, to understand past systems or predict the behavior of future systems.two basic questions of a quantitative approach are:what is to be measured, andhow should those measurements be made?
54 4.6 Collecting Data from Respondents A lot of the research in our discipline (Design, Construction, and Planning) involves asking people questions.This is typical of research in the social and/or management sciences.These take the form of:questionnairesinterviews, andcase studies
55 Each approach is suited to a different class of problems, as follows: first, the stage of development of the research area:research fields that are at an early stage require better understanding at a qualitative levelin this case, use the interview format since it allows for: (i) flexibility during the questioning (the interviewer can adapt to the responses being given or probe if they are hesitating); (ii) requires longer answers (as uses more open-ended questions) which interviewers can transcribe (interviewees may resist giving long answers if they have to write it down);research fields that are at a later stage require better understanding at a quantitative levelin this case, use the questionnaire format since (given the closed-ended question style, and check box responses to many questions) it allows for: (i) more questions to be asked; (ii) larger samples to be taken of the population, to provide the statistical significance
56 * * * second, the scope and depth of the study: the scope is how widely applicable the study is:examples of a broad scope studies: ones covering all geographic areas, or all age groups, or companies from small to large sizethe depth of a study is how much detail it goes into:in depth studies identify more significant variables, and tend to quantify relationships*case studies (one or a very few samples)Depth of Study*Interviews (few samples)*Questionnaires (large sample)Scope of Study
57 The previous diagram is a generalization The previous diagram is a generalization. For example, it is possible for interviews to have a broader scope than questionnaires.
58 Questionnaires: A written or screen based series of questions The questions may be:Closed – asking specific things usually with a set of check-box answers, for example:How many employees does your company employ?: less than 10; l0 to 99; 100 to 1,000, etc…Project planning software helps keep projects on schedule - do you: strongly disagree; disagree; neutral; agree; strongly agree.Open – asking general questions without a prescribed set of answers (useful for getting to understand the issues and key variables in a problem), for example:What do you believe are the most important factors affecting project progress?Explain your answers to the previous question.
59 Some general advice on writing questionnaires: Use as few questions as possible, to maximize response rateKeep questions easy to understand, using simple unambiguous language. Avoid using scientific or technical terms unless you are sure the responder will understand.Make sure the questions cover the scope of the problem under investigationFor multiple choice questions:keep the scale the same from question to question as far as possible (to avoid causing confusion)always keep the direction the same, for example, use lower values for more negative responses, higher values for more positive responses: (disagree, neutral, agree; or -3, -2, -1, 0, +1, +2, +3, or 1-9, 10-99, 100-1,000Where appropriate, give the responder the choice of “other” with a prompt to explain. This should be employed when you are not sure of all the possible responses.
60 Use an appropriate mix of open and closed questions – start with closed questions so that the responder does not get put-off, and starts making a commitment.You can repeat questions to make sure you get the same response, but phrase them differently to disguise this fact.Be very careful to make sure your questions do not lead or imply a preferred answer. The following is a leading, and thus poor question:More women should be employed as project leaders to correct the disparity: disagree / agreeA better way of phrasing this question might be: the ratio of women and men employed as project leaders is: unbalanced / balanced …then follow-up with… if your response was “unbalanced” please explain.
61 Consider performing a pilot study to: check the clarity of questions, and see of they contain any ambiguityto make an initial statistical analysis to compute the number of additional questionnaires that should be sent out (the sample size)to see if the open ended questions suggest things that you had not thought of.Test the questionnaire on people close to you to verify the questions and get ideas for other questions to be included.
62 Questionnaire writing is time consuming and should be performed with the greatest of care: otherwise you will get biased and incomplete responses to the questions, and a poor response rate:postal responses may be as low as 25% to 33%low response rates lead to biased results since there may be something different about those that choose not to respond (maybe they are the busy ones and have a different perspective on the problem you are studying)
63 Follow-up non-responders to try to maximize your response rate Make sure you select an appropriate sample size, balancing cost of the study against statistical valueFollow-up non-responders to try to maximize your response rateTry to determine any pattern in the responders and non-responders to see if there may be some resultant bias in the answersA good reference on designing questionnaires is:“Improving Survey Questions: Design and Evaluation (Applied Social Research Methods)”, by Floyd J. Fowler, Sage Publications Inc., 1995
64 Interviews: Interviews can range from: structured, through semi-structured, to unstructuredthis is concerned with the extent to which the interviewer directs the subject of the interview:
65 Structured interviews: the interviewer asks a set of specific questions without much promptingthe questions leave little room for improvisationit is best suited to problems that are fairly well defined and we are trying to quantify the relationships between system variablesin this sense, it has some similarity to the questionnaire method, noting:structured interviews allow for better response to the individual questions since the interviewer can at least prompt for an answer or clarify a question (however, avoid leading the answer)but, structured interviews (as with any type of interview method) can only be used for a relatively small sample size – due to the human effort involved in obtaining the responseseg: homeowners attitudes towards community issueshere, we may have a good idea of the issues, but we want to quantify themwe can go door to door (or choose a random subset) without too much effort, and thus ensure a maximized responsewe are confined to a small sample size, that of the neighborhood
66 Unstructured interviews: in the extreme case, the interviewer introduces the topic briefly, and then allows the interviewee to indulge in a monolog,prompting from the interviewer may be limited to just encouraging the interviewee to continue, or to prompt them for more information on responses that sound promising;it is particularly well suited to research topics that are not well defined (it is good for probing a topic area to get a feel for the problem);in this sense, it is very much a qualitative tool;eg: we note a large difference in the ability of planners (within a company) to accurately schedule construction work, based on their experience:we could use an unstructured interview to determine what it is that the planners are doing differently,we do not have any specific ideas as to what the experience provides planners to make them better planners, so the interview format should be flexible enough to ascertain thi knowledgeonly a small number of planners are employed making the interview format feasible.
67 Semi-structured interviews: these fall somewhere between the two above extremes:such as, a list of questions, but with some probing for more in-depth answers to interesting responsesor, a list of sub-topics concerning a topic for which the interviewees responses are required.It is good to tape record the respondents’ comments:this reduces the risk of missing something in transcriptionbut get permission from the interviewee first.
68 Case Studies:Typically, these employ a variety of data collection techniques:interviews may be part of this, along with:collection of documentary dataobservation of on-going events:sitting in on meetings;observing design or construction activity.
69 Triangulation:This is the use of two or more methods to investigate the same thing:eg: observing on-going construction operations (using, say, time-lapse photography) and interviewing construction managers about issues they have with these same operationsit has the advantage of allowing one set of data to be validated against another set; andit allows for slightly different perspectives of the same problem to be obtained, thereby maximizing insight to the problem.
70 4.7 Observing System Behavior Observing systems:almost always this involves existing systems that include people:eg: studying an office information handling system (filing, data storage, data exchange) to identify inefficiencies and improved methods or resource allocationsthis may involve modeling the observed system to see how it performs under other circumstanceseg: studying operator’s driving equipment to identify better operating procedures or control layout (ergonomic studies)may set-up the system specifically for the study (such as the ergonomic study, but using benchmark tasks)this is essentially experimentation
71 in special cases, there may be no people involved eg: observations of automated processesA major difference between surveys and observation of human-based systems, is that the latter does not deal with people’s opinionshowever, people may still bias the results by behaving differently under observation – Hawthorne EffectRecording devices:cameras (still, video, time-lapse), etc… possibly done remotely such as over the web…Typical examples of measured parameters:event timing;task durations (from events or spot readings);resource flow bottlenecks;idle timeprocess conflict (such as for safety);user frustrations with aspects of the tasks;etc…