1. PRR 475 Review - Tips Powerpoint Presentations – widgets too Topic Handouts Sample Questions Other Handouts e.g. Statistics Exercises from Lab Trochim chapters Other texts, websites

2. Research/Evaluation Concepts – evaluation.ppt Proposals & Research Design – propdesb.ppt Measurement – measure.ppt Sampling – sample.ppt Survey methods – survey99.ppt Basic Statistics/Data analysis - handouts Experiments – expmt.ppt Observ., qualitative, other methods – QLQN.ppt, econevaluation.ppt Communication & Ethics researchwriting.ppt, ethics.ppt

3. Outline Definitions of Research/Evaluation Purposes of Evaluation / Research Science and Scientific Management Research/Evaluation as Process Types of Research / Evaluation

4. Definitions Evaluation = Process of judging the merit or worth of something Research – application of scientific methods to answer questions –controlled inquiry directed at increasing knowledge/establishing truth Evaluation Research - combine the two

5. Science Body of Knowledge systematic abstract general parsimonious Method of Inquiry logical –induction –deduction self-corrective empirical

6. Scientific Management Application of scientific principles to management and decision making –systematic information gathering –empirical, objective, self-corrective

7. Process -- Steps Research define problem objectives/hypotheses literature review research methods gather data/analysis conclusions Evaluation describe program evaluation criteria program scoping evaluation methods gather data/analysis conclusions

8. Types of Evaluation by Program Stage formative (conceptualization/design)] process (implementation) summative (outcomes, impacts, efficiency)

9. Types - By Approach Standards –norm-based –criterion-referenced Goals and objectives Impacts or effects

10. Evaluation Criteria Effort - qnty and qlty of inputs Performance - qnty and qlty of outputs Adequacy - meet needs? Efficiency - benefits/costs Equity - distributional issues, fairness

11. Process Evaluation Identifies how and why program works –attributes –recipients –conditions –effects single or multiple intended or side effects timing & duration, long/short term cognitive, affective or behavioral

12. Research Process Define Problem, Research Objectives HOW? Overall Method Survey Experiment Case Study Secondary Data What? Concepts Variables Measures Who? Population Sampling Data Gathering Analysis Application

13. Proposal Format 1. Problem Statement - define program to be evaluated/problem to be studied, users & uses of results. Justify importance of the problem/study. 2. Objectives : Concise listing. In evaluation studies, the objectives usually focus on the key elements of program to be evaluated & the evaluation criteria. These are the study objectives NOT the program objectives. 3. Background/Literature Review - place for more extensive history/structure of program. Focus on aspects most relevant to proposed evaluation. Discuss previous studies or the relevant methods. 4. Methods - details on procedures for achieving objectives - data gathering and analysis, population, sampling, measures, etc. Who will do what to whom, when, where, how and why? 5. Attachments - budget, timeline, measurement instruments, etc. NOTE: Most “programs” must be narrowed to specific components to be evaluated. Think of a “Program of studies” rather than a single evaluation study. The proposal should define this specific study & how it fits into a broader program of studies.

14. Purposes of Proposal Communicate with Client Demonstrate your grasp of problem Plan the study in advance, so others can evaluate the study approach –will it work? –have you overlooked something? –will results be useful to client? – Can we afford it?

15. Sample Objectives 1. Estimate benefits and costs of program 2. Estimate economic impacts of program on local community (social, environmental, fiscal). 3. Determine effects of program on target population. 4. Describe users and non-users of program 5. Assess community recreation needs, preferences 6. Determine market/financial feasibility of program 7. Evaluate adequacy or performance of program

16. Methods Choices Overall Approach/Design –Qualitative or Quantitative –Primary or secondary data –Survey, experiment, case study, etc. Who to study - population, sample –individuals, market segments, populations What to study - concepts, measures –behavior, knowledge, attitudes Cost vs Benefit of Study

17. Definition & Measurement “measurement is the beginning of science, … until you can measure something, your knowledge is meager and unsatisfactory” Lord Kelvin Nominal/Conceptual Definition - define concept in terms of other concepts, links concepts without tying them to real world Operational definition - equates definition with measurement, specify procedures/operations to generate the concept.

18. Levels of Measurement

19. Validity vs Reliability

20. Questionnaire Design 1. Preliminary Info Information needed Who are subjects Method of communication 2. Question Content 3. Question Wording 4. Response Format 5. Question Sequencing/Layout

21. What Info? Demographic, Socioeconomic, Physical Cognitive - Knowledge & beliefs Affective - attitudes, feelings, preferences Behavioral - actions

22. Sampling Always define study population first Use element/unit/extent/time for complete definition element - who is interviewed sampling unit - basic unit containing elements extent - limit population (often spatially) time - fix population in time

23. Types of Sampling Approaches Probability vs non-Probability Judgment, Simple Random, Systematic Stratify or Cluster (Area Sample) Time Sampling

24. Sample size Based on four factors Cost/budget Accuracy desired variance in popln on variable of interest subgroup analysis planned Formula: n= Z 2  2 / e 2 n= sample size Z indicates confidence level (95% = 1.96)  = standard deviation of variable in population e = sampling error

25. Sampling errors for binomial (95% confidence interval) percent distribution in population

26. Computing 95% confidence interval N= 100, sample mean = 46%, use p= 50/50, sampling error from table = 10% 95% CI is 46% + or - 10% = (36, 56) N=1,000 sample mean =22% sampling error from table = 2.5% 95% CI is 22% + or - 2.5% = (19.5, 24.5)

27. Research Designs/Data Collection Approaches

28. Major Design Types Surveys Experiments Observation Secondary Data Qualitative Approaches –Focus Group –Case Study

29. General Guidelines on when to use different approaches 1. Describing a population - surveys 2. Describing users/visitors - on-site survey 3. Describing non-users, potential users or general population - household survey 4. Describing observable characteristics of visitors - on-site observation 5. Measuring impacts, cause-effect relationships - experiments

30. Guidelines (cont) 6. Anytime suitable secondary data exists - secondary data 7. Short, simple household studies - phone 8. Captive audience or very interested population - self-administered survey 9. Testing new ideas - experimentation or focus groups 10. In-depth study - in-depth personal interviews, focus groups, case studies

31. Primary or Secondary Data Secondary data are data that were collected for some purpose other than your study, e.g. government records, internal documents, previous surveys Choice between Primary /Secondary Data –Costs (time, money, personnel) –Relevance, accuracy, adequacy of data

32. Qualitative vs Quantitative Purpose Perspective Procedures Quantitative Gen’l Laws Test Hypotheses Predict behavior Outsider-Objective Structured formal measures probability samples statistical analysis Qualitative Unique/Individual case Understanding Meanings/Intentions Insider-Subjective Unstructured open ended measures judgement samples interpretation of data

33. Qualitative vs Quantitative Approaches Qualitative Focus Group In-Depth Interview Case Study Participant observation Secondary data analysis Quantitative Surveys Experiments Structured observation Secondary data analysis

34. Survey vs Experiment Survey - measure things as they are, snapshot of population at one point in time, generally refers to questionnaires (telephone, self-administered, personal interview) Experiment - manipulate at least one variable (treatment) to evaluate response, to study cause-effect relationships (field and lab experiments)

35. 1. Define problem and study objectives 2. Identify information needs & study population(s) 3. Determine basic design/approach - cross sectional vs longitudinal - on-site vs household vs other - self-admin. vs personal interview vs phone - structured or unstructured questions 4. Questionnaire design 5. Choose sample (frame, size, sampling design) 6. Estimate time, costs, manpower needs, etc. STEPS IN A SURVEY

36. 7. Proposal & “Human subjects” review 8. Line up necessary resources 9. Pre-test instruments and field procedures 10. Data gathering and follow-up procedures 11. Coding, cleaning and data processing 12. Analysis: preliminary, then final. 13. Communication and presentation of results. Survey Implementation

37. Characteristics of a true Experiment 1. Sample equivalent experimental and control groups 2. Isolate and control the treatment 3. Measure the effect

38. Pre-test/Post-test with Control R MB1 X MA1 Experimental group R MB2 MA2 Control group R denotes random assignment to groups X denotes the treatment Measure of effect =  Expmt gp -  Control gp = (MA1-MB1) - (MA2-MB2) = with vs without

39. Threats to Internal validity * Pre-measurement (Testing) : effect of pre- measurement on dependent variable (post-test) * Selection: nonequivalent experimental & control groups, (statistical regression a special case) * History: impact of any other events between pre- and post measures on dependent variable * Interaction: alteration of the “effect” due to interaction between treatment & pre-test. Maturation: aging of subjects or measurement procedures Instrumentation: changes in instruments between pre and post. Mortality: loss of some subjects

40. Economic Evaluation Tools Benefit-Cost Analysis Cost Effectiveness Analysis Financial Analysis/Feasibility Fiscal Impact Analysis Economic Impact Analysis Social/Environmental Impact Analysis

41. Ethical Issues Human Subjects Honest and Open Reporting Client confidentiality No selling

42. Human Subjects Voluntary Participation Informed Consent Risk-benefit ratio Anonymity vs Confidentiality

43. Research Writing Like any writing, –Know audience –Know Story/Message/Subject –Follow Formats for Outlet Research Style –Objective, Factual –Impersonal –Dense, concise, logical

44. Sections Abstract – Executive Summary Introd/Problem statement Objectives Lit Review/Background Methods Results Discussion, Implications References

45. Three Audiences/styles Researchers – research journal style –Technical, methods, statistical tests Managers – business style –Results and implications Public – newspaper style –Interesting, no jargon, highlights

46. Research vs Business Reports Written/Research –Problem –Objectives –Methods –Results –Discussion Oral/Business –Objectives –Key Results & Recommendations –Justify from study –Brief methods –Discussion

47. Reminders Final Exam is Friday Dec 15, 7:45-9:45 am, this room Final Papers due by Wednesday Dec 13 See YaYen Sun to finish lab work by end of week.