1. Copyright © Allyn & Bacon 2008 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: any public performance or display, including transmission of any image over a network; preparation of any derivative work, including the extraction, in whole or part, of any images; any rental, lease, or lending of the program. Non-Experimental Quantitative Research Designs Chapter 8

2. Copyright © Allyn & Bacon 2008 Discussion Topics Non-experimental research designs – Descriptive designs – Relationship designs – Causal-comparative and Ex Post Facto designs Using surveys in quantitative research

3. Copyright © Allyn & Bacon 2008 Non-Experimental Designs Research design - the plan and structure of research to provide a credible answer to a research question Purpose of non-experimental designs – Describe current existing characteristics such as achievement, attitudes, relationships, etc.

4. Copyright © Allyn & Bacon 2008 Non-Experimental Designs Four types of designs – Descriptive – Relationships Comparative Correlational – Causal-comparative – Ex Post Facto – Survey

5. Copyright © Allyn & Bacon 2008 Descriptive Designs Studies that describe a phenomena – Statistical nature of the description Frequency Percentages Averages Graphs Importance of these designs in the early stages of the investigation of an area

6. Copyright © Allyn & Bacon 2008 Descriptive Designs Criteria for evaluating descriptive studies – Conclusions about relationships should not be drawn – Participants and instruments should be described completely

7. Copyright © Allyn & Bacon 2008 Relationship Designs Two types of designs – Comparative – Correlational

8. Copyright © Allyn & Bacon 2008 Comparative Designs These designs investigate the relationship of one variable to another by examining differences on the dependent variable between two groups of participants – If math scores for males are significantly higher than those for females, a relationship exists between gender and math achievement – If the academic self-concept scores for ninth graders are significantly different than those for twelfth graders, a relationship exists between grade level and academic self- concept

9. Copyright © Allyn & Bacon 2008 Comparative Designs Criteria for evaluating these designs – Participants and instruments are described completely – Criteria for identifying the different groups is clearly stated – No inferences are made about causation – Graphs and images depict the results accurately

10. Copyright © Allyn & Bacon 2008 Correlational Designs Simple correlation designs – Designs that examine the relationship between two variables – Two variables Predictor and criterion Use caution describing the variables as independent and dependent – Examples Math achievement and math attitudes Teacher effectiveness and teacher efficacy

11. Copyright © Allyn & Bacon 2008 Correlational Designs Simple correlation designs – Cautions in interpreting correlations A relationship between two variables (e.g., achievement and attitude) does not mean one causes the other (i.e., positive attitudes do not cause high levels of achievement) Possibility of low reliability of the instruments makes it difficult to identify relationships

12. Copyright © Allyn & Bacon 2008 Correlational Designs Simple correlation designs – Cautions in interpreting correlations Lack of variability in scores (e.g., everyone scoring very, very low; everyone scoring very, very high; etc.) makes it difficult to identify relationships Large sample sizes and/or using many variables can identify significant relationships for statistical reasons and not because the relationships really exist

13. Copyright © Allyn & Bacon 2008 Correlational Designs Prediction designs – Designs that examine the predictive nature of the relationships between variables – Two types of designs Simple prediction Multiple regression

14. Copyright © Allyn & Bacon 2008 Correlational Designs Prediction designs – Simple predictive studies Performance on one variable (i.e., the predictor) is used to predict performance on a second variable (i.e., the outcome or criterion) Examples – Scholastic Aptitude Test (SAT) scores are used to predict college freshmen grade point averages – Scores from a mathematical attitude scale are used to predict math achievement scores Importance of the time interval between collecting the predictor and criterion variable data

15. Copyright © Allyn & Bacon 2008 Correlational Designs Prediction designs – Simple predictive studies Factors influencing correlations – Possibility of low reliability of the instruments measuring the predictor and criterion variables makes it difficult to identify relationships – Length of time between the predictor and criterion variable data collection – Existence of many factors, not only the one being examined, that influence the criterion variable

16. Copyright © Allyn & Bacon 2008 Correlational Designs Multiple regression – Studies that examine performance on several variables (i.e., predictor variables) to predict performance on a single outcome variable (i.e., criterion) – Examples Scholastic Aptitude Test (SAT) scores, high school grade point average, and high school rank in class are used to predict college freshmen grade point average Math attitude scale scores, academic self-esteem scale scores, and prior math grades are used to predict math achievement scores

17. Copyright © Allyn & Bacon 2008 Correlational Designs Logistic Regression – Another type of multiple regression analysis used to examine the relationship between the predictor variables and dependent variable. – The result is a prediction of whether the participant is a “case” or “non-case” Example: Overall GPA, gender, and special education classification used to determine if a student will pass or fail a standardized test.

18. Copyright © Allyn & Bacon 2008 Correlational Designs Multiple regression – Issues of concern Sample size of at least 10 subjects for each predictor variable Relationships among the predictor variables (i.e., colinearity)

19. Copyright © Allyn & Bacon 2008 Correlational Designs Significance of correlation coefficients – Statistical significance Does a statistical relationship exist? Is the observed correlation significantly different from zero? – Practical significance Does a relationship of practical importance exist? Coefficient of determination (r 2 ) - the percentage of the criterion variable variation that can be explained by the variation in the predictor variable

20. Copyright © Allyn & Bacon 2008 Correlational Designs Guidelines for interpreting the size of correlation coefficients – Much larger correlations are needed for predictions with individual than with groups Crude group predictions can be made with correlations as low as.40 to.60 Predictions for individuals require correlations above.75 – Exploratory studies Correlations of.25 to.40 indicate the need for further research Much higher correlations are needed to confirm or test hypotheses

21. Copyright © Allyn & Bacon 2008 Correlational Designs Guidelines for interpreting the size of correlation coefficients – Multiple correlation coefficients of.20 -.40 are common and usually indicate practical significance

22. Copyright © Allyn & Bacon 2008 Correlational Designs Criteria for evaluating correlational studies – Causation should not be inferred from correlational studies – The reported correlation should not be higher or lower than the actual correlation – Practical significance should not be confused with statistical significance

23. Copyright © Allyn & Bacon 2008 Correlational Designs Criteria for evaluating correlational studies – The size of the correlation should be sufficient for the use of the results – Prediction studies should report the accuracy of predictions for new subjects – Procedures for collecting data should be clearly indicated

24. Copyright © Allyn & Bacon 2008 Causal Comparative Designs Ex-post-facto designs – Studies that investigate the relationships between independent and dependent variables in situations where it is impossible or unethical to manipulate the independent variable Example - what is the effect of pre-kindergarten (Pre-K) attendance on first grade achievement – Cannot mandate Pre-K attendance for children – Characteristics and resources of families who do and do not send their children to Pre-K may influence first grade achievement – Similarities with correlational and experimental research designs

25. Copyright © Allyn & Bacon 2008 Causal Comparative Designs Ex-post-facto designs – Issues of concern Selecting participants who are as similar as possible on all characteristics except the independent variable Generalizing beyond the participants studied

26. Copyright © Allyn & Bacon 2008 Causal Comparative Designs Correlational causal-comparative studies – Use of correlational models to investigate possible cause and effect relationships – Sophisticated statistical models Path analysis Structural equation modeling – Fundamental limitations of all correlational research designs apply

27. Copyright © Allyn & Bacon 2008 Causal Comparative Designs Criteria for evaluating causal-comparative studies – Primary purpose is to investigate causal relationships when experimental designs are not possible – Presumed causal condition has already occurred – Potential extraneous variables are considered – Existing differences between groups being compared are controlled – Causal conclusions are made with caution

28. Copyright © Allyn & Bacon 2008 Using Surveys A data collection method that is very useful in descriptive and correlational studies – Versatile – Efficient – Generalizable Two types of survey designs – Cross sectional designs – Longitudinal designs

29. Copyright © Allyn & Bacon 2008 Using Surveys Cross sectional designs – Information is collected from one or more groups at the same time – Examples Student’s, teacher’s, administrator’s, and parent’s opinions regarding an extended school year Elementary, middle, and secondary teachers’ feelings toward a new school board policy – Issue of concern - comparisons across groups can be the result of differences between participants within the groups Fifth and seventh graders opinions can be affected by a changes in the attendance zones of a school

30. Copyright © Allyn & Bacon 2008 Using Surveys Longitudinal designs - information is collected from the same participants over time – Example - changes in the academic self-concept of students from the sixth to the twelfth grade – Issues of concern Loss of subjects over time Difficulty tracking participants over time

31. Copyright © Allyn & Bacon 2008 Using Surveys Steps in designing a survey – Define a purpose and objectives – Identify the resources needed and the target population Cost of preparation, printing, mailing, and analyzing results Time needed to administer the survey Sample size – Choose the method Paper Electronic Telephone Interview

32. Copyright © Allyn & Bacon 2008 Using Surveys Steps in designing a survey – Develop the items – guidelines Use clear, unbiased, non-ambiguous language Keep it short and simple Use grammatically correct language Do not write leading items Use the same response scale for all items Be consistent with wording – Design the format of the survey White space Font size

33. Copyright © Allyn & Bacon 2008 Using Surveys Steps in designing a survey – Develop directions Make it clear with no ambiguity Indicate clearly how participants are to respond Indicate where responses are recorded Indicate what participants should do when finished – Develop a letter of transmittal Keep it brief Include a statement of the purpose of the research Include a statement of the benefits of the research

34. Copyright © Allyn & Bacon 2008 Using Surveys Steps in designing a survey – Pilot test 15-20 representative participants Identify concerns – Clarity – Format – Responding – Directions – Time to complete

35. Copyright © Allyn & Bacon 2008 Using Surveys Response rates – Low response rates are the major limitation of survey use – Suggestions for increasing response rates Design the survey well Contact the subjects several times especially following-up on non- respondents Include a self-addressed return envelope Use a good transmittal letter Use a telephone for follow-up Use incentives for completing the survey

36. Copyright © Allyn & Bacon 2008 Using Surveys Using electronic surveys – Internet based surveys E-mail attachments Web pages – Evaluations of online university survey research centersonline university survey research – Criteria for evaluating survey research centersevaluating survey research centers

37. Copyright © Allyn & Bacon 2008 Using Surveys Using electronic surveys – Advantages Reduced time and cost Easy access Quick responses Ease of creating data sets – Disadvantages Limited to those with access to the technology Confidentiality and privacy issue