2 Activity – Evaluating and Interpreting Research Literature After reading the materials titled “Evaluating and Interpreting Research Literature, answer the following questions:Have any of your peers, colleagues, or instructors ever stated that a study “proves” something? If so, briefly describe what he or she said, and in light of reading the materials provided, would you be cautious about believing such a statement? Why? (answer in one-two paragraphs)According to the reading materials, what is the primary and secondary purpose for preparing a literature review?What do quantitatively oriented researchers emphasize when sampling that qualitatively researchers do not emphasize?Name two examples of common sampling flaws.
3 Activity – Evaluating and Interpreting Research Literature Name a trait, other than the ones mentioned in this chapter that you think are inherently difficult to measure. Why?Briefly explain why a highly reliable measuring instrument can be invalid.If a common well-known IQ scale is considered to have adequate reliability and validity, does this mean the scale has no flaws? If not, briefly explain why.To study causality, what do researchers need to do? Why?If a difference is statistically significant, does this mean the difference is large? If not, what does the fact that a difference is statistically significant tell you? What else can you look at to indicate the magnitude of a difference?
4 Review In your last assignment you developed a research question Your research question guides your design for the research study methodologyThis session will help you better formulate and articulate the research design and methodology section of your paper
5 Brief Description of Methodology and Research Design This section will define how you are going to address the research question or questionsYou will be able to…Present an overview of the methodologyDescribe the appropriateness of the methodologyExplain the rationale for selecting the methodology
6 Research Design Overview Nonexperimental Research DesignsQuasi-experimental Research DesignsExperimental Research DesignsQualitative Research DesignsProgram Evaluation
7 Nonexperimental Research Designs DescriptivePurpose is to create a detailed description of some phenomenonExample: Satisfaction levels of employers with business graduates’ job skillsCausal-ComparativePurpose is to compare two or more groups in order to explore possible causes or effects of a phenomenonExample: Effects of type of classroom (inclusion vs. non-inclusion) on academic achievement
8 Nonexperimental Research Designs (cont’d) CorrelationalPurpose is to determine the strength and direction between two variables
9 Quasi-Experimental Design Non-Equivalent Control Group 1. The treatment group is composed of students already in the program through self- or external selection. Random assignment to treatment and control groups is not possible. Another group that is as similar to the treatment group as possible is selected for a control group. It does not receive the program.2. Administer the pre-test to both groups. Initial differences can be adjusted later by statistical means.3. Expose the treatment group to the program while withholding it from the control group.4. Administer the post-test to both groups.5. If there is a difference which favors the treatment group, you can be fairly confident (though less so than with random assignment) that it was due to the program. If there is no difference, i.e., if the scores for both groups remained the same or changed equally (up or down), that indicates the program is probably not effective.
10 Quasi-Experimental Designs: Non-Equivalent Control Group Quasi-Experimental Designs: Non-Equivalent Control Group Treatment Group (T) Gets Program (X) Control Group (C ) Does Not. Treatment Group T Pre-Test X Post-Test Already Determined Select Similar C Pre-Test Post-Test Control Group
11 Controlling Non-Equated Variables Through Statistical Analysis (or Making Non-Equivalent Groups Similar) Often times we want to evaluate the effectiveness of a program that is already in place, and we are not able to construct a treatment and a control group. For example, suppose we wanted to evaluate the effectiveness of public schools vs.. private schools on academic achievement; and we looked at the average NAEP math scores for 4th grade students in public and private schools and found the following:
12 But what does the picture look like when we control for SES But what does the picture look like when we control for SES? When we compare public and private students of the same SES, we find there is little difference in their achievement. But because there are more high SES students in private schools, the overall comparison is misleading. (For the precise data on these comparisons, see Lubienski and Lubienski, “A New Look at Public and Private Schools.” Phi Delta Kappan, May 2005.)
13 Quasi-Experimental Design: Interrupted Time Series Design: multiple historical measures only on a treatment group before and after its exposure to the program. In situations where a control group is not possible, if (1) data on the treatment group can be obtained for several periods both before and after the participants are exposed to the program, and (2) there is change in scores immediately following the implementation of the program, and (3) there is a continuation of the change in subsequent time periods, that is considered good evidence that the intervention produced the change Time Experimental Group X X X X New program (X) is introduced.
14 Campbell’s Example of the Interrupted Time Series Design—1
15 Campbell’s Example of the Interrupted Time Series Design—2
17 Experimental Research Designs Involve the introduction of an intervention by the researcher to determine a cause-and-effect relationshipStrongest type of design (pre INT post)!To yield valid findings, these studies must be rigorous!
18 Experimental Research Designs (cont’d) Single-Group DesignsAll individuals in the study receive the treatmentExample: pre WebAchiever postThreats to validityControl-Group DesignsStrongest type of design!!Experimental group: pre INT postControl group: pre NO post
19 Experimental Research Designs (cont’d) Quasi-Experimental DesignsRandom assignment of subjects is not possible (e.g., using an entire classroom, etc.)Biggest problem
20 Experimental Research Designs (cont’d) Single Case DesignsInvolves the intense study of one individual (or more than one individual treated as single group).
21 Validity IssuesCan the change in the posttest be attributed only to the experimental treatment that was manipulated by the researcher?Must be able to control extraneous variables that could have undue influence!
22 Validity Issues (cont’d) Internal ValidityThe extent to which to which extraneous variables have been controlled by the researcher, so that any observed effect can be attributed to the treatment variable.External ValidityExtent to which the findings can be applied to individuals and settings beyond those studiedp. 368
23 Qualitative Research Designs Case StudyResearcher collects intensive data about particular instances of a phenomenon and seek to understand each instance in its own terms and in its own contextHistorical ResearchUnderstanding the present condition by shedding light on the past
24 Program EvaluationMaking judgments about the merit, value, or worth of a programA valuable tool in program management and policy analysisUsually initiated by someone’s need for a decision to be made concerning policy, management, or political strategy.
25 Components Addressed in Research Methodology (See Syllabus for Details) ParticipantsInstrumentsProceduresLimitationsAnticipated OutcomesReferences
26 ParticipantsThis section should include the following elements: (a) the target population or sample to which it is hoped the findings will be applicable should be defined, consistent with the Statement of Problem and the Research Question(s), (b) the population from which the sample will actually be drawn should be specified. This should also include demographic information such as age, gender, ethnicity etc., (c) procedures for selecting the sample should be outlined, including justification for the sampling method, (d) the implications for the generalizability of findings from the sample to the accessible population and then to the target population should be addressed.
27 ProceduresProcedures – the procedures section will be based directly on the research questions. That is, this is the “how-to” section of the study and will introduce the design of the research and how the data will be collected based on the questions of interest. This section should include the approach (i.e., design) to conducting the research (e.g., experimental, quasi-experimental, survey, historical, or ethnographic) and the appropriate procedures to be followed. For example, for an experimental or quasi-experimental study, the proposal should indicate how participants will be assigned to treatments and how the research will be conducted to ensure internal and external validity. If an evaluation project is proposed, the model to be followed should be specified.
28 InstrumentsExamples of data-gathering instruments include standardized tests, teacher-made tests, questionnaires, interview guides, psychological tests, or field-study logsIndicate the source (literature citation) of the instrument and cite appropriatelyInclude validity and reliability information
29 LimitationsLimitations are conditions, restrictions or constraints that may affect the validity of the project outcomesA limitation is a weakness or shortcoming in the project that could not be avoided or corrected and is acknowledged in the final reportCommon limitations are the lack of reliability of measuring instruments and the restriction of the project to a particular organization setting
30 Anticipated Outcomes Description of expected results of the study Detail the importance of conducting the study as well as possible impact on practice and theory
31 Research Question Activity State your research questionWhat are the key variables?What are the Conceptual definitions of your key variables?What are the Operational definitions of your key variables?
32 Conceptual and Operational Variables Concepts (or constructs) = terms that refer to the characteristics of an event, situation, or group being studiedWe need to clearly specify how we define each and every concept in our research studies.
33 Two Types of Definitions Conceptual Definition...the “empirical definition” of a construct
34 Examples of Conceptual Definitions Cognitive dissonance = “the unpleasant state of psychological arousal resulting from an inconsistency within one’s important attitudes, beliefs, or behaviors”Self-esteem = “a person’s overall self-evaluation or sense of self worth”Aggression = “behavior intended to injure another”Stereotype = “a belief about the personal attributes of a group of people”
35 Conceptual definitions offer general, abstract characterizations of psychological constructs. This is exactly why we need operational definitions!Operational definition...the precise specification of how a concept is measured or manipulated in a particular study
36 Operational Definitions How can we operationalize “aggression”?-punching another’s face?-hitting another’s arm?-spreading rumors about another?-verbally insulting another?-throwing glass at another?-etc.* The more specific, the better.
37 Operational Definitions Why are they necessary/important?They force us to think carefully and empirically in precise and specific terms.They make the concept public; they allow for replication.Measurable.