CGT 411 Research Presentation Conducting Research: What Do You Need to Think About? Part 3 – Instruments and Participants

The Nature of Sampling The idea is to examine elements of a population in order to draw conclusions about the population as a whole The (experimental) population is the group of elements about which we want to make inferences A count of all elements of a population would be known as a census Sample elements = data points

Why do we sample? Lower cost  it is normally too expensive to look at everyone Greater accuracy  fewer data points means less to keep track of Faster  this greatly reduces cost Availability of population elements  every population element is often not accessible A census is feasible when populations are small and necessary when elements are not homogeneous

Overview of Research Good research should follow the standards of the scientific method Purpose clearly defined Research process detailed Research design thoroughly planned Limitations frankly revealed High ethical standards applied Adequate analysis for decision-maker’s needs Findings presented unambiguously Conclusions justified Researcher’s experience reflected

Exploratory Research Interviewing Observation Extant data Document analysis Case studies Ethnography Expert interviewing Secondary data analysis Experience surveys Focus groups Two-stage designs Instances of data should be analyzed that provide meaningful information within the context of the organization

Communicating with Participants Observations Interviews Surveys

Getting to the Measurement Questions What type of data is needed to answer the investigator’s dilemma? What communication approach will be used? Should the questions be structured, unstructured, or some combination? Administrative: identify setting characteristics Classification: sociological/demographic Target: address the issues at hand Should the questions be disguised or undisguised?  have to be careful here  potential IRB issues

Constructing and Refining Research Questions Measurement questions should: Consider subject content Consider the wording of each question Consider the response strategy Question development and question sequencing are typically completed at the same time

Drafting and Refining the Instrument The introduction has to hook ‘em and give them the particulars  also has to filter them Some questions depend on other having or not having been asked already Awaken interest and motivation Put sensitive questions at the end Balance complexity of questions and their location General to specific Group questions to minimize shifting frame of reference and subject matter

Drafting and Refining the Instrument Pre-testing/pilot-testing is valuable for having a reliable instrument Discover participants’ reactions to the instrument Ensure meaning is conveyed Why do participants “transform” questions they do not understand? Improve continuity and flow of the instrument Make sure question sequence is appropriate Helps understand variability in population Helps to gauge length and time

Interview Considerations Should the question be asked? Is the question of the proper scope and coverage? Can the participant adequately answer the question as it is asked? Will the participant willingly answer the question as it is asked?

Observation Considerations Much of what we know comes from observation  but the collection process often leaves much to be desired Scientifically based Conducted to answer a specific question Systematically planned and executed Uses appropriate control mechanisms Reliable and valid account of what happened Observation is generally behavioral Nonverbal analysis: body movement Linguistic analysis: patterns/content of speech Spatial analysis: physical relationship to surroundings Extra-linguistic analysis: characteristics of speech

What is Experimentation? The emphasis is towards discovering a causal relationship Accepting the world as it is found (ex post fact, communication, observation) vs. systematically altering a variable and observing the change (experimentation) Is the experimental treatment at the heart of the observed change, or are other factors influencing the outcome?

What is Experimentation? Requires the intervention of a researcher beyond that required for measurement Manipulation of independent variables to affect a hypothesized dependent variable  at least one IV and one DV in an experiment Must be an agreement between variables  presence/absence of one relates to the presence/absence of another Time order of variables must make sense Researchers must be confident that extraneous factors did not cause such a change Often involves the presence of a contrived research scenario  standardized to aid generalizability

Elements of Scientific Method Direct observation of phenomena Clearly defined variables, methods, and procedures Empirically testable hypotheses Ability to rule out rival hypotheses Statistical justification of conclusions Self-correcting process

Descriptive Study Descriptions of phenomena associated with a particular population Estimates of the proportion of the population that would have such characteristics Discovery of associations among different variables The simplest descriptive study concerns a statement about the size, form, distribution, or existence of a variable  no causal relationship implied

Causal Studies Correlation is not causation essentially “A produces B”  an artifact of language We cannot prove that A causes B, but we can find evidence that leads us to believe that A leads to B Covariation of A and B Time order of events in the direction specified No other possible causes Inference-making also requires: Control  no other confounding causes Random assignment along the IV

Advantages Disadvantages Researcher’s ability to manipulate the IV Influence of extraneous variables can be controlled Convenient and cost-effective Ability to replicate across people, places, and time Can use naturally occurring events Contrived setting Generalization from non-probability samples can be problematic Budgets can be deceiving Most effective for problems in the present or the immediate future Ethical issues in the manipulation of people

Conducting an Experiment Select relevant variables Specify levels of treatment Control the experimental environment Choose the experimental design Select and assign the participants Pilot test  revise  test Analyze the data

The Logic of Hypothesis Testing The null hypothesis  a statement that no significant difference exists between a parameter and a statistic being compared to it. Why not state this in a positive form? Evidence would preclude a positive hypothesis from being taken as conclusive evidence for accepting the hypothesis Could be consistent with other hypotheses as well

The Logic of Hypothesis Testing The alternative hypothesis  a statement that there is a significant difference between the parameter and the statistic This is the logical opposite to the null hypothesis May take several forms depending on the objective of the researcher Not equal to (not the same as) Greater than Less than