Sampling and Unit of Analysis EDL 714 Fall 2010

Related but distinct… Sample: the people, places, things, phenomena you will be collecting data about or from. Unit of analysis: that which you seek to make claims about. They are NOT automatically the same thing. Sorting out this distinction is absolutely fundamental. Quite frequently your sample will be strategically used to collect data regarding your unit of analysis, but will not be the same thing. Let’s talk out some examples…

3 research potholes to avoid… Structuring a research question in such a way that you can neither appropriately study it, or make useful claims about. Selection and use of measurements, instruments, or methods that are neither credible nor rigorous. Choosing a sample that does not allow clear and valid links to your research questions, nor generalization or transfer. In research, sampling is destiny (Kemper, et al., 2003)

Guidelines for sampling Apply to qualitative, quantitative, and mixed method designs. Curtis, Gesler, Smith, & Washburn (2000) Kemper, Stringfield, & Teddlie (2003)

#1… logic chain The sampling plan should be clearly linked to the logical argument forged by the relationship of the research questions to the conceptual framework and literature review. Essentially… does the sampling plan logically support answering the research question(s) in a credible and valid manner? Can you actually address your question(s) about your unit(s) of analysis?

#2… sufficient data The sampling plan will generate the necessary and sufficient data, or evidence, to understand and make claims about the phenomenon under study (the latter being your unit of analysis). All claims/conclusions must be evidence-based. Is your study descriptive, interpretive, comparative, etc.? Your research question determines this for the most part.

#3... Credibility (validity) The sample allows the possibility that credible inferences can be drawn from the data. So not only is there a logical basis for making claims, but there is also a chance your claims are credible. (Logic and credibility are related but not the same.) Sample allows for the likelihood that other potential causal factors or variables are accounted for. Conclusions are likely representative of the reality of the participants.

#4… ethics Your sampling strategy does not compromise the rules, regulation, and ethics regarding research with human subjects (participants). Participation in the study is worth the time and involvement of participants.

#5… you can do it The sampling plan is feasible and “do-able” You can actually access the sample/data you plan to access. It is physically, logistically, temporally, spatially, metaphysically possible for the researcher to do what they plan.

#6… extension Conclusions of the study can transfer or generalize to other contexts, circumstances, or participants Analytical generalization is possible (transferability) Statistical generalization is possible (generalization) Practical relevance or applicability is possible Requires a strong connection to practice and/or policy

#7… balance The sampling plan is practical. The sampling plan is efficient. The sampling plan is necessary and sufficient. The sampling plan is logical.

Sampling techniques (use this language) Probability (random) and purposive (non-random) strategies Large-scale or experimental quantitative studies will typically use probability sampling methods Case studies, mixed methods, or qualitative studies will typically use purposive sampling strategies

Probability sampling (random) Sample: the smaller group, from a larger population, from which data is obtained. Population: the entire group from which data could be obtained. Defining the population is an essential task in selecting an appropriate sample.

Random sampling Population: all members of a group of interest. Sample: a segment of that population. Inferential statistics seeks to generalize from samples to populations. Random sampling is the means by which we try to achieve an unbiased sample.

Variations on random sampling Simple random sample Stratified random sample Population divided by stratification variables Equal percentage then randomly drawn from each stratum Multistage random sample Typically used in larger scale studies Strata may be introduced by researcher Random cluster sample Sampling of pre-existing clusters of a population already belonging to a group (i.e. subgroup, grade level, single- parent families)

Random sampling and research in schools Assumptions of “random” assignment in quasi- experimental designs Best approximations of equivalent groups Matched-pair design

Sample size Random samples may still include sampling errors (in fact they likely do). In general, the larger the sample the smaller the sampling errors are. In general, the larger the sample the greater the precision (think consistent replicability) of the results. In general, think of a sample size of as our minimum for inferential stats.

Sample size  Increasing sample size produces diminishing returns.  The smaller the anticipated difference in a population, then the larger the sample size should be.  Even small samples can identify significant differences.  For populations with very limited variability, small samples can be very precise.

Sample size The more variable the population, the larger the sample size should be. When studying something rare (or of low occurrence), then larger samples are usually required. Large sample size cannot control for a biased sample. It doesn’t change. Results from large samples can be potentially misleading, even if statistically significant.

Purposive sampling (non-random) Convenience sampling Extreme/deviant case and/or typical case sampling Confirming/disconfirming cases Homogenous cases Stratified purpose (quota)/random purposive Optimistic and snowball sampling

Convenience sampling Most common (i.e. the teachers in my own building). Sample drawn from group easily accessible to researcher. May not provide the best sample for answering the research question(s). Often result in spurious conclusions.

Extreme/deviant case, Typical case Sampling plans specifically designed to answer the research question at hand through highly strategic sampling decisions. Focus on outliers with extreme cases (i.e. non- Spanish speaking adolescent ELLs with no formal educational experience). Focus on archetypes with typical cases (i.e. Spanish speaking ELLs with prior formal educational experience).

Confirming/ disconfirming cases Confirming case designs seek specific samples or cases that already fit a known pattern (i.e. a 55 year old divorced female superintendent with grown children). Disconfirming case designs seek the opposite; cases that are clear exceptions to a pattern (i.e. a 45 year old married female superintendent with school-age children).

Homogenous cases Identification of cases from a specific subgroup with common characteristics (i.e. female African-American high school students who did well in math).

Stratified purposive (quota), Random purposive Stratified purposive: splitting a non-random sample into smaller units based upon specified criteria (i.e. a non-random matched-pair design). The goal is to compare across groups within the larger group. Random purposive: randomizing within a larger purposive sample. Can be used in quasi-experimental designs. Can add some credibility, but does not contribute to statistical generalizability. Does allow for certain pre/post designs to be used within the context of overall purposive sampling. Both are useful for mixed-methods designs.

Activity Refer to your prospectus self-design as needed. Identify, assess and evaluate your sampling plan based upon the guidelines for sampling we have just discussed (use the worksheet provided). Define your sampling plan using the language we have just covered. Describe how your sampling plan supports investigating your unit of analysis. Share with a colleague. Praise and push.