PSYC512: Research Methods PSYC512: Research Methods Lecture 5 Brian P. Dyre University of Idaho

PSYC512: Research Methods Lecture 5 Outline Questions about material covered in Lecture 4 Questions about material covered in Lecture 4 Scientific Method Scientific Method Proof and disproof Proof and disproof Strong Inference Strong Inference Issues in Measurement Issues in Measurement

PSYC512: Research Methods To Prove or Disprove, That is the Question… Conditional Reasoning: The Propositional Calculus Conditional Reasoning: The Propositional Calculus Two premises and a conclusion Two premises and a conclusion Premise 1) If then Premise 1) If then Premise 2) Affirm/deny Premise 2) Affirm/deny Conclusion) Therefore Conclusion) Therefore Four Possibilities for Premise 2 Four Possibilities for Premise 2 Affirm AntecedentDeny Antecedent Affirm ConsequentDeny Consequent

PSYC512: Research Methods To Prove or Disprove, That is the Question… Confirmational Reasoning Confirmational Reasoning Premise 1 Premise 1 If then If then Premise 2; Conclusion Premise 2; Conclusion AA) theory A is correct; therefore data A will be observed (Valid, but pointless) DA) theory A is incorrect; therefore data A will not be observed (Invalid and pointless) AC) data A observed; therefore theory A is correct (Invalid, but often used) DC) data A not observed; therefore theory A is incorrect (valid, but only if observations are exhaustive—accepting the null)

PSYC512: Research Methods To Prove or Disprove, That is the Question… Disconfirmational Reasoning Disconfirmational Reasoning Premise 1 Premise 1 If then If then Premise 2; Conclusion Premise 2; Conclusion AA) theory A is correct; therefore data B will not be observed (Valid, but pointless) DA) theory A is incorrect; therefore data B will be observed (Invalid and pointless) AC) data B not observed; therefore theory A is correct (Invalid) DC) data B observed; therefore theory A is incorrect (valid, most scientifically useful!)

PSYC512: Research Methods Confirmation and Disconfirmation of Theories: Summary Confirmation (Poor) Confirmation (Poor) if theory correct then observation will occur if theory correct then observation will occur Observation occurs  Support, but not proof Observation occurs  Support, but not proof Observation does not occur  disproof? NO! Observation does not occur  disproof? NO! Disconfirmation (OK) Disconfirmation (OK) if theory correct then observation will not occur if theory correct then observation will not occur Observation does not occur  Support, but not proof Observation does not occur  Support, but not proof Observation does occur  disproof Observation does occur  disproof Strong Inference (BEST!) Strong Inference (BEST!) PROOF?

PSYC512: Research Methods Strong Inference (Platt, 1964) Science is fundamentally based on disconfirmation (Popper) Science is fundamentally based on disconfirmation (Popper) Theories are not evaluated in isolation, rather they compete with one another (relativism) Theories are not evaluated in isolation, rather they compete with one another (relativism) Critical Experiments – results will disconfirm one (or more) theory (theories) while confirming one or more alternative theories Critical Experiments – results will disconfirm one (or more) theory (theories) while confirming one or more alternative theories Disconfirmed theories are discarded (or revised) like logical branches pruned from the tree of understanding, in which only one branch represents truth Disconfirmed theories are discarded (or revised) like logical branches pruned from the tree of understanding, in which only one branch represents truth

PSYC512: Research Methods Strong Inference (Platt, 1964) The Question to ask in your own mind on hearing The Question to ask in your own mind on hearing any scientific explanation or theory put forward: any scientific explanation or theory put forward: “What experiment could disprove your hypothesis?” or on hearing a scientific experiment described: on hearing a scientific experiment described: “What hypothesis does your experiment disprove?” Practicing explicit and formal analytical thinking Practicing explicit and formal analytical thinking the “notebook” containing the logical trees, alternative hypotheses, and crucial experiments the “notebook” containing the logical trees, alternative hypotheses, and crucial experiments include as an appendix to your annotated bibliography include as an appendix to your annotated bibliography

PSYC512: Research Methods Measurement: What Makes Observation Systematic? Careful planning of Careful planning of What will be observed What will be observed How the observations will be made How the observations will be made When the observations will be made When the observations will be made Operational definitions Operational definitions translations of concepts stated in your hypothesis into the operations you will use to manipulate or measure that concept translations of concepts stated in your hypothesis into the operations you will use to manipulate or measure that concept

PSYC512: Research Methods Choosing Measures Research tradition e.g., operant conditioning—lever pressing e.g., cognition—accuracy and reaction time e.g., sensation and perception—discrimination accuracy e.g., personality—surveys, inventories (self-reports) Research tradition e.g., operant conditioning—lever pressing e.g., cognition—accuracy and reaction time e.g., sensation and perception—discrimination accuracy e.g., personality—surveys, inventories (self-reports) Theory e.g., the psychophysical postulate – discrimination accuracy e.g., Serial vs. parallel processes in visual search – RT Theory e.g., the psychophysical postulate – discrimination accuracy e.g., Serial vs. parallel processes in visual search – RT Availability of new techniques Availability of new techniques Availability of equipment Availability of equipment

PSYC512: Research Methods Features of Measures: Scale of Measurement (Stevens, 1946) Four types: nominal, ordinal, interval, and ratio Four types: nominal, ordinal, interval, and ratio Nominal scales Nominal scales set of unique cases, types, or categories with NO ORDER set of unique cases, types, or categories with NO ORDER valid operations are non-parametric: counting frequencies, modes, chi-square, point-biserial correlation valid operations are non-parametric: counting frequencies, modes, chi-square, point-biserial correlation Ordinal scales Ordinal scales different categories that can be ranked along a continuum different categories that can be ranked along a continuum more or less, but not how much more or less more or less, but not how much more or less valid operations are non-parametric: counting frequencies, modes, medians, chi-square, rank-order correlation valid operations are non-parametric: counting frequencies, modes, medians, chi-square, rank-order correlation

PSYC512: Research Methods Features of Measures: Scale of Measurement (Stevens, 1946) Interval Interval intervals of the scale are equal in magnitude intervals of the scale are equal in magnitude valid operations (parametric): all mathematical operations, means and variances, linear and non- linear regression, t-tests, ANOVA valid operations (parametric): all mathematical operations, means and variances, linear and non- linear regression, t-tests, ANOVA no fundamental zero—no ratio statements allowed no fundamental zero—no ratio statements allowed Ratio Ratio Like interval but also has a fundamental zero point— allows ratio statements such as “A is twice as much as B” Like interval but also has a fundamental zero point— allows ratio statements such as “A is twice as much as B” Generally interval or ratio scales should be used if possible Generally interval or ratio scales should be used if possible

PSYC512: Research Methods Features of Measures: Sensitivity Sensitivity: measure must show changes in response to changes in the independent variable Sensitivity: measure must show changes in response to changes in the independent variable Range effects Range effects Ceiling effects: variable reaches its highest possible value and gets truncated (test is too easy) Ceiling effects: variable reaches its highest possible value and gets truncated (test is too easy) Floor effects: variable reaches its lowest possible value and gets truncated (test is too hard) Floor effects: variable reaches its lowest possible value and gets truncated (test is too hard)

PSYC512: Research Methods Features of Measures Reliability Reliability the ability of a measure to produce consistent results when repeated measurements are taken under identical conditions the ability of a measure to produce consistent results when repeated measurements are taken under identical conditions Types of reliability: Types of reliability: precision: physical measurement (1/noise) precision: physical measurement (1/noise) margin of error: sampling in surveys margin of error: sampling in surveys interrater reliability: observers viewing the same behavior interrater reliability: observers viewing the same behavior Test-retest, parallel forms and split-half reliabilities: psychological tests Test-retest, parallel forms and split-half reliabilities: psychological tests

PSYC512: Research Methods Features of Measures Accuracy Accuracy does a measure produce results that agree with a known standard? does a measure produce results that agree with a known standard? Accuracy vs. Precision Accuracy vs. Precision Validity Validity Measurement validity: the extent to which your measure indeed measures what it is intended to measure Measurement validity: the extent to which your measure indeed measures what it is intended to measure Types: Face validity, Content validity, Criterion- related validity (concurrent vs. predictive), Construct validity Types: Face validity, Content validity, Criterion- related validity (concurrent vs. predictive), Construct validity Relationship between reliability and validity Relationship between reliability and validity

PSYC512: Research Methods Next Time… Topic: descriptive statistics, variables, sampling, and more on hypothesis testing Topic: descriptive statistics, variables, sampling, and more on hypothesis testing Be sure to: Be sure to: Read the assigned readings (Howell chapters 3-4) Read the assigned readings (Howell chapters 3-4) Continue searching and reading the scientific literature for your proposal Continue searching and reading the scientific literature for your proposal