1. 1 Reliability and Validity Quality of Data

2. 2 Are we testing what we think we’re testing?

3. 3

4. 4 Quantitative Data Reliability Validity  Face  External  Internal

5. 5 Reliability Implies that the same data would have been collected each time over repeated tests/ observations. Would a particular technique (or survey question) yield the same result each time?  “Did you go to church last week?” vs. “How many times have you been to church in your life?” Reliability does not ensure accuracy.  Taken from Babbie, E.

6. 6 Reliability Problem if interpret questions differently Poorly worded questions Inconsistent coding: coding errors as with open-ended questions Lack of definition of key terms

7. 7 Reliability Poorly worded: Does the library have adequate facilities and equipment for physically disabled students Better: Can patrons in wheelchairs retrieve books from the browsing collection?

8. 8 Reliability (indicators) Pretest Repeat question(s) Test/retest Split half and Parallel Interscore or scorer

9. 9 Validity A term to describe a measure that accurately reflects the concept it is intended to measure. Which is a more “valid” indicator of intelligence- an IQ score, or number of hours spent studying? Ultimate validity cannot be proven, but can be supported by face, internal, and external measures.  Babbie, E.

10. 10

11. 11 Types of Validity Face validity: The quality of an indicator/ question/ test that makes it a reasonable measure of a variable.  Church attendance is an indication of religiosity.  Number of grievances filed is an indicator of worker morale

12. 12 Internal Validity Approximate truth about inferences regarding causal relationships Typically applied to studies using inferential statistics (i.e. quantitative measures) than descriptive or observation studies. Especially useful for studies assessing affects of programs Only applicable to the study in question- not generalizable. Why not? Key question: Whether observed changes can be attributed to your program (the cause) and NOT other possible impacts/ causes.

13. 13 Internal Validity Trochim, W.

14. 14 Internal Validity History or specific events History or specific events: raises the issue that some variable other than the independent variable accounted for the change in the dependent variable. E.G.: the length of time between conducting the pretest and posttest may have a detrimental effect. Maturation Maturation: the change results from biological or psychological processes, which occurred over time, and not from the treatment itself. Maturation becomes more a concern the longer the period between the pretest and posttest Pretesting Pretesting: may affect the dependent variable. Pretesting may alert participants or educate them about the topic under investigation. Therefore if subjects are administered a posttest, their performance may reflect a marked improvement Measuring instruments or observational techniques Measuring instruments or observational techniques: These—not the treatment—may account for the change in the dependent variable. Further, the validity of study findings may have been influenced by the fact that the evaluators as observers, raters, graders, interviewers, and coders gained experience, became tired, obtained a more complete understanding of the project, or eased their expectations of test subjects

15. 15 Internal Validity (continued) nonrandom assignment of subjects A nonrandom assignment of subjects to groups may signify that the groups were dissimilar from the beginning. Therefore any change might be attributed to the differential selection of subjects, rather than the actual treatment. Statistical regression Statistical regression refers to the tendency for extreme scores to regress or move toward the common mean of subsequent measures. The assignment of subjects to a particular test group on the basis of extreme views may affect study findings.

16. 16 Internal Validity (continued) Mortality Mortality refers to the possibility that some subjects may have dropped out of the study after completion of the pretest but before the administration of the posttest. In such instances, every effort should be made to identify any common patterns or characteristics to ensure that any difference between a group’s pretest and posttest scores cannot be attributed to the loss of subjects. Interaction Interaction refers to the fact that more than one of the previous threats might be in play. This is especially likely in those cases where subjects were not randomly assigned to groups and the evaluation was based on existing, intact groups.

17. 17 External Validity The approximate truth of generalizations drawn from a study. The degree to which conclusions drawn from your study sample would hold true to other persons in other places at other times  Trochim, W.

18. 18 External Validity Trochim, W.

19. 19 External Validity Example: institutions of higher education in Massachusetts: control, highest degree offered, and some characteristics of library (staff number, budget, and volume number) Return rate? Do respondents differ from non- respondents as a group?

20. 20 Validity Content validity (for achievement test): How well does the test sample what the students learned? How well does a standardized test cover what was taught in the information literacy program?

21. 21 Validity (continued) Criterion-related (predictive) (attitude test to predict performance in a library skills program): Who well does the test predict achievement for college freshmen? Criterion-related (diagnostic): How well does the test diagnose current problems with library use?

22. 22 Validity (continued) Construct validity: How well does the test measure comprehension of library use? Does a test on the use of an OPAC really measure effective and efficient use rather than one’s ability to read test items?

23. 23 Qualitative Study Equivalent Credibility Dependability Confirmability Transferrability

24. 24 Qualitative Reliability Researcher is the “instrument”- how to test for reliability?  Provide details of method, and abundance of evidence  Provide evidence of qualifications as observer  Make assumptions (and possible biases) clear  State research questions clearly  Use early stages of study to generate focus  Observe for an adequate period of time, across a full range of activities  Collect data from multiple sources  Save data for reanalysis

25. 25 Qualitative Validity Depends upon reliablity. Like reliability, asserted by documenting steps  Triangulation- data from different sources/ methods  Full documentation of data- “chain of evidence”  Logical connections between data and conclusions  Conscious and deliberate inclusion of data that might not support thesis  Preparedness to entertain alternatives  Self-reflection, acknowledgement of own biases  Review of preliminary reports by objective observers  Awareness of limitations  Gorman and Clayton

26. 26 Qualitative Study: Increasing Reliability and Validty Inquiry affected by Results inAccount for by: To lead toFor findings that are: DuringAfter Factor patternings Non- interpretability Prolonged engagement Persistent observation Peer Debriefing Triangulation Member checks Establish structural corroboration (coherence) CredibilityPlausible Situational Uniqueness Non- comparability Collect thick descriptive data Do theoretical/ positive sampling Develop thick description TransferabilityContext relevant -Gorman and Clayton

27. 27 Qualitative Study: Increasing Reliability and Validity cont’d Inquiry affected by Results inAccount for by: To lead toFor findings that are: DuringAfter Instrumental changes InstabilityUse overlap methods Use stepwise replication Leave audit trail Do dependability audit (process) DependabilityStable Investigator Predilections BiasDo triangulation Practice reflexivity (audit trail) Do confirmability audit (product) ConfirmabilityInvestigator- free -Gorman and Clayton

28. 28 Example For a sweeping study When conduct it? For how long? How deal with reliability and validity? Course evaluation

29. 29 References Babbie, E. (2005). The basics of social research. Belmont, CA: Wadsworth Publishing. Gorman, G.E. & Clayton, P. (2005). Qualitative research for the information professional: A practical handbook. London: Facet Publishing. Trochim, W. M. K. (2006). Research methods knowledge base. Retrieved July 8, 2008 from http://www.socialresearchmethods.net/kb/intval.php http://www.socialresearchmethods.net/kb/intval.php http://www.socialresearchmethods.net/kb/external.p hp

30. 30 Data Collection Intro to Methods

31. 31 Methodology Quantitative, Qualitative, Mixed, and Triangulation How to select an appropriate method?  What does quantitative data yield?  What does qualitative data yield?  What is the purpose of our study?

32. 32 Data Collection Right kind of data (available sources) Enough data (sample size) Goes back to problem statement

33. 33 Data Collection Primary data collection  Information you gather directly  Reduce bias and increase accuracy  Measure, don’t ask Secondary data collection  Using data collected by someone else for a different purpose  Report information or,  Conduct secondary analysis

34. 34 Primary Data Quantitative or Qualitative  Surveys/ Questionnaires  Interviews  Observations  Case Studies  Diaries/ Journals  Tests  Experiments

35. 35 Secondary Data Can be useful for  Supplementing your own research  Offering background  Providing larger scope/ scale Must consider  Appropriateness  Where it has come from  Currency  If you are going to compare it to other data- are they alike (same units etc.)

36. 36 Secondary Data Main sources  Governments  General business resources  International/ national organizations