Presentation is loading. Please wait.

Presentation is loading. Please wait.

Data Collection and Reliability All this data, but can I really count on it??

Similar presentations


Presentation on theme: "Data Collection and Reliability All this data, but can I really count on it??"— Presentation transcript:

1 Data Collection and Reliability All this data, but can I really count on it??

2 Agenda Data collection – review and continue Reliability What is it? How do we achieve?

3 Questionnaires Controlling problems Equal numbers of positively and negatively worded statements Alternating positive and negative statements Providing confidentiality or anonymity to respondents

4 Designing Questionnaires Online resources www.surveymonkey.com http://pareonline.net/getvn.asp?v=5&n=3 http://www.peecworks.org/PEEC/PEEC_Inst/I0 004E536http://www.peecworks.org/PEEC/PEEC_Inst/I0 004E536 http://www.statpac.com/surveys/

5 Questionnaires Take the following questionnaire http://www.surveymonkey.com/s/PNBKFJ7 Strengths? Weaknesses?

6 Observations Observations - direct observations of behaviors Provide first hand account (ameliorates issues of self-reporting in questionnaires) Natural or controlled settings Ex – classroom vs. lab (child attachment studies) Structured or unstructured observations Ex – frequency counts vs. narrative record Detached or involved observers

7 Observations Inference Low inference - involves little if any inference on the observers part On-task/Off-task behavior instrument High inference - involves high levels of inference on the observers part Teacher effectiveness – PDE form 430

8 Observations Controlling observer effects Observer bias Training Inter-rater reliability (Cronbach’s alpha) Multiple observers Contamination - knowledge of the study influences the observation Training Targeting specific behaviors Observers do not know of the expected outcomes Observers are “blind” to which group is which

9 Observations Observer effects Halo effectHalo effect - initial ratings influence subsequent ratings Hawthorne effectHawthorne effect - increased performance results from awareness of being part of study LeniencyLeniency - wanting everyone to do well Central TendencyCentral Tendency - measuring in the middle Observer DriftObserver Drift - failing to record pertinent information

10 Examples to critique Measures Questionnaire – Psychological School Membership Survey used with middle school students Interview protocol – for teachers & counselors regarding professional development issues Observation instrument – PDE 430 for student teachers What are 2 benefits and 2 limitations of this measure?

11 Validity/Reliability and Trustworthiness Why do we need validity and reliability in quantitative studies and “trustworthiness” in qualitative studies? We can’t trust the results if we can’t trust the methods!

12 Thought Question On the ACT and SAT assessments, there is a definitive script that test administrators are required to follow exactly. What measurement issue are the test makers addressing?

13 Reliability of Measurement Reliability - The extent to which measures are free from error Error is measured by consistency

14 Reliability of Measurement Sources of error Test construction and administration Ambiguous questions, confusing directions, changes in scoring, interrupted testing, etc. Subject’s characteristics Test anxiety, lack of motivation, fatigue, guessing, etc.

15 Reliability of Measurement Reliability Measurement 0.00 indicates no reliability or consistency 1.00 indicates total reliability or consistency <.60 = weak reliability >.80 = sufficient reliability

16 Reliability of Measurement Types of reliability evidence Test-retest Testing the same subject using the same test on two occasions Limitation - carryover effects from the first to second administration of the test Equivalence (i.e. parallel form) Testing the same subject with two parallel (i.e. equal) forms of the same test taken at the same time Limitation - difficulty in creating parallel forms

17 Reliability of Measurement Internal consistency Testing the same subject with one test and “artificially” splitting the test into two halves Limitations - must have a minimum of ten (10) questions Often see “Chronbach’s alpha” for reliability coefficient

18 Reliability of Measurement Agreement / Inter-rater reliability Observational measures Multiple observers coding similarly

19 Reliability of Measurement Enhancing reliability Standardized administration procedures (e.g. directions, conditions, etc.) Appropriate reading level Reasonable length of the testing period Counterbalancing the order of testing if several tests are being given

20 Next Week Tuesday – Validity and Review Thursday – Opportunity to display brilliance! Multiple choice Short Answer Essay – TAKE HOME. Critique Triplett article (see next slide).

21 Take Home Exam Critique the article based on the following: Introduction and research problem Review of literature/ theoretical framework Methods of data collection (including participants) and data analysis Results and conclusions including issues of trustworthiness. Be sure to address whether we should trust the claims that the authors have made and why we should or should not trust the claims.


Download ppt "Data Collection and Reliability All this data, but can I really count on it??"

Similar presentations


Ads by Google