Presentation is loading. Please wait.

Presentation is loading. Please wait.

Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 1 Presented by AmeriCorps Program Staff and JBS International.

Similar presentations


Presentation on theme: "Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 1 Presented by AmeriCorps Program Staff and JBS International."— Presentation transcript:

1 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 1 Presented by AmeriCorps Program Staff and JBS International

2 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 2 Learning Outcomes As a result of this session, participants will: Understand core concepts associated with data collection and aggregation; Better understand how to assess the quality of your current data collection tools and systems; and Identify upgrades you need to make to insure rigorous data collection and meaningful reporting.

3 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 3 Agenda What are You Trying to Measure? Are You Measuring it the Best Way? Measuring Outcomes – Data Collection Method Considerations – Instrument Considerations

4 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 4 Food for Thought Why are you doing the intervention? What change do you want to create? Can you measure the change/outcome? If you cant measure the outcome, are you sure you are doing the right thing?

5 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 5 Now What? You have identified the desired change (outcome) through your program design and theory of change…. Are you clear about WHAT to measure? AND Are you measuring it the BEST way?

6 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 6 What are Your Desired Outcomes? What question are you trying to answer? What change are you expecting? – Did you select one of the CNCS priority measures? – Did you create your own program-specific outcome? Either way…the data collection and aggregation basics are the same.

7 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 7 Driving School Example Intervention: ABS driving school gives a 10 week course that meets twice a week for 60 minutes that include classroom-based and on road lessons on driving skills Desired Outcome: Students have basic driving proficiency.

8 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 8 Real Life Example: Driving Test Question: Do you like driving? Answer: I LOVE driving!! Do you like driving? gets information about an attitude – not skill level.

9 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 9 Real Life Example: Driving Test Question: Do you think you are a skilled driver? Answer: I think I am a GREAT driver! Are you a skilled driver gets information about self perception, a thought – not actual skill level. Self-ratings are subjective NOT objective.

10 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 10 Real Life Example: Driving Test Question: Do you know the state driving laws ? Answer: I got 100% correct on my written drivers test! Knowing state driving laws reflects knowledge – not actual skill level even though it is objective.

11 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 11 Real Life Example: Driving Test Question: Did you pass your road test? Answer: YES! An on-road driving test DOES measure skill level or proficiency and is objective.

12 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 12 What kind of information do you want? Subjective - includes an element of opinion or personal feeling. – Example: how someone feels about driving, confidence Objective - is not dependent upon opinions or personal feelings. It is based on facts that are observable and measurable. – Example: knowledge of driving laws and skill driving

13 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 13 How Do You Choose Your Method? Choice depends on: – what you want to measure; and. – the situation (i.e. resources for data collection/aggregation, site/partner agreements/restrictions, etc.) Each method is more appropriate in some situations than others (e.g., age, language, content sensitivity, etc.) Will it get you high quality data?

14 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 14 Does Your Method Measure Your Outcome? Commonly used data collection methods – Surveys – Pre/post tests – Observations – Standardized tests – Interviews – Focus Groups – Diaries, Journals, Self-reported Checklists – Available secondary data

15 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 15 How Do You Choose Your Instrument? Whichever method you select, what instrument will you use? Borrow vs. develop Does it ask the right questions to get at your desired outcome? Does it have all the necessary components? What information will each question yield? How will you use information, if not related to outcome?

16 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 16 What is Qualitative Data? Describes or characterizes through words Focuses on meaning, experience or attitudes Collected through focus groups, interviews, opened ended questionnaire items, and other less structured situations. Not the same as anecdotal Information Qualitative Quality

17 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 17 What is Quantitative data? Focus on numbers and frequencies. Data which can be measured Length, height, area, volume, weight, speed, time, temperature, humidity, sound levels, cost, ages, scores, etc. Quantitative Quantity

18 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 18 Example: How Do You Feel? Qualitative DataQuantitative Data Have you felt sad or depressed at all lately, or have you generally been in good spirits? Thinking about the past week, how depressed would you say you have been on a scale from 0 to 10, where 0 means "not at all" and 10 means "the most possible Im not at all depressed. I feel great! I love my new job. Ive lost 20 pounds and feel healthier than I have in years. 0

19 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 19 Are You Asking the Question You Mean to Ask? Does your instrument or data collection method help you measure your desired outcome? Example: – If desired outcome = improved academic performance DONT measure attendance or attitude toward school DO measure improved proficiency in a subject VALIDITY

20 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 20 Validity Example Desired Outcome (O11): Economically disadvantaged individuals transitioned into safe, healthy, affordable housing Data to collect (from NOFO): – An inspection report and certificate of occupancy, – proof of residence such as lease or mortgage, or – other verification from an external agency that the work was completed and is being occupied might be used.

21 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 21 Validity Example (cont.) Which questions are most likely to yield the desired information? – Do you feel safe in your new home? – Can you afford this house? – Do you like this house? – Do you have a lease or mortgage for this house? Who holds the lease/mortgage? Please share a copy of the signed paperwork.

22 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 22 What is a Standardized Test? A test that: – is administered and scored in a consistent or standard manner. – has been validated externally on a randomly-selected population Need not be high-stakes tests, time-limited tests, or multiple choice tests – e.g., state administered proficiency tests generally should NOT be used

23 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 23 Can It Be Repeated? 1.Does your instrument measure the same thing, the same way every time it is used? 2.Does every person collecting data use the instrument the same way? Have they been trained? 3.Are your instrument instructions clear so respondents have a similar frame to answer? RELIABILITY

24 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 24 Reliability Example Intervention: Preschool children attend early childhood education programs providing school readiness activities in three areas (social/emotional development, literacy and numeracy skills) 4 days a week for 4 hours/day Desired Outcome (ED23): Children demonstrate gains in social and/or emotional development.

25 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 25 Reliability Example (cont.) Observations: – Open-ended Question: Does the child seem well adjusted and ready to attend kindergarten?

26 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 26 Instrument Pilot Testing Test before using - with small number of people similar to those who will respond Pilot test analysis should look at: – Were the questions clear enough? Did people understand what you were asking? – Do the answers seem appropriate given what you were asking? Make revisions based on results of pilot

27 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 27 Is Something Systematically Off? Bias: Problems with WHO you ask – sampling bias – response rates, etc. Problems with HOW you ask – method inappropriate – construction of your instrument

28 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 28 What is Bias? Measurement bias occurs when information collected for use is inaccurate. Bias may be introduced by poor measurement design or poor data collection. Bias cannot be controlled for at the analysis stage. Bias risks readers drawing conclusions that are systematically different from the truth. Bias can lead to an over or underestimation of an effect.

29 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 29 Revisit: What to Look For? Are you measuring what counts/matters? Is your measurement approach credible? Are your instruments valid? Are your instruments reliable? Are your measurements precise?

30 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 30 Instrument Mapping Look at each question on your data collection tool and ask: – Does this help us measure the desired outcome? Is there one question? More than one? – What kind of data will we get? Subjective? Objective? Quantitative? Qualitative? – If it doesnt measure the outcome, do we really need to ask it? How will we use the answer? Nice to know but wont use it? Internal use? – How will we analyze this? – What is our target? How much change is enough?

31 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 31 Target Example: Pre/Post test Mentored children will enhance developmental assets in the areas of social competence and positive identity.

32 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 32 Do You Have a Summary Sheet w/Target?

33 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 33 After You Get the Data, then What? Do you have a plan for who and how the data collected will be aggregated and summarized? – Given the types of questions on your tool(s), is this realistic? Which data DIRECTLY relate to the desired outcome? NOTE: Just because you asked it, doesnt mean it helps you report on your outcome…dont confuse people!

34 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 34

35 Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 35


Download ppt "Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 1 Presented by AmeriCorps Program Staff and JBS International."

Similar presentations


Ads by Google