2. The most important aspects of collecting data is that you know 1.What it is you are looking for 2.Where to look for it (who to ask or where it already exists.) 3.How best to collect it (what methods are most suitable.)

3. On the next slide is an example of a “particularized measure”. It was constructed by the evaluator – with the help of the staff – because –The data collection needed to be ongoing to answer the questions –The clients had chronic and severe diagnoses, so it was believed that the staff working with them could answer the questions best –much of the data either did not exist or was kept in different places –It created a way to collect the same data for all clients –even if they were in different agencies. –It allowed staff who were involved in the evaluation and who worked in different agencies to collect the data that was needed to answer the research questions

4. Example Of a Data Collection Instrument Constructed For a Specific program

5. Although in the previous example, we had one major data collection instrument –one that “technically” answered all of the research questions – we also knew that we needed ways to ‘check’ the reliability of the data. This meant that we needed to use some other measures to make sure we were getting accurate data. We then used –Focus groups with staff every 4 months to insure that staff felt we were capturing what was needed. –We also planned to do in depth semi-structured interviews with a random sample of 6 successful clients and 6 ‘not so successful’ clients in order to find out if they saw the outcomes in the same way as staff. The use of multiple methods of collecting data is Called “triangulation”. Most research questions require more than one method of data collection

6. You might use the following grid to determine you method of data collection First, take each research question and use this decision tree. Evaluation question #1___________________________________ 1.What is it I’m looking for_____ 2.Is it Quantitative or qualitative data needed______ 3.need in-depth info or breadth of info_____ 4.Where am I looking 5.few people or documents or many people or documents_____ 6.What would be best for the people involved given where I have to look Then go to the grid on the next slide and begin to narrow down

7. COLLECTION METHOD QuantitativeQualitativeReliabledepthbreadthfewmanyeasyhard Mail surveyxxx Phone surveyxxx Net surveyxxx Structured in-depth interview xxxxxx ?xxx Existing numerical program records xxx ? xxx Standardized scales or measures xxx ? Particularized measures Xxx?? ?? Secondary data Program charts/notes Xxx ? ??xxxXxxxxx Unobtrusive measure type B Xxx????xxx Public recordsxxx Social indicatorsxxx Xxx Direct observationXxx Xxx ?Xxx xxx Indirect observation Inc videotape XxxxxxXxx ?xxx Semi-structured Dramaturgical inter xxx ? Xxx xxx Focus groupsxxx? Key informantsxxx?Xxxxxx Agency forumsXxx xxxXxxxxx Nominal group.Xxxxxx

8. If you have followed this, You have begun to answer these three general questions. 1.What it is you are looking for 2.Where to look for it (who to ask or where it already exists.) 3.How best to collect it (what methods are most suitable.)

9. Another way to state the 3 general questions these are more precise and might give better direction.Begin to answer them now! 1. IS the research question measurable in words or numbers Recall that outcome evaluations are almost always Quantitative; process evaluations are primarily qualitative. 2. What data or information could answer the research question.. what things do I need to know in order ton provide an answer to that question? 3. Are there sources for that info or data already exist? If not, 4. What would be the best way to get that information; For example, surveys, focus groups, scales, direct observation etc. in other words how would that data be best collected? ORWhat would be the best way to get that information; For example, surveys, focus groups, scales, direct observation etc. 5. Once I determined the best way to collect it, where might I go to collect it and who might I talk to, interview etc in order to collect it? Who might have that information or be able to provide that info. 6. Once I have answered the questions above, what specifically would I be looking for, asking about or observing? This question asks you to begin to develop questions if using a survey, rating systems if using scales or find instruments if using standardized measures or develop observational techniques if directly observing. 7a. If measurable in numbers, what would I have to do in order to make sense of or analyze the numbers? 7b. If measurable in words, what must I do in order to make sense of or analyze the words