1. PPA 502 – Program Evaluation Lecture 5b – Collecting Data from Agency Records

2. Introduction  The source of data most often used by evaluators is records kept either by the agency delivering the service being evaluated or by other agencies that have records relevant to the work of the program being evaluated.  It is tempting to evaluators to accept agency records at face value and not look critically at the information they contain.  Don’t!

3. Potential Problems  Missing or incomplete data. –Possible ways to alleviate the problem. Go back to the records and related data sources (such as by interviewing program staff) to fill in as many gaps as possible. Determine whether part or all of the evaluation needs to be modified or terminated. Exclude missing data or provide a “best estimate” of the missing values.

4. Potential Problems  Data available only in overly aggregated form. –Where feasible, go back into the records to reconstruct the needed data. –Do new, original data collection. –Drop the unavailable disaggregations from the evaluation.

5. Potential Problems  Unknown, different, or changing definitions of data elements. –Make feasible adjustments to make the data ore comparable. –Focus on percentage changes rather than absolute values. –Drop analysis of such data elements when the problem is insurmountable.

6. Potential Problems  Data that are linked across time and clients. –Be sure that the outcome data apply to the particular clients/ work elements covered by the evaluation. –Track the clients/ work elements between agencies/ offices using such identifiers as social security numbers. –Look for variations in spellings, aliases, etc.

7. Potential Problems  Confidentiality and privacy considerations. –Secure needed permissions from persons about whom individual data are needed. –Avoid recording client names. Instead, use code identifiers. –Secure any lists that link code identifiers to client names. Destroy these after the evaluation requirements are met. –Obtain data without identifiers from agency employees.

8. Quality Control  Check data for reasonableness. –Assign ranges of possible values for each data element, and check to see if any data fall outside the range. –Check consistency across data elements. –Look for missing data, flagging these instances so that decisions can be made on how to deal with them.  Staffing considerations. –Staff collecting the data should be given sufficient training in what to look for.

9. Other Suggestions  Before actual field data collection begins. –1. Make friends with those agency staff who originated the data. –2. Try to deal directly with the persons most familiar with the data records. –3. If evaluators ask the agency to provide data rather than requesting access to agency files, they should make the task as easy as possible for the agency staff: Given the agency as much advance notice as possible. Put the request in writing and provide clear, full descriptions of the data needed. Tell the agency why the data are needed but be flexible.

10. Other Suggestions  Before actual field data collection begins (contd.). –4. Request samples of the data formats and definitions before you go into the field to gain a better perspective. –5. In some cases, it may be necessary and appropriate to compensate the agency for the extra time and effort required to generate the requested information.

11. Other Suggestions  In the field. –6. Whether they are collecting completed agency records or are extracting data from agency files, the evaluators should talk with the persons who are providing the data and who know something of its content. –7. The evaluators should learn the form and detail in which the data are available. –8. For each item of data collected, the evaluator should identify the time covered by that item. –9. Similarly, for data elements intended to cover specific geographical areas, the evaluators should identify what geographical areas apply to each data element.

12. Other Suggestions  After initial data have been obtained. –10. Determine for each data element how missing or incomplete data should be handled. –11. Check for illogical, inconsistent data. –12. Send data back for verification – in situations where the originators are likely to be able, and willing, to make such verifications. –13. Thank agency sources for their assistance. –14. Document and provide appropriate caveats in the evaluation report.