1. Assuring the Quality of your COSF Data

2. What factors work to improve the quality of your data?
What factors work to lessen the quality of your data?
How to address these factors?
You may wish to start or end this presentation with a discussion of factors that participants feel have influenced the quality of the data they are generating and reporting. Find a worksheet to facilitate this discussion at this link:

3. Take Home Message
If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else.
What do you do if the data are not as good as they should be?
Answer: Continue to improve data collection through ongoing quality assurance

4. Many steps for ensuring quality data
Before completing the COSF
Good data collection/training
During the completion of the COSF
Ongoing supervision
Feedback
Refresher training
After completing the COSF
Review of COSF records
Data analyses for validity checks
One way to think about quality assurance is in terms of activities that occur before, during, and after data collection.

5. Many Steps for ensuring quality data
Before
Good data collection/Training
Before collecting data, COSF users need to know how to use the COSF, therefore training is critical.

6. Promoting quality data
Through training and communication related to:
Understanding the COSF process
Functional assessment
Age expectations for typical child development
Ongoing staff development will help maintain the skills of the COSF user. Skills that underlie the correct use of the COSF include the ability to conduct functional assessment and ongoing progress monitoring. Functional assessment gives the team the information needed to determine a COSF rating both at entry and at exit. Ongoing progress monitoring is particularly useful for completing the exit rating. Knowledge of typical child development allows the team to ‘age anchor’ a child’s skills, on which an accurate rating can be determined.

7. Promoting quality data
Through data systems and verification, such as:
Good data entry procedures
Data system error checks
In addition to training, the data system should be set up with good data entry procedures and ways to check for error. What is the process for entering and checking data in your CDSA?

8. Many steps for ensuring quality data
During
Ongoing supervision
Feedback
Refresher training
Once data collection has begun, COSF users need supervision and feedback. Staff development should occur periodically to make sure everyone is still on the same page (‘refresher’ training).

9. Promoting quality data
Ongoing staff development:
Video team and child examples
Written child examples
“Quizzes” for ensuring learning
Refresher trainings –
Beware of Drift!!
Ongoing staff development/refreshers help keep COSF users from ‘drift’ away from the common understanding of the 7 points on the rating scale, as described in Unit 2.

10. Ongoing supervision Review of the process Methods
Is the process high quality?
Are teams reaching the correct rating?
Provide feedback to the team
Methods
Observation
Videos
Supervisors should observe COSF team discussions and provide feedback to the team.

11. Indicators of a quality COSF team discussion
All team members participate in the discussion
2. Parent input is considered in the rating
3. The team documents the rating discussion
4. The team discusses multiple assessment sources

12. Indicators of a quality COSF team discussion
5. The team describes the child’s functioning, rather than just test scores
6. The discussion includes the child’s full range of functioning, including skills and behaviors that are age appropriate, immediate foundational, and leading to immediate foundational
Either at this point or after the slide show presentation, have participants assess the quality of the team discussion video with the materials found at this link To emphasize the role of the families in the team discussion, also observe a team discussion and look for the quality indicators described at this link --

13. Quality review through process checks
Provider surveys
Self assessment of competence
Knowledge checks
Process descriptions
Identification of barriers
The CDSA may wish to assess the needs of its own staff, as the state EI office assessed the needs of the CDSAs. Sample questions are at this link:

14. Many steps for ensuring quality data
After
Review of COSF records
Data analyses for validity checks
After completion of the COSF, quality checks can involve reviewing the completed COSF and analysis of aggregated COSF data – looking for patterns and ‘outliers.’ See COSF training unit 4 for training materials related to data analysis.

15. Indicators of quality COSFs
The COSF is complete
Evidence
matches the appropriate outcome area
is based on functional behaviors
considers the child’s functioning across settings and situations
Ratings are consistent with the evidence

16. Supplemental Group Activities
Discussion of factors that influence data quality
Quality review of a COSF team discussion
Quality review of family participation in a COSF team discussion
Review a completed COSF for errors
COSF documentation using supporting evidence
Customize this slide to fit your group’s agenda and available time. See trainer notes for specifics about each activity (handouts, answer keys, time estimates, etc)