1. Assessment Group for Provincial Assessments, June 6 2013 Kadriye Ercikan University of British Columbia

2. Outline  Purposes of assessments from different perspectives  Sample versus census testing  Some principles we can discuss, agree, disagree on

3. Purposes of Provincial Assessments  To provide system level achievement data Examine effectiveness of curriculum – is the curriculum appropriate for different age and cultural groups As a measure of system level achievement, for monitoring learning To identify inequities and help inform policy and resource allocation  To provide classroom and school level data Provide information to teachers Serve as an external measure of achievement that is comparable across the province  To provide individual student level data Inform teaching and learning, possibly starting earlier than grade 4 Provide information about learning and growth of children in care and Aboriginal children Provide information to parents about how children are doing

4. Concerns  Consequences of use of assessment results Unfair comparisons of students Inappropriate interpretations of assessment results, such as school rankings as indicators of school effectiveness Narrowing curriculum and negative effects on classroom teaching  Confidentiality when data are released

5. Random Sample versus Census Testing  Random Sample Testing What it is Motivation  Can it meet all needs of the province  Comparison to census testing  Continuing to inform key policy decisions  What will it look like?

6. Random Sample Testing  Sampling the way in which students are selected for the assessment  Sample Selected group of students for the assessment  Population Population refers to all eligible students within a jurisdiction  Representative sample All students in the population need to have an equal probability of being selected Large enough number of students to allow for adequate representation of the population’s performance

7. Census Testing  Test ALL grade 4 and 7 students and make inferences about grade 4 and 7 students in the province

8. Inference in Census Testing Population

9. Random Sample Testing  Test a representative sample of grade 4 and 7 students  Make inferences about all grade 4 and 7 students

10. Inference in Random Sample Testing Representative sample Population

11. Motivation for Random Sample Testing  Reduce burden on teachers and students  Cost savings  Cover a broader range of curriculum  Include performance assessment tasks  Faster scoring and reporting of scores  Reduce chances of inappropriate use of assessment data

12. Does Random Sample Testing Fit with FSA’s Purposes?  Purposes of FSA “The main purpose of the assessment is to help the province, school districts, schools and school planning councils evaluate how well students are achieving basic skills, and make plans to improve student achievement.”.  Two key purposes Evaluate Inform planning  For both of these purposes the assessment provides information about achievement levels in relation to the provincial curricular and learning goals.  Assessment results based on a representative sample of Grade 4 and 7 students can be generalized to all Grade 4 and 7 students in BC.  Sampling will determine if district level information is obtained

13. Trade-offs of Random Sample Administration  No individual student level scores to students, teachers and parents  No classroom level scores to teachers  Schools won’t be able to track their yearly progress based on FSA  If special considerations are not made, no district level scores  Unless special sampling is considered no scores by aboriginal, ESL, students with special needs groupings

14. Example: Pan-Canadian Assessment Program (PCAP)  A national assessment that measures the knowledge and skills of 13-year- old students in reading, mathematics, and science.  In 2007, over 30,000 students from across Canada wrote PCAP.  Students from all ten provinces and the Yukon participated.  In BC, around 2,500 students from more than 150 schools participated.

15. Sampling in PCAP  Two-step stratified sampling process 1. Random selection of schools from each jurisdiction, drawn from a complete list of publicly funded schools provided by the jurisdiction 2. Random selection of students, drawn from a list of all eligible participants within each school  In the case where numbers were smaller than the desired size, all schools and/or all students meeting the criteria within the jurisdiction were selected.  This assured an adequate number of participants to allow for reporting on their achievement as if all students within the jurisdiction had participated

16. Important Sub-group Information  Aboriginal, ESL, special needs students for example  Students in special programs  These can be achieved by over- sampling, for example including all aboriginal students in a particular district.

17. Scenario 1: Province Level Reporting of Scores Only  Two-stage random sampling  Approximately 150 schools are randomly selected  Within each selected school, approximately 25 students are randomly selected OR  Within each selected school one classroom is randomly selected.

18. Scenario 2: District Level Reporting of Scores  Three-stage random sampling  Within each district, randomly select a number of schools proportional to the size of the district.  Within each selected school approximately 25 students are randomly selected OR  Within each selected school one classroom is randomly selected.

19. What’s Reported  Currently scores are broken down by Gender, Aboriginal/Non-aboriginal, ESL and Special Needs  Gender does not need special sample considerations  Aboriginal students will require over- sampling of this population  ESL and Special Needs, depending on the purposes of reporting, over sampling may be needed

20. Future Directions  Principles for the assessments  Addressing Concerns

21. Principles  Assessments need to assess valued outcomes  Assessments need to be aligned with learning, instructional and curricular goals  Assessments need to provide accurate estimates of student knowledge and competencies  Score reports need to be informative and developed in a timely manner  Consequential aspects of assessment results need to be an important consideration

22. Concerns, Issues to be Addressed  Use, misuse, and interpretation of assessment data  Confidentiality  Validity investigations  Explicit theories of actions  Support and resources to address inequities