1. 2012–2013 Holistic Rating Training Requirements Texas Education Agency Student Assessment Division
This PowerPoint summarizes the TELPAS holistic rating training requirements and processes. It is designed to provide testing coordinators and bilingual/ESL coordinators with a general understanding of the requirements for the 2012–2013 school year.

2. Disclaimer
These slides have been prepared by the Student Assessment Division of the Texas Education Agency.
If any slide is amended or revised for local use, please remove the TEA footer at the bottom of the slide.
TEA

3. Topics TELPAS Raters Holistic Rating Training Credentials
Responsibilities
New & returning rater definitions
Holistic Rating Training
Overview
Training requirements
Basic training, calibration, and supplemental support
TEA

4. Rater Credentials Each teacher selected to rate an ELL must
have the student in class
be knowledgeable about the student’s ability to use English in instructional and informal settings
hold valid education credentials such as a teacher certificate or permit
be appropriately trained, as required by TEA
TELPAS raters may include:
Bilingual education teachers
English as a second language (ESL) teachers
Elementary general education teachers
Middle or high school general education teachers of foundation (core) subjects
Special education teachers
Gifted and talented teachers
Teachers of enrichment subjects
Paraprofessionals may not be designated as raters.
TELPAS ratings reflect the ability of students to understand and use English during content area instruction. Therefore, teachers who have ELLs in content area classes are integral to the assessment.
Districts are encouraged to provide TELPAS holistic rating training to as many teachers as possible. The training offers professional development for teachers and promotes increased instructional support for students. TELPAS training helps teachers understand and use the proficiency level descriptors (PLDs) component of the Texas English Language Proficiency Standards (ELPS), which should be incorporated in instruction all year long.
Training more rather than fewer teachers also helps districts ensure that they can effectively meet TELPAS rating needs in the spring of the year should there be unforeseen changes in student schedules or teaching assignments.
TEA 4

5. TELPAS Rater Responsibilities
A student’s TELPAS rater is the teacher designated by the district as the official rater of the student’s English language proficiency.
The student’s rater must rate the student in all domains for which the student is eligible. A student is not permitted to have one rater for some domains and another rater for other domains.
For example, it is not permitted for one rater to have the responsibility of rating the writing domain while another rater has the responsibility for rating the listening and speaking domains.
TEA 5

6. New and Returning Rater Definitions for K–1 and 2–12
Raters who complete K–1 training but not 2–12 training will be new raters if they switch to 2–12, and vice versa. Training for K–1 and 2–12 is separate training because of differences in the tested components and rating rubrics (PLDs).
“Completed” means that the rater took all required training. It does not necessarily mean that the rater was successful in meeting training requirements.
Examples (not exhaustive) of when new-rater training might be required of returning raters:
A rater who did not successfully meet calibration requirements in the previous year might be required by the district to complete new-rater training.
A district might require a rater who completed training requirements years ago but who has not been a TELPAS rater recently to complete new-rater training.
Returning Rater Planning Roster:
A roster of individuals within a district who meet the returning rater definition according to the online training histories will be available to assist coordinators in making training assignments.
The training webpage informs raters that districts may sometimes require a returning rater to complete new-rater training and to consult with their testing coordinator if they need clarification.
TEA

7. Holistic Rating Training
It is recommended that districts and campuses determine in the fall who their TELPAS raters will be in the spring.
A training flowchart is provided on the next slide.
It is recommended that campus personnel identify all ELLs early in the school year in order to determine which teachers would benefit from fall ELPS training and which teachers will need to be trained as TELPAS raters.
TEA

8. Training Fall ELPS-TELPAS Foundational (Awareness) Training
For teachers who will be trained as new TELPAS raters in the spring if they lack this foundation
Spring TELPAS Administration Procedures Training
As a key part of this training, information from the TELPAS Manual for Raters and Test Administrators is reviewed to prepare raters to proceed with online holistic rating training
New Raters
Returning Raters
Online Basic Training Course
Teachers unfamiliar with the ELPS and/or TELPAS should receive foundational training in the fall as a best practice. Spring training of new TELPAS raters will be a smoother process for teachers who already have this foundation.
Summary of Spring Rater Training Process:
In the spring, all new raters complete an online basic training course, which includes practice rating activities for each language domain – listening, speaking, reading (K–1 only), and writing.
New raters and returning raters complete calibration activities to ensure that they are prepared to apply the PLD rubrics consistently and accurately.
Returning raters have the option of reviewing the online basic training course, if desired, before completing calibration activities.
There are two sets of initial calibration activities. Each set contains 10 students. Each language domain is represented in each set. Raters successful on the first set do not calibrate further.
Only individuals who do not rate accurately enough on the first set will complete the second set. Those not successful on the second set will receive supplemental training, followed by a third and final calibration set.
Individuals not successful on the final set will either not be used as raters (a district decision) or will be provided rater support in accordance with test administration regulations.
The following slides review and provide details about the training process.
Online Calibration
(Sets 1 and 2*)
Online Calibration
(Sets 1 and 2*)
If not calibrated: Supplemental Holistic Rating Training
Final Online Calibration (Set 3)
*Set 2 required only if not successful on Set 1
TEA

9. TELPAS Administration Procedures Training
As part of annual spring TELPAS administration procedures training, holistic rating training requirements are reviewed with raters, as well as information about how to access the online training and calibration components.
In addition, raters receive training on assessment procedures such as how to assemble writing collections, how to record students’ proficiency ratings, etc.
TEA

10. Two Types of Training
Online basic training course This course is for new raters. It provides instruction on using the PLD rubrics and gives raters practice rating students in each language domain. There are separate courses for K–1 and 2–12.
Online calibration This is for all raters. Raters use the PLDs to rate students in each language domain. Raters have three opportunities to calibrate on assigned grade cluster.
TEA 10

11. Grade Clusters of Training and Calibration
Raters must know their assigned grade cluster to select the appropriate online training.
Raters should consult with their campus testing coordinator if they are unsure of their assigned cluster.
Grade Clusters
Grades K–1 Grade 2
Grades 3–5
Grades 6–8
Grades 9–12
Raters need to be careful to select the appropriate cluster.
K–1
The K–1 online basic training course and calibration activities are separated from the other grade clusters in the online system.
2–12
There is one online basic training course for grades 2–12, but within the course raters must make grade-cluster specific selections when they practice rating students. For calibration, raters choose the appropriate grade cluster from the start.
TEA

12. Raters with Students in Multiple Grade Clusters within Grades 2–12
Raters are required to train and calibrate in the cluster in which they have the most ELLs.
As a best practice, these raters should also review online basic training course practice activities in the additional cluster(s).
Example: A grade 2–5 ESL teacher has most of her ELLs in grade 3. She must complete training and calibration in grades 3–5. As a best practice, she should review the online practice activities for grade 2 to check her readiness to apply the rubrics appropriately.
These instructions apply to both online basic training and calibration.
TEA

13. Raters with Students in Multiple Grade Clusters that Include Grades K–1
Because of differences in the instructional content and rating rubrics, these raters must complete training and calibration for K–1 and at least one other cluster in 2–12.
Raters with more than one additional cluster should train in the cluster in which they have the most ELLs.
As a best practice, these raters should also review online basic training course practice activities in the additional cluster(s).
These instructions apply to both online basic training and calibration.
Example: A grade 1–3 ESL teacher has most of his ELLs in grade 2. He must complete training and calibration in grades K–1 and 2. As a best practice, he should review the online practice activities for grades 3–5 to check his readiness to apply the rubrics appropriately for his 3rd grade students.
TEA

14. When and where do raters take the online training?
Depending on campus arrangements, raters may either complete training and calibration during school hours, after school, or on weekends.
The online training system allows raters to access the training from any computer that meets the minimum system requirements.
Campuses inform raters of the details during annual administration procedures training.
TEA

15. Why is calibration necessary?
Calibration helps ensure that raters have adequate training, including ample practice and feedback, before they assess their students in the spring.
Calibration ensures that raters clear their heads and consider only the elements of student performance included in the PLDs.
Calibration supports assessment validity and reliability and is an important part of holistically scored assessment processes.
Standardized testing programs include processes to ensure that all individuals assessing students interpret the scoring rubrics the same way. Scorers of written compositions for the STAAR program complete calibration activities.
Yearly calibration is a necessary aspect of administering a holistically scored assessment. When holistic assessment processes are used, even the most experienced scorers need to make sure they are calibrated to score accurately.
Over time, calibration activities serve to give raters more examples that help expand their knowledge and help them rate students who are near the border between two proficiency levels or who exhibit less typical language characteristics.
TEA

16. Preparing for Calibration Sets
New raters must complete the online basic training course before beginning calibration.
Returning raters have the option to review the online basic training course (which includes rating activities) before beginning calibration.
TEA

17. Other Things to Know About Calibration
Raters affirm online that they will keep the contents of the calibration sets secure and confidential.
Calibration activities are taken from a bank and randomized. Trainees will rate different sets of students.
Raters can work at their own pace, go back and review students, and change ratings as they work.
For grades K–1, each set includes students to rate in listening, speaking, reading, and writing.
For grades 2 and up, each set for each grade cluster includes students to rate in listening, speaking, and writing.
TEA

18. Other Things to Know About Calibration
Raters can exit and return later to finish. They click a “submit” button when they are finished with a set.
After completing a calibration set, raters immediately see results. Results show the rating assigned by the rater and the correct rating.
Raters see annotations explaining the ratings. Raters should use the annotations to go back and review any incorrectly rated students. .
TEA

19. Is calibration a test?
No, it is a training method that ensures that raters have enough guidance, practice, and support to assess students consistently and accurately.
TEA

20. Will raters be able to refer to any resources during calibration activities?
Yes, raters should use their rating rubrics (PLDs) and refer to, as needed, information from the:
online basic training course
holistic rating PowerPoints produced by TEA
TELPAS Manual for Raters and Test Administrators
Educator Guide to TELPAS
The practice activities in the online basic training course illustrate how to interpret and apply the rubrics. The student examples serve as anchors for raters to use to ensure they are all interpreting the rubrics the same way when evaluating students. The PLDs, explanatory information provided during training, and student examples are all used to support rating consistency.
TEA

21. How many students must be rated successfully?
To be successful, raters need to rate students in their assigned grade cluster with at least 70% accuracy.
TEA

22. Supplemental Holistic Rating Training
Raters not successful after sets 1 and 2 must receive supplemental training.
The rater will meet with a district-appointed supplemental support provider.
After the rater has received supplemental training, he or she will be able to access the third and final calibration set.
This supplemental support training model has been established so that districts (regardless of the size of their ELL population) can provide this direct holistic rating support without the need to travel to an education service center.
Districts select individuals to serve as supplemental support providers. The number of supplemental support providers needed depends on the size of the district and the number of ELLs it serves. Reviewing the supplemental training needs of a previous school year might be a good indicator of the current year’s needs.
More information about the supplemental training process and criteria for individuals chosen to provide supplemental support will be included in the spring 2013 manuals for coordinators and raters.
TEA

23. Recap of Calibration Process
There are 3 sets of 10 students.
Raters who calibrate on set 1 are done.
Raters who don’t calibrate on set 1 go on to set 2. Raters who calibrate on set 2 are done.
Raters who don’t calibrate on set 2 receive supplemental training.
Raters attempt third and final calibration set.
Raters who calibrate on set 3 are done.
Set 2 can be accessed immediately after set 1. Raters who are not calibrated after set 2 will wait until they receive supplemental support to complete set 3.
TEA

24. What happens if a rater is unsuccessful in calibrating?
Two outcomes are possible for individuals who complete the calibration activities but are not successful:
The district may choose not to assign the individual to be a TELPAS rater.
If the individual is needed to serve as a rater, the district must implement rater support procedures to ensure that the rater’s students are evaluated consistent with the rating rubrics.
Individuals are not authorized to serve as raters unless they complete the required training components.
TEA

25. Course and Calibration Certificates
Raters who take the basic training course get a certificate from the online TrainingCenter after completing the course components.
Raters will receive a certificate of successful calibration when they calibrate.
Continuing education credit hours will be indicated in the rater’s online training history. Five continuing education credit hours will be available for completing the online basic training courses (K–1 and 2–12), and two hours will be provided for calibration. For calibration, raters will receive a certificate only if they calibrate.
District testing coordinators have access to reports to monitor completion and performance on the online basic training course and calibration activities.
TEA