1. STAR Training of Trainers

2. What is a computer adaptive test?
Too
high
low
Appropriate content
Before this slide – I’m thinking of a number between 1 and 100, guess the number.

3. Traditional Assessments are Imprecise Above and Below Grade Level
Sixth grade reading assessment
Another benefit to computer adaptive testing is its ability to accurately test students that are far above or below grade level better than other assessment types. On traditional assessments, there are many test items for the grade level the assessment was designed for. Only a few items are designed to test material above or below that grade level. For example, on this traditional sixth grade test, you can see there is a large number of questions written at the sixth grade level and relatively few written above or below that grade level.
(click) Students who are achieving at their grade level have a large pool of questions to answer to help determine their abilities.
(click) Students performing outside their grade levels have many fewer questions to answer, decreasing the accuracy with which you can determine their abilities. The result is an incomplete picture of the achievement of these students.
Below sixth grade
Sixth grade test items
Above sixth grade

4. CAT: Precise at All Levels
Sixth grade reading assessment
However, with computer adaptive technology, students are able to be tested on items for other grade levels, allowing for a more accurate categorization of their abilities. Therefore, another benefit to computer adaptive testing is its ability to accurately test students that are far above or below grade level better than other assessment types.
Why is this important? (to measure growth of low performing students you need to have accurate data)
Below sixth grade
Sixth grade test items
Above sixth grade

5. STAR Early Literacy General readiness Graphophonemic awareness
Phonics
Vocabulary
Comprehension
Structural analysis
STAR Score Definitions (Emergent Reader, Probable Reader, etc.) HANDOUT

6. STAR Reading Vocabulary in context Comprehension
Instructional reading level
Independent reading range
SEL – up to Grade 1 (typically) – transition to STAR-R when a student scores as “Probable Reader” – 100 sight vocab. – STAR –R, 1-12
Vocabulary as a proxy for comprehension (just like fluency is a proxy) because you can’t measure comprehension directly, you measure its manifestations. Vocabulary is

7. STAR Reading tests comprehension
Background
knowledge
Background
knowledge
Vocabulary
knowledge
Construct
meaning
from text
The items used in STAR Reading are designed to test students’ reading comprehension skills. The assessment items require students to call upon background information combined with vocabulary knowledge in order to construct meaning from text. These cognitive tasks are consistent with what researchers and educational experts describe as reading comprehension. In other words, background knowledge and vocabulary intersect to create meaning from text.
STAR Reading assesses comprehension directly with robust, holistic questions that tell you a lot about how all the reading skills relate to each other and work in concert.
Comprehension

8. STAR Math tests competency in eight mathematical strands
Numeration
Computation
Estimation
Algebra
Geometry
Word
Problems
STAR Math is a well-rounded math assessment. It combines computation and numeration items with word problems, estimation, statistics, charts and graphs, geometry, measurement, and algebra to help you pinpoint students’ math levels more accurately and efficiently.
Measurement
Data analysis
& statistics

9. Scaled Score (SS) 615 0 SS 900 SS 1400 SS Scaled Score (SS)
Based on question difficulty and number of correct responses
Best score to use for measuring growth because can be used to compare student performance over time and across grade levels
Useful in setting goals, measuring progress within a school year, and growth from year-to-year
The scaled score is used to derive the other STAR scores

10. Percentile Rank (PR) 52 1 PR 99 PR
To understand growth, we need to understand the scores that are used to measure that growth.
Quickly review:
Percentile Rank, a norm-referenced score
Gives the best measure of the student’s reading ability relative to peers
Ranges from 1 to 99
(click) Example a PR of 52 means a student’s reading skills are greater than 52% of same-grade students
PR is useful when comparing students at the same grade level. It is also helpful when looking at student growth relative to peers. If they are growing in PR they are surpassing average growth rates and experiencing an acceleration in learning. ???

11. Grade Level Equivalent Score (GE)
4.8

12. Scaled Score: Absolute Growth Percentile Rank: Relative Growth
Two types of growth
When using STAR Reading to monitor progress, you’ll need to be familiar with both scaled score and percentile rank. The scaled score will tell you about the absolute growth of the student: the raw amount of growth they have made. This is similar to a doctor measuring the height of a child in inches.
(click) The percentile rank will tell you about the relative growth of a student: how his growth compares to other students in the same grade. A doctor looks at relative growth by using a growth chart to compare the child’s height with other children the same age.
Scaled Score: Absolute Growth
Percentile Rank: Relative Growth

13. Estimated Oral Reading Fluency
A New Measure in STAR-EL
and STAR Reading

14. STAR and Est. ORF Resulted from a correlational study looking at
STAR (EL and Reading) scores
and DIBELS Oral Reading Fluency
Statistically HUGE sample
N=12,220

15. What a correlation!

16. What a correlation!

17. Notice What Happens
G4 – R=0.71
G3 – R=0.78
G2 – R=0.84
G1- R=0.87

18. Diagnostic Information

22. Reasons for score fluctuation
Standard Error
of Measurement
(chance)
Fluctuation in
student performance
(good day, bad day)
There are three main reasons for this normal fluctuation. First is the standard error of measurement or the extent to which scores would be expected to fluctuate due to chance. Standard error of measurement can be calculated, and a chart listing the SEM can be found in the Understanding Reliability and Validity document in the software.
(click) Next there are fluctuations in student performance. A student may perform at her best during one administration of a test, and somewhat less than her best on another occasion. These fluctuations may be related to such things as illness, distractions, anxiety, or motivation.
(click) A final reason for score fluctuation is a statistical phenomenon called regression to the mean. This is what happens when students with the highest and lowest scores on a first test tend to score closer to the average on the next test.
More information can be found in the Understanding Reliability and Validity guide under the resources tab in the software.
Regression
to the mean
(statistical phenomenon)

23. Fidelity of Administration
Pre-Test Instructions

24. Extended comprehension
Item Time Limits
Grades K - 2
3 minutes
per item
60 seconds
per item
Grades 3 +
STAR READING
K-2 60 seconds per item
Grades 3+ have 45 seconds per short comprehension item
Grades 3+ have 90 seconds per extended comprehension item
STAR MATH
All items have a 3 minute time limit
STAR EARLY LITERACY
All items have a 90 second time limit
ALL ASSESSMENTS
Testing time does not affect scores and there is no overall test time limit. They are just in place to maintain test security and keep the test moving.
Warning clock appears in upper right corner during last 15 seconds
When time limit expires, test continues to the next item and counts the timed out item as incorrect unless students have selected the correct answer before time runs out.
45 seconds
Short comprehension
90 seconds
Extended comprehension
90 seconds
per item

25. Using STAR in an RtI Setting

26. Core Instructional Program (Tier 1)
Progress
shown
Lack of
progress
Data Review
Progress
Monitoring
with STAR
Intervention B
Intervention A
(Tier 2)
[This diagram is on page 2 of the handouts.]
The process schools follow when implementing RtI can vary.
The core instructional program is the foundation. This is where learning takes place everyday for all students, so it is important that students are engaged and best practices are followed. No matter what difficulties individual students may have, creating and maintaining a strong core instructional program is essential.
(Click) Periodically, all students in the core instructional program are assessed to determine their degree of success within that program. This is called universal screening, and STAR can be used to accurately and efficiently screen all students.
(Click) The information from that screening is reviewed, generally by a team made up of teachers, administrators and specialists. The team determines which students need additional opportunities to learn the material.
(Click) An intervention begins for students who needed extra help. This is done in addition to the core instructional program, not in place of it.
(Click) The progress of the students receiving the intervention is monitored on a regular basis, usually between weekly and monthly. The STAR assessment is an appropriate tool for progress monitoring.
(Click) The data is reviewed to determine if the students are experiencing success with that intervention or are still having difficulties.
(Click) Students showing progress, may continue with the intervention or (click) stop with the intervention and return to receiving just core instruction.
(Click) If students are not showing progress, check that the intervention is being administered with fidelity. Also, consider intensifying the intervention, for instance, increasing the amount of time the student receives intervention services.
(Click) Or, try a different intervention, perhaps a more intensive intervention, such as individualized instruction.
Data Review
Screening with STAR
Core Instructional Program (Tier 1)

27. Screening Report The report is a hyperlink to a full pdf.
Talk through the report format:
Black bar represents benchmark – 40th percentile – can be adjusted
Each bar represents a student – green, at or above benchmark, 40th – blue, on watch, between 40th and 25th – yellow, intervention, between 25th and 10th – red, urgent intervention, 10th or below
Briefly discuss the data on this specific report sample
Fall screening – Grade 6 – Reading
Time permitting, ask the audience, “If this report was for a school you supervised, would you be pleased with their performance? Why or why not?:
Answer: yes – 72% meet or exceed benchmark (target is 80, so they are close) – also percentages in yellow and red fall close to the RTI rules of thumb for T2 and T3 (15% and 3-5% respectively).

28. A Math Report 28

29. No hyperlink 29

30. Are you satisfied with the number/percent of students in your class/grade who are at or above proficiency?  Is core instruction effective?
Some students are in the “red.”  Based on your opinion of the student(s) and other information you have, is this student in need of “urgent intervention?”
Some students (blue) are “on watch” or “almost” at benchmark.  Which of these are you “worried about” and which do you think will progress with continued core instruction?
What is being done or what do you think needs to be done for those students in need of “intervention” (Yellow)?
The questions are from George Batsche (Univ. of South FL). A noted expert in RTI, Batsche wrote these to help people interact with the data presented on the Screening Report.
George Batsche

31. Screening
“The testing needs to be brief, easy to administer, reliable, and valid” (Applebaum, 2009, p. 4)
“Screening is a type of assessment that is characterized by providing quick, low-cost, repeatable testing of age-appropriate critical skills or behaviors” (NRCLD, 2006, p. 1.2)
“For a screening measure to be useful, it should. . . be practical” (NRCLD, 2006, 1.2)
More kids don’t need screening than do.

32. Clean the Fish Tank!
If the water in the fish tank is dirty, you don’t start taking our individual fish and diagnosis their needs. You clean the fish tank!
Heather Diamond, FL Department of Ed.

33. Progress Monitoring

36. What’s wrong? Nothing!
From a statistical and systems standpoint, things are fine with this report. The teacher, however, felt “something must be wrong” because there are so many test scores under the line. This is a clear case of where we need to ignore the blue marks and trust the line. Statistically the blue dots are irrelevent. They mean nothing. It’s only when they combine to form a line that meaning begins to emerge – and the formula for that is far more complex than “eyeballing” it.

38. Student Growth in PR and SS
(these are STAR Reading equivalents)
Why do you need to look at both? It is possible, even common, that the SS might change but PR might not. Each PR at each grade level is equivalent to more than one scaled score.
Example: (click) A fifth grade student has a 22 PR with a scaled score of If she scores 484 SS on the next STAR assessment, (click) she will have made absolute growth as reflected by the SS, but her PR is still 22. She did not make growth relative to her peers.

39. Meta-skill assessments & single skill probes
The STAR assessments are the way you can determine if a student’s reading or math skills have improved overall. Teachers may also choose to monitor progress in individual skills, but ultimately, because you want better readers or math students, you’ll want to monitor improvement in reading or math as a whole.
STAR Assessments
measure skills
as a whole
Individual probes
measure skill on
isolated tasks

40. Progress Monitoring
It is a common misconception that accurate measure of the isolated skills requires an explicit, separate test for each skill. On the contrary, since the sub skills in a given domain are highly interrelated, sub skills scores can be derived more accurately—and more efficiently—from student’s overall test performance, which provides far more data from more items than a short probe.

41. Goal Setting

43. Star Hosted Data

45. The Statistics Behind the Goal-Setting WizardK 1 2 3 4 5 6 7 8 9 10 11 12

46. Goal-Setting Rate to Maintain 11PR = 2.0 Rate to Meet Bench-
mark = 7.0
Moderate Goal
= 3.3
Ambitious Goal =
5.5

47. Not responding to the intervention
Check the fidelity
of implementation
Give the intervention
more time to work
Increase the intensity
of the intervention
Try a new
intervention
We’ve determined that Y is not responding to the intervention. What could we do next? (Give participants time to discuss/record ideas.)
Here are a few options. (click) First, if you stick with Intervention A, you want to make sure that the intervention was implemented with fidelity. Was it done right? Was the student engaged and motivated?
(click) If you determine it was implemented correctly, you might want to give it more time to work if you notice positive changes in student learning that have not been reflected in testing yet. Is the student more on task? Completing homework that wasn’t done before? Seeking help when needed? This may be an indication learning growth is on its way.
(click) You might decide to increase the intensity of the intervention. If the student was previously receiving extra instruction three times a week for 20 minutes, perhaps boosting it to five times a week or 30 minute sessions would result in significant growth.
(click) If you do determine this intervention is just not the right fit for the student, you could try another intervention. This could be another Tier 2 intervention or a Tier 3 intervention depending on how intense, individualized, or diagnostic the intervention is.

48. Editing a goal Give the intervention more time to work
If you make a change to an intervention you’re trying with a student may mean that you want to edit the previous goal or start a new goal. For example, if you decide to give the intervention more time to work, you may need to edit the goal end date in the software to reflect this change.

49. Starting a new intervention
Try a new
intervention
If you decide to try a new intervention, you will want to start a new intervention and use the Goal Setting Wizard to start a new goal. This will result in a vertical red line being drawn to indicate a change in intervention. This allows you to see how the student’s response to the new intervention differs from the previous intervention. (screen shot to show this?)

50. How is Y responding to the new intervention?
(New mock ups will be created for this screen. The new sample would be similar to this with two interventions shown, or perhaps with just one successful intervention.)
For now let’s assume that you started a new intervention with the student. A new goal was set in the software, resulting in a vertical red line to be drawn and a new goal and goal line to appear on the Student Progress Monitoring report. After seven subsequent tests, a new picture emerges. What do you notice about how the student is responding to the new intervention? Is the student on track to meet her goal? How do you know?
Note the growth rate on page two. Here we can see student growth in scaled scores per week and compare it to the expected growth rate above.

51. STAR Early Literacy STAR Math STAR Reading
Very Highly Rated
for Both Screening
and Progress Monitoring
by the National Center for Response to Intervention

52. High rating from National Center on RTI
Screening Tools Chart (partial)

53. High rating from National Center on RTI
Progress Monitoring Tools Chart (partial)
STAR Reading, STAR Early Literacy, and STAR Math received the highest ratings in eight of the nine categories, each receiving significantly higher ratings than several other tools.

55. Resources Software: manuals, Live Chat Phone: (800) 338-4204
Renaissance Training Center:
Getting the Most out of STAR Guides
Help in the software (Software and technical manuals, live chat)
Technical support by phone, , knowledge base
Renaissance Training Center: On-Demand Sessions
Getting the Most out of STAR Reading guide (STAR Math and STAR Early Literacy guides to come)

56. RtI Experts Dr. George Batsche University of South Florida
Dr. Matt Burns
University of Minnesota
Dr. Ted Christ
University of Minnesota
Dr. Joe Kovaleski
Indiana Univ of PA
Dr. Jim Ysseldyke
University of Minnesota
Dr. Ed Shapiro
Lehigh University
Dr. Amanda VanDerHeyden
Consultant