Can Third Graders Type? An examination of the fairness by grade of requiring constructed- response items on large- scale assessments administered via the computer By Steve Martin Connecticut State Department of Education Connecticut Assessment Forum 8/13/12

Good Afternoon Today we will Practice using responders. Use responders to get a sense of where the group is. Review where SBAC is on this question and cover standards and other background on this issue. Discuss what other states are doing. Examine typed-response data from a recent CMT pilot test. Discuss possible contributing factors to trend data. Review where Connecticut is going in the short-term. Discuss reporting and further research possibilities. Q&A We will finish by exploring some free sample online resources designed to help students learn to type.

Practice Using Responders Start->programs->2know toolbar ① To start your responder, first hit the on/off button on the bottom.

Practice Using Responders Start->programs->2know toolbar ① To start your responder, first hit the on/off button on the bottom. ② You will know it is on when words appear in this screen.

Practice Using Responders Start->programs->2know toolbar ① To start your responder, first hit the on/off button on the bottom. ② You will know it is on when words appear in this screen. ③ Wait for the 2Know Toolbar to appear on this screen before you enter the session.

Practice Using Responders Start->programs->2know toolbar ① To start your responder, first hit the on/off button on the bottom. ② You will know it is on when words appear in this screen. ③ Wait for the 2Know Toolbar to appear on this screen before you enter the session. ④ Once you see the toolbar press ENTER. ⑤ When you see the network “Steve” on your responder, press ENTER again. ⑥ Once everyone is on we will practice this ABC question. When instructed, press A if you are MALE, and press B if you are FEMALE; and then press ENTER.

Question #1 Q:What age were you when you first learned to type? A. Age nine or younger B. Age C. Age D. Age E. Age 18 or older

Question #2 Please answer this question only if you have children of your own. Q:At what age was your child(ren) when he/she/they first learned to type? A. Age nine or younger B. Age C. Age D. Age E. Age 18 or older

Question #3 If you have a child or children who are at least 13 years of age, (or at least grade 8) please answer this question about typing speed. Q: I think that my child (or my children) can type… A. Faster than I can. B. About the same speed as I can. C. Not as fast as I can. D. I don’t know how fast my child(ren) can type.

Question #4 Please answer this question if you have recent experience with third graders using the computer. Q:Which of the following is the fairest assessment of typical third-grade computer skill? A. They cannot use the keyboard or mouse with any reliability. B. They can use the mouse to point and click reliably C. They can use the mouse and keyboard reliably, but their typing is very slow. D. They can use the mouse and keyboard reliably and they can type words using the computer keyboard at least as fast as they can write with a pencil.

Question #5 Q:Given that the statewide test will be administered to third graders via computer beginning in the spring of 2015, which of the following test environments for them would you currently support? A. Third graders should only need to point and click. B. Third graders should point and click AND type responses to short-answer items. C. Third graders should point and click, type responses to short answer items, AND type responses to extended response items (like the CMT writing sample).

Where SBAC Stands Steve: What research is planned on this issue? SBAC Cognitive lab research is planned to learn how students across a range of grades approach and interact with different types of computer-administered items. This is part of a larger effort to eliminate or minimize factors that could distract students from the ability to demonstrate their knowledge and skills. For more information, please see the SBAC accessibility guidelines

Where SBAC Stands Steve: What about keyboarding skills specifically? SBAC: The Common Core State Standards emphasize proficiency with technology. For example, the 4th grade writing standard calls for students to “demonstrate sufficient command of keyboarding skills to type a minimum of one page in a single sitting.” Administering assessments by computer is a part of ensuring that schools and teachers can measure mastery of the full range of the standards.

Technology Standards for Grade 3 Students What exactly are the CCSS Standards on the keyboarding issue for grades 3 and 4? CCSS: Grade 3 Standard—W.3.6—With guidance and support from adults, use technology to produce and publish writing (using keyboarding skills) as well as to interact and collaborate with others. Grade 4 Standard—W.4.6.—With some guidance and support from adults, use technology, including the Internet, to produce and publish writing as well as to interact and collaborate with others; demonstrate sufficient command of keyboarding skills to type a minimum of one page in a single sitting.

What is Age Appropriate? Differing perspectives “There is no right or wrong time to introduce your child to the family computer. Much like deciding when to buy the first bike, or when your child can go out alone with friends, the choice will be based on your parenting style, your values and, most importantly, your child’s readiness.” Source:

What is Age Appropriate? Differing perspectives What about computers and Waldorf education? Source: “Waldorf teachers feel the appropriate age for computer use in the classroom and by students is in high school. We feel it is more important for students to have the opportunity to interact with one another and with teachers in exploring the world of ideas, participating in the creative process, and developing their knowledge, skills, abilities, and inner qualities. Waldorf students have a love of learning, an ongoing curiosity, and interest in life. As older students, they quickly master computer technology, and graduates have successful careers in the computer industry.”

General Typing Speed Data Source: Lynda S. Hartman rev 1/20/05 Permission to use if credit is retained.

General Typing Speed Research Source: Lynda S. Hartman rev 1/20/05 Permission to use if credit is retained Study In the Illinois School District U  Researchers suggest that wpm is sufficient in later elementary school (grades 3 – 6).  Some teachers, though, have made lower targets:  Others made their goal to be 2-3 times the students handwriting speed. GradeTarget in WPM

Source: Lynda S. Hartman rev 1/20/05 Permission to use if credit is retained. Buhler, Kansas USD – 313 Conducted Research of student typing speed. Task Force Report:  No recommendations for keyboarding speed until grade 8.  At this point, though, students are expected to type at a rate of 30 WPM with high accuracy. General Typing Speed Research

Source: Lynda S. Hartman rev 1/20/05 Permission to use if credit is retained. Utah State Office of Education Utah provides a detailed Web site in which keyboarding speed requirements are laid out very explicitly: GradeWPM by end of keyboarding course ards.html General Typing Speed Research

What Are Other States Doing? I asked states that were listed in a State Educational Technology Directors Association 2011 report as having computer delivered assessmentsalready the following six survey questions: 1. Is your state’s summative test administered mostly via computer at any grade? 2. Is your state’s summative test administered mostly via computer to grade 3 students? 3. For how many years has your state allowed for grade 3 students to be administered a large-scale exam on computer? 4. Are there any computerized, grade 3, open-ended items requiring a keyboarded short-answer response on the statewide exam?

What Are Other States Doing? Survey Questions 5 and 6: 5. Are there any grade 3, open-ended items requiring a keyboarded extended-answer response on the statewide exam? (For purposes of this question, consider an “extended” response as typically requiring more than two sentences to receive the highest score.) 5. Has the issue of keyboard competence for younger students in any way shaped your state’s decisions in the selection of computerized item-types?

What Are Other States Doing? 1. Is your state’s summative test administered mostly via computer at any grade? StateResponse DelawareYes GeorgiaNo HawaiiYes IdahoYes IndianaYes KansasYes MarylandGr 5 &8science MinnesotaYes NebraskaYes OklahomaNo grade 3 test TexasNo West VirginiaYes  9/12 state’s surveyed administer some portion of their summative test via computer.  For Georgia, the computer version is an option for the grade 3 reading retest only.  Oklahoma reports that there is no grade 3 computer assessment in the state at all.  Texas administers an assessment in grades 2-12 online. TELPAS (Texas English Language Proficiency Assessment System) is for LEP students only. That is the only online assessment at grade 3 in Texas.

What Are Other States Doing? 2. Is your state’s summative test administered mostly via computer to grade 3 students? StateResponse DelawareYes GeorgiaNo* HawaiiYes IdahoYes IndianaNo KansasYes MarylandNo MinnesotaYes NebraskaYes OklahomaNo TexasNo* West VirginiaYes*  7/12 state’s surveyed administer their summative test via computer to grade 3.  For Georgia, the computer version is an option for the grade 3 reading retest only.  Texas LEP students only.  Limited to the Direct Assessment of Writing. The rest of the Westest 2 is administered via paper and pencil at this grade. First draft can be done in pencil, but then it must be copied into the computer for a score.

What Are Other States Doing? 3. For how many years has your state allowed for grade 3 students to be administered a large-scale exam on computer? StateResponse Delaware2years Georgia4 years* Hawaii2years Idaho3years Indiana2years Kansas6years MarylandWas modified/ no longer Minnesota3years Nebraska3years OklahomaNA TexasNA West Virginia3years  9/12 state’s surveyed have administered some test to grade 3 students for 2 years or more.  Maryland gave a small number of students a modified assessment online, but they no longer administer a modified exam.  Kansas has been at this the longest having six years of computer- delivered testing.  Georgia makes this an option for grade 3 retesters.  Indiana—not the summative test.

What Are Other States Doing? 4. Are there any computerized, grade 3, open-ended items requiring a keyboarded short-answer response on the statewide exam? StateResponse DelawareNo all m-c and grid GeorgiaNo HawaiiYes* IdahoNo IndianaNo KansasNo MarylandNo MinnesotaYes* NebraskaNo OklahomaNo TexasNo West VirginiaNo*  Only 2/12 state’s surveyed require keyboarded short-answer responses from students at grade 3.  Hawaii gave a tentative “Yes” to this question. This still needs confirmation.  Minnesota administers the MCA in Mathematics to 80% of the students. The short-answer items are only numerical. MN ran cognitive labs and decided to limit students to just the numbers.  West Virginia only gives the Writing Sample online. No short-answer open- ended.

What Are Other States Doing? 5. Are there any grade 3, open-ended items requiring a keyboarded extended-answer response on the statewide exam? StateResponse DelawareNo GeorgiaNo HawaiiYes* IdahoNo IndianaNo KansasNo MarylandNo MinnesotaNo NebraskaNo OklahomaNo TexasNo West VirginiaYes*  Only 2/12 state’s surveyed require keyboarded extended-answer responses from students at grade 3.  Hawaii gave a tentative “Yes” to this question. This still needs confirmation.  West Virginia direct assessment of writing is keyboarded at grades 3-11, the rough draft can be by paper/pencil but final must be keyboarded by grade 3 students. When surveyed, 85% of 3 rd graders said that they preferred the online version of the test over the traditional paper and pencil test.

What Are Other States Doing? 5. Has the issue of keyboard competence for younger students in any way shaped your state’s decisions in the selection of computerized item-types? StateResponse DelawareNo GeorgiaNo HawaiiYes* IdahoNo IndianaNo KansasPossibly* MarylandYes* MinnesotaYes* NebraskaYes* OklahomaNo TexasNA West Virginia Yes*  6/12 state’s surveyed have made decisions based, at least in part on, keyboard competence.  Hawaii gives students three opportunities to test.  Kansas has statewide standards for keyboarding that begin in kindergarten.  MD moved away from administering online.  MN’s Cognitive Labs indicated that they should limit keyboarding to just the numbers.  Nebraska does paper/pencil until grade 4.  West Virginia did a survey, 85% of 3 rd graders said that they preferred the online version of the test over the traditional paper and pencil test.

A Question for Your Responder Given what you just learned about the experience in other states… Q: which of the following test environments for third- grade students would you currently support? A. Third graders should only need to point and click. B. Third graders should point and click AND type responses to short-answer items. C. Third graders should point and click, type responses to short answer items, AND type responses to extended response items (like the CMT writing sample).

March 2012 CMT DAW Pilot Test Over a thousand CT students per grade (range 1,381 – 1,581) participated in an online Direct Assessment of Writing pilot which required them to keyboard their responses. These responses are not reported back to schools or students. They don’t “count.” The purpose of the pilot test is to determine the difference in difficulty between the two modes. If the two modes can be “post-equated” then a choice of modes can be offered before the 2015 online test. The tests were recently scored. Both the pilot scores, and the observations of the scoring leads at the various Measurement Incorporated scoring centers were reported to the state.

Anecdotal Evidence From the Scoring Center Mathematics: (Marianne Nasiatka, Grade 3 Scoring Director) We saw evidence that some third graders don’t know how to use the shift key. An equal sign would appear in their responses instead of an addition symbol.

Anecdotal Evidence From the Scoring Center Reading and Science (Steve Zrimec, project monitor) The quality of the typewritten responses appeared to be equal to the handwritten responses from past projects. Keyboard unfamiliarity presented no obvious problems. There did not appear to be any obvious advantages for the students (i.e. spelling, grammar, and even ideas weren’t improved because of the format). And actually, this seemed true across all grades, not just Grade 3.

Anecdotal Evidence From the Scoring Center Direct Assessment of Writing (Sharon Williams, Grade 3 scoring director) We were pleasantly surprised at the typing skills. There were few responses that indicated that the students did not know how to type. In other words, we saw very few "fingers on the wrong keys" gibberish responses but rather a fairly competent sample. Readers noted (as did I) that an unusually high number of responses ended abruptly, as if the writers had run of time before finishing their narratives. We wondered if this was because the third graders are unable to type as fast as they can write. We also noted that many of the responses seemed brief and undeveloped compared to the responses we saw in the spring assessment. Very few typed responses went on to the second page, and many were only a half page long. This could be an issue with time and typing speed too.

A Question for Your Responder Now that you have heard of the observations from the Scoring Centers… Q: which of the following test environments for third- grade students would you currently support? A. Third graders should only need to point and click. B. Third graders should point and click AND type responses to short-answer items. C. Third graders should point and click, type responses to short answer items, AND type responses to extended response items (like the CMT writing sample).

CMT Typed Response Data Grade ScorePointScorePoint Difference 80.3

CMT Typed Response Data Grade ScorePointScorePoint Difference

CMT Typed Response Data Grade ScorePointScorePoint Difference

CMT Typed Response Data Grade ScorePointScorePoint Difference

CMT Typed Response Data Grade ScorePointScorePoint Difference

CMT Typed Response Data Grade ScorePointScorePoint Difference

CMT Typed Response Data Grade ScorePointScorePoint

Possible Contributing Factors to This Trend Were the piloted (keyboarded) prompts more difficult? Was the novelty of computer testing a factor? Are the scorer’s showing a bias in favor of handwritten text? Because of a lack of clear standards on the pilot, did the time of administration of the pilot generally rob students of the full 45-minutes intended? If time on task is a factor, will lengthening the standard time limit for a handwritten response be sufficient to overcome the differences for students who are typing? Is keyboard familiarity (typing skill) a contributing factor?

A Question for Your Responder Now that you have seen the data from the CMT DAW Pilot Test… Q: which of the following test environments for third- grade students would you currently support? A. Third graders should only need to point and click. B. Third graders should point and click AND type responses to short-answer items. C. Third graders should point and click, type responses to short answer items, AND type responses to extended response items (like the CMT writing sample).

Connecticut’s Short-Term Plan 1) The Department would like to encourage equipped districts to administer more statewide tests on the computer before ) Pilot data will show which of the test sessions can be equated to take into account the different difficulties of taking tests on the computer. 3) The Department plans to give districts some level of choice about which mode (paper-and-pencil vs. computer submitted) for students taking the census test.

Connecticut’s Short-Term Plan 4) Because selected forms will be equated, this means that a resulting scale score for the same test may have a different raw score as a result of mode. (written vs. computer submitted) =

Further Research RESPONDER TIME: Should these CT findings be reported to SBAC? Please use “TRUE” for YES and use “FALSE” for NO. A repeated study could be done to see if the patterns are persistent over time. (Students may be improving on typing skill each year.) It is possible to design a factor analysis to try to find out how much of the difference between the two modes is attributable to typing skill alone.

Q & A

Online Resources ing.cfm Mrs. Yollis’ blog (above) sends you to these fine resources: ing.cfm Steve’s Review: Learn to type AND Dance Mat Typing Game (same site)— By the BBC the best one here. I learn Tech…Website—hit the drop down “typing games.” the best ones are:  typing chef  keyboard climber game  super hyper I also got addicted to “Typing Defense” Typer Shark game???—could not find it nor “Typing Lessons”

Dance Mat Typing  Sequential lessons  Animal guides  Reward Screens  10-minute lessons  20 small typing tasks.  About a dozen lessons in the sequence.

Typing Chef  Type the word in the bubble before it pops.  In later, more challenging screens other food falls into the pan.  Game works on speed of typing target words that get increasingly complex.

Keyboard Climber Game  Help the monkey to climb ever higher by correctly typing the letter above him.  Clue to which hand to use based on left or right of screen.  Monkey falls to bottom of current level with each mistake.

Super Hyper Spider Typer  Three levels of difficulty.  Each progressively difficult within level.  Type the letters or words on the chameleons’ backs to protect the spider.  Different colored chameleons have different numbers of lives.

Typing Defense defense.swf defense.swf  Type the word associated with each asteroid before it collides with the earth.  Planetary defense ray destroys correctly spelled asteroids.  Additional key words are earned like “slow, ice, and strike” as you increase in levels.

A Question for Your Responder Now that you attended this session… Q: Will you fill out an evaluation sheet? A. Yes! B. No! C. There is no evaluation sheet.