1. New Visions for Public Schools “A Guide to Using the Regents Item Analysis Tool to Analyze Student Data” for Teachers and School Leaders October 2011

2. © New Visions for Public Schools, 2010 Overview of the Item Analysis Tool The Regents Item Analysis Tool enables teachers to gauge their students’ readiness for the New York State Regents Exams. The tool allows teachers to input student response data from both mock Regents assessments and from actual previous exams. Through an analysis of Regents Exam questions and students responses, the tool provides teachers and students with a breakdown of: The student responses to each question, Which questions students found the most difficult and what were the most common mistakes, The skills and content knowledge required to answer each exam question correctly, and What scores are needed to be career and college ready. The Regents Item Analysis Tool will allow teachers to assess student strengths and weaknesses across individuals and classes. The following guide will assist you in orienting yourself to using the Regents Item Analysis Tool. 2

3. © New Visions for Public Schools, 2010 Goals of Guide The purpose of this guide is to provide step by step information about how to use the New Visions Item Analysis Tool. This guide will provide: Instructions for how to open and navigate around the New Visions Regents Item Analysis Tool. Detailed descriptions of all of the components the tool. Instructions for how to use each part of the tool to analyze the Regents exams and your own student data. 3

4. Table of Contents 1.Accessing the Regents Assessment Tool 2.Setup 3.Regents Grid 4.Summative Graph 5.Summary by Unit 6.Summary by Student 7.Answer Key 8.Conversion Chart 4

5. 1. Accessing the Regents Assessment Tool The Regents Assessment Tool has been provided for your school on a CD that includes the tool itself as well as this instruction manual for how to operate the tool. To download the tool... 1.Place the CD into the drive, 2.Locate “NV Regents Item Analysis Tool” in the “My Computer” section of your computer 3.Double click and download the file. Our assessment tool currently includes the Global, US History, Integrated Algebra, Living Environment, English Language Arts, and Earth Science Regents Exams, combined into one spreadsheet. You will find the student responses from the latest exam has been input in each of these subjects. Please Note: This is the first year that the New Visions Regents Item Analysis tool includes the English Language Arts and Earth Science Exams. For this reason, only the most recent exam tab will be available. 5

6. © New Visions for Public Schools, 2010 Setting up the Item Analysis Tool Now--open the Item Analysis Tool that has been downloaded onto your computer. It is a Microsoft Excel file. 6 At the bottom of the Excel page, you will find that the tool contains a number of worksheet tabs. Make sure you start on the Interim Assessment Setup worksheet by clicking on that tab at the bottom of the page. The rest of the worksheet tabs will be explained later in this guide. The first part of the Item Analysis Tool is the Interim Assessment Setup worksheet. This worksheet will allow you to select up to three tests to analyze and compare at once.

7. © New Visions for Public Schools, 2010 Setup: Enabling Macros Before you select the tests to analyze and compare, Macros must be enabled. To enable Macros, follow these steps: 1. Click the Office Button in the top left corner. 2. Choose the Excel Options button. 7 1 2

8. © New Visions for Public Schools, 2010 Setup: Enabling Macros 3. Click on Trust Center on the left. 4. Click on Trust Center Settings on the right. 5. Click on Macro Settings on the left. 6. Select Enable All Macros. 7. Click on OK. 8 3 4 56 7

9. © New Visions for Public Schools, 2010 Setup: Selecting Exams to Compare After enabling macros, follow these steps to select up to three exams to analyze and compare: 1.Click on the box next to Subject in the middle of the Setup worksheet. 2.Click on the down arrow that appears. 9 1 2

10. © New Visions for Public Schools, 2010 Setup: Selecting Exams to Compare 3.Select the Subject from the drop-down menu. 4.Click on the box next to First Exam, click on the down arrow, and select an exam from the drop-down menu. 5.Repeat step 4 for the Second Exam and Third Exam if you wish (If you only wish to examine one Exam select “none” for the Second Exam and/or Third Exam tabs). 6.Click on the button that says Click Here to Change Settings 7.Wait for the macros to run. You will see the screens and tabs changing automatically, and when the macro is finished running you will only see the worksheet tabs relating to the tests that you selected in the Setup worksheet. 10 3 4 6 5

11. © New Visions for Public Schools, 2010 Setup: Moving Across Sheets 1.To move across worksheets to different parts of the tool, click on the tabs at the bottom of the page with the names of the different sheets. 2.If you need to scroll to see more worksheet tabs, use the arrows to the left of the tabs. *Note: If you have not enabled macros and/or selected exams to compare in the Setup tab, you will see worksheet tabs for all of the exams at the bottom of page. 11 12

12. © New Visions for Public Schools, 2010 Setup: Moving Across Sheets There are three types of worksheets that present information about the Regents exams: 1.The light green worksheet tabs indicate the Regents Grids for the individual exams. These grids will allow you to analyze student performance on the Regents Exams. 2.The blue tabs indicate the Summative Graph, the Summary by Unit, and the Summary by Student for each subject area. These charts will allow you to compare how students performed on the tests you selected in a graphical summary and in more detailed tables. 3.The yellow tabs indicate the Answer Keys and Conversion Charts for each subject area. The Answer Keys contain more detailed information about the content and skills of each exam question. The Conversion Chart will help you convert raw exam scores into final scores on the Regents Exam. It also shows you what scores are needed to be on track for graduation. 12 123

13. © New Visions for Public Schools, 2010 The Regents Grid The first part of the Item Analysis Tool is the Regents Grid. Again, these will be the light green tabs at the bottom of the screen. This grid will allow you to analyze student performance on the Regents Exam. The Regents Grids are named according to their subject area and date (Such as Global Jane 09 or Algebra June 10 etc.) 13

14. © New Visions for Public Schools, 2010 Regents Grid: Components The Regents Grid will allow you to analyze student level performance on the Regents Exam. It contains three main sections: 1.Content of Exam Questions 2.Student Exam Responses 3.Distracter Analysis 14 1 2 3

15. © New Visions for Public Schools, 2010 Regents Grid: Exam Question Content Section 15 The top section of the Regents Item Analysis Grid contains information about the content of each individual question on the exam. The default view only shows all information for each question. To show less details, click on the minus arrow on the top left of the worksheet. The language used to describe the questions varies across subjects, and includes information such as units or strands, content or key ideas, skills or performance indicators, point values, and correct responses for each question.

16. © New Visions for Public Schools, 2010 Regents Grid: Student Response Section 16 The middle part of the Regents Item Analysis Grid contains student level response data. This contains the following: 1.Rows with Student-level Identification Information 2.Columns for Multiple Choice Raw Score, Open Response Raw Scores (DBQ, Essay, or Extended Response, as applicable), and Final Score 3.Columns for Each Multiple-Choice Exam Question 4.Cells with Student Responses 1 2 3 4

17. © New Visions for Public Schools, 2010 Regents Grid: Student Response Section 17 The multiple choice column is based on the results of student data that you have entered (see next page for directions on inputting student data). The open response column is based on the data that you enter under the open response raw score section of the Regents Grid (see page 20). The final score column adds up the points a student has earned on the multiple choice and open response sections. The color of that column is linked to the conversion chart (see page 39), and indicates Student Readiness. When a student answers a question correctly, the cell with the student response turns green. Incorrect answers have no color. Multiple Choice Open Response Final Score

18. © New Visions for Public Schools, 2010 Regents Grid: Inputting Raw Data 18 Student identification information Multiple choice student responses The Regents Grids require you to input your own student level data. The Regents Grid allows you to input multiple choice and open response student data for analysis. Please note: This year we have pre-entered the most recent exam in each subject with actual student scores for the purpose of comparison. IMPORTANT! The Regents Grid is defaulted at 40 rows for student responses. If you have data for fewer or more than 40 students (very likely) follow these steps on pages 24 and 25 to remove or add the number of students for your data.

19. © New Visions for Public Schools, 2010 Regents Grid: Inputting Raw Data-Multiple Choice For the multiple-choice questions, the data can be entered in the Regents Grid by copying and pasting the scores from a Scranton results spreadsheet or by entering all of the scores manually. Follow these steps to enter the data. (Your Scantron data spreadsheet may differ slightly from the pictures below). 1.First, highlight all of the Students Names and OSIS numbers (For example, the information for the first student will read “First 1”, “Last 1”, and “111111111”) in the Regents Grid, then right click and select Clear Contents. 2.Input your student names and ID numbers manually, or copy and paste them from your Scantron data spreadsheet. Make sure that the Grade column is filled in correctly on your Regents Grid. *If your Scantron spreadsheet has student last names and first names combined into one column, you can simply copy the whole Student Name column from the Scantron spreadsheet into the Last Name column in the Regents Grid. 19

20. © New Visions for Public Schools, 2010 Regents Grid: Inputting Raw Data-Multiple Choice 3.Highlight all of the data you would like to replace, then right click and select Clear Contents. 4.If you are using data from a Scantron, copy your data from the Scantron Excel file and paste it into the empty cells. 20 *If your Scantron answers are displayed in letters instead of numbers, see next page for directions on converting letters to numbers. If you are entering data manually, begin typing the information for each student into the empty cells.

21. © New Visions for Public Schools, 2010 Regents Grid: Inputting Raw Data-Multiple Choice If your Scantron spreadsheet displays the answers as letters instead of numbers, you will have to convert the letters in your Scantron spreadsheet into numbers before you copy and paste it into the Regents Grid. To convert the letters to numbers, follow these steps: 1.With the Scantron spreadsheet open, press CTRL+H on your keyboard. You will see the Find & Replace screen pop up. 2.In the box next to “Find what,” enter “A.” 3.In the box next to “Replace with,” enter “1.” 4.Click on the Options button. 21 2 3 4

22. © New Visions for Public Schools, 2010 Regents Grid: Inputting Raw Data-Multiple Choice 5.Check the box that says “Match entire cell contents.” 6.Click on the button that says “Replace All.” 7.Repeat steps 2, 3, and 6 to replace B with 2, C with 3, and D with 4. 22 5 6

23. © New Visions for Public Schools, 2010 Regents Grid: Inputting Raw Data-Open Response To see the open response data input section, scroll to the far right of the Regents Grid using the bar on the bottom right of the worksheet. The open response scores have to be entered manually on the right side of the worksheet. 23 Open Response Scores Scrolling Bar

24. © New Visions for Public Schools, 2010 Regents Grid: Raw Data Input With Less than 40 Students 24 The Regents Grid is defaulted at 40 rows for student responses. Should you have data for fewer than 40 students, follow these steps to input your data. 1.Enter your student data following the steps from the previous page. 2.Locate the rows that are blank. 3.Highlight these rows (note: highlight the entire rows by clicking on the row header on the left side of the Excel Sheet.) 4.Right click and select Delete. 5.The Distracter Analysis Section in the Regents Grid will update to reflect your data. Use the side row headers to highlight the rows empty of student data.

25. © New Visions for Public Schools, 2010 Regents Grid: Raw Data Input With More than 40 Students 25 If you have data for more than 40 students, follow these steps to add rows that will allow you to input more data. *Please note that you must have macros enabled to input data for more than 40 students. 1.Click on the Add Students button*. 2.Type in the number of students you want to add and click on OK. 3.Wait for the macro to add rows automatically. 4.Add student data as explained above. *Note: The Add Students button will only work if macros are enabled.

26. © New Visions for Public Schools, 2010 Regents Grid: Distracter Analysis Section The third part of the Regents Grid Sheet is the Distracter Analysis. It looks like this and is found at the bottom of the Light Green Regents Exam Tab 26 The Distracter Analysis Section provides a breakdown on how all of the students performed on each question of the Regents Exam. Each column displays information on an individual exam question. You will find information about: What percentage of the students answered each question correctly- % Correct How many students left each question blank- Stdts. Left Blank The Most Common Incorrect response- Most common Inct. A breakdown of how many students chose each possible multiple choice response as their answer- Stdts. Choosing 1, Stdts. Choosing 2, etc. Each column represents one exam question.

27. © New Visions for Public Schools, 2010 Summative Graph 27 The Summative Graph is the first of the Light Blue worksheet tabs. The Summative Graph allows you to easily visualize student performance data. * If you have enabled Macros and have selected up to three exams to analyze and compare in the Setup worksheet, you will see only one summative graph that compares student performance data across the exams. *Please note that if you do not have Macros enabled and/or have not selected exams to analyze and compare, you will see a number of graphs that may not necessarily cover all exams, since this part of the tool was only intended to be used with Macros. In the Summative Graph worksheet, you will see bar graphs representing average student scores in various categories, depending on the exam, such as Score by Unit, Score by Strand, or Score by Literary Element.

28. © New Visions for Public Schools, 2010 Summary by Unit 28 The Summary by Unit is the second of the Light Blue worksheet tabs. This worksheet gives you more detailed information about average student performance across exams for a number of exam variables. *If you have enabled Macros and have selected up to three exams to analyze and compare in the Setup page, you will see one Summary by Unit worksheet that compares student performance data across the exams that you selected. * Please note that if you do not have Macros enabled and/or have not selected exams to analyze and compare, you will see a number of Summary by Unit worksheets that may not necessarily cover all exams, since this part of the tool was only intended to be used with Macros. In the Summary by Unit worksheet, you will see tables representing average student scores in various categories, depending on the exam, such as Unit, Strand, or Literary Element. Please note that if you see a plus sign on the left side of the screen, you can click on it to see more rows.

29. © New Visions for Public Schools, 2010 Summary by Unit 29 The tables in the Summary by Unit worksheet show the number of questions and the percentage of correct student responses in each category, across the exams that you selected in the Setup worksheet. For the Percent Correct columns, red boxes indicate figures under 55%, yellow boxes indicate figures between 55% and 64%, and green boxes indicate figures above 65%. The Summary by Unit worksheet also allows you to compare the percentage change in average student scores for two exams in the Change column. Red boxes indicate negative change, while Green boxes indicated positive change, from one exam to the next. The exams that are compared in the Change column can be selected in the Set Preferences form on the right side of the worksheet (see next page). *Please note that you must have Macros enabled to use the Set Preferences form.

30. © New Visions for Public Schools, 2010 Summary by Unit: Setting Preferences 30 What is shown in the Summary by Unit tables can be altered by changing your “preferences” in the Set Preferences form on the right side of the Summary by Unit worksheet. You can change the following preferences: The default view for the Summary by Unit worksheet is “All Students”. To select information for only “one student”, 1. click on the box that says All Students, 2.click on the down arrow that appears to the right of the box, 3.click on the student you wish to select, scrolling down or up if necessary. 4.Then click on the Show Student button. The default sorting of information in the Summary by Unit is alphabetical. To choose a different method of sorting, such as By First Exam, By Second Exam, By Third Exam, or By Change, select the sorting option you prefer under the “Sort” heading. To select the exams to compare in the Change column, select an option under the “Show Change From” heading. To show or hide individual or the Change column, check or uncheck the appropriate boxes under the “Show or Hide Exams” heading.

31. © New Visions for Public Schools, 2010 Summary by Student 31 The Summary by Student is the third of the Light Blue worksheet tabs. The Summary by Student worksheet shows you how individual students performed on the exams that you selected in the Setup worksheet, in categories such as unit, strand, or text type. *Please note that if you do not have Macros enabled and/or have not selected exams to analyze and compare, you will see a number of Summary by Student worksheets that may not necessarily cover all exams, since this part of the tool was only intended to be used with Macros. Here, you will see your students’ most recent overall test scores. You will also see the maximum possible points students could earn in the category you selected, as well as the percentage of those points that each student actually earned. You must choose the category to view in the Set Preferences form on the right side of the worksheet (see next page).

32. © New Visions for Public Schools, 2010 Summary by Student 32 Like the Summary by Unit worksheet, the Summary by Student worksheet allows you to compare the percentage point change for two exams in the Change column. The exams that are compared in the Change column can be selected in the Set Preferences form on the right side of the worksheet (see below). Red boxes indicate negative change, Green boxes indicated positive change. You can select what data is shown in the Summary by Student table by changing your preferences in the Set Preferences form. Please note that you must have Macros enabled to use this form. You can change the following preferences: To select a method of sorting, such as Alphabetically, By First Exam, By Second Exam, By Third Exam, or by Change, click on the sorting option you prefer under the “Sort” heading. To select the exams to compare in the Change column, select an option under the “Show Change From” heading. To show or hide exams or the Change column, check or uncheck the appropriate boxes under the “Show or Hide Exams” heading.

33. © New Visions for Public Schools, 2010 Summaries by Student 33 The category and subcategory of student scores that will be shown in the table can also be changed. The categories vary depending on the exam, and include Unit, Strand, and Literary Element. To select a category and subcategory, follow these steps: 1.Click on the box under the “Choose a Category” heading. 2.Click on the down arrow that appears to the right of the box.

34. © New Visions for Public Schools, 2010 Summaries by Student 34 3.Click on a category, scrolling up or down if necessary. 4.Repeat steps 1-3 for the box under the “Choose a Subcategory” subheading. *Please note that you must select both a category and a related subcategory for the preferences to go into effect.

35. © New Visions for Public Schools, 2010 35 Conversion Chart The Conversion Chart is the first of the Yellow worksheet tabs. This chart will help you convert raw exam scores into final scores on the Regents Exam. It also shows you what scores are needed to be on track for graduation. *Please note that if you selected exams to analyze and compare in the Setup worksheet, you will see Conversion Charts for the selected exams side by side. Otherwise, there will be a separate Conversion Chart for each exam.

36. © New Visions for Public Schools, 2010 Components of the Conversion Chart 36 The Conversion Chart allows you to convert points earned from the multiple choice, short answer DBQ questions and essay questions into final exam scores. It connects these scores to the New Visions Student Readiness metric. The conversion chart has three major parts: 1.Rows indicating possible scores on multiple choice and DBQ short answer sections of the exam. 2.Columns indicating possible scores on the essay portion of the exam 3.Cells with possible student final exam scores based on a combination of the multiple choice, DBQ short answer and essay scores. 1 2 3

37. © New Visions for Public Schools, 2010 Using the Conversion Chart The conversion chart is used to determine what final score on the exam a student would receive based on the points received on the three parts of the exam. To use the chart: 1.Locate the row with the score a student received on the multiple choice and short answer DBQ section. These scores run down the first column of the chart. 2.Look across the top row of the chart and locate the column with the score that the student received on the essay portion of the exam. 3.Where the column and row meet in the middle of the chart indicate the final score the student received on the exam. A student who received 56 points on the multiple choice and short answer DQ sections, and 3 points on the essay section would receive a score of 80 on the entire exam. 37 Points on multiple choice and DBQ short answer sections. Final Score on Entire Exam Points on essay section.

38. © New Visions for Public Schools, 2010 Conversion Chart Scores and Student Readiness The conversion chart can also be used to determine if a student is on track for graduation. The colors used match the College Readiness Metric. Blue indicates that the student scored a 75 or above and is on track for College Readiness (note: This only applies to the Integrated Algebra exam.) Green indicates that the student has scored a 65 or above and is on track for Graduation with a Regents Diploma. Yellow indicates that the student has scored above 55 and is on track for Graduation with a Local Diploma (note: this only applies to students cohorts 2011 or earlier; all younger students must score a 65 or higher on all Regents). Red indicates that the student has scored below a 55 and must retake the exam. 38 Score of 55+ On track for graduation with Local Diploma Score below 55 Student must retake exam. Score of 65+ On track for graduation with Regents Diploma

39. © New Visions for Public Schools, 2010 39 The Answer Key The last part of the Item Analysis Tool (the second Yellow worksheet tab) is the Answer Key. It contains more detailed information about the content and skills of each exam question.

40. © New Visions for Public Schools, 2010 Components of the Answer Key 40 The Answer Key contains information about how exam questions are aligned to content standards as well as a breakdown of the skills needed to answer the questions. The answer keys vary somewhat across the different subjects of the exam. But overall, you will find question-specific information in the answer key about: The unit or strand each question pertains to The specific content or key idea of each question The skill or performance indicator each question employs Additional information about the question such as geographical area, text type, literary elements, and signal words The point value of each question The correct response for each question Student performance data, including the percent of students who answered the question correctly, the most common incorrect answer, and the number of students choosing each answer.

41. © New Visions for Public Schools, 2010 Components of the Answer Key 41 The answers and point values are in a table to the right of the main Answer Key, presented horizontally in the same format as the Regents Grid for ease of browsing. *Please note that if you have selected exams to analyze and compare in the Setup worksheet, you will see Answer Keys for all of the selected exams stacked vertically. Otherwise, there will be a separate Answer Key for each exam. Answers in horizontal format Multiple exams stacked vertically

42. © New Visions for Public Schools, 2010 Thank you! 42 This concludes the guide to using the New Visions Regents Item Analysis Tool. If you have any questions, please contact us at: datatools@newvisions.org