Presentation on theme: "S IZING U P S TANDARDIZED A SSESSMENTS : B REAKING D OWN THE ETS P ROFICIENCY P ROFILE, M AJOR F IELD T ESTS, AND THE R OO W RITER By Nathan Lindsay and."— Presentation transcript:
S IZING U P S TANDARDIZED A SSESSMENTS : B REAKING D OWN THE ETS P ROFICIENCY P ROFILE, M AJOR F IELD T ESTS, AND THE R OO W RITER By Nathan Lindsay and Dan Stroud January 30, 2013
A SSESSMENT AT UMKC Consists of a multifaceted approach: Formative and Summative Assessments Direct and Indirect Assessments Direct assessments include embedded assessments (in the classroom), and standardized assessments
UMKC S TANDARDIZED T ESTS O VERVIEW For many years, undergraduate students at UMKC have participated in the following standardized tests: The ETS-Proficiency Profile test that measures General Education learning outcomes Major Field Tests (MFTs) that measure learning outcomes for students in their disciplines. The WEPT (soon to be the RooWriter) that measures students writing ability For the ETS tests, only this past year did we purchase a reporting license that allows us to closely analyze the data
Q UESTION What are some of the pros and cons of standardized tests?
W HY S TANDARDIZED T ESTS M IGHT BE USEFUL A LONG SIDE IN - HOUSE A SSESSMENTS o Standardized tests allow comparisons to both national and peer data (which are requested by external constituents). o There are benefits in the use of longitudinal data. o ETS test design includes strong: 1) Validity 2) Reliability 3) Fairness/Equity 4) Representation
H OW ETS D EVELOPS T EST Q UESTIONS Below is the process by which tests and test questions are developed for future ETS Proficiency Profile tests and MFTs: http://www.ets.org/s/understanding_testing/flash/h ow_ets_creates_test_questions.html
T EST S CORING Two Types of Test Scores Norm-referenced (scaled) scores - These scores pull their meaning from comparing the scores between and individual student or group of students with those of either the same individual or group or another at different points of time. The majority of test results are of this nature. Criterion-referenced (proficiency classified) scores - These pertain to the level of proficiency students obtained within a particular skill set. This type of score stands on its own – with no need to compare to another group, though they can still use specific comparisons as well.
ETS – P ROFICIENCY P ROFILE (I N 2011-2012, N = 1448 STUDENTS ) Test given to all students after they have completed 70 credit hours. Completion is a requirement of graduation. Scores were above benchmark peers for all four areas: Reading Critical Thinking Writing Mathematics Scores on individual items exceeded the national average 51 of 53 times. Higher scores may be due to the fact that most UMKC students take the ETS-PP right before they graduate (which is not always true at other institutions)
M AJOR F IELD T ESTS (MFT S ) UMKC students take the MFTs in the following areas: Biology Business Psychology Political Science Chemistry Computer Science Mathematics Physics 20 students have to take the test to have sufficient comparative data, so results are only available for some of these disciplines.
T WO R EPORTS AVAILABLE FOR M AJOR F IELD T ESTS The Custom Comparative Data Report indicates how UMKCs students' skills and knowledge compare with the skills and knowledge of students at similar institutions. In the Item Information Report, the items are prioritized first by content, and subsequently by sub-content to make comparisons easier for grouping. Scores compared in this report are referenced to the overall national percentage for each item.
M AJOR F IELD T ESTS R ESULTS AT UMKC Biology (N = 87 Students) – UMKC scored above average on 8 of 9 Assessment Indicators. Business (N = 192 Students) – UMKC scored above average on 4 of 9 Assessment Indicators. Psychology (N = 81 Students) – UMKC scored above average on 2 of 6 Assessment Indicators. Political Science (N = 20 Students) – UMKC scored above average on 1 of 3 Assessment Indicators.
D ISSEMINATING THE ETS R ESULTS The General Education scores have been reviewed by the General Education Committee, the University Assessment Committee, and the Assessment Academy Team. These MFT reports have been sent to all relevant departments, which have been encouraged to discuss how they can enhance their curriculum to address the areas of weakness within each test. A summary of our standardized tests will be shared with the Higher Learning Commission. We will collect data during the 2012-2013 academic year to determine any changes over time.
D EPARTMENTAL D IALOGUE : C ONVERTING THE DATA TO A CTION Use the data to stimulate conversation in the department about the curriculum and pedagogy surrounding these topics. Address the following questions: 1. How do the content areas of the test relate to our degrees learning outcomes? Should test items be added? 2. Are there any areas of the curriculum that could/should be emphasized more strongly? [curriculum mapping] 3. Do the results have face validity? How do the results compare to our findings from course-based assessments and our program/degree assessments? 4. How did different student groups/demographics score on the test? Should more sub-groups for analysis be created? 5. What are other ways that we could address/assess the areas that were lower? Should we develop our own test? 6. What trends do the scores show longitudinally? 7. How could we increase student motivation for the test?
H OW TO CUSTOMIZE MFT R EPORTING G ROUPS Cohorts - Determine which students to put in what groups as you design report emphasis Subgroups - You can create up to 16 subgroups - For each subgroup, you can design two questions with up to 8 possible responses. There must be a minimum of 5 students in each group, cohort, or subgroup. Minimums for reporting cohorts or subgroups must add up together to 20 as a minimum to report.
D ISCUSSION Are there other ways that your unit/department could develop test questions that would evaluate the learning outcomes for your degree(s)? Questions to consider: o Whats the point/purpose of the test? o What are the intended uses of this test? o What aspects of each unit or department do we wish to measure? o What scores will be reported? o Is the test in question statistically reliable, valid, and fair?
S TUDENT M OTIVATION FOR S TANDARDIZED A SSESSMENTS Key Issue - High stakes for institutions - Low stakes for students Incentivizing is two tiered - Get students to take the test - Get students to do their best Best PracticeConnect to Academic Outcomes - Participation grade for a class - Students who attain a score of _______ or higher receives _______?
D ISCUSSION What incentives could be used to help students to take standardized testing more seriously?
O THER W AYS TO E NCOURAGE M EANINGFUL P ARTICIPATION Inform students well in advance Include the MFT in the syllabus Send letter or email from Dean/ Department Chair Develop faculty understanding of the purposes of the test Create interdepartmental competition Develop an Assessment Day (Long range goal)
WEPT This is a Pass/Fail test, and approximately 5% of students have failed the test the last few semesters Each test is rated on 4 areas: Point/Thesis/Idea – Evaluated on the clarity of purpose throughout the essay. Development/Support/Evidence – Evaluated on the relevance and convincingness of the proof offered in support of the thesis. Organization/Logic/Coherence – Evaluated on the ease of understanding and logical arrangement of the essay. Usage/Mechanics/Documentation – Evaluated on the ease of reading and grammatical and style correctness The test is graded on a four-point scale The average scores for area one (i.e., Point/Thesis/Idea) were the highest at 3.10, up from the 3.07 in the previous semester. The remaining three scores average fell near 2.82 (2.82, 2.83, and 2.80, respectively).
T HE R OO W RITER To replace the WEPT for incoming freshmen in Fall 2013 An online diagnostic/formative test with many benefits for students, faculty, and the institution (see attached handout) The University Writing and Reading Board is currently working on finalizing all of the logistics for the test Please refer to the RooWriter website for an overview: http://dev.umkc.edu/roowriter/index.cfm
Q UESTIONS Are there other standardized tests that UMKC should be considering? What other questions about standardized tests do you still have?