Presentation on theme: "Assessing Quality and Sharing Best Practices for the Undergraduate Senior Thesis Collaboration Across Three Liberal Arts Colleges Student Success and the."— Presentation transcript:
Assessing Quality and Sharing Best Practices for the Undergraduate Senior Thesis Collaboration Across Three Liberal Arts Colleges Student Success and the Quality Agenda, AAC&U 2013
Teagle Senior Thesis Evaluation Capstone as most significant student artifact No systematic evaluation Developed Senior Thesis Rubric Performed cross-institution thesis evaluation Administered Thesis Experiences Survey Will perform second thesis evaluation on individual campuses using rubric Collaborators New College of Florida Bard College Bennington College Colorado College Hampshire College Smith College Wellesley College
Key Questions for Thesis Evaluation Are there common evaluative criteria across analytic disciplines? Can a common rubric be developed? Can a rubric improve on the validity, reliability and systematicity of grades for evaluating analytic work? Can using a rubric help institutions improve pedagogy? To what extent do student input characteristics predict high-quality senior theses? Can we identify student experiences that predict high- quality theses? Do the same experiences lead to high-quality work no matter whether theses are required, optional, or honors-only?
Timeline at Bard College June 2009: first reading/evaluation of senior theses using the rubric. Three Bard faculty participated. June 2010: full consortium meeting at Bard. Faculty discussed results from the June 2009 reading and planned for improvement. February 2012: meeting at Bard for Bard faculty to further disseminate the work of the grant. Led by Steve Weisler and Carol Trosset in conjunction with Bard Center for Faculty and Curricular Development. Steve and Carol also held a focus group with students. April 2012: meeting at Bard with faculty in the performing and practicing arts from Bard, Bennington, and New College. Explored possibilities for rubrics in the arts. May 2012: first survey of graduating seniors at Bard about their Senior Project experiences. The response from the class of 2012 was encouraging. ~160 students responded.
Results at Bard College The results from the June 2009 meeting showed that Bard senior theses were not theses that stand alone well with respect to the rubric, in comparison with the other colleges in consortium. The June 2010 meeting produced enough momentum to focus the discussion of the Senior Project on campus for first time in many years. Indeed, the discussion has shifted away from anecdotal stories about individual students to a discussion focused on talking about how we prepare and evaluate all students. A number of programs have instituted practices that help students prepare for the Senior Project and provide a scaffolding for students as they progress through the Senior Project. In addition, several programs are experimenting with rubrics to advise and evaluate students. Take home lesson for Bard: make the Senior Project more like a course.
New College of Florida Senior Thesis Practices Public honors college, narratives instead of grades, contracts not credit hours Senior thesis/project required of each student. 6 th semester Thesis Prospectus form Thesis advisor and 2 additional faculty serve as Baccalaureate Committee for oral exam. Thesis is a major focus of final semester.
Results:NCF Assessment and Priorities for Discussion 2009 Thesis Assessment Results In 2009 Thesis Assessment, New College compared favorably with others New College theses weakest areas Scholarly Context and Position 2012 Faculty and Student Priorities for Discussion Students & Faculty: Preparing students for thesis success before thesis year Faculty: Managing thesis workload Students: regular meetings with advisor, time to revise drafts, support for interdisciplinary and non- standard projects
Evolving NCF Thesis Discussions Initial skepticism/resistance to rubric assessments Later, rubric seen as possible tool: set expectations, guide theses, replace SAPA Positive and energizing conversations with Bard and Hampshire Faculty Hampshires workload depiction and assigning advisors Bards midway and moderation practices Reinvest in projects as well as theses
NCF Enduring Insights Learning process can be highly successful even when the senior thesis itself is disappointing. No matter how well we prepare students, there will always be a distribution from low to high quality. There is tension in expecting students to be flexible, independent, risk-takers and also meet honors standards for senior thesis.
Thesis Rubric Introduction: Rational
Most theses get As. GradeNumber of thesesPercent of theses A18566% A-5420% B+2810% B72% B-21% C+10.3% C1 D1
Using a Rubric Diversifies and Focuses the Evaluation Process Level 1Level 2Level 3Level 4Mean Rationale 8%36%41%13%2.6 Complexity 13%37%38%12%2.5 Approach 7%46%29%8%2.3 Context 18%43%33%6%2.3 Position 14%51%25%10%2.3 Argument 13%44%37%6%2.4 Evidence 6%41%42%10%2.6 Insight 11%47%33%8%2.4 Usage 1%10%57%31%3.2 Organization 6%37%45%11%2.6 Style 6%37%47%10%2.6
Scores Can Show Institutions How to Improve ABCDEFG Rationale Complexity Approach Context Position Argument Evidence Insight Usage Organization Style Grade
Grades and Self-Rated Quality
External readers are more discriminating.
Student judgments correlate with external readers more than with grades.
Do inputs predict quality?
Are theses better at more selective institutions?
Different experiences are more common at different institutions. Smallest Value Largest Value Average # courses taken while writing thesis Worked on thesis 6 or 7 days in a typical week10%32% Worked on thesis 1 to 3 days in a typical week22%51% Worked on thesis more than 6 hours per day13%34% Worked on thesis less than 4 hours per day26%54% Met with adviser weekly52%100% Topic was closely related to advisers research7%53% Student usually follows advisers advice19%48% Thesis topic was settled during junior year24%63% Student plans to seek an external audience19%50%
What makes for a good thesis? Predicted Grade Student felt prepared to find and evaluate relevant scholarship Student felt prepared to fairly consider conflicting points of view Predicted Self-Rated Quality Student felt prepared to form their own position or argument on a subject Student felt prepared to manage their own time Hours per day spent on the thesis Student usually took the advisers advice
The Best Predictors of Quality Only two survey items predicted* both the adviser-assigned grade and self-rated quality: –Student views the thesis as the most important work done in college –Student feels challenged by the adviser to do his/her best work *in a stepwise regression equation, for more than one institution
Next Steps Survey will be administered again this spring. Each institution will conduct a thesis-reading applying the rubric to theses written by students who completed the survey. A new analysis will discover whether particular experiences predict higher rubric scores for the same theses.