We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byDylan Kelly
Modified about 1 year ago
© 2012 Boise State University Marcia Belcheir, Ph.D. Robert Anson, Ph.D. James A. Goodman, Ph.D. Boise State University On-line Course Evaluation Implementation and Improvement of Response Rates
© 2012 Boise State University Goals of the Session Offer suggestions for implementing on-line course evaluations Report research results of how faculty can improve response rates in their courses
© 2012 Boise State University A short history on implementing on-line course evaluations at Boise State Diffused ownership – collaboration of stakeholders from all parts of campus examined viability of the move Most departments wanted to participate; some skeptical, needed to see it work on campus No university-wide question set; instead, from college, department, and a variety of academic support programs Purchased CollegeNet, “WDYT” product
© 2012 Boise State University Academic-Oriented Team Led by Institutional Research Teamed with 2 faculty—one in Center for Teaching and Learning, one teaches IT Minimized OIT involvement (only prepare data source and set up user authentication) Implementation required: –No new FTE resources (“other duties as assigned”) –Need “sensitive touch” for faculty buy-in and participation To Opt-In To encourage student involvement To trust data for annual performance evaluations
© 2012 Boise State University Coordinated many hands (and no FTE body) to run on- line evaluations –Developed upload files and checked for accuracy –Handled help questions –Prepared survey calendar each semester Recommendations –Reduce # of sessions = more sessions = more cycles to support –Don’t assume course registration data is accurate –Communication is key Implementing On-Line Evaluations in a Streamlined Environment
© 2012 Boise State University Opt-In for colleges, departments (down to course level) created challenges to selectively load course enrollment data Also opt-in for trans-departmental academic programs-- “matrix university” Solution –Developed MS Access Database to check, filter and restructure data flexibly and systematically Recommendation –Overcomes potential faculty and department resistance to how teaching is evaluated –(Optional) University-wide question set should be universally adopted early in the process Implementing On-Line Evaluations in a Voluntary Environment
© 2012 Boise State University Implementing On-Line Evaluations in a Decentralized Environment Questions controlled by colleges, departments, and academic support programs; no required university question-set Solution –Vendor (CollegeNet, WDYT) selected for flexibility to create and bind specific question sets to type of course –Centralized question review, assistance, creation in Center for Teaching and Learning Recommendations –Decentralized requires centralized editing & question support –Helps to incorporate academic support programs (Honors? On-line programs? SL?) in on-line evaluations to reduce extra survey demands
© 2012 Boise State University Part 2: Improving response rates The key to good student response rates for on-line evaluations often lie with the faculty ➢ What does the literature say? ➢ What do our faculty do? ➢ How well do different tactics work to raise response rates? ➢ Are there other factors?
© 2012 Boise State University Background and literature review Response rates for online evaluations tend to be lower than of pen/paper evaluations (~70-80%/~50- 60%) (Avery et al, 2006; Dommeyer 2002; many others) Likert-type quantitative ratings remain constant even with lower response rates (Anderson et al, 2005; Dommeyer, 2004; many others)
© 2012 Boise State University Background and literature review Qualitative responses tend to increase (Handwerk, 2000; Heath, et al, 2007) Institutions and instructors that do something tend increase response rates (Nulty, 2008) The more tactics an institution and/or instructor uses, the higher the response rate (Nulty, 2008)
© 2012 Boise State University Response Rates and Adoption of On-line Evaluations * Estimate based on literature
© 2012 Boise State University ➢ Online Evaluations –76,172 enrollments (88% of total were online evaluations) 1,134 instructors 3,213 classes ➢ Response Rates –Overall: 54.9% –Colleges High: 73% in Engineering Low: 52% in Arts & Sciences –Departments High: 82% in Construction Mgmt Low: 31% in Respiratory Care –Classes: 0% to 100% Fall 2012
© 2012 Boise State University University Measures/Tactics No iPad lotteries, web advertising, or grade hooks Sent 5 student s –Start –Reminders (3) –End Sent 5 instructor s –Pre-eval (enter custom questions) –Start –Reminder –End –Results available
© 2012 Boise State University 1. Studied Fall 2012 evaluation –Used individual response rate data –Received anonymous data set from vendor 2. Surveyed all 1,134 instructors – 678 (60%) instructors responded about tactics used in 1 or 2 classes taught that Fall –1,128 (35%) classes studied –Survey asked Tactics used to increase student response rates or otherwise obtain student feedback Basic demographic questions Research Methods: Data
© 2012 Boise State University Applied series of t-tests and ANOVAS on individual variables for statistical differences Applied regression for modeling the best set of factors for predicting response rates Research Methods: Analysis
© 2012 Boise State University Research Questions Addressed What tactics do instructors use to improve their response rates? Does it help to use more tactics? What is the impact of course and instructor characteristics? What is the best model for explaining variation in response rates? Are there interactions between tactics used and course or faculty characteristics?
© 2012 Boise State University Which Tactics Were Instructors Most Likely to Use? Reminded students during class (61%) Explained to class how I use the results to improve teaching (57%) Sent personal s to students as reminders (33%) Posted a reminder or assignment on Blackboard (32%) Provided incentives to complete the evaluation (22%)
© 2012 Boise State University What instructor tactics improved response rates? Tactics Used tactic Response rate Didn’t use Response rate Provided incentives 79%57% Provided time in class to complete 70%61% Sent personal s as reminders 66%60% Reminded students during class time 65%57% Explained to class how I use the results 65%57% Posted reminder/assignment on BlackBoard 64%61% I did nothing 50%63%
© 2012 Boise State University Incentives: Does the kind of incentive and approach affect response rates? Basis for Awarding Incentive Type of Incen- tive Individual% of classTotal Points78% (n=62) 79% (n=156) 79% (n=218) Other incentive77% (n=15) 82% (n=18) 80% (n=33) Total78% (n=77) 79% (n=174) 79% (n=251)
© 2012 Boise State University Point-based class-wide rewards –If 80% complete the eval, everyone gets 1 point added to final grade –If 90% complete eval, all students gain back all participation points Non-Point-based class-wide rewards –If 70% complete eval, instructor will bring cupcakes to final –If 90% complete eval, everyone can bring a 3 X 5 notecard to the final Class-based Incentive Examples
© 2012 Boise State University Impact of threshold on response rate Incentive thresholdN who used that threshold Average response rate 100%889% 90%+1087% 80%+2286% Sliding scale873% 70%+1268% 60%+362% Unknown974%
© 2012 Boise State University Did using more tactics improve response rates?
© 2012 Boise State University Course and instructor characteristics: Response Rate N Small courses (<20) vs larger (>40) 64% vs 59% 477 vs 154 In-person courses vs On-Line 63% vs 50% 955 vs 119 Graduate courses vs Undergraduate 65% vs 61% 153 vs 974 Tenure/tenure-track faculty vs Adjunct 64% vs 57% 748 vs 379 Higher response rates were associated with:
© 2012 Boise State University Final model (R 2 =.32): VariableStandardized estimate Provided incentives0.41 Number of tactics used0.43 Took class time to complete evaluation0.07 Small class0.09 Taught on-line-0.09 Undergraduate course0.04 Taught by adjunct-0.12 Interaction: Number of tactics & undergraduate course level -0.32
© 2012 Boise State University Response rates based on number of tactics and course level
© 2012 Boise State University Conclusions Incentives provide the biggest boost –If you use incentives, set the threshold for the class (not individual) and set it at 80% or more –If using incentives, best to set high threshold (80% or greater) for class as a whole Use multiple tactics On-line and large section classes have a built in disadvantage—use even more tactics
© 2012 Boise State University Future Changes Institutionalize small core set of university- wide questions to enable comparisons Pushing grade hook proposal Keep expanding use (now 91.5%) Explore means to impact response rates for on-line courses
© 2012 Boise State University Questions? Marcia Belcheir - Robert Anson – James A. Goodman –
The Redesigned Elements of Statistics Course University of West Florida March 2008.
MAP the Way to Success in Math: A Hybridization of Tutoring and SI Support Evin Deschamps Northern Arizona University Student Learning Centers.
End of Course Evaluation Taimi Olsen, Ph.D., Director, Tennessee Teaching and Learning Center Jennifer Ann Morrow, Ph.D., Associate Professor of Evaluation,
Quality Online Preparation: Qualities of Faculty, Courses, and Preparation Programs By Dr. Erin O’Brien Valencia College League of Innovation Conference,
The Use of Formative Evaluations in the Online Course Setting JENNIFER PETERSON, MS, RHIA, CTR DEPARTMENT OF HEALTH SCIENCES.
University of Florida Academic Activities Reporting (Formerly IWF)
Institutional Course Evaluation Solution Faculty Senate Executive Committee September 12, 2012 Carol VanZile-Tamsen, Ph.D.; Associate Director, Office.
University-Wide Course Evaluation Committee Peter Biehl, Chair, Department of Anthropology Krissy Costanzo, Committee Staff Support; Academic Affairs March.
Online Course Evaluations Is there a perfect time? Presenters: Cassandra Jones, Ph.D., Director of Assessment Michael Anuszkiewicz, Research Associate.
Maximizing Learning Using Online Assessment 2011 SLATE Conference October 14, /12/ P. Boyles, Assistant Professor, Chicago State University,
Shimon Sarraf Center for Postsecondary Research, Indiana University Bloomington Using NSSE to Answer Assessment Questions Regional User’s Workshop October.
NewTextbook Tools M-Pathways and Ctools. Background Fall 2006: Student concerns about rising textbook costs results in Provost’s Textbook Task Force.
Teaching Thermodynamics with Collaborative Learning Larry Caretto Mechanical Engineering Department June 9, 2006.
Results of the Faculty Survey on Internationalization at Villanova: A Preliminary Report Prepared for the International Leadership Committee Prepared by.
Advising to Make the Grade FYE Assignments that Benefit Students and Advisers Veronica Giguere Florida Institute of Technology.
Get Your Feedback Into GEAR League of Innovations, 2013 Rio Salado College Patricia Case, Ph.D Tristan Marble, MA Corey Pruitt, MS.
Bieber et al., NJIT © Internship/Ph.D. Opportunity PLA: Participatory Learning Approach Michael Bieber Jia Shen, Dezhi Wu, Roxanne Hiltz Information.
Taeho Yu, Ph.D. Ana R. Abad-Jorge, Ed.D., M.S., RDN Kevin Lucey, M.M. Examining the Relationships Between Level of Students’ Perceived Presence and Academic.
St. Louis Public Schools Human Resources Support for District Improvement Initiatives (Note: The bullets beneath each initiative indicate actions taken.
ONLINE VS. FACE-TO-FACE: EDUCATOR OPINIONS ON PROFESSIONAL DEVELOPMENT DELIVERY METHODS BY TERESA SCRUGGS THOMAS Tamar AvineriEMS 792x.
Dual Credit and Advanced Placement: Do They Help Prepare Students for Success in College? Mardy Eimers, Director of Institutional Research & Planning Robert.
Early Identification of Introductory Major's Biology Students for Inclusion in an Academic Support Program BETHANY V. BOWLING and E. DAVID THOMPSON Department.
Not Just for Students: An Archives Workshop for Faculty Rachel Grove Rohrbaugh Chatham University Archives | Pittsburgh, Pa. | (412)
Data Management for Large STEP Projects Michigan State University & Lansing Community College NSF STEP PI Meeting March 15, 2012 Workshop Session I-08.
Flexible Scheduling Mrs. Smith's class leaves Mr. Brown's class enters Improving Student Learning Through Expanded Use of the Library Media Center Library.
ASDP Infusing Institute Project Development of a Foundational World Languages and Cultures course To encourage students to study non-western languages.
Developing An Online Information Literacy Course Nancy O’Hanlon Ohio State University Libraries Wuhan University, China March 2007.
Faculty Instructional Support Moving beyond putting courses online to curriculum redesign Leila Lyons, University of Delaware.
Using Blended Learning to Increase Student Engagement Fiona Fui-Hoon Nah, Ph.D. Professor of Business & Information Technology (BIT) March 14, 2014.
Implementing Active Learning Strategies in a Large Class Setting Travis White, Pharm.D., Assistant Professor Kristy Lucas, Pharm.D., Professor Pharmacy.
Student Learning Outcomes Assessment Montgomery College Fall 2009 Orientation Workshop.
LibQUAL+™ Process Management: Using the Web as a Management Tool Amy Hoseth Massachusetts LSTA Orientation Meeting Boston, MA October 21, 2005 old.libqual.org.
Improving Campus IT Accessibility Dr. Jonathan Lazar Dept. of Computer and Information Sciences Towson University Copyright Jonathan Lazar This work.
ASEE Profiles and Salary Surveys: An Overview Department of Assessment, Evaluation, and Institutional Research Brian L. Yoder, Ph.D., Director Amlan Banerjee,
Student Development The Gateway to … Opportunities Accomplishment Financial Solutions Student Success Achievement Student Life.
Increasing Online Survey Response Rates April 7, 2015 Institute for Faculty Development.
Defining Success Understanding Statistical Vocabulary.
Old.libqual.org What will ARL do for you? What will you do? January 2005 Shrivenham, UK.
Guidance Expo 2010 Developing, Organizing and Maintaining High School Advisory Programs Workshop Session V 1:15 – 2:10 p.m.
The Integration of Embedded Librarians at Tuskegee University Juanita M. Roberts Director Library Services Ford Motor Company Library/Learning Resources.
Redesign of PSYC 1101 into a 50% Online (Hybrid) Course Sue Spaulding, UNC Charlotte Pearson Education March 9, 2012 Boston Office.
SLIDE 1 Innovations Conference March SLIDE 2 The National Lone Star Report Aligning Technology with Student Success.
June 7, After the year and review, the FOE committee made multiple recommendations for work. These activities have begun and we intend.
Blogging the First Year… Using blogs in Elmhurst Colleges First Year Seminar Program Peg Cook, Assistant Reference Librarian A.C. Buehler Library, Elmhurst.
Practices and Predictors of the Use of Accommodations by University Faculty to Support College Students with Disabilities Leena Jo Landmark, M.Ed., and.
Making Progress with Academic Progress Reporting A Customized Tool for Faculty and Advisors.
Technology Support on a University Campus Contingency Theory and Collaboration.
Bassam Hammo (Ph.D) Department of Computer Information Systems King Abdullah II School for Information Technology Jordan University Amman Jordan e- mail:
Assessment of Course-Level Learning Outcomes in Psychology.
© 2017 SlidePlayer.com Inc. All rights reserved.