Teaching Effectiveness Past, Present, Future

Ken Keefner, R.Ph., PhD Associate Professor Vice Chair Department of Pharmacy Sciences School of Pharmacy and Health Professions Tracy Chapman, M.Ed. Executive Director for eLearning and Technology School of Pharmacy and Health Professions

Today’s Session… Overview of the challenges we faced and prompts to make changes Process employed to affect change Current status and future plans Our context

“ I never teach my pupils; I only attempt to provide the conditions in which they can learn.” Albert Einstein

Challenges and Prompts Course evaluations as sole data source for dossiers to document teaching Faculty consistently received the poorest ratings (P&T) No validity/reliability tied to tools used for course evaluations No consistency in tools/questions used Looking for a new tool for course evaluations Very little formative assessment Death by survey

Existing Structure Administration School Assessment Committee Pharmacy Ass’t Cmte PT Ass’t Cmte OT Ass’t Cmte Office for Faculty Development & Assessment OASA OLAT

Assessing Teaching Effectiveness Change in Institutional Culture Build Shared Trust Build Shared Motivation Build Shared Language Build Shared Guide- lines Transforming the Culture Course Evaluation Teaching Effectiveness Adapted from Angelo 1999

Process Stakeholder involvement External expertise Collaboration & Communication Tenacity Someone ‘leading the charge’ Worked at multiple levels

Process Administration School Assessment Committee Pharmacy PT OT Internal Consultants Peer Groups External Consultants Peer Coaches SGID Facilitators IDEA School of Medicine Office for Faculty Development & Assessment

“A change is something you do, and a fad is something people talk about.” Peter Drucker Managing in the Next Society

What did we end up with? Process/Plan for using multiple data points Multiple Tools for data collection Agreement on data ownership Beginning to think about student learning in a more robust way Cross disciplinary examination/discussion on teaching effectiveness Decreased ‘death by survey’

Tools Developed/Adapted Teaching Effectiveness Plan Teaching Effectiveness Table – what to use when and for what purposes Annual Course Report form Small Group Instructional Diagnosis (SGID) Peer Mentoring Policies/Procedures Individual Development & Educational Assessment (IDEA)

Individual Development Educational Assessment & New tool – opportunity for change Appropriate use of student ratings & student comments Shared language

Shared Language Course evaluations → Student ratings of instruction Teaching evaluation → Teaching effectiveness Faculty focus → Student (learning) focused

New Tool: An Opportunity for Change Student Learning Focus Diagnostic Component Scores Adjusted for Extraneous Influences What was instructor’s influence on learning? Documented Validity and Reliability National Comparative Data Group Summary Reports Program Review Accreditation

Appropriate Use of Student Ratings & Student Comments Student Ratings can have a positive impact if... The instrument : Is “learning focused” and provides a diagnostic The emphasis for “summative” faculty evaluation is appropriate: 30%-50% of the overall evaluation of teaching Use additional sources of evidence Results are not over-interpreted 3-5 performance categories Appropriate use of student comments Results are representative of instruction: 6 to 8 classes representing all or nearly all classes taught More if classes are small

Faculty expertise Appropriateness of goals, content, and organization of course Materials used in delivery How student work is evaluated, including grading practices Students Are Not Qualified to Assess

Assessing Teaching Effectiveness Change in Institutional Culture Build Shared Trust Build Shared Motivation Build Shared Language Build Shared Guide- lines Transforming the Culture Course Evaluation Teaching Effectiveness Adapted from Angelo 1999

References Selected References: Arreola R. Developing a Comprehensive Faculty Evaluation System. Bolton, MA; Anker Pub, Hutchings P. Ethics of Inquiry: Issues in the Scholarship of Teaching and Learning. Menlo Park, CA; The Carnegie Foundation for the Advancement of Teaching, McKeachie W. Teaching Tips. 11th Edition, Boston, MA; Houghton Mifflin Co, Penny A, Coe R. Effectiveness of consultation on student ratings feedback: a meta- analysis. Review of Educational Research, 74(2): , Seldin P. Changing Practices in Evaluating Teaching, Bolton, MA; Anker Pub, Shulman L. Teaching as Community Property: Essays on Higher Education, San Francisco, CA, Wergin J. Departments that Work: Building and Sustaining Cultures of Excellence in Academic Programs. Bolton, MA; Anker Pub, 2003.