“Strategies for Harnessing Information Technology to September 5, 2000 8th Improving Student Learning Symposium Improving Student Learning Strategically “Strategies for Harnessing Information Technology to Facilitate Institutional Assessment” Gloria M. Rogers, Ph.D. Institutional Research, Planning, and Assessment Rose-Hulman Institute of Technology Terre Haute, Indiana USA G. Rogers, Rose-Hulman Institute of Technology gloria.rogers@rose-hulman.edu

Overview Use of models to guide institutional strategies for improving student learning Assessing student learning Best practices for student assessment Brief history of RHIT process Assessment model/taxonomy A case study - demonstration Benefits to teaching/learning Assessment method truisms Barriers to faculty involvement Advice from the field

Use of Principles of Best Practice for Assessment of Student Learning in guiding development of assessment “system” Value of using models to guide practice Recognition of local constraints OUTCOMES INPUTS

Rose-Hulman Institute of Technology September 5, 2000 Rose-Hulman Institute of Technology Terre Haute, Indiana, USA 1600+ undergraduate students B.S. degrees in engineering, science, and mathematics Median SAT scores 1350 (700M,650V) 80%+ engineering students G. Rogers, Rose-Hulman Institute of Technology gloria.rogers@rose-hulman.edu

BRIEF History Presidential Commission of faculty, staff and students appointed in Spring of 1996 to develop a plan for the assessment of student outcomes Provide for continuous quality improvement Meet outcomes-based accreditation standards Regional (NCA) Program (ABET)

Rose-Hulman Institute of Technology Gloria Rogers - Rose-Hulman Institute of Technology Accreditation Requirements Institutional Mission Constituents Educational Goals & Objectives Measurable Performance Criteria Educational Practices/Strategies Feedback for Continuous Improvement Program Outcomes Accreditation Assessment: Collection, Analysis of Evidence Evaluation: Interpretation of Evidence Assessment for Continuous Improvement

X X X Taxonomy of Approaches to Assessment Placement “Gatekeeping” B E H A V I O R A T I U D E S & V L S K I L K N O W L E D G Competency-Based Instruction Assessment-Based Curriculum Individual Perf. Tests Placement Advanced Placement Tests Vocational Preference Tests Other Diagnostic Tests X X “Gatekeeping” Admissions Tests Rising Junior Exams Comprehensive Exams Certification Exams Individual Level of Assessment (Who?) X Program Enhancement Individual assessment results may be aggregated to serve program evaluation needs Campus and Program Evaluation Program Reviews Retention Studies Alumni Studies “Value-added” Studies Group Object of Assessment (What?) Learning/Teaching (Formative) Accountability (Summative) Purpose of Assessment (Why?) (Terenzini, JHE Nov/Dec 1989)

Rose-Hulman’s Mission To provide students with the world’s best undergraduate education in engineering, science, and mathematics in an environment of individual attention and support.

Input Recruit highly qualified students, faculty, and staff Provide an excellent learning environment Quality Encourage the realization and recognition of the full potential of all campus community members Climate Instill in our graduates skills appropriate to their professions and life-long learning Outcomes Resources Provide resource management & development that supports the academic mission

Instill in our graduates skills appropriate to their professions and life-long learning Outcomes Ethics and professional responsibility Understanding of contemporary issues Role of professionals in the global society and ability to understand diverse cultural and humanistic traditions Teamwork Communication skills Skills and knowledge necessary for mathematical, scientific, and engineering practice Interpret graphical, numerical, and textual data Design and conduct experiments Design a product or process to satisfy a client's needs subject to constraints

Why portfolios? Authentic assessment Capture a wide variety of student work Involve students in their own assessment Professional development for faculty

Why “electronic” portfolios? Student-owned laptop computer program since 1995 Classrooms, residence halls, common areas, library, fraternity houses all wired Access Efficient Cost effective Asynchronous assessment

RosE-Portfolio Structure Advisor ADMIN Student Rater Employer User Management Group Management System Configuration Criteria Tree Activity Managment Faculty Submit Review Search Dynamic Resume Access Control View Advisee’s portfolio Search Advisee’s portfolio Inter-rater Reliability Rating sessions Feedback Rating management Curriculum Map PTR Portfolio Submit Review Search View Search

Show Me!

Assessment of student material Faculty work in teams Each team assesses one learning objective Score holistically Emerging rubrics Does the reflective statement indicate an understanding of the criterion? Does the reflective statement demonstrate or argue for the relevance of the submitted material to the criterion? Does the submitted material meet the requirements of the criterion at a level appropriate to a graduating senior at R-HIT?

Show Me!

Example of Results Understand criterion? Submission relevant to criterion? Meet standards for R-HIT graduate?

Example of Results Does submission meet the standards for a graduate of R-HIT? Appropriate for audience Organization Content factually correct Test audience response Grammatically correct

Linking results to Practice Development of Curriculum Map Linking curriculum content/pedagogy to knowledge, practice and demonstration of learning outcomes Show Me!

Curriculum Map Results Fall 1999-2000 (181 courses/labs) Communication Skills

Curriculum Map Results Fall 1999-2000 (181 courses/labs) Ethics

Institute acts on the recom-mendations of the Eval. Comm. Closing the loop JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC WINTER SPRING SUMMER FALL Eval Committee receives and evaluates all data; makes report and refers recom-mendations to appropriate areas. Institute acts on the recom-mendations of the Eval. Comm. Reports of actions taken by the Institute and the targeted areas are returned to the Eval Comm. for iterative evaluation. Institute assessment cmte. prepares reports for submission to Dept. Heads of the collected data (e.g. surveys, e-portfolio ratings).

Primary focus It is not about electronic portfolios. It is about: supporting teaching and learning faculty and student development the transformation of the teaching/learning environment

Benefits to teaching Faculty are asked to reflect on learning outcomes in relation to practice Consider the value of stated outcomes Right ones? Right performance criteria? Individual faculty role in creating the context for learning Develop a common language and understanding of program/institutional outcomes Explicit accountability Promotes interdisciplinary discussions/collaborations

Benefits to learning Students review their own progress as it relates to expected learning. Portfolios provide a way for students to make learning visible and becomes the basis for conversations and other interactions among students and faculty. Learning is viewed as an integrated activity not isolated courses. Students learn to value the contributions of out-of-class experiences. Student reflections are metacognitive as they appraise their own ways of knowing. Promotes a sense of personal ownership over one’s accomplishments.

Assessment method truisms There will always be more than one way to measure any outcome No single method is good for measuring a wide variety of different student abilities Consistently inverse relationship between the quality of measurement methods and their expediency Importance of pilot testing to see if method is good for your program (students & faculty)

Barriers to implementation Faculty current workload lack of incentive to participate in the process (rewards) “what’s in it for me” (cost/benefits) Institutional/program leadership Lack of vision for the program/institutional assessment process (no existing, efficient models) Cost/benefit unknown Difficulty of restructuring the reward system to facilitate faculty participation

Portfolio deficiencies Process deficiencies Lack of understanding of the dynamics of organizational change Absence of “tools” to facilitate collaborative work Portfolio deficiencies Ill-defined purpose Lack of efficient ways to manage the portfolio process Systematic review of portfolio contents is ill-defined or non-existent Student and faculty roles not clear Portfolio process not integrated into the teaching/learning environment Resource deficiencies Expertise in portfolio development Development of “authentic” portfolio

Advice from the field E=MC2 You cannot do it all - prioritize All assessment questions are not equal One size does not fit all It’s okay to ask directions Take advantage of local resources Don’t wait until you have a “perfect” plan Decouple from faculty evaluation

DEMO Site http://www.rose-hulman.edu/ira/reps/