Presentation on theme: "1 Gloria M. Rogers, Ph.D. Institutional Research, Planning, and Assessment Rose-Hulman Institute of Technology Terre Haute, Indiana USA 8th Improving."— Presentation transcript:
1 Gloria M. Rogers, Ph.D. Institutional Research, Planning, and Assessment Rose-Hulman Institute of Technology Terre Haute, Indiana USA 8th Improving Student Learning Symposium Improving Student Learning Strategically Strategies for Harnessing Information Technology to Facilitate Institutional Assessment
2 Overview 4 Use of models to guide institutional strategies for improving student learning 4 Assessing student learning 4 Best practices for student assessment 4 Brief history of RHIT process 4 Assessment model/taxonomy 4 A case study - demonstration 4 Benefits to teaching/learning 4 Assessment method truisms 4 Barriers to faculty involvement 4 Advice from the field
3 OUTCOMES Use of Principles of Best Practice for Assessment of Student Learning in guiding development of assessment system 4 Value of using models to guide practice 4 Recognition of local constraints INPUTS
4 Rose-Hulman Institute of Technology 4 Terre Haute, Indiana, USA undergraduate students 4 B.S. degrees in engineering, science, and mathematics 4 Median SAT scores 1350 (700M,650V) 4 80%+ engineering students
5 BRIEF History 4 Presidential Commission of faculty, staff and students appointed in Spring of 1996 to develop a plan for the assessment of student outcomes 4 Provide for continuous quality improvement 4 Meet outcomes-based accreditation standards Regional (NCA) Program (ABET)
Assessment for Continuous Improvement Educational Goals & Objectives Constituents Program Outcomes Assessment: Assessment: Collection, Analysis of Evidence Evaluation: Evaluation: Interpretation of Evidence Feedback for Continuous Improvement Gloria Rogers - Rose-Hulman Institute of TechnologyAccreditation Measurable Performance Criteria Educational Practices/Strategies Accreditation Requirements Institutional Mission
7 Competency-Based Instruction Assessment-Based Curriculum Individual Perf. Tests Placement Advanced Placement Tests Vocational Preference Tests Other Diagnostic Tests Gatekeeping Admissions Tests Rising Junior Exams Comprehensive Exams Certification Exams Campus and Program Evaluation Program Reviews Retention Studies Alumni Studies Value-added Studies Program Enhancement Individual assessment results may be aggregated to serve program evaluation needs Level of Assessment (Who?) Individual Group KNOWLEDGEKNOWLEDGE SKILLSSKILLS ATTITUDES&VALUESATTITUDES&VALUES BEHAVIORBEHAVIOR Object of Assessment (What?) Learning/Teaching (Formative) Accountability (Summative) Purpose of Assessment (Why?) (Terenzini, JHE Nov/Dec 1989) Taxonomy of Approaches to Assessment X X X
8 Rose-Hulmans Mission Rose-Hulmans Mission To provide students with the worlds best undergraduate education in engineering, science, and mathematics in an environment of individual attention and support.
9 Input Recruit highly qualified students, faculty, and staff Provide an excellent learning environment Quality Encourage the realization and recognition of the full potential of all campus community members Climate Instill in our graduates skills appropriate to their professions and life-long learning Outcomes Resources Provide resource management & development that supports the academic mission
10 4 Ethics and professional responsibility 4 Understanding of contemporary issues 4 Role of professionals in the global society and ability to understand diverse cultural and humanistic traditions 4 Teamwork 4 Communication skills 4 Skills and knowledge necessary for mathematical, scientific, and engineering practice 4 Interpret graphical, numerical, and textual data 4 Design and conduct experiments 4 Design a product or process to satisfy a client's needs subject to constraints Instill in our graduates skills appropriate to their professions and life-long learning Outcomes
11 Why portfolios? 4 Authentic assessment 4 Capture a wide variety of student work 4 Involve students in their own assessment 4 Professional development for faculty
12 Why electronic portfolios? 4 Student-owned laptop computer program since Classrooms, residence halls, common areas, library, fraternity houses all wired 4 Access 4 Efficient 4 Cost effective 4 Asynchronous assessment
13 RosE-Portfolio Structure Student Advisor ADMIN Employer RaterFaculty Submit Review Search Dynamic Resume Access Control View Advisees portfolio Search Advisees portfolio User Management Group Management System Configuration Criteria Tree Activity Managment View Search Inter-rater Reliability Rating sessions Feedback Rating management Curriculum Map PTR Portfolio Submit Review Search
14 Show Me!
15 Assessment of student material 4 Faculty work in teams 4 Each team assesses one learning objective 4 Score holistically 4 Emerging rubrics Does the reflective statement indicate an understanding of the criterion? Does the reflective statement demonstrate or argue for the relevance of the submitted material to the criterion? level appropriate to a graduating senior Does the submitted material meet the requirements of the criterion at a level appropriate to a graduating senior at R- HIT?
16 Show Me!
17 Example of Results 1 Understand criterion? 2 Submission relevant to criterion? 3 Meet standards for R-HIT graduate?
18 Example of Results Does submission meet the standards for a graduate of R-HIT? 1 Appropriate for audience 2 Organization 3 Content factually correct 4 Test audience response 5 Grammatically correct
19 Linking results to Practice 4 Development of Curriculum Map 4 Linking curriculum content/pedagogy to knowledge, practice and demonstration of learning outcomes Show Me!
20 Curriculum Map Results Fall (181 courses/labs) Communication Skills
21 Curriculum Map Results Fall (181 courses/labs) Ethics
22 Closing the loop JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC Eval Committee receives and evaluates all data; makes report and refers recom- mendations to appropriate areas. Institute acts on the recom- mendations of the Eval. Comm. Reports of actions taken by the Institute and the targeted areas are returned to the Eval Comm. for iterative evaluation. Institute assessment cmte. prepares reports for submission to Dept. Heads of the collected data (e.g. surveys, e-portfolio ratings).
23 Primary focus 4 It is not about electronic portfolios. 4 It is about: supporting teaching and learning faculty and student development the transformation of the teaching/learning environment
24 Benefits to teaching 4 Faculty are asked to reflect on learning outcomes in relation to practice Consider the value of stated outcomes –Right ones? –Right performance criteria? –Individual faculty role in creating the context for learning 4 Develop a common language and understanding of program/institutional outcomes 4 Explicit accountability 4 Promotes interdisciplinary discussions/collaborations
25 Benefits to learning 4 Students review their own progress as it relates to expected learning. 4 Portfolios provide a way for students to make learning visible and becomes the basis for conversations and other interactions among students and faculty. 4 Learning is viewed as an integrated activity not isolated courses. 4 Students learn to value the contributions of out-of-class experiences. 4 Student reflections are metacognitive as they appraise their own ways of knowing. 4 Promotes a sense of personal ownership over ones accomplishments.
26 Assessment method truisms 4 There will always be more than one way to measure any outcome 4 No single method is good for measuring a wide variety of different student abilities 4 Consistently inverse relationship between the quality of measurement methods and their expediency 4 Importance of pilot testing to see if method is good for your program (students & faculty)
27 Barriers to implementation äFaculty äcurrent workload älack of incentive to participate in the process (rewards) äwhats in it for me (cost/benefits) äInstitutional/program leadership äLack of vision for the program/institutional assessment process (no existing, efficient models) äCost/benefit unknown äDifficulty of restructuring the reward system to facilitate faculty participation
28 äProcess deficiencies äLack of understanding of the dynamics of organizational change äAbsence of tools to facilitate collaborative work äPortfolio deficiencies äIll-defined purpose äLack of efficient ways to manage the portfolio process äSystematic review of portfolio contents is ill- defined or non-existent äStudent and faculty roles not clear äPortfolio process not integrated into the teaching/learning environment äResource deficiencies äExpertise in portfolio development äDevelopment of authentic portfolio
29 Advice from the field You cannot do it all - prioritize All assessment questions are not equal One size does not fit all Its okay to ask directions Take advantage of local resources Dont wait until you have a perfect plan Decouple from faculty evaluation E=MC 2