1. MEASURES OF SUCCESS: An Evaluators Perspective Carol L. Colbeck Director & Associate Professor Center for the Study of Higher Education The Pennsylvania State University 400 Rackley Building University Park, PA 16802 clc15@psu.edu

2. Colbeck 2003 LEVELS OF EVALUATION Participation Satisfaction Learning Application Impact

3. Colbeck 2003 EXAMPLE: Classroom Teaching Course Focus on Teaching as Research – Deliberate, systematic, and reflective use of research methods to develop and implement teaching practices that advance the learning experiences and learning outcomes of students as well as teachers.

4. Colbeck 2003 PARTICIPATION Who? Why? Demographics – Discipline – Gender – Position – Where earned degrees – Country of origin Experience – # of years/classes taught – Other professional development experience Motivation – Own goals – External encouragement

5. Colbeck 2003 SATISFACTION Value for participants Content Instruction/facilitation Did program address participants reasons for coming? – Own goals – External encouragement Were learning objectives specified and addressed? Did process foster participant interest? – Instructor style – Teaching methods

6. Colbeck 2003 LEARNING Knowledge gains Conceptual change Skill development Has awareness increased? – Information – Resources – Theoretical foundations – Empirical research Are there changes in? – Attitudes – Beliefs – Confidence Can tasks be performed? – Behaviors

7. Colbeck 2003 APPLICATION Attempt Implement Adjust Are new knowledge, attitudes, skills tried independently? Does application follow desired processes & procedures? Does participant analyze & adjust appropriately for context?

8. Colbeck 2003 IMPACT On undergraduate students – Increased satisfaction – Deeper learning – Application: in lab, on job, on GREs On graduate student participants – Increased confidence in teaching – Wider array of career opportunities – More integration of teaching and research On institution – Improved recruitment/retention of graduate students – Improved recruitment/retention of undergraduate students – Stronger community around research, teaching, and learning

9. Colbeck 2003 SAMPLE EVALUATION QUESTIONS What types of students are most likely to participate? What are barriers to others participation? What are most effective ways to overcome barriers? How do relationships between participant characteristics,experience, and teaching methods affect their satisfaction? Learning? How do relationships between institutional context, class content, and individual characteristics affect participants independent application of learning? Under what conditions are undergraduate students most likely to report gains in satisfaction and learning from a participant who has undergone professional development training?