1. Tom Lewis, Director, Ed-Tech Development Group Educational Partnerships & Learning Technologies Scott Macklin, Director Program for Educational Transformation through Technology Copyright Tom Lewis and Scott Macklin, 2003. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author. The Catalyst E-Portfolio: Using Assessment to Shape Enterprise-Wide, Learner-Centered Technologies

2. How do you scale beyond the early adopters to reach wary adopters and encourage innovation?

3. PETTT Educational Technology Surveys Faculty, Spring 2001 2000 respondents Students, Fall 2002 1000 respondents

4. 20% of college students began using a computer between 5 and 8 years old. By the time they are 16-18, all had used a computer. 80% of UW undergraduates use a computer daily. Technology Adoption: Student Driven

5. 72% of students think all courses should have a Web site Students particularly want online: course syllabi (81%) grading criteria (79%) problem sets or exercises (78.6%) links to electronic course reserves (72%) 7% of students stated that educational technology is used consistently through UW Students…

6. Im graduating now and Im pretty upset with how little I know about technology, I really wish that I was more confident and knew more after four years of my education at a university. A lot of it is probably independent work that you can do to learn it but were busy enough with jobs and school that I just really havent felt like I had the time to actually independently learn it…it would have been really helpful to have not been expected to know it but to have actually been taught it. -Undergraduate Student

7. Which of the following concern faculty when using information technology to do university work?

8. Scaling Innovation with the 4-Tiered Catalyst Initiative Lesson Learned: Scale support through partnerships, making disparate resources and innovative practices visible and available to all. http://catalyst.washington.edu

9. On an average day in Autumn 2002, instructors viewed 5000 pages of Catalyst content every day. 8557 instructors have created 18,592 implementations of Catalyst Web Tools. Catalyst Web Tools were used for 66,000 learning activities during Autumn 2002. The Catalyst Initiative

10. Catalyst Web Site

11. Participatory Design & Catalyst Web Tools Lesson Learned: Create tools that meet pedagogical needs expressed by faculty and constantly engage with faculty to make sure the tools remain current and easy to use.

12. Collaborative Development Assessing what the educator wants is central: Get feedback at every step of design. Process is more important than product. Direct communication between developers and educator. Make feedback and interaction easy. Value educators ideas and time.

13. SimpleSite Create Course Web pages without HTML or an editor Publish Web pages without FTP Keeps usage statistics for you

14. VirtualCase For case studies and problem-based learning activities Contains a message board and collaborative file sharing Allows for group decision making

15. Partnerships as a support strategy to bring easy to use, flexible & campus-wide support. Lesson Learned: Make sure that all the units that should be shaping technology innovation are shaping innovation.

16. Portfolio NEW!

17. Resources Catalyst group (under Vice-Provost of Educational Partnerships & Learning Technologies) supplied main funding Student Technology Fee committee gave financial support. Program for Educational Transformation through Technology provided financial and assessment support. Computing & Communications furnished server space and maintenance Office of Undergraduate Education offered the Freshman Interest Group program as a test-bed

18. Project Assumptions Tool cannot be built without input from students, instructors, and academic and career advisers Tool should be flexiblestudents should be able to use it for classes, extra-curricular activities, job searches, or personal reflection Tool should be useful for students in all colleges and schools

19. How did we build it? Asked everyone on campus who has ever uttered, or is likely to utter the word Portfolio: 200 + Looked at other universities tools and commercial products Read scholarly articles on the uses of portfolios in education Conducted focus groups and usability tests with UW instructors and students

20. Basic Decisions: Students should be able to collect anything they want to and have exclusive access to those collections Students should be able to publish many different portfolios of their work on the Web Instructors and advisors should be able to easily lead students through the process of building certain types of portfolios, but students still own these portfolios

21. What is a portfolio tool? There is a distinction between collecting artifacts, reflecting upon them, and thoughtfully presenting them in a portfolio Students will be collecting artifacts throughout their academic careers and should have exclusive access to their collections Using portfolios for reflection is a process that is important in all phases of an academic career, not just at the end of senior year

22. What is a portfolio tool (cont.)? Instructors, advisers, and peers should collaborate with students to help them present artifacts in a reflective way Students should be able to present a variety of portfolios for different objectives: completing coursework or demonstrating academic, personal, or career development

23. Lesson learned: Need research and assessment to inform the development of learning technologies and teaching strategies. Instructors need help in effectively integrating technologies into teaching.

25. 1.Perform studies in real settings to investigate what is learned, who learns, and how it happens, 2.which in turn inform the design of appropriate tools. 3.These new tools are integrated into educational settings, 4.in which new studies are conducted to understand how these new tools have affected learning and learners. The process

26. The FIG Program and Portfolios: Pilot project in Autumn 2001 with three Freshman Interest Group (FIG) sections (~60 students). Development of the Catalyst Portfolio Tool in 2001-2002. Introduction of the tool into the FIG program in Autumn 2002, with a study by PETTT researchers.

27. Our Goals for Autumn 2002: Introduce the concept of portfolios to 3,200 freshman students. Within the goals of the existing University Community class, encourage reflective learning. Learn lessons that can guide future portfolio projects and inform future revisions of the tool.

28. What was our experiment? Collect data on the use of the tool by FIG leaders and students through: Surveys Focus groups Classroom observation Examination of student portfolios

29. Study design: 162 Freshman Interest Groups 162 FIG leaders, each with a class of 10- 25 students Special group: 20 FIG leaders, paid to give regular, individualized feedback to students

30. Study questions: 1)What method(s) of instruction help students feel most competent using Portfolio? 2)How can Portfolio be best integrated into the GENST 199 curriculum? 3)What instructional supports (in-class discussion, feedback, prompts within tool) might best support reflective thinking?

31. Data collection: 1)First day classroom observations 2)Student surveys 3)FIG leader surveys 4)Focus groups with students and FIG leaders 5)Classroom observations & student interviews 6)Review of student portfolios

32. Three themes from preliminary analysis: Students found Portfolio easy and convenient to use. However, they largely perceived answering the questions as busywork. FIG leaders' attitude toward the tool and the work affected students' response to the tool and to class. Feedback was important--Students wanted to know that someone was reading what they wrote. These are issues to consider as we head towards a redesign and make recommendation for how to use Portfolio. NOT reasons to abandon the project.

33. Quotes… I feel its really fun to me, I mean maybe its weird but I think its almost like a penpal letter kind of to our FIG leader and we can answer the questions and sometimes theyre really in-depth and we actually have to think about what we think about life and on these certain subjects, we have to think about what our beliefs are, and I think thats helped me a lot to really figure out what I believe and why I believe it, and apply that to the discussion that weve had in class and stuff. -FIG Student

34. Its nice when they return it and they add comments so you know that theyre actually looking at it. (student) -FIG Student If the FIG leaders did give feedback… it does kind of make it more interesting and you do feel like your FIG leader is paying a little bit of attention to you. It might even create discussions that you could have in your FIG class. -FIG Student

35. [Portfolio and the feedback function] helps you know more about your students, share and get ideas. -FIG leader Because I comment so much, they [students] see me as a friend and come talk to me. I learned a lot from them. -FIG leader

36. Building Catalyst through Co-Branding Lesson Learned: Renew and refresh anytime-anywhere technology support resources by co- branding and co-developing them with campus teaching practitioners.

37. Students learn through portfolios when they… assemble a collection of relevant artifacts select items from that collection for a specific purpose reflect on the selected items and/or process of selection design the presentation of their portfolio for a particular audience/purpose receive feedback on the quality of their selection, reflection, and/or design

38. Considerations for teaching with portfolios… specific purpose/audience assignments to produce relevant artifacts guidelines for item selection guidelines for and examples of good reflection guidelines for and examples of good design time for feedback on quality of selection, reflection, and/or design

39. More considerations for teaching… Evaluation individual items? whole? reflection? establish clear criteria & SHARE with students ahead of time. will the evaluation count toward the grade? if students dont have opportunity to learn from feedback, a grade isnt a measure of what they learnedonly innate ability.

40. http://catalyst.washington.edu Tom Lewis > Scott Macklin > http://depts.washington.edu/pettt