Tom Lewis, Director Ed-Tech Development Group Educational Partnerships & Learning Technologies Scott Macklin, Director Program for Educational Transformation through Technology The Catalyst Initiative: Using Assessment to Drive Innovation in Teaching & learning with Technology

How do you scale beyond the early adopters to reach wary adopters and encourage innovation?

Scaling Innovation with the 4-Tiered Catalyst Initiative Lesson Learned: Scale support through partnerships, making disparate resources and innovative practices visible and available to all.

On an average day in Autumn 2002, instructors viewed 5000 pages of Catalyst content every day instructors have created 18,592 implementations of Catalyst Web Tools. Catalyst Web Tools were used for 66,000 learning activities during Autumn The Catalyst Initiative

Catalyst Web Site

Catalyst Profiles... Stories of Innovation

Teaching Guides start with your teaching goals

Action Plans help you use technology

The How-To section has detailed instructions for specific tasks

The learning section provides information about workshops and clinics

Participatory Design & Catalyst Web Tools Lesson Learned: Create tools that meet pedagogical needs expressed by faculty and constantly engage with faculty to make sure the tools remain current and easy to use.

Collaborative Development Assessing what the educator wants is central: Get feedback at every step of design. Process is more important than product. Direct communication between developers and educator. Make feedback and interaction easy. Value educator’s ideas and time.

SimpleSite Create Course Web pages without HTML or an editor Publish Web pages without FTP Keeps usage statistics for you

WebQ Create online surveys, quizzes & questionnaires Download data, or look at it on the Web Many security options for password protecting your data Approved by UW Human Subjects Division

Peer Review Create a forum for to review each others’ documents Encourage group projects with the new group features Supports any kind of Web content: (text, graphics, sounds, movies) Comment on sentences, paragraphs or entire document Complete participant reports accessible to instructor

UMail An anonymous feedback tool Messages come to your regular inbox

E-Submit Students can turn in homework online Saves time and frustration with attachments Students receive a digital “receipt”

EPost Create an online discussion board Many sort options: by thread, subject, author, and date Generates “participation reports”

Create 1-question surveys for your Web site Results tabulated automatically Numeric and graphic displays Many submission controls available Can be linked to EPost QuickPoll

Partnerships as a support strategy to bring easy to use, flexible & campus-wide support. Lesson Learned: Make sure that all the units that should be shaping technology innovation are shaping innovation.

Portfolio NEW!

Resources Catalyst group (under Vice-Provost of Educational Partnerships & Learning Technologies) supplied main funding Student Technology Fee committee gave financial support. Program for Educational Transformation through Technology provided financial and assessment support. Computing & Communications furnished server space and maintenance Office of Undergraduate Education offered the Freshman Interest Group program as a test-bed

Project Assumptions Tool cannot be built without input from students, instructors, and academic and career advisers Tool should be flexible—students should be able to use it for classes, extra-curricular activities, job searches, or personal reflection Tool should be useful for students in all colleges and schools

How did we build it? Asked everyone on campus who has ever uttered, or is likely to utter the word “Portfolio”: Looked at other universities’ tools and commercial products Read scholarly articles on the uses of portfolios in education Conducted focus groups and usability tests with UW instructors and students

Basic Decisions: Students should be able to collect anything they want to and have exclusive access to those collections Students should be able to publish many different portfolios of their work on the Web Instructors and advisors should be able to easily lead students through the process of building certain types of portfolios, but students still own these portfolios

What is a portfolio tool? There is a distinction between collecting artifacts, reflecting upon them, and thoughtfully presenting them in a portfolio Students will be collecting artifacts throughout their academic careers and should have exclusive access to their collections Using portfolios for reflection is a process that is important in all phases of an academic career, not just at the end of senior year

What is a portfolio tool (cont.)? Instructors, advisers, and peers should collaborate with students to help them present artifacts in a reflective way Students should be able to present a variety of portfolios for different objectives: completing coursework or demonstrating academic, personal, or career development

Lesson learned: Need research and assessment to inform the development of learning technologies and teching strategies. Instructors need help in effectively integrating technologies into teaching.

1.which in turn inform the design of appropriate tools. 2.These new tools are integrated into educational settings, 3.in which new studies are conducted to understand how these new tools have affected learning and learners. 4.Perform studies in real settings to investigate what is learned, who learns, and how it happens, The process

The FIG Program and Portfolios: Pilot project in Autumn 2001 with three Freshman Interest Group (FIG) sections (~60 students). Development of the Catalyst Portfolio Tool in Introduction of the tool into the FIG program in Autumn 2002, with a study by PETTT researchers.

Our Goals for Autumn 2002: Introduce the concept of portfolios to 3,200 freshman students. Within the goals of the existing University Community class, encourage reflective learning. Learn lessons that can guide future portfolio projects and inform future revisions of the tool.

What was our experiment? Collect data on the use of the tool by FIG leaders and students through: Surveys Focus groups Classroom observation Examination of student portfolios

Study design: 162 Freshman Interest Groups 162 FIG leaders, each with a class of students Special group: 20 FIG leaders, paid to give regular, individualized feedback to students

Study questions: 1)What method(s) of instruction help students feel most competent using Portfolio? 2)How can Portfolio be best integrated into the GENST 199 curriculum? 3)What instructional supports (in-class discussion, feedback, prompts within tool) might best support reflective thinking?

Data collection: 1)First day classroom observations 2)Student surveys 3)FIG leader surveys 4)Focus groups with students and FIG leaders 5)Classroom observations & student interviews 6)Review of student portfolios

Three themes from preliminary analysis: Students found Portfolio easy and convenient to use. However, they largely perceived answering the questions as busywork. FIG leaders' attitude toward the tool and the work affected students' response to the tool and to class. Feedback was important--Students wanted to know that someone was reading what they wrote. These are issues to consider as we head towards a redesign and make recommendation for how to use Portfolio. NOT reasons to abandon the project.

Quotes… “I feel it’s really fun to me, I mean maybe it’s weird but I think it’s almost like a penpal letter kind of to our FIG leader and we can answer the questions and sometimes they’re really in-depth and we actually have to think about what we think about life and on these certain subjects, we have to think about what our beliefs are, and I think that’s helped me a lot to really figure out what I believe and why I believe it, and apply that to the discussion that we’ve had in class and stuff.” -FIG Student

“It’s nice when they return it and they add comments so you know that they’re actually looking at it.” (student) -FIG Student “If the FIG leaders did give feedback… it does kind of make it more interesting and you do feel like your FIG leader is paying a little bit of attention to you. It might even create discussions that you could have in your FIG class”. -FIG Student

“[Portfolio and the feedback function] helps you know more about your students, share and get ideas”. -FIG leader “Because I comment so much, they [students] see me as a friend and come talk to me. I learned a lot from them.” -FIG leader

Building Catalyst through Co-Branding Lesson Learned: Renew and refresh anytime-anywhere technology support resources by co- branding and co-developing them with campus teaching practitioners.

Students learn through portfolios when they… assemble a collection of relevant artifacts select items from that collection for a specific purpose reflect on the selected items and/or process of selection design the presentation of their portfolio for a particular audience/purpose receive feedback on the quality of their selection, reflection, and/or design

Considerations for teaching with portfolios… specific purpose/audience assignments to produce relevant artifacts guidelines for item selection guidelines for and examples of “good” reflection guidelines for and examples of “good” design time for feedback on quality of selection, reflection, and/or design

More considerations for teaching… Evaluation individual items? whole? reflection? establish clear criteria & SHARE with students ahead of time. will the evaluation count toward the grade? if students don’t have opportunity to learn from feedback, a grade isn’t a measure of what they learned—only innate ability.

Extending Innovation through Assessment Lesson Learned: Reach out to instructors and students in as many ways and as often as possible.

PETTT Educational Technology Surveys Faculty, Spring respondents Students, Fall respondents

20% of college students began using a computer between 5 and 8 years old. By the time they are 16-18, all had used a computer. 80% of UW undergraduates use a computer daily. Technology Adoption: Student Driven

72% of students think all courses should have a Web site Students particularly want online:  course syllabi (81%)  grading criteria (79%)  problem sets or exercises (78.6%)  links to electronic course reserves (72%) 7% of students stated that educational technology is used consistently through UW Students…

“I’m graduating now and I’m pretty upset with how little I know about technology, I really wish that I was more confident and knew more after four years of my education at a university. A lot of it is probably independent work that you can do to learn it but we’re busy enough with jobs and school that I just really haven’t felt like I had the time to actually independently learn it…it would have been really helpful to have not been expected to know it but to have actually been taught it.” -Undergraduate Student

Which of the following concern faculty when using information technology to do university work?

Top 5 desired faculty uses for the Web

Faculty preferred methods for learning

depts.washington.edu/pettt Tom Lewis > Scott Macklin >