1. Copyright © 2000. SRI International. All rights reserved. Session 6 The Unfinished Revolution - II Networked Improvement Communities in Education Roy Pea Bootstrap Colloquium Stanford University, Feb 10, 2000 Networked Improvement Communities in Education Roy Pea Bootstrap Colloquium Stanford University, Feb 10, 2000

2. Copyright © 2000. SRI International. All rights reserved. Center for Technology in Learning MISSION: Improving learning and teaching through innovation and inquiry in computing and communications MISSION: Improving learning and teaching through innovation and inquiry in computing and communications http://www.sri.com/policy/ctl/

3. Copyright © 2000. SRI International. All rights reserved. How does CTL do this? Advance theory and research on effective learning and teaching. Innovative design, use, assessment of interactive learning environments. In short, learning sciences and engineering. Advance theory and research on effective learning and teaching. Innovative design, use, assessment of interactive learning environments. In short, learning sciences and engineering.

4. Copyright © 2000. SRI International. All rights reserved. What’s the need? Education in the U.S. is decentralized and fragmented Education improvement efforts are isolated and incremental Technology can leverage the knowledge “in the system” in revolutionary ways Education in the U.S. is decentralized and fragmented Education improvement efforts are isolated and incremental Technology can leverage the knowledge “in the system” in revolutionary ways

5. Copyright © 2000. SRI International. All rights reserved. A “powerful idea” for CTL... Networked Improvement Community (NIC) … is a coalition of organizations each engaged in a similar improvement process... networks these organizations and crafts new mechanisms for improving their isolated improvement processes...builds value and leverages knowledge for distributed communities Thanks to Douglas Engelbart for fruitful discussions about NICs Networked Improvement Community (NIC) … is a coalition of organizations each engaged in a similar improvement process... networks these organizations and crafts new mechanisms for improving their isolated improvement processes...builds value and leverages knowledge for distributed communities Thanks to Douglas Engelbart for fruitful discussions about NICs

6. Copyright © 2000. SRI International. All rights reserved. The Many NICs of CTL

7. Copyright © 2000. SRI International. All rights reserved. Sponsors: An Education Community-of-Practice Model for Scalable, Sustainable Teacher Professional Development http://www.tappedin.org Leader: Mark Schlager Contact: schlager@unix.sri.com TM

8. Copyright © 2000. SRI International. All rights reserved. What? Web-based Virtual Conference Center Who? A growing community of over 7000 K-16 education professionals, 15 partner organizations and agencies, and scores of small groups who create on- line learning experiences from any computer, anytime. Multi-user virtual environment (MUVE) Synchronous and asynchronous communication Storing, sharing of Websites and documents (permanence) Community-wide activities and support services TAPPED IN Overview TM

9. Copyright © 2000. SRI International. All rights reserved. Private Public Pepperdine School of Education Student Activities Center TAPPED IN Building Gathering Place: TAPPED IN Campus TM

10. Copyright © 2000. SRI International. All rights reserved. TM 19971998 1999 Jan-00 7000+

11. Copyright © 2000. SRI International. All rights reserved. University Schools of Education  Pepperdine University (PT3)  California State Univ., Sacramento  University of Wisconsin (NSF)  George Mason University Local and State Education Agencies  New Haven Unified School District, California (PT3, BTSA)  Joint Venture: Silicon Valley 21st Century Education Initiative (TICG)  Kentucky Department of Education  Los Angeles County Office of Education TFL, TEAMS, LAAMP (NSF) Teacher/Librarian Professional Development Organizations and Websites  American Association of School Librarians  Institute for Research on Learning MMAP program (NSF)  Lawrence Hall of Science, University of California, Berkeley (NSF)  Math Forum Website at Swarthmore College (NSF) University Schools of Education  Pepperdine University (PT3)  California State Univ., Sacramento  University of Wisconsin (NSF)  George Mason University Local and State Education Agencies  New Haven Unified School District, California (PT3, BTSA)  Joint Venture: Silicon Valley 21st Century Education Initiative (TICG)  Kentucky Department of Education  Los Angeles County Office of Education TFL, TEAMS, LAAMP (NSF) Teacher/Librarian Professional Development Organizations and Websites  American Association of School Librarians  Institute for Research on Learning MMAP program (NSF)  Lawrence Hall of Science, University of California, Berkeley (NSF)  Math Forum Website at Swarthmore College (NSF) TM Some TAPPED IN users

12. Copyright © 2000. SRI International. All rights reserved. What can educators do on-line? Educators engage in professionally relevant activities: Planning and conducting projects with colleagues and students Leading or joining topical discussions Conducting and attending courses Finding resources, experts, and new colleagues Serving as resources for other educators Trying out new ideas in a safe supportive environment Using the Internet in new ways Extend capacity to interact online through: Real-time and e-mail Help Desk After School Online discussion sessions Free offices for individuals and small groups Newsletter, events calendar, and MeetMe list How-to Guides (e.g., lead a discussion; conduct training) Consulting for TPD organizations TM

13. Copyright © 2000. SRI International. All rights reserved. What are we learning? Content is important, but learning occurs through discourse around content Membership diversity is essential for innovations to germinate and spread – Preservice and new teachers learn about the profession and veterans become valued resources – Members form groups that cross agency, program, project, stakeholder group, and geographical boundaries Both private and public places are needed to support the community TM

14. Copyright © 2000. SRI International. All rights reserved. No single organization can satisfy teachers’ changing TPD needs Yet sustainable on-line communities are hard & costly to establish TPD organizations must understand the teachers’ challenge begins where their efforts leave off Teachers need ownership of an on-line workplace that will be there after TPD courses, projects, workshops, or grants end Teachers are ready for on-line learning; many leaders are not Leaders see their projects as unique and view going on-line as a costly risk, not a strategic cost- and time-saving tool Traditional TPD approaches do not transfer well to on-line learning TPD pedagogy, staffing, and budgeting must be rethought Challenges? TM

15. Copyright © 2000. SRI International. All rights reserved. Performance Assessment Links in Science Leader: Edys Quellmalz (quelmalz@unix.sri.com) http://www.ctl.sri.com/pals Focus: An interactive resource bank for improving standards-based science assessment Size: $1.2M for 3 years Funder: National Science Foundation –Elementary, Secondary, Informal Education (EISE) –Instructional Materials Development Leader: Edys Quellmalz (quelmalz@unix.sri.com) http://www.ctl.sri.com/pals Focus: An interactive resource bank for improving standards-based science assessment Size: $1.2M for 3 years Funder: National Science Foundation –Elementary, Secondary, Informal Education (EISE) –Instructional Materials Development

16. Copyright © 2000. SRI International. All rights reserved.

17. Project Participants Assessment Consortia Council of Chief State School Officers (CCSSO) California Systemic Initiatives Assessment Consortium State Departments of Education Connecticut Department of Education Illinois Department of Education Kentucky Department of Education School Districts Springfield, Illinois Vancouver, Washington Science Curriculum Projects Teachers Assessment Consortia Council of Chief State School Officers (CCSSO) California Systemic Initiatives Assessment Consortium State Departments of Education Connecticut Department of Education Illinois Department of Education Kentucky Department of Education School Districts Springfield, Illinois Vancouver, Washington Science Curriculum Projects Teachers

18. Copyright © 2000. SRI International. All rights reserved. Performance Assessment Links in Science Lessons Learned Start by addressing priorities voiced by key participant groups (state science assessment programs, professional development groups) Partner with representatives of participant groups Involve participants, contributors, and funders in prototype design, seed content, and user testing Lessons Learned Start by addressing priorities voiced by key participant groups (state science assessment programs, professional development groups) Partner with representatives of participant groups Involve participants, contributors, and funders in prototype design, seed content, and user testing

19. Copyright © 2000. SRI International. All rights reserved. Performance Assessment Links in Science Addressing Challenges Obtain commitment of partners to content development and review Engage partners in ongoing interactions and community events Design incentives for users to provide feedback Build on models for maintenance and growth Collect systematic evidence of quality, usability and impact Balance public service and commercial functions Addressing Challenges Obtain commitment of partners to content development and review Engage partners in ongoing interactions and community events Design incentives for users to provide feedback Build on models for maintenance and growth Collect systematic evidence of quality, usability and impact Balance public service and commercial functions

20. Copyright © 2000. SRI International. All rights reserved. Online Evaluation Resource Library Leader: Edys S. Quellmalz (quelmalz@unix.sri.com) http://oerl.sri.com Focus: Improving Project Evaluation Size: $1.75M for 7 years Funder: National Science Foundation –Research, Evaluation, and Communication Leader: Edys S. Quellmalz (quelmalz@unix.sri.com) http://oerl.sri.com Focus: Improving Project Evaluation Size: $1.75M for 7 years Funder: National Science Foundation –Research, Evaluation, and Communication

21. Copyright © 2000. SRI International. All rights reserved. Online Evaluation Resource Library Project Participants NSF Current and Prospective Principal Investigators and Evaluators NSF Program Officers Evaluation Professors, Students Evaluation Training Programs The Evaluation Community Project Participants NSF Current and Prospective Principal Investigators and Evaluators NSF Program Officers Evaluation Professors, Students Evaluation Training Programs The Evaluation Community

22. Copyright © 2000. SRI International. All rights reserved. Online Evaluation Resource Library Lessons Learned Exemplary content may be hard to find Develop navigation supports for novice users Involve key stakeholders in design and reviews Seek multiple dissemination paths early on Lessons Learned Exemplary content may be hard to find Develop navigation supports for novice users Involve key stakeholders in design and reviews Seek multiple dissemination paths early on

23. Copyright © 2000. SRI International. All rights reserved. Online Evaluation Resource Library Addressing Challenges Enable participants to take multiple paths to access information and resources; “no one size fits all” Engage participants in active learning experiences aimed at extending their knowledge base and strategies Support and provide incentives for content contributions Support collaboration and peer review Addressing Challenges Enable participants to take multiple paths to access information and resources; “no one size fits all” Engage participants in active learning experiences aimed at extending their knowledge base and strategies Support and provide incentives for content contributions Support collaboration and peer review

24. Copyright © 2000. SRI International. All rights reserved. Center for Innovative Learning Technologies (CILT) Leaders: Roy Pea & Barbara Means SRI International Marcia Linn UC Berkeley John Bransford Vanderbilt University Bob Tinker Concord Consortium Mission: To serve as a national resource for stimulating research on innovative technology- enabled solutions to critical problems in K-14 learning Funding sources: –NSF ($6M total for 4 years) –Industry partners (Intel: $100K/year) Leaders: Roy Pea & Barbara Means SRI International Marcia Linn UC Berkeley John Bransford Vanderbilt University Bob Tinker Concord Consortium Mission: To serve as a national resource for stimulating research on innovative technology- enabled solutions to critical problems in K-14 learning Funding sources: –NSF ($6M total for 4 years) –Industry partners (Intel: $100K/year) http://cilt.org

25. Copyright © 2000. SRI International. All rights reserved. CILT Activities and Participants Annual workshop for harvesting knowledge and leveraging diverse efforts on “theme teams” of high-priority and breakthrough opportunity Visualization and Modeling Ubiquitous Computing Community Tools Assessments for Learning 400-500 Participants set priorities for partnership projects; CILT later “seed funds” promising efforts and promotes synergy efforts “CILT Knowledge Network” for People, Places, Projects, Bibliography, Collaborations, Syllabi http://kn.cilt.org Postdoctoral training program Annual workshop for harvesting knowledge and leveraging diverse efforts on “theme teams” of high-priority and breakthrough opportunity Visualization and Modeling Ubiquitous Computing Community Tools Assessments for Learning 400-500 Participants set priorities for partnership projects; CILT later “seed funds” promising efforts and promotes synergy efforts “CILT Knowledge Network” for People, Places, Projects, Bibliography, Collaborations, Syllabi http://kn.cilt.org Postdoctoral training program

26. Copyright © 2000. SRI International. All rights reserved. CILT as a Multi-level NIC SRI Berkeley Vanderbilt Concord

27. Copyright © 2000. SRI International. All rights reserved. SRI Berkeley Vanderbilt Concord Research Community CILT as a Multi-level NIC

28. Copyright © 2000. SRI International. All rights reserved. SRI Berkeley Vanderbilt Concord Research Community Industry Policy Practice CILT as a Multi-level NIC

29. Copyright © 2000. SRI International. All rights reserved. Lessons Learned: Getting Started NIC-ing takes a huge amount of coordination - plan for it! –Tasks that are often viewed as “overhead” by partners become core NIC functions Work with all groups of stakeholders early Provide a concrete focus or venue for sharing and collaboration (e.g., improving learning technology R&D projects; workshops) NIC-ing takes a huge amount of coordination - plan for it! –Tasks that are often viewed as “overhead” by partners become core NIC functions Work with all groups of stakeholders early Provide a concrete focus or venue for sharing and collaboration (e.g., improving learning technology R&D projects; workshops)

30. Copyright © 2000. SRI International. All rights reserved. Indicators of CILT Success Informal results as important as formal ones Formal Industry Alliance partnerships are difficult, but ad-hoc industry/research interactions are fruitful Metrics include: CILT-sponsored collaborations that resulted in sustained outside funding Ideas we promoted that took hold in research, policy, education, product development communities Isolated individuals who got the connections/support they needed to succeed Informal results as important as formal ones Formal Industry Alliance partnerships are difficult, but ad-hoc industry/research interactions are fruitful Metrics include: CILT-sponsored collaborations that resulted in sustained outside funding Ideas we promoted that took hold in research, policy, education, product development communities Isolated individuals who got the connections/support they needed to succeed

31. Copyright © 2000. SRI International. All rights reserved. Lessons Learned: CILT Challenges Contracting: Universities and other institutions like to own, not share (public-domain) Financing: Under traditional models, the process of collaboration is rarely funded Time and attention: NICs bring together the best and brightest - and therefore “the busy” Design: It’s easier to share curricular resources when they were designed for adaptability from the start Contracting: Universities and other institutions like to own, not share (public-domain) Financing: Under traditional models, the process of collaboration is rarely funded Time and attention: NICs bring together the best and brightest - and therefore “the busy” Design: It’s easier to share curricular resources when they were designed for adaptability from the start

32. Copyright © 2000. SRI International. All rights reserved. Leaders: Jeremy Roschelle, Chris DiGiano, Roy Pea (SRI); Jim Kaput (U. Mass., Dartmouth) http://www.escot.org Focus: Improving quality, interoperability and reuse of software components for teaching middle school mathematics Size: $2 million for 2 years Funder: National Science Foundation

33. NEED: Without a NIC, disparate math education efforts have their own isolated improvement processes, resulting in ‘application islands.’ CTL SW Univ. of CO SW Key Press SW SimCalc SW ShowMe Curricula MathForum Resources Math Teachers Redundant Non-cumulative Unconnected Copyright © 2000. SRI International. All rights reserved.

34. ESCOT as a NIC Evolving knowledge network and interoperable software library for continuous improvement by accumulating, integrating, sharing, and testing work. CTL Components Univ. of CO Components Key Press Components SimCalc Components ShowMe Curricula MathForum Resources Math Teachers ESCOT Testbed Cumulative Integrated Continual testing Copyright © 2000. SRI International. All rights reserved.

35. Growing Partnerships and Results Overall: 8 funded institutional partners, plus 25+ volunteer institutional partners; 23 software developers Non-Profits: SRI International; U. Massachusetts; U. Colorado; Swarthmore College; Stanford University; UC-Berkeley; many middle schools Small Companies: Key Curriculum Press, DesignWorlds International: Centre for Constructive and Experimental Mathematics (Canada); Computer Technology Institute (Greece) Results: ESCOT now has 37 components, inc. spreadsheet, web browser, grapher, scripting languages, simulation engines, geometry sketchpad... ESCOT database of 5 major math textbooks indexes 3000 places to use technology components to enhance learning

36. Copyright © 2000. SRI International. All rights reserved. Lessons Learned (startup) Vet ideas for NIC in extensive planning process Pick motivated, complementary, communicative core partners Support extensive face-to-face contact for first 6 months Clear tasks / expectations for partner roles Create good multi-channel communications infrastructure (extranet site, e-mail lists, phone conferences, software configuration management) Vet ideas for NIC in extensive planning process Pick motivated, complementary, communicative core partners Support extensive face-to-face contact for first 6 months Clear tasks / expectations for partner roles Create good multi-channel communications infrastructure (extranet site, e-mail lists, phone conferences, software configuration management)

37. Copyright © 2000. SRI International. All rights reserved. Lessons Learned (Ongoing) Structure IP cross licensing carefully; hard to both build trust and create the most value from synergy across developers Focus NIC around frequent integration task all participate in (Problem of the Week) Bi-weekly NIC newsletter serves to keep members motivated, yet hard to keep loosely coupled and busy partners on-task Challenge of finding $$ for project coordination Key metric: How easy is it for stakeholders to adopt techniques coming from the NIC? Structure IP cross licensing carefully; hard to both build trust and create the most value from synergy across developers Focus NIC around frequent integration task all participate in (Problem of the Week) Bi-weekly NIC newsletter serves to keep members motivated, yet hard to keep loosely coupled and busy partners on-task Challenge of finding $$ for project coordination Key metric: How easy is it for stakeholders to adopt techniques coming from the NIC?

38. Copyright © 2000. SRI International. All rights reserved. A guiding principle for CTL Developing NICs powerfully amplifies your ability to solve important interdisciplinary problems in education no one can solve alone, and... Requires and grows champions Establishes Centers of Excellence Seeds business development (partners and funders come back to us) Grows name and leadership recognition Supports recruitment

39. Copyright © 2000. SRI International. All rights reserved. Cross-NICing and Meta-NICs? Large challenges due to demanding natures of NICs and domain differences NOTE: each of our education NICs had different mechanisms and solutions: TAPPED IN: virtual places; member services PALS: web-based digital library and training tools OERL: digital library of case-study models, and tools CILT: F2F workshops; CILT-KN; partnership seed funding ESCOT: interoperability testbed; POW integration teams All needed activities for integrating knowledge and incenting contributions and participation. Large challenges due to demanding natures of NICs and domain differences NOTE: each of our education NICs had different mechanisms and solutions: TAPPED IN: virtual places; member services PALS: web-based digital library and training tools OERL: digital library of case-study models, and tools CILT: F2F workshops; CILT-KN; partnership seed funding ESCOT: interoperability testbed; POW integration teams All needed activities for integrating knowledge and incenting contributions and participation.