Ron Sokol, M.D. Principal Investigator, CCTSI Bonnie Walters Executive Director, The Evaluation Center Kathryn Nearing, Ph.D. Associate Director, The Evaluation Center CCTSI is supported in part by Colorado CTSA grant 5UL1RR from NCRR/NIH

The Evaluation Center 2010  Two report forms  Meeting report form (specific to each core: features list of meetings, associated participants, aims and “active” indicators)  Director report form (generic; completed by directors quarterly)  Performance dashboards  Feature aims, indicators specific to each core, current phase of grant  Each indicator has a status designation that supports progress monitoring  Master Calendar

The Evaluation Center 2010 meeting event master calendar with link to form

The Evaluation Center 2010

 Three summaries (generated monthly)  Meeting summary (how many of which meetings occurred)  Summary of indicators addressed and key decisions  Summary of participants within and across cores, as well as affiliate institutions (to support SNA to examine integration, collaboration)  Automated sends a record of what was submitted via electronic forms  Archival summaries to respond to NIH reporting requirements (in development)

The Evaluation Center 2010  Performance Dashboard  Real-time snapshot of how work is progressing during the current phase of the grant (updated monthly)  Stakeholder briefs  Highlight major accomplishments, emerging issues, future priorities, evaluation updates (often reviewed by steering committees)  Other reports may be generated “on the fly” by creating tailored views of data in SharePoint  Customized views can be created by any user of the system (with the proper permissions)

The Evaluation Center 2010 Document meetings (ongoing) Review summaries (monthly), director reports (quarterly) Update dashboards (monthly) Generate stakeholder briefs (quarterly)

The Evaluation Center 2010  Briefs created a responsive feedback mechanism  Master calendar  Became a central repository for meeting, event information  Streamlined efforts to advertize CCTSI events  Indicators became more visible as an organizing framework to structure reporting  Underscored the importance of well-written (i.e., “track-able”, “actionable”) indicators  Agendas became more responsive to aims, indicators and the content of briefs

The Evaluation Center 2010  Specific indicators -- critical benchmarks that indicate meaningful progress -- should be:  Written with the long-view in mind  In language understandable to diverse stakeholders  Timelines must be specified so that status designations are meaningful

The Evaluation Center 2010  Buy-in from leadership is essential  Orientation/ training is an ongoing process  Ongoing support should be provided in multiple ways  Ongoing access to tech support is essential  Administrators are integral to the success of the system

The Evaluation Center 2010 The PTS is more than an evaluator’s tool  The system is designed to:  Support leaders and administrators at multiple levels proactively manage (and reflect on) their work  Keep diverse stakeholders informed, thus enhancing their capacity to contribute meaningfully to organizational change

The Evaluation Center 2010 The PTS does not run on its own  The system requires ongoing commitment, time and attention  Requires compliance with reporting requirements  A commitment to adequate, accurate and timely reporting (by both evaluators and program representatives)  A commitment to review reports and utilize information generated

The Evaluation Center 2010  Microsoft Office InfoPath© 2007 (optional)  Microsoft Office SharePoint© server and client access licenses (CALs)  Internet Explorer browser (preferred)

The Evaluation Center 2010  Dedicated PTS available for each CTSA  PTS site is self-contained and self-managed  Each CTSA would need an in-house administrator  We would provide:  Kick-off training and some level of ongoing support  Hosting and some application support We are currently working with NCRR to explore whether funding might be available to support these efforts

The Evaluation Center 2010 Kathryn Nearing Jeff Proctor On the Web