1. Janet E. Halliwell

2. Presentation overview The big picture - measuring the return on S&T investments (ROI) Context, challenges and issues, variables, methodologies, trends Innovation and what our understanding means for assessing ROI The data challenge Wrap-up

3. The ROI context Increasing pressure to measure impacts of public S&T investments (nothing new here!) What do researchers tend think about (especially for P&T)? Inputs (e.g., funding raised) Research productivity Numbers of HQP Perhaps commercialization What are institutions most interested in? Competitiveness for students, prestige and funds Costs - impacts on their bottom line What is the larger public interest? Quality of the PSE system Larger social and economic impacts from R&D and service

4. ROI measurement challenges Very diverse languages and expectations on what is meant by ROI No universal framework or universally applied methodologies Measurement of ROI needs to encompass diverse dimensions of impact: Economic (e.g. jobs, new products and services, spin off companies, business process change) Social and health (e.g. changes in policy and practice, improved outcomes, costs avoided) Environmental (e.g. reduced footprint and environmental impact, branding Canada green) In addition to practical stumbling blocks of measurement of these, interpretation of any measures is non-trivial And all of the above does not necessarily measure the full impact on innovation or the innovation system

5. Some issues ROI measurement requires us to think about what happens down the value chain as a result of the research and research related activities - beyond the quality, volume and influence of research on other research – e.g. what difference did this S&T investment make in the real world ROI measurement is NOT a classical economic I/O study (which measures the flow of monies resulting from an activity regardless of what that activity is) A theoretically-sound ROI method is poor if key stakeholders are not consulted or dont understand it

6. Variables to think about Your audience/target Scope and level of aggregation Distance down the value chain of outputs, outcomes and impacts Time scale (how far back) Methodologies Desired detail; how to communicate (e.g. visualize) Balancing accuracy, longevity, comparability and ease of collection of metrics

7. Downstream measurment … Categories relevant to innovation include (at least): Direct – Using research findings for better ideas, products, processes, policies, practice etc. Indirect – From participation in the research, including HQP training, KT, tacit knowledge, better management, etc. Spin-off – Using findings in unexpected ways and fields Knock-on – Arising far after the research is done Also very important – outcomes that foster an environment in which innovation flourishes

8. Example methodologies Quantitative Surveys Bibliometrics, including publication counts, citation analysis, data mining, international collaboration analysis, social network analysis Technometrics, including hotspot patent – publication linkages Economic rate of return – micro and macro levels Sociometrics Benchmarking Qualitative Peer Review, Merit/Expert Review Case study method – exploratory, descriptive, explanatory Retrospective documentary analysis Interviews Mixed models (e.g. Payback, OMS, CAHS)

9. Trends Mixed methods Increasing attention to networks, linkages, collaborations Global frame of reference Involvement of stakeholders inside and external to R&D unit External focus – e.g. short-term external impacts for industry or government, rather than immediate outcomes for the R&D organization

10. What is innovation? Doing new things in new ways Tom Jenkins Innovation is (or should be) a very broad term, BUT … Many studies focus only on the easiest metrics to measure – not innovation relevant issues Or, they focus exclusively on industrial impacts such as sales of new products So …. Encourage other routes and types of impacts Attempt to measure them, including cost savings Encourage end-goal thinking

11. And remember Innovation comes in many different guises, e.g. Incremental innovation Integrated innovation Open innovation Informal innovation Social innovation Design as innovation Consider measurement in the innovation context

12. The macro level messages Measurement is important – but NOT just any measurement Need to ground measurement in a strong conceptual framework connecting activities ultimate goals, intended uses, and both targeted users (logic models) Then look at relationships of outcomes and impacts with innovation in your sector or sphere of activity Measurement is BOTH qualitative and quantitative Proper measurement often takes deliberate effort and time

13. Why quantitative and qualitative Quantitative for understanding reach, scope, & importance of impacts Qualitative for the how and why of impacts, barriers & solutions, incrementality and attribution, government, societal and environmental effects, etc. There is no such thing as a purely quantitative system that measures full impacts of S&T

14. Reporting SE impacts ImpactInvestigated and Described in: Narrative fashionQuantitative termsDollar terms Qualitative Quantitative Economic

15. And … Consider outcome mapping a la IDRC – where the focus is on people and organizations. Go beyond simply assessing the products of a project/program to focus on changes in behaviours, relationships, actions, and/or activities of the people and organizations with whom a program works This is a learning-based and use-driven approach Recognizes the importance of active engagement of players in adaptation and innovation – productive interactions

16. The data challenge (1) Need a good MIS at levels of researcher and project/activity - one that connects with your Network/Centre goals Need to integrate in the MIS the needs of your reporting requirements, accountability plans and Centre/Network self monitoring/self-learning Tie the MIS to performance measurement system by having automatic reports produced, red flags etc Need a foundation of data standards

17. The data challenge (2) Standards can help: Reduce burden on researchers Facilitate the interface with funders Access cost effective software solutions Comparisons with self over time Comparisons with other institutions International benchmarking CASRAI is a large part of the standards picture

18. Customized and flexible methodologies There are plenty of metrics and methods available No need to invent any more (although you will likely need to intensify your data collection) Its how metrics and narrative are used and combined that make the difference No one size fits all methods or metrics work for all types of S&T, for all types of organizations, or for all uses and users All methods have substantial strengths and substantial weaknesses Involve key stakeholders Remember that innovation requires many players, not just the R&D team

19. Measurement can make a difference Accountability and advocacy - Making the case on the basis of outcomes and impacts: For overall program funding For the nature and dynamics of the staff complement Self awareness and understanding: Internal - Strengths, weaknesses, gaps External – Threats, opportunities Forward directions/areas needing attention Fine tuning the strategic vision Fostering sustainable relationships

20. Finally … To achieve these objectives, you need: Good (and visionary) governance Good management - capable staff complement Robust database with in-house expertise Active engagement of players in using the outcomes measures for adaptation and innovation Measurement is quantum– It changes the system; you tend to get what you ask people to measure

21. Thank you jehalli@telus.net