1. Follow-up Monitoring and Evaluation Systems for National R&D at NEDO -Part 1- ~Results from follow-up monitoring~ American Evaluation Association Conference October 28, 2005 Kazuaki Komoto, kohmotokza@nedo.go.jp Shuji Yumitori, Hiroshi Sano, Takahisa Yano

2. Follow-up Monitoring & Evaluation ・ Introduction ・ Outline of the Method Used ・ Activities in FY2004 1) Results of Simplified Monitoring 2) Results of Detailed Monitoring ・ Activities in FY2005 1) Results of Simplified Monitoring ・ Conclusion of Monitoring ・ Evaluation CONTENTS

3. This presentation explains the following: ☞ An outline of the method of follow-up monitoring & evaluation. ☞ The results of follow-up monitoring and the effects of NEDO’s projects. ☞ The evaluation of the results of such monitoring. Introduction

4. The objective of this new initiative are: To assess the impact of NEDO’s R&D projects on the technological progress in the relevant fields, by monitoring post-project activities of project participants toward practical applications of R&D achievements, To provide feedback to improve R&D management, by reviewing previous post-project evaluations in view of post-project activities, and To contribute to the planning future of R&D strategies, by understanding technology trends in post-project research activities. Follow-up Monitoring & Evaluation Outline of the Method

5. Grasp the current situation of an organization’s activities 1) Basically annually for five years 2) Through data collection 3) In addition, if there are changes in contact person in the organization Step 1 Simplified monitoring Detailed monitoring based on simplified monitoring Questionnaires or interviews (sales amount, market share, reason for discontinuance, other spin-off effects, etc.) Step 2 Detailed monitoring Implementation under the evaluation committee 1 ） To assess the impact of national R&D projects 2) Improved R&D management 3) Planning future R&D strategies Step 3 Evaluation Follow-up Monitoring & Evaluation Process Key questions: ・ Has this R&D accomplished the final target ?, ・ Has there been delay of 2 years against the pre-set target?, or ・ Are there any other necessity?

6. ☞ Target projects All 27 projects completed in FY2001 All 29 projects completed in FY2002 ☞ Target organizations Private companies, research institutes, public entities, universities ☞ Questionnaires  For companies ・ Progress of the following activities: ☞ Research ☞ Development ☞ Practical application ☞ Commercialization ・ Paper, awards received, names of products, Far-reaching effects ・ Competition situation (precedence/equal/delay)  For universities ・ Research activities Total:56 projects Total:642 organizations -Areas of monitoring- ○ Electronics and information ○ Nanotechnology and materials ○ Biotechnology and medical technology ○ Machinery systems ○ Environment ○ Fuel cell and hydrogen technology ○ New energy ○ Energy conservation Follow-up Monitoring & Evaluation Simplified Monitoring

7. ◆ 541 responses have been received to date ◆ Disaggregated data are as follows; *Organizations being monitored Private Companies Research Institutes Public Entities Universitie s Total 333 (351) * 17 (24) * 48 (50) * 143 (217) * 541 (642) Follow-up Monitoring & Evaluation Results of Simplified Monitoring in FY2004 Response Rate by Respondents Public Entities (96%) Total (84%) Private Companies (95%) Universities (66%) Research Institutes (71%)

8. New Energy & Industrial Technology Organization Entity (Private Company) Basis Analysis 1) 64% of the private companies are currently conducting post-project activities. However, 36% of the private companies have no post- project activities. 2) 22% of the private companies have reached the stage of practical application or commercialization of R&D achievements. Follow-up Monitoring & Evaluation Results of Simplified Monitoring in FY2004 (cont.) ３３３ No Post-project Activity* １２０ （３６％） Post-project Activity ２１３ （６４％） Research ７６ (36%) Discontinued １２ (5%) Development ７９ (37%) Practical Application & Commercialization ４６ (22%) Total Number of Responses Post-project Status Stage of Post-project Activity Number of private companies *No Post-project activity : No follow-up activities planned after NEDO ’ s project is completed.

9. ☞ Motive for participation in NEDO ’ s project Practical application & commercialization (n=78) No post-project activity (n=246) Follow-up Monitoring & Evaluation Results of Detailed Monitoring in FY2004 (cont.) ◆ Motivations of companies carrying out practical application and commercialization matched with a main purpose of the company. ◆ Companies with no post-project activity aimed to challenge a new technical target and to improve the level of technology. ※ Multiple answers allowed Matched a main purpose of the organization Challenge a new technical target Invitation of other organization Matched a main purpose of the organization Challenge a new technical target Invitation of other organization Improvement of technology Create network Personnel training Other Personnel training Other

10. Follow-up Monitoring & Evaluation Results of Detailed Monitoring in FY2004 (cont.) ☞ Timing of technical seed occurrence ◆ Companies in the practical application and commercialization stage had developed technology seeds before the project started. ◆ However, most companies which have discontinued the post-project activity had tried to develop the technical seeds in the project phase. Discontinued the Post- Project Activity During NEDO’s project (29%) Before NEDO’s project started (67%) Practical Application & Commercialization During NEDO’s project (70%) Before NEDO’s project started (30%) (n=48)(n=10) Other (4%)

11. ◆ After achieving practical application or commercialization, 37% of monitored companies established new enterprises such as new R&D departments, spinoffs and/or joint ventures. Contribution to establishment of a new enterprise Effect on R&D policy of companies ◆ Growth of participating company R&D departments was recognized in 32% of monitored companies, and related R&D projects have been launched in 29% of monitored companies. Follow-up Monitoring & Evaluation Results of Detailed Monitoring in FY2004 (cont.) (n=38) (n=41) Others No effect Establish a related R&D project Growth of participating R&D departments Effect on R&D activities of companies (n=41)

12. New Energy & Industrial Technology Organization ◆ More than 90% of monitored companies recognized some effect on job creation after achieving practical application or commercialization. However, the effect was limited. Contribution to job creation Impact on academic field ◆ More than 70% of monitored companies recognized that the results obtained from NEDO’s R&D project had some effect on the academic field, and 34% answered it was significant. Follow-up Monitoring & Evaluation Results of Detailed Monitoring in FY2004 (cont.) (n=38) (n=36)

13. New Energy & Industrial Technology Organization Expectations regarding NEDO’s R&D Project ◆ Expectations of monitored companies regarding NEDO’s project were different before and after the project. ◆ Before NEDO’s project, company expectations were focused on technology improvement and/or product performance, as well as engaging in high-risk R&D. ◆ However, during the project, companies indicated that NEDO’s project was useful for technology improvement and/or product performance, as well as formation of a network of personal contacts. ◆ After completing NEDO’s project, companies saw providing support for high-risk R&D as the most important role of NEDO’s projects. Follow-up Monitoring & Evaluation Results of Detailed Monitoring in FY2004 (cont.) Improvement of quality of researchers Improvement of technology /products Support for high-risk R&D Formation of network of personal contacts Other/Nothing Special Before NEDO project During NEDO project After NEDO project ※ Multiple answers allowed

14. Follow-up Monitoring & Evaluation Results of Simplified Monitoring in FY2005 Number of Projects and Entity Monitoring in FY2005 FY2001 & FY2002 FY2003FY2004Total Number of Projects completed 563321110 Number of Organizations 5294692041202 Remarks ・ FY2005 was the second year of monitoring. ・ Monitoring was for post- project activities. ・ FY2005 was the first year of monitoring. ・ Monitoring was for post- project activities. ・ FY2005 was the first year of monitoring. ・ Monitoring was for final targets of post- project. - Preliminary results

15. Total number of responses Post-project status Stage of Post- project activity No Post- project Activity Research Discontinued Development Practical Application Post-project activity Commercializat ion Research Unanswered Development Practical Application Commercializat ion Number of private companies Private company monitoring completed in FY2001 & FY2002 Monitoring in FY2004 ◆ Four companies with no activities in 2004 started post-project activities in 2005. ◆ Differences between 2004 and 2005 regarding the stage of post- project activities are explained in the next chart. Monitoring in FY2005 Follow-up Monitoring & Evaluation Results of Simplified Monitoring in FY2004 & FY2005 Progress of Post-project Activities of Private Companies No Post- project Activity Post-project activity Post-project status Stage of Post- project activity Preliminary results

16. Follow-up Monitoring & Evaluation Progress of Post-project Activity at Private Companies Monitored in FY2004 & FY2005 Progress of Post-project Activities of Private Companies ◆ Relatively many companies, that is 22% of the total reach the practical application stage or the commercialization stage. ◆ Out of 69 companies in the research stage and 67 companies in the development stage in the FY2004, only 2% went up to the practical application and commercialization stage in FY2005 ◆ Progress of post-project activities toward practical application and commercialization stage does not necessary correlate with the time, meaning there is no linear relationship between time and progress of activities. Preliminary results

17. Follow-up Monitoring & Evaluation Results of Simplified Monitoring in FY2005 (cont.) 255 174 81 15 17 63 41 15 23 0 100 200 300 企 業 数 回答数継続事業の有無継続事業の内訳 Total number of responsesPost-project status Stage of post-project activity No Post- project activity Research Unanswered Development Practical Application Post-project activity Commercialization Discontinued Number of private companies Private company monitoring Completed in FY2003 There is no significant difference between FY2003 and FY2001/FY2002 Progress of Post-project Activities of Private Companies Preliminary results

18. Follow-up Monitoring & Evaluation Preparation for Simplified Monitoring in FY2005 80 59 19 22 14 15 8 0 100 企 業 数 回答数継続事業の有無継続事業の目標設定 Total number of responses Post-project status Target of post-project activity No Post-project activity Research Unanswered Development Practical Application Post-project activity Commercialization Number of private companies Private company monitoring Completed in FY2004 ◆ More than 70% of companies have plans to carry out some post- project activity. ◆ 43% of these companies plan to continue to attain practical applications or commercialization. Targets after a Project Preliminary results

19. Collaboration after NEDO ’ s Project (Responses from university participants) 42% Private companies participating in NEDO projects 52% 58% Other private companies 48% Industry-Academia Collaboration Follow-up Monitoring & Evaluation Results of Simplified Monitoring in FY2005 No Yes Monitoring completed in FY2003, 2004 & 2005 ◆ After the completion of NEDO projects, half of the university participants still collaborate with private companies, half of which participate in new NEDO projects. ◆ These results indicate that NEDO’s projects may provide a good opportunity for collaboration between industry and academia. Preliminary results

20. Follow-up monitoring of post-project activities to find out the effect of NEDO projects For FY2001 & FY2002 projects, 64% of private companies are carrying out post-project activities, and 22% of them have reached practical application & commercialization. There is no significant difference between FY2003 and FY2001/ 2002. Expectations of monitored companies regarding NEDO’s projects ☞ Before : Improvement of technology, challenge high-risk R&D ☞ During : Improvement of technology, Formation of a network of personal contacts ☞ After : Promotion of high-risk R&D 1) 2) Follow-up Monitoring & Evaluation Conclusion of monitoring (1/2)

21. The progress of post-project activities of private companies monitored in FY2004 and FY2005 is : 3) Follow-up Monitoring & Evaluation Conclusion of monitoring (2/2) ☞ Relatively many companies, that is 22% of the total reach the practical application stage or the commercialization stage in FY2004. ☞ Out of 69 companies in the research stage and 67 companies in the development stage in the FY2004, only 2% went up to the practical application and commercialization stage in FY2005. ☞ Progress of post-project activities toward practical application and commercialization stage does not necessary correlate with the time, meaning there is no linear relationship between time and progress of activities.

22. Follow-up Monitoring & Evaluation Evaluation Assess the impact of government-supported R&D projects As a result of monitoring of the post-project activities of project participants, we were able to clarify the effect of NEDO ’ s projects. Improve R&D management This will be described in Part 2. Plan R&D strategies Continuation of monitoring and data collection.

23. Thank you very much for your kind attention!

24. N O T E