Presentation is loading. Please wait.

Presentation is loading. Please wait.

Using Logic Models in Managing Performance of Research and Technology Programs: An example for a Federal energy efficiency and renewable energy program.

Similar presentations


Presentation on theme: "Using Logic Models in Managing Performance of Research and Technology Programs: An example for a Federal energy efficiency and renewable energy program."— Presentation transcript:

1 Using Logic Models in Managing Performance of Research and Technology Programs: An example for a Federal energy efficiency and renewable energy program IAMOT 13 th International Conference on Management of Technology April 4, 2004 Gretchen Jordan, Sandia National Labs John Mortensen, Independent Consultant John Reed, Innovologie George Teather, Independent Consultant Work presented here was completed for the U.S. DOE Office of Energy Efficiency and Renewable Energy by Sandia National Laboratories. Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy ( DOE) under Contract DE-AC04-94AL8500. Opinions expressed are solely those of the authors.

2 April 4, 20042 Presentation Overview Managers are under increasing pressure to demonstrate the value of their programs Logic models can help programs identify and explain their value Logic models can include strategies that represent research, development, demonstration and deployment (RDD&D) stages Logic model example for federal energy program Logic models help programs identify: –Indicators –Performance targets –Evaluation questions

3 April 4, 20043 There is increasing pressure on RDD&D programs to demonstrate value U.S. Requirements: Government Performance and Results Act of 1993 (GPRA) requires agencies to develop: –Strategic plans –Annual performance plans –Annual performance reports OMBs Program Assessment Rating Tool (PART) scores programs on: –Program purpose and design –Strategic planning –Program management –Program results

4 April 4, 20044 Logic models help programs identify value A logic model is a diagram or table describing how a program will solve identified problems Elements of a logic model include: –Resources (Inputs) –Activities –Outputs –Customers reached –Outcomes Short Intermediate Long-term –External influences

5 April 4, 20045 Basic logic model structure Resources (Inputs) ActivitiesOutputs Short-Term Outcomes Intermediate Outcomes (through customers) Long-Term Outcomes & Problem Solution for Customers Reached External Influences and Related Programs (mediating factors)

6 April 4, 20046 A logic model example A logic model was developed for the U.S. Department of Energys Office of Energy Efficiency and Renewable Energy (EERE) EEREs goals are to: –Modernize energy conservation –Increase energy supplies –Modernize our critical energy infrastructure

7 April 4, 20047 EERE has 7 strategies for achieving its goals 1.Plan and assess programs 2.Develop and maintain program infrastructure 3.Conduct research 4.Develop technology 5.Demonstrate technology 6.Develop government and market infrastructure 7.Deploy technology

8 April 4, 20048 The 7 strategies cover stages in the RDD&D spectrum and are not necessarily linear Program planning & assessment Conduct research Develop technology Demonstrate technology Deploy technology Develop & maintain program infrastructure Developing government & market infrastructure Feedback Loops RDDD

9 April 4, 20049 The strategies are represented as activities in the draft logic model Program planning & assessment Conduct research Develop technology Demonstrate technology Deploy technology Develop & maintain program infrastructure Developing government & market infrastructure Feedback Loops Inputs Activities Outputs Outcomes For

10 April 4, 200410 EEREs draft logic model shows how its strategies/activities are linked to its goals Inputs Activities Outputs Outcomes Program planning & assessment Conduct research Develop technology Demonstrate technology Deploy technology Federal, state & local government funding Private funding, Personnel, Facilities, Past R&D results Develop & maintain program infrastructure Developing government & market infrastructure Benefit estimates, Priorities identified, Budget requests, Program plans New knowledge, proof of concepts as represented by data, publications Technology prototypes -initial -intermediate -commercial Performance analysis Test, improve, & validate commercial- scale technology, Give industry hands-on experience Government purchases, Information disseminated, Early seeding of technologies Public & private labs and test beds, Knowledge bases, Trained S&T personnel, Partnerships Codes and standards, Trained personnel, Audits tools, State programs Concepts & designs with possible applications, Knowledge spill-over Investment by industry in innovative or advanced commercial products Favorable policies, capable delivery channels for EERE products Widespread adoption of EERE products; More productive use of energy For Economic, security, and environmental benefits Technology leadership Programs, CFO, OMB, Congress Programs, partners R&D community R&D Community, Industry Relevant industries Relevant markets Potential purchasers Potentially commercializ- able technologies to replace existing or fill a system need Political environment Quality of R&D proposals Unpredictable nature of R&D Cost and performance of competing technologies Industry willingness to take risk Energy prices State of the economy Government policies and regulations External Influences Spin-off products and their associated benefits New products & businesses Program funding in appropriate areas; Efficiency, Fiscal responsibility Relevant S&T expertise, capabilities and facilities to deliver programs Feedback Loops National R&D capabilities, including options if circumstances change

11 April 4, 200411 Logic models help programs identify: Measurement areas for which indicators (metrics) may be developed Performance targets for each indicator Evaluation questions

12 April 4, 200412 Each box in the logic model is a potential measurement area Inputs Activities Outputs Outcomes Program planning & assessment Conduct research Develop technology Demonstrate technology Deploy technology Federal funding (millions of nominal $)Federal personnel (FTEs) Private funding (millions of nominal $)# of RD&D facilities Develop & maintain program infrastructure Developing government & market infrastructure % programs w/benefit estimates % program w/program plans # of journal articles # of presentations # prototypes -initial -intermediate -commercial Prototype cost & performance # and % of commercial- scale technologies validated # of techs purchased by govt, # of materials disseminated, # of website hits # of partnerships # codes and standards, # personnel trained, # audits, # state programs # journal article citations # of innovative or advanced commercial products with improved cost & performance # of recommenda- tions for using advanced commercial products and practices # and % of advanced commercial products by adoption stage For Energy saved (quad. Btu), GW of addl RE capacity, Expenditure savings ($) Carbon saved (mmtce) Programs, CFO, OMB, Congress Programs, partners R&D community R&D Community, Industry Relevant industries Relevant markets Potential purchasers # of potentially commercializ- able technologies Cost and performance of competing technologies (varies by technology) Oil prices ($/barrel) NG prices ($/tcf) Electricity prices (c/kWh) Coal prices ($/ton) GDP (billion 1996 $) RE production tax credit (c/kWh) EE/RE tax credits ($) CAFÉ standards (mpg) Vehicle & power plant emission standards (varies by pollutant) External Influences # of technology spinoffs Feedback Loops

13 April 4, 200413 Inputs Activities Outputs Outcomes Program planning & assessment Conduct research Develop technology Demonstrate technology Deploy technology Federal funding (millions of nominal $)Federal personnel (FTEs) Private funding (millions of nominal $)# of RD&D facilities Develop & maintain program infrastructure Developing government & market infrastructure % programs w/benefit estimates % program w/program plans # of journal articles # of presentations # prototypes -initial -intermediate -commercial Prototype cost & performance # and % of commercial- scale technologies validated # of techs purchased by govt, # of materials disseminated, # of website hits # of partnerships # codes and standards, # personnel trained, # audits, # state programs # journal article citations # of innovative or advanced commercial products with improved cost & performance # of recommenda- tions for using advanced commercial products and practices # and % of advanced commercial products by adoption stage For Energy saved (quad. Btu), GW of addl RE capacity, Expenditure savings ($) Carbon saved (mmtce) Programs, CFO, OMB, Congress Programs, partners R&D community R&D Community, Industry Relevant industries Relevant markets Potential purchasers # of potentially commercializ- able technologies Cost and performance of competing technologies (varies by technology) Oil prices ($/barrel) NG prices ($/tcf) Electricity prices (c/kWh) Coal prices ($/ton) GDP (billion 1996 $) RE production tax credit (c/kWh) EE/RE tax credits ($) CAFÉ standards (mpg) Vehicle & power plant emission standards (varies by pollutant) External Influences # of technology spinoffs Feedback Loops Performance targets may also be developed for each box in the logic model Cost of Hydrogen ($/gallon gasoline equivalent) 200320062010 Non- renewables 5.003.001.50 Renewables6.205.303.90

14 April 4, 200414 Arrows between the boxes help identify evaluation questions Activities Outputs Outcomes For Demonstrate technology Test, improve, & validate commercial- scale technology, Give industry hands-on experience Investment by industry in innovative or advanced commercial products Relevant industries Evaluation Questions: Has industry experience lead to investment in these innovative or advanced commercial energy products? Have relevant industries gained hands-on experience with the technologies? Have EEREs efforts to test and demonstrate energy technologies led to validated commercial scale technologies? (Activities to outcomes) Has EERE engaged the right partners in tests and done it efficiently?

15 April 4, 200415 Potential management / evaluation questions for EERE ask if strategies are working 1.Have program planning and assessment activities increased potential program benefits per federal dollar spent? 2.Has developing and maintaining EERE program infrastructure provided the scientific and technical expertise and facilities required to carryout program activities? 3.Has research conducted by EERE yielded energy-related concepts and designs with possible commercial applications?

16 April 4, 200416 Management / evaluation questions (cont.) 4.Has EERE technology development yielded potentially commercializable energy technologies? 5.Have EERE efforts to test and demonstrate energy technologies led to investment by industry in these innovative or advanced commercial energy products? 6.Have EERE efforts to develop government and business infrastructures led to favorable policies and capable delivery channels for EERE products and practices?

17 April 4, 200417 Management / evaluation questions (cont.) 7.Have EERE deployment activities with end users increased the awareness, appreciation, and adoption of EERE products and practices? And, altogether has the EERE portfolio of RDD& D programs led to adoption of EERE products and practices and yielded economic, security, and environmental benefits that would not have occurred otherwise?

18 April 4, 200418 Inputs Activities Outputs Outcomes Program planning & assessment Conduct research Develop technology Demonstrate technology Deploy technology Federal funding (millions of nominal $)Federal personnel (FTEs) Private funding (millions of nominal $)# of RD&D facilities Develop & maintain program infrastructure Developing government & market infrastructure % programs w/benefit estimates % program w/program plans # of journal articles # of presentations # prototypes -initial -intermediate -commercial Prototype cost & performance # and % of commercial- scale technologies validated # of techs purchased by govt, # of materials disseminated, # of website hits # of partnerships # codes and standards, # personnel trained, # audits, # state programs # journal article citations # of innovative or advanced commercial products with improved cost & performance # of recommenda- tions for using advanced commercial products and practices # and % of advanced commercial products by adoption stage For Energy saved (quad. Btu), GW of addl RE capacity, Expenditure savings ($) Carbon saved (mmtce) Programs, CFO, OMB, Congress Programs, partners R&D community R&D Community, Industry Relevant industries Relevant markets Potential purchasers # of potentially commercializ- able technologies Cost and performance of competing technologies (varies by technology) Oil prices ($/barrel) NG prices ($/tcf) Electricity prices (c/kWh) Coal prices ($/ton) GDP (billion 1996 $) RE production tax credit (c/kWh) EE/RE tax credits ($) CAFÉ standards (mpg) Vehicle & power plant emission standards (varies by pollutant) External Influences # of technology spinoffs Feedback Loops Evaluations can explain why some goals were met and others were not (1) Funding was about what was expected (2) R&D yielded expected results (3) But, competing technologies improved more than expected (4) Energy prices were lower than expected (5) Thus, market penetration was slower than expected (6) And benefits were lower than expected

19 April 4, 200419 In summary, logic models for RDD&D programs help identify… Inputs Activities Outputs Outcomes Program planning & assessment Conduct research Develop technology Demonstrate technology Deploy technology Federal, state & local government funding Private funding, Personnel, Facilities, Past R&D results Develop & maintain program infrastructure Developing government & market infrastructure Benefit estimates, Priorities identified, Budget requests, Program plans New knowledge, proof of concepts as represented by data, publications Technology prototypes -initial -intermediate -commercial Performance analysis Test, improve, & validate commercial- scale technology, Give industry hands-on experience Government purchases, Information disseminated, Early seeding of technologies Public & private labs and test beds, Knowledge bases, Trained S&T personnel, Partnerships Codes and standards, Trained personnel, Audits tools, State programs Concepts & designs with possible applications, Knowledge spill-over Investment by industry in innovative or advanced commercial products Favorable policies, capable delivery channels for EERE products Widespread adoption of EERE products; More productive use of energy For Economic, security, and environmental benefits Technology leadership Programs, CFO, OMB, Congress Programs, partners R&D community R&D Community, Industry Relevant industries Relevant markets Potential purchasers Potentially commercializ- able technologies to replace existing or fill a system need Political environment Quality of R&D proposals Unpredictable nature of R&D Cost and performance of competing technologies Industry willingness to take risk Energy prices State of the economy Government policies and regulations External Influences Spin-off products and their associated benefits New products & businesses Program funding in appropriate areas; Efficiency, Fiscal responsibility Relevant S&T expertise, capabilities and facilities to deliver programs Feedback Loops National R&D capabilities, including options if circumstances change Performance Targets Cost of Hydrogen ($/gallon gasoline equivalent) 200320062010 Non- renewables 5.003.001.50 Renewables6.205.303.90 Indicators Cost of Hydrogen ($/gal gasoline equivalent) –Non-renewables –Renewables Evaluation Questions Did development activities lead to potentially commercializable technologies?

20 April 4, 200420 … and tell the performance story Inputs Activities Outputs Outcomes Program planning & assessment Conduct research Develop technology Demonstrate technology Deploy technology Federal, state & local government funding Private funding, Personnel, Facilities, Past R&D results Develop & maintain program infrastructure Developing government & market infrastructure Benefit estimates, Priorities identified, Budget requests, Program plans New knowledge, proof of concepts as represented by data, publications Technology prototypes -initial -intermediate -commercial Performance analysis Test, improve, & validate commercial- scale technology, Give industry hands-on experience Government purchases, Information disseminated, Early seeding of technologies Public & private labs and test beds, Knowledge bases, Trained S&T personnel, Partnerships Codes and standards, Trained personnel, Audits tools, State programs Concepts & designs with possible applications, Knowledge spill-over Investment by industry in innovative or advanced commercial products Favorable policies, capable delivery channels for EERE products Widespread adoption of EERE products; More productive use of energy For Economic, security, and environmental benefits Technology leadership Programs, CFO, OMB, Congress Programs, partners R&D community R&D Community, Industry Relevant industries Relevant markets Potential purchasers Potentially commercializ- able technologies to replace existing or fill a system need Political environment Quality of R&D proposals Unpredictable nature of R&D Cost and performance of competing technologies Industry willingness to take risk Energy prices State of the economy Government policies and regulations External Influences Spin-off products and their associated benefits New products & businesses Program funding in appropriate areas; Efficiency, Fiscal responsibility Relevant S&T expertise, capabilities and facilities to deliver programs Feedback Loops National R&D capabilities, including options if circumstances change

21 April 4, 200421 For more information contact: Gretchen Jordan Sandia National Laboratories 950 L'Enfant Plaza, SW Suite 110 Washington, DC 20024-2123 Phone:(202) 314-3040 Email:gbjorda@sandia.govgbjorda@sandia.gov John Mortensen:jmort@prodigy.netjmort@prodigy.net John Reed:jreed@innovologie.comjreed@innovologie.com George Teather:gteather@sympatico.cagteather@sympatico.ca


Download ppt "Using Logic Models in Managing Performance of Research and Technology Programs: An example for a Federal energy efficiency and renewable energy program."

Similar presentations


Ads by Google