1. 1 A Generic Theory Based Logic Model for Creating Scientifically Based Program Logic Models An Expert Lecture Presented to the American Evaluation Association The Research, Technology, and Development Evaluation TIG Portland, Oregon November 4, 2006 John H. Reed Innovologie, LLC jreed@innovologie.com Work presented here was completed for the U.S. DOE Office of Energy Efficiency and Renewable Energy by Sandia National Laboratories, Albuquerque, New Mexico, USA under Contract DE-AC04-94AL8500. Sandia is operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Opinions expressed are solely those of the authors. This work is preliminary. This work may be cited or used only with permission.

2. 2 Acknowledgements Co-authors –Gretchen Jordan (Sandia National Laboratories) –Ed Vine (Lawrence Berkeley National Laboratory) US Department of Energy Sponsor –Jeff Dowd

3. 3 Overview Next generation logic models The problem US DOE’s Office of Energy Efficiency and Renewal Energy Diffusion of innovations - a theory of change Placement of a theory of change in the logic model A generic theory of change and more domain focused versions Researchable issues deriving from the model Summary and conclusions

4. 4 This Generation of Logic Models The logic model framework is a heuristic based on the performance spectrum (resources, activities, outputs, outcomes) Users apply the heuristic to their understanding of a program The resulting logic model with its description of how target actors and partners respond is dependent on the knowledge and skills of the model developer and may or may not closely resemble reality and may or may not be grounded in accepted social science theory.

5. 5 A Theory Based Logic Model Focuses more attention on outcomes Increases attention and understanding to what target actors and partners do in response to program outputs Grounds this in widely accepted social theory imbuing the logic model with a theory of change rather than asking the user to create one. Makes it easier for program managers and evaluators to: –understand effects their programs are having –develop more realistic models –design and conduct better evaluations

6. 6 The Problem A governmental organization with multiple offices and programs conducting energy research and delivering a broad range of products and services to induce the adoption of efficient technologies and practices The organization regularly measures and tracks large numbers of outputs There are numerous partners and target audiences forming a complex market mosaic The need for accountability requires that outputs of the programs be causally linked to the actions of partners and target audiences which in turn produce effects that result in the desired impacts (outcomes) It is important to predict and measure the effects and impacts and attribute them to the program More consistency is needed in the metrics and measures across programs allowing better defense of the program and roll-up of effects

7. 7 Thereby Reducing energy use and emissions, enhancing productivity, and global security Actions by End-users Actions by Manufac- turers and Business Actions by Public Entities Actions by Knowledge Workers Analyze and Plan Develop Technical Information Assist Public Entities Assist Businesses Outreach and Partner Provide Tools and Technical Assistance Assist and Fund Purchases Build Infrastructure Reviewing and Reporting The Office of Energy Efficiency and Renewable Energy of the US Department of Energy Undertakes Activities to Fund and Promote Adoption Federal, state, local agencies, and public organizations Manufacturers distributors retailers End-user organizations, firms and individuals Technical and other personnel in laboratories, government, firms, colleges, universities To target So that there will be

8. 8 Examples of Actors in Each of Four Domains Knowledge firms / orgs Policy makers / public entities BusinessEnd-users Laboratories Universities Research labs Consultants Media specialists Web developers Modelers Regulatory commissions State energy offices Legislatures Public Goods Charge programs Materials suppliers Product manufacturers Distributors/ wholesalers Utilities Retailers / Builders Service providers i.e., auditors, consultants, Installers / contractors Maintenance contractors Households Commercial building owners Tenants in commercial buildings The buildings of retailers The facilities and processes of industrial firms

9. 9 Examples of Actions in Each of Four Domains Knowledge workers: Assemble a steam system audit program Establish a website Run 10,000 building simulations to assess the potential for zero energy commercial buildings Business: Manufacture an efficient product Distribute a new product line Promote, sell, and install an efficient product (retailer) Train to service a product Public Entities: Efficient public purchasing Adopt efficient codes and standards Establish green ordinances End-user examples: Purchasing and installing an energy efficient household product Make an industrial process more efficient Build a “zero energy” office building

10. 10 Analyze and Plan Develop Technical Information Assist Public Entities Assist Businesses Outreach and Partner Provide Tools and Technical Assistance Assist and Fund Purchases Build Infrastructure Reviewing and Reporting The Office of Energy Efficiency and Renewable Energy of the US Department of Energy Undertakes Activities to Fund and Promote Adoption Federal, state, and local agencies Manufacturers distributors retailers End-user organizations, firms and individuals Technical and other personnel in laboratories, government, firms, colleges, universities Create, advance, and package market and technical knowledge to make energy efficiency more accessible and implementable Change the policies, structure and operation of public entities to smooth the advance energy efficiency Create and enhance products, create and align market channels, enhance marketing, and develop installation and support infrastructures Adopt, replicate, institutionalize, and enculturate energy efficient technologies and practices To target So that these target audiences will Thereby Reducing energy use and emissions, enhancing productivity, and global security

11. 11 Analyze and Plan Develop technical information Assist Public Entities Assist businesses Outreach and Partner Provide tools and technical assistance Assist and fund purchases Build Infrastructure Reviewing and Reporting The Office of Energy Efficiency and Renewable Energy of the US Department of Energy Undertakes Activities to Fund and Promote Adoption Federal, state, and local agencies Manufacturers distributors retailers End-user organizations, firms and individuals Technical and other personnel in laboratories, government, firms, colleges, universities Create, advance, and package market and technical knowledge to make energy efficiency more accessible and implementable Change the policies, structure and operation of public entities to smooth the advance energy efficiency Create and enhance products, create and align market channels, enhance marketing, and develop installation and support infrastructures Adopt, replicate, institutionalize, and enculturate energy efficient technologies and practices To Target So that these target audiences will Thereby Reducing energy use and emissions, enhancing productivity and global security, creating sustainable habitats We need to describe and measure the expected response of: Knowledge workers Public Entities Manufacturing and Business End-users Knowing these activities and their corresponding outputs

12. 12 The Solution A theory based logic model that encompasses the range of program delivery activities from which program managers can choose relevant elements to develop a logic model customized to their own program The inclusion of a widely accepted theory of social change in the outcome space of the logic model that will aid program managers to systematically describe how their program outputs will lead to their desired long term goals

13. 13 Diffusion of innovations model Confirmation Awareness Persuasion Decision Implementation Feedback Continued adoption Later adoption Discontinuance Continued rejection Adoption Rejection Product Characteristics Relative advantage Compatibility Complexity Trialability Observability Characteristics of the decision-making unit Adopter type Personality type Communication behavior Socio-economic status Socio-cultural/market environment Market structure Market segments Prior practice Culture and norms Innovativeness Communication field Broadcast Contagion Source: Everett Rogers 2003 as modified by Innovologie, LLC. 2005

14. 14 Decide to partici- pate in a program Decide to imple- ment practices / measures BroadcastContagion Innovators Early Adopters Early and late majority Participants replicate changes Widespread acceptance Institutional change Incorporate into standard operating procedures Increased knowledge, promotion & advocacy  Government policy, codes, programs support adoption  Product, sales, service are available, high quality, profitable  Production and transaction costs decline External households/ facilities/firms observe and emulate changes A Version of Diffusion of Innovation to Link Outputs and Impacts Energy and non- energy impacts Changed practices directly due to program -Finance / contract -Design/ specification -Implement -Operation / maintenance Awareness of resources, EERE opportunities, technologies and practices Seek information / become persuaded Product value accepted -Relative advantage -Compatibility -Complexity -Trialability -Observability Value confirmed

15. 15 So that there will be actions Analyze and Plan Develop Technical Information Assist Public Entities Assist Businesses Outreach and Partner Provide Tools and Technical Assistance Assist and Fund Purchases Build Infrastructure Reviewing and Reporting The Office of Energy Efficiency and Renewable Energy of the US Department of Energy Undertakes Activities to Fund and Promote Adoption Federal, state, and local agencies Manufacturers distributors retailers End-user organizations, firms and individuals Technical and other personnel in laboratories, government, firms, colleges, universities To Target Thereby Reducing energy use and emissions, enhancing productivity, and global security By End-users By Manufac- turers and Businesses By Public Entities By Knowledge Workers Appropriate versions of the model can be developed for knowledge workers, public entities, and businesses as well

16. 16 BroadcastContagion Innovators Early Adopters Early and late majority Value confirmed -Production goals met -Financial goals met -Demand stable / rising -Profit goals met -Product leadership recognized Increased promotion -Peer firms -Advocacy networks -One-to-one interactions External firms observe changes learn through peer interaction Long term impacts Sustained institutional change -Production meets demand -Economies of scale occur -Product becomes permanent part of product line -Product captures market share -Product development path identified A Generic Version of the Diffusion Model Appropriate to Manufacturers and Businesses Implement production/distribution of products -Finance Capital set aside Subsidies -Design / plan Production facility Distribution channels Showrooms Training -Implement Production facility built Product network established Products produced Train personnel / distributors Products distributed Products stocked Products sold Products serviced -Operation / maintenance Service network established Service / maintenance workers trained Awareness of: -information resources -Technology potential - Program opportunities Information seeking/ persuasion -Product and market research -Persons trained -KSA’s learned -Advice received from peers -Participated in codes and standards setting -Participate in demonstrations Supply production value accepted -Relative advantage -Profit -Goodwill -Market leadership -Compatibility -Manufacturing facility -Existing business lines -Complements business environment -Complexity -Low complexity -Ease of manufacture -Trialability -Demonstration -Pilot Test -Observability Decide to implement practices / measures Decide to participate in a program External firms emulate changes

17. 17 Increased promotion -Advocacy groups -Peer networks -One-to-one interactions Value confirmed -Energy benefits observed -Non-energy benefits supporting value propositions observed Program participants emulates changes within household/facility/firm -Changes at the same site -Changes at another site Widespread acceptance of product among target group -Recognition -Market tipped -Increased market penetration -Product costs decline External households /facilities/firms emulate changes Sustained change -Households Develop energy efficiency orientation Adopts new energy efficiency habits Take advocacy stance -Firms Adopt corporate energy policy Appoint corporate energy manager Incorporate energy efficiency as a criteria for personal performance Adopt energy efficiency as part of competitive strategy Incorporate energy efficiency into branding Take advocacy stance External households /facilities/firms observe changes learn through peer interaction Information seeking/ persuasion -Literature/web search -Audit -Peer recommendation evaluation -Obtain and try tools -Training -Attendance at Showcase / demonstration attendance -Hands-on experience Changed technologies/ practices directly due to the program -Finance Incentives Buy down Life cycle costing used Budgeting / capital set aside Performance contracts Loans -Design/redesign/specif ication System/whole system approaches Purchasing specifications -Implement system redesigns and upgrades Install efficiency measures (insulation, control systems, etc) One-to-one replacement with efficient equipment at failure Early one-to-one replacement -Operation / maintenance Awareness of: -information resources -potential/ efficiency opportunities - program opportunities (program offerings/ labels, etc) -EERE opportunities Product value accepted -Relative advantage -Compatibility -Complexity -Trialability -Observability BroadcastContagion Innovators Early Adopters Early and late majority Decide to participate in a program Decide to implement practices / measures A Generic Version of the Diffusion Model Appropriate to End-users Long term impacts

18. 18 Increased promotion -Advocacy groups -Peer networks -One-to-one interactions Implementation -Finance Incentives Grants Performance contracts Loans Buy downs -Design/redesign/specification Codes and Standards Internal policies Internal efficiency programs - Coordination Coordinated federal, regional, state, local policies Coordinated codes and standards –Appliances –Buildings –Net metering -Trained government infrastructure -More efficient public policies -Energy efficiency programs  Demonstrations  Pilot programs  Audits  Performance contracts  Information  Purchasing  Operation / maintenance Value confirmed -Measure energy savings -Measure and observe non-energy benefits Government entity promotes at other sites or promotes other types of activities Widespread acceptance of product among target group -Recognition -Market tipped -Increased market penetration Awareness of: -information resources -Policy / implementation needs -Information gaps -Training opportunities -Opportunities for coordination -Opportunities to provide citizen services / meet needs -Potential energy efficiency opportunities Other government entities observe and emulate Sustained institutional change Make program a permanent offering Adopt energy efficiency as governmental goal Appoint governmental energy manager Incorporate energy efficiency as a criteria for personal performance Incorporate energy efficiency into branding Incorporate energy efficiency purchasing and building into government related purchases, new construction, remodeling Take advocacy stance Other government entities observe and seek information Product value accepted -Relative advantage Protects commons Social and economic benefits for large numbers of citizens Low cost Low political cost -Compatibility Fits existing policy / legal / administrative frameworks Culturally acceptable Fills market gap or need Constituency or easily developed constituency -Complexity Easily understood Simple administration -Trialability Can be piloted Demonstrated -Observability Effects easily measured Effects measured in a timely way BroadcastContagion Innovators Early Adopters Early and late majority Information seeking/ persuasion -Literature/web search -Technical analyses completed -Policies analyzed -Peer contacts made -Peer recommendations received -Market assessments completed -Training completed -Evaluations completed Decide to participate in a program Decide to implement practices / measures Energy and nonenergy impacts A Generic Version of the Diffusion Model Appropriate to Public Entities

19. 19 Persuasion / information search -More knowledge sought in this area -New research ideas developed -Obtained and tried knowledge tools (e.g., software, curriculum) -Demonstrated/ saw demonstration -Hands-on experience Access Information and are Aware of: -Knowledge gaps -information resources (curriculum, tech reports, etc.) -Program opportunities (program offerings/ labels, etc) -EERE potential/ opportunities Product filter: Knowledge/tool has value -Relative advantage More credible information Information confers - economic benefit - efficiency benefit - production benefit - quality benefit -Compatible With other knowledge/tools Production requirements Organizational goals/systems Culture -Complexity Easy to understand Simplifies existing knowledge -Can be tried by target audience -Benefits of use can be Observed BroadcastContagion Innovators Early Adopters Early and late majority Advocacy Spreads word to others Awareness of confirmed value spreads through -Advocacy groups -Peer networks -One-to-one interactions Implemented: Behavior/ Practices Changed Capital -Larger, stronger knowledge community/ critical mass, core competencies -Invest in more research in this area Design/Plan -Changed knowledge seeking behavior (e.g. research approach, how calculate savings, use different procedures, consider labeling criteria) -Considered limitations, benefits/ opportunities (e.g. in setting standards) Implement -Changed attitude, perception of value used in negotiations, text books, technical specifications -Used tool to measure, choose option, shift loads Maintain - keep data base current, correct, consistent over time (e.g. climate data, wind resources data) Confirm Value Knowledge valued in community Knowledge cited by others Knowledge used by others Replication: Emulate changes within Organization/house hold/ facility/firm -Changes at the same site -Changes at another site Replication by those not directly involved with the program -Widespread acceptance of product among target group - Tip the market - Market penetration increases Sustained Knowledge R&D community, Policy makers, Households, Firms use and share the knowledge routinely in textbooks, technical manuals, regulations, etc. Others observe changes and learn through peer interaction A Generic Version of the Diffusion Model Appropriate to Knowledge Entities

20. 20 Supply production value accepted -Relative advantage Profit Goodwill Market leadership -Compatibility Manufacturing facility Existing business lines Complements business environment -Complexity Low complexity Ease of manufacture A Comparison of three Product Characteristics BusinessPublicKnowledge Knowledge/tool has value -Relative advantage More credible information Information confers - economic benefit - efficiency benefit - production benefit - quality benefit -Compatible With other knowledge/tools Production requirements Organizational goals/systems Culture -Complexity Easy to understand Simplifies existing knowledge Product value accepted -Relative advantage Protects commons Social and economic benefits for large numbers of citizens Low cost Low political cost -Compatibility Fits existing policy / legal / administrative frameworks Culturally acceptable Fills market gap or need Constituency or easily developed constituency -Complexity Easily understood Simple administration

21. 21 AttributeResearchable QuestionMeasure Progress of technology through the stages of diffusion Awareness 1.Has the percentage of the people/firms that are aware of the program’s brand increased, decreased or remained about the same? 2.Has the percentage of people/firms aware of the program’s targeted technologies or practices increased, decreased, or stayed the same, in response to the program 3.How does this compare to non-participants? ･ Percent of people or firms that are brand aware ･ Percent of people aware of the technology or practice Generic Researchable Issues and Questions Matrix for All Concepts

22. 22 An Example of Tracking Projects Through Stages

23. 23 EERE’s Impact Evaluation Framework

24. 24 What we have done Developed a generic model that incorporates a theory of change Adapted that tool to a specific institutional setting Laid it all out in an impact framework that users can use to model their programs and develop an impact evaluation

25. 25 What the tool does - I Provides a common framework for implementation and evaluation across programs in a complex multi- program environment Moves us from simplistic, ad-hoc, and post-hoc explanations of behavior to a scientifically grounded understanding of programs It moves logic modeling from a descriptive analytic posture to a program defining posture Managers in a multi-program environment will share a common understanding of how programs work

26. 26 What the tool does - II Program managers will be better able to understand their programs and define and manage evaluations Common language and approach should help to increase communication about programs and evaluation among all players The model should help to detect details missing from our understanding of programs It should focus attention on the important details rather than all the details In multi-program environments should move us in the direction of: –Common researchable questions –Common metrics –Shared evaluations and evaluation resources

27. 27 Contact information John H. Reed, Principal and Owner, Innovologie, LLC., 305 Summer Garden Way, Rockville, MD 20850-2887, 301 340-8701, jreed@innovologie.com, www.innovologie.com jreed@innovologie.com,