How Logical Is Your Logic Model How Logical Is Your Logic Model? Developing Useful and Robust Project Logic Models 2/27/2014

The work of TEAMS is supported with funding provided by the National Science Foundation, Award Number DRL 1238120. Any opinions, suggestions, and conclusions or recommendations expressed in this presentation are those of the presenter and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content. 2/27/2014

Strengthen the quality of the MSP project evaluation and build the capacity of the evaluators by strengthening their skills related to evaluation design, methodology, analysis, and reporting 2/27/2014

Online Help-Desk for submitting requests Website at: teams.mspnet.org Webinar series targeted to specific evaluation topics Tiered technical assistance for differentiated services Instrument review and sharing 2/27/2014

RMC Research Corporation How Logical Is Your Logic Model? Developing Useful and Robust Project Logic Models Dave Weaver, Director RMC Research Corporation Portland, Oregon February 27, 2014 2/27/2014

Webinar Objectives Provide a rationale for the need for a well-defined logic model Summarize characteristics of a well-defined theory of action and logic model Provide an opportunity to examine your current theory of action and logic model Discuss strategies for developing useful and robust logic models 2/27/2014

Why Are Logic Models Important? Common Guidelines for Education Research and Development U.S. Department of Education, National Science Foundation. (August, 2013). http://ies.ed.gov/pdf/CommonGuidelines.pdf “The proposal should include a description of the initial concept for the planned investigation, including a well-explicated theory of action or logic model. The concept and logic model should identify key components of the intervention and should describe their relationships, theoretically, and operationally.” 2/27/2014

What does it mean? “well-explicated theory of action or logic model” 2/27/2014

It starts with the Theory of Action A logic model is only as good as the theory of action that it is based on 2/27/2014

What Is a Theory of Action? Confronting the Elephant in the Room North Central RTI TA CollaborativeNext Steps in Building Collaborative Technical Assistance in the North Central Region April 2011 February 2009February 2009 April 2009February 2009February 2009 What Is a Theory of Action? Collective belief about causal relationships between action and desired impacts Simple: If . . .  Then . . .  Complex: If  . . .  and . . .  and . . .  and . . . Then  . . .  Collaborative interpretation of the literature 2/27/2014 NWRCC Next Steps in Building Collaborative Technical Assistance in the North Central RegionNorth Central RTI Technicl Assistance Collabortive National Center on Response to InterventionNext Steps in Building Collaborative Technical Assistance in the North Central RegionNorth Central RTI Technicl Assistance Collabortive 10 10

Characteristics of a Robust Theory of Action Confronting the Elephant in the Room April 2011 Characteristics of a Robust Theory of Action Describes project impact as close to the primary target audience as possible Example: A description of what students do to learn Recognizable when it is going on Observable Defines fidelity of project implementation Can be a testable hypothesis Believable Interpretation of current literature 2/27/2014 NWRCC

Math Example (Common Core State Standards) If teachers use developmentally appropriate yet challenging tasks and activities that engage students in: Justifying—Explaining and justifying their reasoning mathematically Generalizing—Identifying and verifying conjectures or predictions about the general case Representing—Using representations (symbolic, notation, graphs, charts, tables, and diagrams) to communicate and explore mathematical ideas Applying—Applying mathematical skills and concepts to real-world applications Then student achievement and interest in mathematics will increase. 2/27/2014

Does it describe project impact as close to the primary target audience as possible? If teachers use developmentally appropriate yet challenging tasks and activities that engage students in: Justifying—Explaining and justifying their reasoning mathematically Generalizing—Identifying and verifying conjectures or predictions about the general case Representing—Using representations (symbolic, notation, graphs, charts, tables, and diagrams) to communicate and explore mathematical ideas Applying—Applying mathematical skills and concepts to real-world applications Then student achievement and interest in mathematics will increase. 2/27/2014

Is it recognizable when it is going on? If teachers use developmentally appropriate yet challenging tasks and activities that engage students in: Justifying—Explaining and justifying their reasoning mathematically Generalizing—Identifying and verifying conjectures or predictions about the general case Representing—Using representations (symbolic, notation, graphs, charts, tables, and diagrams) to communicate and explore mathematical ideas Applying—Applying mathematical skills and concepts to real-world applications Then student achievement and interest in mathematics will increase. 2/27/2014

Does it define fidelity of project implementation? If teachers use developmentally appropriate yet challenging tasks and activities that engage students in: Justifying—Explaining and justifying their reasoning mathematically Generalizing—Identifying and verifying conjectures or predictions about the general case Representing—Using representations (symbolic, notation, graphs, charts, tables, and diagrams) to communicate and explore mathematical ideas Applying—Applying mathematical skills and concepts to real-world applications Then student achievement and interest in mathematics will increase. 2/27/2014

Can it be a testable hypothesis? If teachers use developmentally appropriate yet challenging tasks and activities that engage students in: Justifying—Explaining and justifying their reasoning mathematically Generalizing—Identifying and verifying conjectures or predictions about the general case Representing—Using representations (symbolic, notation, graphs, charts, tables, and diagrams) to communicate and explore mathematical ideas Applying—Applying mathematical skills and concepts to real-world applications Then student achievement and interest in mathematics will increase. 2/27/2014

Is it believable? If teachers use developmentally appropriate yet challenging tasks and activities that engage students in: Justifying—Explaining and justifying their reasoning mathematically Generalizing—Identifying and verifying conjectures or predictions about the general case Representing—Using representations (symbolic, notation, graphs, charts, tables, and diagrams) to communicate and explore mathematical ideas Applying—Applying mathematical skills and concepts to real-world applications Then student achievement and interest in mathematics will increase. 2/27/2014

Science Example Students learn science when they: Articulate their initial ideas, Are intellectually engaged with important science content, Confront their ideas with evidence, Formulate new ideas based on that evidence, and Reflect upon how their ideas have evolved 2/27/2014

Online Survey 1 Does your project have a theory of action statement? If yes, to what extent does it meet the criteria? Criteria for a robust theory of action: Describes impact close to the primary audience Is recognizable in practice Defines fidelity of implementation Can be a testable hypothesis Is believable—Shared interpretation of literature 2/27/2014

Confronting the Elephant in the Room April 2011 What Is a Logic Model? A diagram that shows the logical connection between project resources, activities, outcomes, and expected impacts Incorporates a primary theory of action Can be viewed as a collection of theories of actions Answers the question: Why would the planned activities be expected to have the desired impacts? 2/27/2014 NWRCC

Basic Logic Model Black Box Issues Resources Distal Impacts Activities Inputs Activities Outputs Outcomes Impacts Resources Distal Impacts Activities Proximal Impacts 2/27/2014

Logic Model Columns Resources—What resources are or could reasonably be available? Strategies and Activities—What will the activities, events, etc. be? Outputs—What are the initial products of these activities? Short-Term Outcomes—What changes are expected in the short-term? Long-Term Outcomes—What changes are wanted after initial outcomes? Impacts—What are hoped for changes over long haul? 2/27/2014

Characteristics of a Robust Logic Model Incorporates a primary theory of action Shows the logical connection between project resources, activities, outcomes, and expected impacts Is a collection of If … Then… statements Answers the question: Why would the planned activities be expected to have the desired impacts? Serves as a framework for project evaluation Guides fidelity of implementation 2/27/2014

Making Mathematical Reasoning Explicit (MMRE) Logic Model

Does it incorporate the primary theory of action for the project? If teachers use rich mathematical tasks combined with purposeful and probing questions to engage students in discourse and the use of the tools (notation, symbolization, graphs, charts, etc.) of mathematics to: Explain and justify their mathematical reasoning (justification) Develop and verify mathematical generalizations Then students’ reasoning skills and student achievement in mathematics will increase.

Does it show the logical connections?

Does it show why the planned activities would be expected to have the desired impacts?

Does it serve as a framework for project evaluation?

Does it guide fidelity of implementation?

Online Survey 2 Does your project have a logic model? If yes, to what extent does it meet the criteria? Criteria for a robust logic model: Incorporates the project theory of action Shows the logical connections between project resources, activities, outcomes, and expected impacts Explains why the planned activities would be expected to have the desired impacts Serve as a framework for the project evaluation Guides fidelity of project implementation 2/27/2014

Strategies for Logic Model Development Engage project stakeholders Begin by drafting a primary theory of action statement Go back to the characteristics of a robust theory of action Ask probing questions to help stakeholders think about why activities should work Take notes and work toward consensus 2/27/2014

Suggested Process First half-day session with stakeholders Learn about project activities, objectives, and goals Introduce logic model development process Gather input for project theory of action Between Sessions First draft of Theory of Action and Logic Model Second half-day session Gather input on draft Establish commitment to theory of action statement “Do you believe that this statement accurately states your collective belief about why your project will achieve its desired impacts?” 2/27/2014

Go Back to the Characteristics of a Robust Theory of Action Confronting the Elephant in the Room April 2011 Go Back to the Characteristics of a Robust Theory of Action Describes project impact as close to the primary target audience as possible Example: A description of what students do to learn Recognizable when it is going on Observable Defines fidelity of project implementation Can be a testable hypothesis Believable Interpretation of current literature 2/27/2014 NWRCC

Some of My Favorite Questions! What would it look like in the classroom if your professional development was effective? What kind of cognitive engagement would students be involved in? How would you know your project has been successful? When this project is over what will you be able to point to as a legacy of the project? What is it about your professional development that you think is effective? 2/27/2014

Using the Technology Taking Notes Drafting the Logic Model Capture the language of the stakeholders Don’t be afraid to paraphrase Use outliner view Drafting the Logic Model Don’t use a table format—too linear and constraining Use graphic software Visio MS Word 2/27/2014

Please use your chat box to submit questions for Dave 2/27/2014

RMC Research Corporation 633 17th Street Suite 2100 John T. Sutton, PI Dave Weaver, Co-PI RMC Research Corporation 633 17th Street Suite 2100 Denver, CO 80202-1620 111 SW Columbia Street Suite 1030 Portland, OR 97201-5883 Phone: 303-825-3636 Toll Free: 800-922-3636 Fax: 303-825-1626 Email: sutton@rmcdenver.com Phone: 503-223-8248 Toll Free: 800-788-1887 Fax: 503-223-8399 Email: dweaver@rmccorp.com 2/27/2014