Presentation on theme: "Leveraging Data For Action April 8, 2014. Background Nonprofit organizations are no strangers to data, in fact, many are experts in consolidating lots."— Presentation transcript:
Background Nonprofit organizations are no strangers to data, in fact, many are experts in consolidating lots and lots of data points. We try to collect it, we try to analyze it, and we try to respond to it. More often than not, the time and effort is just that – time and effort with little to show for how our data can be utilized to assess and improve our programs. What’s on the horizon… The demand for data will continue to increase as organizations try to link effective organizational and fiscal stewardship over: – Community impacts – Resources – Assets – Budgets 2
Learning Objectives Participants will learn how to develop a sustainable data management strategy. Participants will learn how to set and evaluate data targets. Participants will increase their understanding of data analysis techniques and tools that will help them make sense of their data. Participants will learn how to prioritize and apply a quality improvement processes in response to their data findings. Participants will learn how to tell a compelling story utilizing NPI and agency performance indicators. 3
Where does Data Management Strategy belong? Approaches we will take How we will measure our success What we are trying to achieve Who we are Vision, Mission, Values Strategic Aims / Objectives Goals Performance Measures 5
What is a Data Management Strategy? It is a … Strategic approach by which data is acquired, validated, stored, protected, and analyzed, and by which its accessibility, reliability, and timeliness is ensured to satisfy the needs of the data users. When the data management strategy is working well it … Provides the right data at the right time, by ensuring data are regularly collected, analyzed, reported, and effectively used to make improvements. 6
Intentional data management begins below... 7 Acquisition: How is the data acquired? Validation: How is the data validated? Format: What format is the data in? Accessibility: Is the data easily accessible? Reliability: How reliable is the data? Timeliness: How timely is the data? Storage & Protection: How is the data stored & protected?
Acquisition What data is currently manually entered into the system? – Interviews, observations, tracking Is the data automatically tracked by a system? Is the data coming from multiple sources? – Community Action Agencies – Public Sources Are staff able to to collect additional data about actions, perceptions, or processes that they aren’t currently collecting? Are there consistent data templates? Tip: Templates assist in consistent data delivery. 8 Data acquisition can be a challenge with different partner agencies, however there is a shared goal – we all want to showcase the impact our program is having in our communities.
Data Format Clear definition of the desired data format (%s, #s, $s). – Percentages (%s) – Numbers (counts, cumulative, ratios) – Dollars (Budgets, Expenses, Revenues, Assets) Clear understanding of what comprises the numerator and denominator. Data is submitted in uniform format. Tip: Develop a Measure Definition Sheet that details goals, performance indicators, and format of the data. (see examples in Appendix) 9 A consistent data format is essential if we are going to compare years, counties, or similar programs.
Validation Historical Trend – Does it make sense? Outliers – Is there just one? Or are there many? Spot Checking / Quality Control – Small random sample to validate the accuracy of data findings. – Do we need to spend significant amount of time cleaning it up? Tip: Line graphs are your friends in visually assessing potential issues with the data. 10 Different programs or partners will require more scrutiny until the validation process is more consistent.
Accessibility Is the data readily available? – Monthly, Quarterly, Bi Annually, Annually Is the data cloud-based? Or server-based? What type of system houses the data? – CAP 60, ServTraq – Microsoft Excel, Access – MySQL, Oracle – Attendance Logs Who runs the reports? Who is responsible for pulling the data? – Do we have a backup that can run the reports? 11 Data accessibility is critical for tracking and measuring community impact. The sooner you have the information the faster you can make the necessary course corrections.
Timeliness What is the frequency in which I will be able to collect the information? – Monthly, Quarterly, Bi Annually, Annually Do we have to consistently chase down the data? – Same program – Same individuals Tip: Try to minimize the use of indicators where data are only available at the end of the year. Find or develop a more timely source. 12 Utilize a balanced approach when dealing with data availability. Some data may be available on a daily basis posing its own challenges, while other data may not be available until the very end of the fiscal year which is truly too late.
Reliability How reliable is the data? Is the data consistently of good quality? Is there too much variability in the numbers to accurately tell the story? Tip: Quality data is your strongest point of view in telling your story and rebuffing critics. It’s unerring proof that the organization is making progress and delivering on its mission. 13 Spend the necessary time ensuring the data is consistently of good quality. The old adage is correct ….“Garbage in equals garbage out.”
Storage and Protection Can I access the historical record? Where do we store and backup the data? What types of protection protocols do we have in place? – Do we have a Business Continuity Plan? – What is the procedure if the data is inaccessible, erased, corrupted, or we’ve experienced a natural disaster? 14 There is nothing worse than losing or not having access to all of the data and corresponding analysis at a time when you need to. Plan for it.
Let’s Assess Our Data Management Strategy Assess and prioritize those measures with issues that need to be improved first. The Key to Success in implementing a data management strategy is to prioritize and focus on the metrics that are mission critical to your community impact. 15 Quarterly Review
Clearing Up Terminology Confusion 17 Setting and Evaluating Targets TermDefinitionNPI Examples Aim Statement Strategic Objective A clearly articulated statement provides both the foundation and the focus of the problem-solving effort. “What are we trying to achieve?” Address causes of poverty by implementing programs and services that empower low-income families and individuals. Performance Standard Performance Goal Objective standards or guidelines that are used to assess an organization’s performance. “What approaches will we utilize?” Low income people become more self- sufficient. The conditions in which low income people live are improved. Performance Measure Performance Indicator Quantitative measures of capacities, processes, or outcomes relevant to the assessment of a performance standard/goal. “What quantitative data do we have available to measure progress?” Number of participants pursuing post- secondary education. Number of participants demonstrating ability to complete and maintain a budget for >90 days. Performance Targets Specific and measureable goals related to agency or system performance. “What end of year performance outcome do we want to achieve?” Expected outcomes in reporting period. 85% of participants will pursue post-secondary education.
Step 1: Setting Annual Targets A target is the desired end of the year performance outcome which is typically determined by past performance. What needs to be in place before setting targets? Aim Statements – Typically 4 to 5 statements that encompass community and organizational needs. Goal statements – Often range from 3 to 6 goal statements in support of 1 Aim Performance Indicators – Typically 2 to 5 quantitative measures are selected in assessing progress against defined goals. 18
Step 2: Setting Annual Targets Scenario: We have defined our organizations aim statements, goals, and have aligned on which performance indicators to track, measure, and report on. How do I determine what should be the target? 19 Performance Indicator Funding Source Sets Target Is this doable?Yes Apply target or set one higher No Set interim targets that bridge the gap Organization Sets Target Do we have the data? Yes Evaluate your history and set benchmark year No Extrapolate to build initial target Leverage other program targets as starting point Leverage expert input
Step 3: Setting Annual Targets In addition to annual targets, Red Line targets are just as important because they help determine the minimum level of acceptability. Red line targets are unacceptable outcomes at any point and time that require immediate program or organizational attention and intervention. Setting redline targets is essential, and often left out, to help your organization gauge and prioritize your program improvement efforts. 20 Achieved or Exceeded Target Fell Short of Red Line Target Performed Above Red Line & Below Annual Target Red Line TargetAnnual Target
Visual Analysis: Leveraging SQVID 23 Series of 5 questions that we walk our initial idea through in order to bring visual clarity. Helps us imagine what visual messages we’d like to convey before we start worrying about which picture we’re going to use. Simple Elaborate vs Quality Quantity vs Vision Execution vs Individual Attributes Comparison vs Delta (or Change) Status Quo vs Source: Roam, Dan. The Back of the Napkin, Solving Problems and Selling Ideas with Pictures.
Visual Model 24 Source: Roam, Dan. The Back of the Napkin: Solving Problems and Selling Ideas with Pictures.
Visual Model (cont’d) 25 Source: Roam, Dan. The Back of the Napkin: Solving Problems and Selling Ideas with Pictures.
Visual Model (con’td) 26 Source: Roam, Dan. The Back of the Napkin: Solving Problems and Selling Ideas with Pictures.
Rules of Thumb for Visual Modeling Portrait – Think simple. – Illuminate lists. – Visually describe. Charts – It’s the data that matters, so let it show. – Pick the simplest model to make your point. Bar charts: comparing absolute quantities Lines & Area charts: For comparing absolute quantities between two different criteria or times. – If you start with one model, stay with one model. Maps – Everything has a geography. – North is a state of mind. – Look beyond the obvious hierarchy. 27 Source: Roam, Dan. The Back of the Napkin: Solving Problems and Selling Ideas with Pictures. Timelines – Time is a one-way street. – Repeating timelines create life cycles. – Round versus linear. (clock or ruler) Flowchart – Showcasing cause and effect. Plots – Multiple-variable plots aren’t hard to make, but they require patience, practice, and a point. – Create a scale model of an entire business universe or business problem. – The biggest danger of multiple- variable plots is that they invite the layering of many data types.
An integrative process linking.. …throughout an organization. Allows us to ask, “given our resources and authority, are we improving the lives within our communities in the most efficient way possible? Ultimately, it is a journey of continuous improvement, allowing organizations to maximize their impact and outcomes. “Doing the right thing for the right reason at the right place in the right ways at the right time.” 29 What is Quality Improvement? KnowledgeStructuresProcessesOutcomes Source: NNPHI, Proctor & Gamble
How does ROMA fit in a Performance Management System? 30 Source: Turning Point, From Silos to System, “Embracing Quality in Public Health: A Practitioner’s Performance Management Primer”
When Do I Launch a Quality Improvement Project? QI is an essential piece of performance management. When your performance management system shows a gap that is not closing between a performance target and your programs current performance, you have an opportunity to implement QIP to improve program and service impacts. Popular QI Methodologies Plan-Do-Study-Act (PDSA) Lean / Six Sigma 5s FADE 31
Prioritize and Select Quality Improvement Projects Review the measures that performed at the redline or below. Review the measures that performed above redline and below the annual target. As a team, leverage a matrix exercise to help you prioritize and evaluate which measures are candidates for a QIP. (See Appendix for QIP Matrix) Out of the prioritization matrix, select a max of 3 potential measures that may be candidates for a QIPs, – Three may be too many if there are significant resource constraints or process complexity, in which case target just 1 measure for a QIP. 32
Closing Gaps: Plan-Do-Study-Act (PDSA) Request corrective actions on significant differences between actual and planned results. Analyze the differences to determine their root causes. Determine where to apply changes that will include improvement of the process, product, or service. 33 Iterative four-step management method used in business for the control and continuous improvement of processes and products. Establish objectives and processes necessary to deliver results in accordance with the expected target or goals. When possible start on a small scale to test possible effects. Implement the plan, execute the process, deliver the service. Study the actual results (measured and collected in "DO" above) and compare against the expected results (targets or goals from the "PLAN") to ascertain any differences or gaps. Plan Do Study Act
Closing Gaps: Lean 34 Muri (Load) Concept of not overloading the process Muda (Flow) Concept of keeping the flow smooth and event Mura (Waste) Concept of removing the non-value adding activities Overproduction Transportation Motion Correction Over Processing Inventory Waiting Unused Employee Ideas & Talent Photos www.alverdyan.com
Telling the story 35 Source: Non Profit Data Decision-Making Workbook, January 2013
Telling the story “If a metric measures in a forest with no one around to hear it, it does not make a sound…” Who will be the recipients of the story? In what meetings will they be used? What decisions shouldn’t be made without them? How frequently will you plan to adjust your actions based on what the metrics say? 36 Source: Non Profit Data Decision-Making Workbook, January 2013
Effective Ways to Structure Your Story Beginning Given the short attention spans and the need to move from one activity to the next as quickly as possible, you want to highlight the momentous and game changing results first. Middle The body of your story should focus on those areas where we are improving and making significant strides. End The end of the story is where you will want to talk about the proactive steps that will be taken to bring under performing metrics back on track. 37 Filter down the most important themes you wish to convey. Ask your self what do we need to know to ensure we are making progress and report on those measures.
Leverage a combination of text and graphic to reinforce your story making it more memorable. 38 Source: Ontario’s Health System Yearly Report, 2013
Resources Getting Started with Data-Driven Decision Making: A Workbook. http://www.nten.org/research/2013-data-workbook-download. http://www.nten.org/research/2013-data-workbook-download Organizational Research Services, A Handbook of Data Collection Tools: Companion to “A Guide to Measuring Advocacy and Policy.” http://www.organizationalresearch.com/publicationsandresources/a_handbook_of_data _collection_tools.pdf. http://www.organizationalresearch.com/publicationsandresources/a_handbook_of_data _collection_tools.pdf Sample Size Estimator. http://www.surveysystem.com/sscalc.htm.http://www.surveysystem.com/sscalc.htm Roam, Dan. The Back of the Napkin, Solving Problems and Selling Ideas with Pictures. Measuring Your Impact: Using Evaluation to Demonstrate Value. http://nnlm.gov/evaluation/workshops/measuring_your_impact/DataCollectionHandout.p df. http://nnlm.gov/evaluation/workshops/measuring_your_impact/DataCollectionHandout.p df Turning Point, From Silos to System, “Embracing Quality in Public Health: A Practitioner's Performance Management Primer.” http://www.phf.org/resourcestools/Documents/silossystems.pdf. http://www.phf.org/resourcestools/Documents/silossystems.pdf “10 Steps to Creating the Perfect Infographic.” http://www.creativebloq.com/design/10- steps-creating-perfect-infographic-3145672.http://www.creativebloq.com/design/10- steps-creating-perfect-infographic-3145672 10 Free Tools for Creating Infographics. http://www.creativebloq.com/infographic/tools- 2131971.http://www.creativebloq.com/infographic/tools- 2131971 39
Further Questions or Comments firstname.lastname@example.org 40 This presentation was created by California/Nevada Community Action Partnership in performance of the California Department of Community Services and Development CSBG Discretionary Grant #13F- 3122. Any options, findings, and conclusions, or recommendations expressed in this materials are those of the author(s) and do not necessarily reflect the views of the U.S. Department of Health and Human Services, Administration for Children and Families.