1. Defining Data Literacy for Educators Ellen Mandinach, WestEd Edith Gummer, Education Northwest and NSF A Collaborative Project of WestEd and Education Northwest Sponsored by the Bill & Melinda Gates Foundation STATS-DC 2012 Data Conference July 12, 2012

2. Problem Space No common definition of data literacy Professional development providers Teacher/Administrator preparation Confusion between data literacy and assessment literacy Lack of understanding of what it means to be data literate and use data effectively

3. The Project Bring together a diverse group of expert stakeholders who know data-driven decisionmaking Scope out a common definition of data literacy Discriminate between data literacy and assessment literacy, understanding the requisite skills and knowledge Perform a content analysis of major professional development books and models. Identify the key knowledge and skills. Identify commonalities, differences, and perform a gap analysis.

4. The Stakeholder Groups Experts from: Research Professional development Government Policy Funding agencies Assessment Other stakeholders

5. Data Sources Pre-conference data definitions from experts Professional development text analyses A hands-on activity to distinguish between data literacy and assessment literacy Transcripts from the conference

6. Pre-Conference Definitions Really interesting and diverse Some succinct; others quite involved Some quite philosophical Focus on data; focus on assessment Plethora of skills and knowledge components

7. A Wordle for Data Literacy

8. Some Overarching Thoughts and Select Quotes from Pre-Conference Differentiation between declarative, procedural, schematic and strategic knowledge; the content versus the performance/process components Data literacy is more than the sum of the component skills and knowledge Continuum of novice to expert “Defining data literacy will require careful consideration of the level of ability necessary to define literacy – somewhere between basic skills to complete fluency.”

9. Conflicting Perspectives (in one quote) “At the state department and college/university levels we are doing a damn lousy job of teaching data literacy. Too few states demand courses on testing and measurement. Too few colleges have those courses taught (when taught at all) by people who could plausibly be described as psychometrically competent. We spend too much time on the statistics of data and too little on the ‘how to make constructive use of data’ and the ethics of data gathering and use.”

10. A Hands–On Activity Purpose – to differentiate between data and assessment literacy Outcomes Data literacy is seen as broader than assessment literacy – but contested Assessment literacy is a component of data literacy – but has unique components The question remains what are the unique components – open quandary

11. Analysis of the Materials Purpose – to characterize the terrain of professional development in data literacy What knowledge and skills are suggested by the materials as being the core of what educators need to know and be able to do? Representative rather than exhaustive search Provide an initial operational framework

12. Analysis of the Materials Process for analysis Had to start somewhere – 2 texts we knew were being used extensively in professional development » Love, Stiles, Mundry, & DiRanna (2008) The data coach's guide to improving learning for all students. » White (2011) Show me the proof: Tools and strategies to make data work with the Common Core Standards. Read through the texts identifying the key concepts and skills being presented on each page – tons of post-it notes

13. Analysis of the Materials Grain size decisions – 30,000 feet versus weeds Examined commonality in language across the 2 texts Distilled 4 pages of codes to 8 key terms – negotiated process Sorted remaining codes into 8 key terms

14. Analysis of the Materials Coding process 2 research associates used the coding spreadsheets – negotiated practice Code values » Mentioned – page # and m » Whole page addressed code – page #-1 » More than one page addressed code page # + » Multiple pages addressed code – page #-page # Relative emphasis of the text material

15. Findings – Common Elements Collaborative processes Multiple forms of data – triangulation Key focus on student achievement Little emphasis on statistical knowledge General analysis processes described with examples Templates for examining and displaying data and making interpretations Connection to instruction overly general

16. Findings – Variations Level of description and instruction – a number of texts provide broad general description and few details Data use versus formative assessment Focus on multiple forms of data beyond achievement and student performances Educator versus teacher

17. Analysis of the Conference Transcripts Overarching theme: It’s complex Data literacy is more than just its definition There is an entire educational contextual landscape that surrounds the definition The issue is systemic

18. Working Definitions for Data Literacy Data Literacy is……… Complexity 95/5 issue – importance of the 5% Consensus definition

19. Complexity of Defining Data Literacy Literacy – generic to specific Messaging Language Role or stakeholder influence/perspective Systems nature Technical knowledge and skills Data Pedagogical/administrative

20. Knowledge and Skills 95%/5% Problem Focus Identify purpose Frame questions Understand context Apply data to instructional or administrative action Data Focus Select the right data Know how to access data Design instruments Collect data Organize data Use multiple sources of data Understand data properties Understand different kinds of data (formative, summative, diagnostic) Use a variety of data sources (qualitative, quantitative)

21. Knowledge and Skills 95%/5% Data Focus - continued Use of statistics and measurement concepts - understand reliability, validity, sources of error Understand scaled scores, percentiles, performance levels Drill down to different levels of data Summarize data Synthesize information Analyze data Analyze different levels of data – cohort, course, grade Examine patterns and trends Make interpretations Troubleshoot data

22. Knowledge and Skills 95%/5% Process Focus Formulate hypotheses Engage in collaborative inquiry Transform data into actionable knowledge Know how to use data systems, tools, and applications Comprehend data displays and data reporting Draw inferences Consider consequences Test assumptions Evaluate outcomes Critique arguments Habits of Mind - Dispositions

23. Educational Context The need for a conceptual model Account for organizational factors Issues around data properties and adequacy for use Issues around professional development The need for best practices and models The role-based nature of data literacy Roles of schools of education, state agencies, and policymakers The SLDS Other technologies and supporting resources The role of research The need for instrumentation The role of funding organizations The need for collaboration Messaging

24. A Road Map for Next Steps For the researchers For the professional development providers For the policymakers For the funders