1. Choose Your Own (Data) Adventure: Perils and Pitfalls, and Lots of Promise
April L. Zenisky
University of Massachusetts Amherst
Address at NERA - Trumbull, CT: October 2015 1

2. 2

3. 3

4. 4

5. 5

6. 6

7. 7

8. Use the Data!
Lots of talk lately about data and data use in educational settings
Data is good, and using data is better!
My way in - communication and use of assessment data
Disclaimer: I like data!
In educational settings the expectation of ‘use the data’ is interesting to me 8

9. Setting the (historical) stage
Reporting data and results has largely been an enterprise straight out of the school of “you’ll take what we give you and you’ll like it”
A Report was generated by agency charged with reporting, disseminated to users intact
Control of the data, and hence control of the story about how the data should be interpreted, resided with the agency who owned the data
Here’s where to look, what you’ll see, and here are prepped answers to the questions you’ll probably ask 9

10. The winds of change
Mechanisms for the ‘use the data’ expectations are online dashboards, tools, and portals
Paradigm shift from “Here’s the Story” to “You’re in Charge”
Allow users to customize which results to display based on interests / needs
Some blurring of line between reporting and secondary analysis
Not disseminating results, but data (& access to it) 10

11. Disseminating data, not results
Drawing from my work in score reporting
Potential pitfalls in assumptions about Use, Access, and Understanding
What is it that we hope to achieve when data is shared publicly through portals, tools, and dashboards?
Is ‘use the data’ actually a reasonable expectation? 11

12. 12 12

13. Use the data in educational settings
This means different things to different people
Often depends on where you fit in “Education”
Administrative / managerial capacity (leaders)
Instructional / delivery capacity (teachers)
Personal capacity (families)
Peripheral capacity (researchers, The Public)
We all have thoughts about how others should use data 13

14. Assumption 1: Use the data
It connotes action
Go forth with numbers and do stuff
Action (which includes inaction) is a function of who they are
Very specific, given why they are there
Use needs to take users into account 14

15. Avoiding the Use pitfall
Needs analysis (Research!)
Consideration of the users
What do they want to know?
What, if any, are the actions they want to take?
What can data managers do to enable users?
Balance of keeping tools flexible but supporting specific real-life use cases
Necessary challenge for research and practice 15

16. Assumption 2: Access the data
I have instructors and families squarely in mind here
Issue 1: equity concerns
Issue 2: opportunity concerns
“I don't think it matters how easy they are to use.... time is the issue. Time to read them, time to figure out how to integrate results into classroom practice with multi levels and where differentiated teaching is necessary.”
Issue 3: familiarity concerns 16

17. Avoiding the Access pitfall
Access needs to be recognized as a problem!
Equity, opportunity, familiarity - these impact data use in ways that are real, not theoretical
No easy solutions here - problems often present themselves at the local level
One strategy is use-case sharing and collaboration to build a toolbox of approaches
Your ideas? 17

18. Assumption 3: Understand the data
A lot gets made of a lack of relevant types of “literacies”
Quantitative, graphical, data, assessment
Step back from labels
Working with data is like anything else - it requires investment of time and energy
Cost-benefit ratio of effort to perceived outcome
Hope is that if data makes sense = has value! 18

19. Assumption 3: Understand the data
Understanding builds on process and knowledge
Can users obtain results and work within the data to answer their questions to their own satisfaction?
Are statistical caveats and limitations of data communicated clearly to support appropriate interpretations (and curtail errors)?
How do we promote understanding in this choose- your-own-adventure approach, where control resides in the hands of the users? 19

20. Avoiding the Understanding pitfall
Coming at this from the communication and systems side
Quality dashboards, portals, and tools are a must
Collaboration in systems development
Consideration of use cases must be at the forefront 20

21. So, where does that leave us?
I’ve described plenty of pitfalls related to use, access, and understanding
Focusing on assumptions feels … negative.
Go back to Slide 5.
What is it that we hope to achieve when data is shared publicly through portals, tools, and dashboards?
Is ‘use the data’ actually a reasonable expectation? 21

22. Disseminating data isn’t going away…
Reflective of evolution in how people interact with information today
These orientations toward information must inform the choices we make in advancing public use of data
Explanatory and Instructive
Exploratory and Constructive 22

23. 23

24. 24

25. “Use the data” is not unreasonable
Must be accompanied by good choices behind the scenes, to empower users in their choices
“Use the data” can be more than a buzzword
Let’s build tools to support users and advance research and practice - that’s the real adventure! 25

26. Enjoy the rest of the conference!
Thanks!
Enjoy the rest of the conference! 26