Regional Educational Laboratory at EDC relnei.org Research Agenda Workshop [District/State Name] Session 1: [Date and time] Session 2: [Date and time] [Facilitator Name] 1

Regional Educational Laboratory at EDC relnei.org Welcome and Purpose Our purpose in this workshop series is to Engage in a collaborative process to identify district/state priorities, and Develop a set of research questions that comprise a coherent research agenda that will be the work of our district/state for next 3-5 years. 2

Regional Educational Laboratory at EDC relnei.org Goals for the Session Session 1 (today) will focus on reviewing different types of research, including levels of evidence, making causal claims, and linking questions to methods. We will leave today having prioritized initial research topics. Next session, we will generate research questions, and refine and prioritize our identified topics and research questions. We will leave with an initial research agenda for our district/state. 3

Regional Educational Laboratory at EDC relnei.org Agenda for the Session Introduce workshop participants Investigate types of research and levels of evidence, including reviewing the pre-work and additional research examples Discuss examples of researchable questions and research agendas Review district/state goal and generate some possible topics Prioritize possible research topics Closing 4

Regional Educational Laboratory at EDC relnei.org Introductions Please share your: –Name –Agency/where you work –Your role there –[if there are pictures of the participants you can insert an additional slide with the name and pictures while introductions are taking place]. 5

Regional Educational Laboratory at EDC relnei.org Primary Types of Research Different types of research allow us to answer different types of questions and to make different types of claims. What’s Happening –Descriptive What’s Known –Descriptive 6 Making Connections –Correlational Making an Impact –Impact/Causal

Regional Educational Laboratory at EDC relnei.org What’s Happening Descriptive studies do not examine cause and effect but look at what is happening: trends; baselines; and experiences of individuals, groups, or programs –What is the trend in science NECAP scores over past 5 years? –What are the certification characteristics of beginning teachers in the region? –What are the features of district teacher evaluation systems? Methods tend to include secondary data analysis, document and records review Tend to use descriptive statistics: averages, frequencies, and percentages 7

Regional Educational Laboratory at EDC relnei.org What’s Known Another type of descriptive study focusing on reviews of the literature and other research that has already been conducted. –Literature review –Meta-analysis 8

Regional Educational Laboratory at EDC relnei.org Making Connections Correlational studies look at the relationship between two or more variables but one cannot infer causality. –How do math course taking patterns vary race and gender? –What factors are related to differences in graduation rates across districts? Methods tend to be secondary data analysis of administrative or other state, district, or school datasets Statistical methods include: differences in means, ANOVA, multivariate regression models 9

Regional Educational Laboratory at EDC relnei.org Making an Impact Causal/Impact studies usually examine questions about impact, building on descriptive studies –Does this afterschool program increase student engagement? –Does this math intervention lead to improved math achievement? –Do new teacher evaluation policies improve student achievement outcomes? Many “opportunistic” possibilities in implementation According to IES, an impact study: –examines the effectiveness of a particular policy, program, or practice –can be small-scale or large-scale –can be used in a formative way (e.g. a short-term pilot study) or a summative way 10

Regional Educational Laboratory at EDC relnei.org Making an Impact Only certain types of designs can be used to assert causal claims or claims about program impact IES-funded impact studies must meet What Works Clearinghouse standards –Randomized Control Trials (RCTs) = “the gold standard” –Randomization of treatment and control groups (e.g. those who get the intervention and those who don’t) –Other designs, including quasi-experimental, matched comparisons or others may not yield unbiased estimates of impact 11

Regional Educational Laboratory at EDC relnei.org Investigating Research Examples Refer to the 2 research summaries you read during pre- work You will be placed into breakout rooms where small groups will focus on one study and discuss –Research questions –Data sources –What was answered/found –Study design—discuss and add this information –What else there is to learn/ future research questions 12

Regional Educational Laboratory at EDC relnei.org Discussion: Research Design Back in the large group we will share out from our small group discussions What were the studies about? What is and is not answered by research questions/designs? –What related research questions might you ask? 13

Regional Educational Laboratory at EDC relnei.org Researchable Questions What is a researchable question and is not? –Reasonable –Appropriate –Answerable –Measurable How do you develop researchable questions? –Questions, concerns, and values of stakeholders/community –Important issues raised in the field or literature –Professional standards or guidelines –Views and knowledge of experts –One’s own views and judgement 14

Regional Educational Laboratory at EDC relnei.org Research Agendas What is a research agenda? What does a research agenda look like? –Linear –Topical –Examples 15

Regional Educational Laboratory at EDC relnei.org Example: Research agenda that is less coherent TOPIC: COLLEGE READINESS –Descriptive study of high school dropout rates for key subgroups –Descriptive study of policies for assigning students to college math vs. general math –RCT of the impact of classroom coaching for new high school English teachers in 5 large urban districts (Source: REL Kickoff Meeting) 16

Regional Educational Laboratory at EDC relnei.org Example: Coherent, linear research agenda TOPIC: HIGH SCHOOL GRADUATION Action Goal: Increasing the percentage of District X students who graduate from high school –Descriptive study of high school dropout rates for key subgroups –Descriptive study of re-enrollment rates in traditional high schools and education trajectories of re-enrollees –RCT of the impact of alternative programs for degree completion on high school graduation 17 (Source: REL Kickoff Meeting)

Regional Educational Laboratory at EDC relnei.org Example: Coherent, topical research agenda TOPIC: MATHEMATICS LEARNING Action Goal: Increasing mathematics achievement in the Northeast region –RCT of the impact of student access to Algebra I in grade 8 –Descriptive study of math education practices for students with disabilities in six schools in two Northeast states –Descriptive studies of math performance patterns for students with disabilities in MA and NY 18

Regional Educational Laboratory at EDC relnei.org Our Process to Identify Topics Work from district/state goal to identify topics and narrow focus to two topics Identify subtopics and questions within top two topics Prioritize questions [or subtopics] 19

Regional Educational Laboratory at EDC relnei.org Example: Gender in Schools Research district/state Goal: Provide research that informs and promotes gender equity in schools 20 TopicSubtopicQuestionsTiming Girls in STEMTracking achievement differences Encouraging interest and enrollment in STEM Persistence in STEM majors Attainment for Boys Structured Inequality

Regional Educational Laboratory at EDC relnei.org Moving from Subtopics to Questions 21 SubtopicQuestionsTiming Tracking achievement differences What are gender differences in STEM achievement in K-12 and how have they changed over time? Do gender differences in STEM achievement vary among districts and schools? Encouraging interest and enrollment in STEM What can teachers do? Are single-sex schools and classrooms better? What programs help promote STEM for girls? Persistence in STEM majors

Regional Educational Laboratory at EDC relnei.org Moving from Subtopics to Questions 22 SubtopicQuestionsTiming Tracking achievement differences What are gender differences in STEM achievement in K-12 and how have they changed over time? Do gender differences in STEM achievement vary among districts and schools? Encouraging interest and enrollment in STEM What can teachers do? Are single-sex schools and classrooms better? What programs help promote STEM for girls? Persistence in STEM majors A fully specified research question A general question that can be used to develop a set of research questions

Regional Educational Laboratory at EDC relnei.org District/State Goals 23

Regional Educational Laboratory at EDC relnei.org Goals & Research Topics Reflecting on our goals, brainstorm topics that are important to you and that fit under the goals. Record in your workbook on page 5. –If you are unsure whether a topic fits under the goal, include it in the “possible” column of the page. The ideas don’t need to be refined, but we will be narrowing the list later in the session, so focus on topics that are most important to you. You will have 5 minutes to think and write by yourself. 24

Regional Educational Laboratory at EDC relnei.org Research Topics-Breakout Rooms You will be placed into a virtual breakout room to discuss the topics you brainstormed (15 minutes) Share your topics with your small group On your breakout room whiteboard: –Write the topics that you would like to bring back to the whole group for consideration as priority topics These topics will be brought back to the whole group, when the breakout time ends. 25

Regional Educational Laboratory at EDC relnei.org Grouping our topics Together, we will engage in a process of grouping the topics. Our identified topics are diverse but some are related. What topics generated in the groups do you think go together? After we review the grouping of topics we will name the topics. –The name becomes the “topic” and the items grouped under that topic name are “subtopics” 26

Regional Educational Laboratory at EDC relnei.org Narrowing the List of Topics The Goal is to narrow the list of topics to 2 priority topics for our work—we will do this using a modified Focusing Four process –Brainstorm: We did earlier and grouped our ideas into topics –Clarify: We will review the groupings. We can clarify what any topic is, suggest moving topics or splitting groupings. –Advocate: Think about narrowing to 2 topics that could comprise a good agenda when taken together. –Canvass: You will each have 3 votes to use to indicate the topics that you think should be a research priority for our district/state. 27

Regional Educational Laboratory at EDC relnei.org Clarify Review the “groupings” of topics Do any topics on any list need to be clarified? Do the groupings make sense? Should we move any items or split any groupings? Can anything be combined or removed? 28

Regional Educational Laboratory at EDC relnei.org Advocate Think about which cluster of topics, when taken together, provide the best research direction for the district/state. Share your preferred priority topic and provide a few positive reasons why you think the topic grouping should be a priority for the district/state. 29

Regional Educational Laboratory at EDC relnei.org Break 5–10 minute break 30

Regional Educational Laboratory at EDC relnei.org Polls Vote once in each Poll to indicate your priority research topic for the district/state. –You can vote for the same topic in each poll if you like, or vote for different topics. At the conclusion of the canvassing, we will review the results of our Polling and make some decisions about our 2 topics. –This will be your take-away from Workshop session 1. 31

Regional Educational Laboratory at EDC relnei.org Prepare for Next Session For one or both of the topics, think about: –What research questions interest you under the priority topic areas? (these do not have to be at the level of researchable questions but they should be questions that are a finer grain of detail than the topics themselves). questions to facilitators before Session 2. –Begin to think about the data sources that are appropriate to answering these questions and where they reside, if they exist. 32

Regional Educational Laboratory at EDC relnei.org Thank you Thank you for your participation and engagement today. See you [INSERT DATE AND TIME OF NEXT SESSION]! 33

Regional Educational Laboratory at EDC relnei.org Research Agenda Workshop Session 2 [District/State Name] Session 1: [Date and time] Session 2: [Date and time] [Facilitator Name] 34

Regional Educational Laboratory at EDC relnei.org Welcome and Purpose Our purpose in this workshop series is to Engage in a collaborative process to identify district/state priorities, and Develop a set of research questions that comprise a coherent research agenda that will be the work of our district/state for next 3-5 years. 35

Regional Educational Laboratory at EDC relnei.org Goals for the Session Session 1 focused on reviewing different types of research, including levels of evidence, making causal claims, and linking questions to methods. We left with some prioritized topics. This session, we will: –Generate, refine and prioritize research questions, and –Leave with an initial coherent research agenda 36

Regional Educational Laboratory at EDC relnei.org Agenda for the Session Review topics from session 1 Generate questions (review questions sent in by and generate additional questions) Prioritize questions –We will prioritize all the questions generated in our two topic areas. Generate preliminary research agenda Next steps 37

Regional Educational Laboratory at EDC relnei.org Review Research Topics and Begin to Generate Questions Highlights of topics from last session and review of research questions sent in by . In virtual breakout rooms, work on the white boards for 30–40 minutes to generate questions related to the priority topic (and any subtopics within it) Groups should think comprehensively within their topic/subtopic 38

Regional Educational Laboratory at EDC relnei.org Share Out Small Group Work/Refine Questions Re-group into whole group and share white boards from small group work. Discuss questions generated and refine by: –Edit, add, and/or delete questions –Check: any additional questions—is something missing? 39

Regional Educational Laboratory at EDC relnei.org Prioritize Research Questions Focusing Four process –Brainstorm: Review our list of questions, generated earlier –Clarify: Do any questions on the list need to be clarified? –Advocate: Think about narrowing a few questions in each topic/subtopic area that could comprise a good agenda –Last step will be done after the break 40

Regional Educational Laboratory at EDC relnei.org Break 5–10 minute break 41

Regional Educational Laboratory at EDC relnei.org Poll Poll: Vote for the research questions. –Vote once in each Poll –You can vote for the same question in each Poll or different questions 42

Regional Educational Laboratory at EDC relnei.org Preliminary Research Agenda Review identified priority topic, subtopics, and questions How many top rated questions should be on the agenda? Consider: –Types of work (data alignment, workshops, studies, etc.) –Sequence –Short- and long-term nature of work Remember… 43

Regional Educational Laboratory at EDC relnei.org Research Agenda- continued Refinement of the research agenda should continue over time. The agenda should: –encompass the work that is most important to the district/state, –Align with the district/state goal and –Be the work that the district/state will do over the next 3-5 years. 44

Regional Educational Laboratory at EDC relnei.org Next Steps Continued refinement of agenda Involvement in prioritization and study development Work to conduct research or identify research support Use agenda to recruit research and funding partners for this work. 45

Regional Educational Laboratory at EDC relnei.org Thank you Thank you for your participation and engagement today. 46