Presentation on theme: "Research Agenda Workshop"— Presentation transcript:
1 Research Agenda Workshop [Insert relevant information. Consider:Alliance or group nameFacilitator’s name(s)DateLocation][Customize this slide with information relevant to your workshop. Have this slide up on the screen as participants enter the room]
2 Welcome and purpose Workshop objectives: Engage in a collaborative processIdentify alliance research prioritiesDevelop a set of research questionsDevelop a coherent research agenda for the next 3–5 yearsWelcome to the Research Agenda Workshop. We look forward to a productive day. Our purpose for coming together today is to engage in a collaborative process where we identify research priorities for our alliance/group and develop a set of research questions aligned to these priorities. Together these will comprise a coherent research agenda that will guide our work in the next 3–5 years.
3 Session goals Morning: Afternoon: Review different types of research Identify and prioritize research topicsGenerate related research questionsAfternoon:Refine and prioritize research questions to form initial research agendaOur work today will focus on three things:Reviewing different types of research, types of evidence, and types of research agendasIdentifying and prioritizing possible research topicsGenerating, refining, and prioritizing research questions
4 Agenda Introduce participants Identify possible priority research topicsInvestigate types of research and levels of evidenceExplore research questions and research agendasPrioritize research topicsGenerate research questionsShare, refine, and prioritize research questionsPut it all together into a research agenda[Facilitator reviews agenda for the workshop]
5 Introductions Name Where you work Your role Let’s go around the room and introduce ourselves. Please share your:NameWhere you work, andYour role
6 Alliance goal[Insert your alliance or group’s goal. For example: The USVI College and Career Readiness Research Alliance will support the Virgin Islands Department of Education in their efforts to prevent and reduce the number of students dropping out of schools by providing applied research and analytic technical support on how best to use available data to both establish robust early warning systems and identify interventions to help improve outcomes for students at risk.]Let’s start by reviewing our Alliance Goal/Mission/Purpose [facilitator reads goal]
7 Alliance goal and research topics Record in your workbook on page 5:Topics that fit under your alliance’s goalIf you are unsure of a topic, include it in the “possible” columnFocus on topics that are most important to you.You will have five minutes to think and write by yourself.Our first task today is identify some possible research topics that are relevant to our alliance’s goal. We are going to first work individually and then in small groups.First, turn to page 6 of your workbook; working on your own, take five minutes to brainstorm topics that are potentially relevant to the alliance goal. You will see the worksheet on page 6 is divided into two columns; use the left column to list topics that you think fit under the alliance’s goal; use the right column to list topics that you think might fit under the alliance goal.Don’t worry right now about the “grain” size of your topics/ideas—we will revise and narrow the topics later.
8 Small-group discussions Break up into small groupsOn large Post-It notes, write down the topics your group would like considered as priority topicsOne topic per Post-It noteNow in small groups, we will share and discuss the topics we came up with individually, thinking about such things as relevance and importance. Again, you don’t need to worry at this point about the grain size of the topic; it’s okay if some are narrow and some are broad.Group members should write down topics on the large Post-it notes. These will be shared with the whole group, which will then group the topics.
9 Grouping our topics Affinity grouping process. What topics generated in the groups go together?Review the topic groupings and name them.Items grouped under each topic name are now considered subtopics.[Bring the small groups back together, have participants stick the Post-It notes on the wall, and in whole group review the topics. This is not a discussion but simply a review of what’s written on the post it notes.]Together, we will group our topics using an affinity grouping process; everyone will have a say. Now that we’ve had a chance to review the topics listed, does anyone have ideas about which topics are “like” or kind of go together? Do any seem to be the same thing? Can someone help me by moving the Post-it notes around per the discussion?Let’s see how many topic groups we have. Do we have any that don’t seem to go with anything else? That’s okay, they can be a group by themselves.Before we continue, let’s review our grouping and see if we need to make more changes to it.[When satisfied, we can now name our topic groupings]Now that we have grouped our topics, let’s “name” the groups –this will be the “topic” and the items listed in the group will be the “subtopics.”
10 Types of research Typology used by the REL program at IES includes: What’s HappeningDescriptiveWhat’s KnownMaking ConnectionsCorrelationalMaking an ImpactImpact/causalFor the next little while, we will step back from the particular topics of interest to our alliance and agenda, and focus on research in general, as well as research questions and agendas in general. Don’t worry, we will come back to prioritize our topics in just a little bit.First, here’s a brief overview of different types of research. There are many ways of organizing types of research. Here we are using the REL language and typology for research and research products. One type of research is not necessarily better than the other, but different types of research allow us to answer different types of questions and with more or less certainty.We think it’s important to review information on different types of research because it may inform the types of questions we ask or possibly the sequence of our work. Note that more information than we will cover is in your workbook.
11 What’s HappeningDescriptive studies look at what is happening: trends; baselines; and experiences of individuals, groups, or programsMethods include secondary data analysis and document and records reviewThe studies use descriptive statistics: averages, frequencies, and percentagesFirst is Descriptive Research. The REL program describes descriptive research by the product category of What’s Happening.This type of research examines baselines, trends, and may look at experiences or characteristics of individuals, groups or programs. This type of research DOES NOT inform cause and effect but it can be a useful first step in a larger research agenda or set of questions, and it can inform hypotheses that can be tested using more rigorous methods.Descriptive studies can use primary or newly collected data, or secondary data analysis from existing data sets, and even document review.
12 What’s KnownAnother type of descriptive study focuses on reviews of the literature and other previous research. Examples include:Literature reviewsMeta-analysesWhat Works Clearinghouse reviewsThe What Works Clearinghouse is an IES-funded initiative that provides reviews of existing education research and evaluates the credibility and reliability of the research evidence.Another type of descriptive research focuses on reviews of literature and reviews of other research, known as a meta-analysis.The REL program classifies research conducted in this category as “What’s Known.”The What Works Clearinghouse, another program of IES, provides these types of reviews. Established in 2002, the WWC is a resource for informed education decision-making. The WWC provides reviews of existing education research and identifies studies that provide credible and reliable evidence of the effectiveness of a given practice, program, or policy (referred to as “interventions”). Summary information and reports, including REL reports, are available on the WWC website.With over 700 publications available and more than 6,000 reviewed studies in the online searchable database, the WWC aims to inform researchers, educators, and policymakers as they work toward improving education for students.
13 Making ConnectionsCorrelational studies look at the relationships among two or more variables or characteristics but do not imply that one causes another.Correlational research tests whether relationships between variables are “statistically significant,” meaning they are not likely due to chanceMethods include analysis of existing data from administrative or other state, district, or school datasets.Another type of research is correlational research. It’s about connections, but not causality. This type of research explores the relationships and trends between two or more variables. For example, a correlational study may look at how students’ math course-taking patterns vary by race and gender. While the results may show that course-taking patterns differ by race and gender, they cannot prove what causes the differences.Correlational research can use primary or secondary data analysis. In education research, we frequently see secondary data analysis of school, district, or state data.Statistical techniques test for relationships between two or more variables or differences between two or more variables. These techniques include: difference in means testing, Analysis of Variance (ANOVA), and regression techniques.
14 Making an ImpactCausal/impact studies usually examine questions about impact; they can build on descriptive studiesLook for “opportunistic” possibilitiesAn impact study:Examines the effectiveness of a particular policy, program, or practiceCan be small or largeCan be used in a formative way (for example, a short-term pilot study) or a summative wayThe fourth REL product category is called “Making an Impact. ” These types of products are reserved for the most rigorous types of research, which allow researchers to explore questions of cause and effect, impact, and effectiveness.Many educational practices—such as the piloting of new curriculum or program—provide opportunities for this type of research.Impact studies examine the effective of educational policies, programs, or practices. They can be small-scale or large-scale, and can be used for formative or summative purposes.
15 Making an ImpactOnly certain types of designs can assert causal claims or claims about program impactREL impact studies must meet What Works Clearinghouse standardsRandomized controlled trials (RCTs) = the “gold standard”Other designs, including quasi-experimental and matched comparisons, may not yield unbiased estimates of impactCausal or impact research designs require the use random assignment to treatment and control groups; this allows researchers to be more confident that any resulting differences between groups are not due to other factors. One generally embarks on this type of rigorous research only after extensive descriptive research.On page 11 in your workbook, you can find a table titled The Continuum of Rigor in Impact Evaluation Designs. This is excerpted from a guide on evaluating educational reforms and provides information on and examples of the different types of impact study designs, including their level of rigor.
16 Investigating research examples Review the research summary you read during prework (5 minutes)Divide into small groups by summary and discuss (15 minutes)What the research questions wereWhat data sources were usedWhat the research design wasWhat else there is to learnWhat you would need to answer the remaining questions (data, RCT, etc.)?Whole-group discussion (10 minutes)Now that we’ve had a primer on types of research and designs, let’s dig a little deeper into some research examples we read as part of preparation for the workshop.First, you will individually review the research summary that you read during pre-work. This will take about 5 minutes.Then, we will divide the large group into small groups according to the study summary you read and have you discuss the studies.Finally, we will have a brief whole group discussion about what you noticed about the research designs, questions, data and such from the studies.You may want to use the Investigating Research Examples organizer on page 3 of your workbook to assist in organizing your thoughts about the research study.[After first 5 minutes, divide up into groups, for another 15 minutes][When whole group comes back together prompt discussion by asking such things as – what was your study about? What did you note about the research questions or design? What were some of the strengths and weaknesses of the approach?]
18 Researchable questions What is a researchable question?ReasonableAppropriateAnswerableSpecificWhere do researchable questions come from?Questions, concerns, and values of stakeholdersImportant issues in the field or research literatureProfessional standards or guidelinesViews and knowledge of expertsOne’s own views and judgmentNow that we’ve reviewed some research study summaries, we have a sense of what research questions are. Earlier today, we identified some possible research topics; a research topic is a general statement about the research interest (e.g., the impact of early education on later achievement).A research question, by contrast, is more specific and gives us an idea about the method or approach used and what exactly was studied. Descriptive and correlational questions tend to be “How” or “What is” questions, whereas impact or causal questions tend to ask “Does…” For example, “Does participating in this program change behavior or increase knowledge?”Some guidelines about research questions include:They are reasonable – That is, exploring the research question is do-able, given time and budgetThey are appropriate – The research question or questions fits with the program or issue being studied; in this case, they questions fit with the alliance goals or purposeThey are answerable – This means appropriate data can be gathered or found that will actually answer the question or questions.They are specific – This means they include clearly defined and measurable indicators of success or the desired outcome.Research questions are more difficult to come up with than topics. And you might not immediately know if your questions are answerable or measureable. Sometimes, we need to do preliminary work or investigating to determine if a question is researchable.Research questions come from many places–Issues or questions of stakeholders or the communityPrevious research or the literatureViews of expertsProfessional standards and guidelinesYour own experiences and views
19 Research agendas What is a research agenda? Identifies research priorities and questions for groupIs coherentLeads to rigorous and relevant research that is actionableMay include current or future projectWhat does a research agenda look like?LinearTopicalFor our purposes here, we define a research agenda as 2 to 4 focused research topics each with a set of 3 to 5 coherent research questions that will help to achieve the alliance’s goal. A research agenda also can include a set of possible studies or projects that align with the research questions.Essentially, a research agenda identifies the research priorities for the group and can be specific enough to include questions and possible studies that provide a direction or path for the group’s work.There is no right way to organize an agenda. Research agendas can be linear or topical. We include a few examples on the next slides and in your workbook.
20 Example: Research agenda that Is less coherent Topic: College readinessResearch questions:What are the high school dropout rates for particular subgroups of students?Does coaching high school English teachers impact their students’ performance?Studies:Descriptive study of high school dropout rates for key subgroupsDescriptive study of policies for assigning students to college math versus general mathRCT of the impact of classroom coaching for new high school English teachers in five large urban districtsThis first example is of a LESS coherent research agenda. The topic is College Readiness, but as you can see the identified studies are only tangentially related and don’t really inform each other in any way.
21 Example: Coherent, linear research agenda Topic: High school graduationResearch questions:What are the high school dropout rates for key subgroups of students?What are the effects of alternative high school programs on students’ degree completion and graduation?Studies:Descriptive study of high school dropout rates for key subgroupsDescriptive study of re-enrollment rates in traditional high schools and education trajectories of re-enrolleesRCT of the impact of alternative programs for degree completion on high school graduationThis is an example of a coherent linear agenda. This agenda is linear because the studies build on one another, with one being completed before the next is undertaken.
22 Example: Coherent, topical research agenda Topic: Mathematics learningResearch questions:What is the impact of grade 8 students’ access to algebra I on their math achievement?How do students with disabilities perform in math?Studies:RCT of the impact of student access to algebra I in grade 8Descriptive study of math education practices for students with disabilitiesDescriptive study of math performance patterns for students with disabilitiesResearch agendas also can be coherent and arranged more topically rather than sequentially or linear, as this example indicates.
23 Prioritizing research topics Narrow the list of research topics to 24 priority topics for our alliance, using a modified Focus Four:BrainstormClarifyAdvocateCanvassNow that we’ve reviewed different types of research, and learned about research questions and agendas, we are going to return to our alliance focus and the possible research topics that we identified earlier this morning. Specifically, we are going to narrow the list of topics to identify two to four PRIORITY topic areas for our alliance. We will get to these priority topics using a modified Focus Four process. The Focus Four includes four steps:Brainstorm the topics; we have already done thisClarify – clarify the topics by asking questions, reviewing our affinity groupings, and making any necessary changesAdvocating – participants have an opportunity to advocate, in a brief positive way, for a topic. We are not arguing against topics!Canvass the group to see where interest lies.Let’s begin with clarifying since we have already brainstormed our topics.[Facilitator works through clarifying and advocating with the participants. When it comes time to canvass there are different ways to do it.The topics should be posted around the group or listed on chart paper. Give each participant three ‘sticky’ dots to use to indicate their preferences. Ask participants to place sticky dots next to their topic or topics they believe should be the alliance’s research priority. Note that they can spread their dots among 2 or 3 topics or even use all three on one topic.][After canvassing, take a few minutes to review the results and identify the top-rated topics and the subtopics (if any) that fall under the topic. These will be the topics for which the group will generate research questions.]
24 Example: Gender in Schools Research Alliance Alliance goal: Provide research that informs and promotes gender equity in schoolsTopicSubtopicQuestionsTimingGirls in STEMTracking achievement differencesEncouraging interest and enrollment in STEMPersistence in STEM majorsAttainment for boysStructured InequalityNow that you have identified your priority research topics, we will move to generating research questions for the topics. We have already reviewed the characteristics of good research questions, but we also want to quickly review how one can move from topics to subtopics to research questions.This slide shows an example of a research alliance focused on gender in schools. Some of the priority topics this group identified include Girls in STEM, Attainment for Boys, and Structured Inequality. These topic areas are pretty broad and could lend themselves to many different foci and research paths. So we’ve identified subtopics. The subtopic examples provided here fall under the Girls in STEM topic.24
25 Moving from subtopics to questions This slide shows specific questions that were developed for the Girls in STEM subtopics. You can see that the subtopics each have multiple questions and different types of questions, with different levels of specificity.
26 Moving from subtopics to questions A fully specified research questionSubtopicQuestionsTimingTracking achievement differencesWhat are gender differences in STEM achievement in K12 and how have they changed over time?Do gender differences in STEM achievement vary among districts and schools?Encouraging interest and enrollment in STEMWhat can teachers do?Are single-sex schools and classrooms better?What programs help promote STEM for girls?Persistence in STEM majorsA general question that can be used to develop a set of more specific research questionsHere we see the same subtopics and questions. While all are “research questions,” they represent different types of research questions. Some are fully specified research questions and some are more general.We don’t expect the questions generated today to be necessarily fully specified. General research questions will be sufficient for us to work with to develop a set of fully specified questions. As we move to generating questions for our research topics, we don’t want you to get hung up on developing fully specified questions; if you come up with some great, but if not, the general research questions will be helpful in later development of more specific questions.
27 Generating research questions In pairs, work for 30 minutes to generate research questions related to identified priority research topic(s).Think about:What are some possible research questions for these topics and subtopics?What data would be needed to research these questions?Now we will dive into our priority topics and begin to generate research questions. We will work in pairs to generate research questions for one of the priority research topics or the subtopics within a topic. You may want to use the organizer on page 19 of your workbook to take notes.In pairs, discuss potential research questions that the topics and subtopics point to. Use the chart paper to keep track of the questions you develop. We will work in pairs for 30 minutes. Save some time at the end to list the questions you generated on newsprint. We will be sharing the newsprint in the next segment.
28 Lunch break45 minutes[Before going off for lunch have pairs post their newsprint of questions around the room.]
29 Sharing questions Post newsprint with questions from each group Review the posted newsprint and consider:What would you change or edit?What would you delete?What would you add?Briefly discuss with a partnerThe morning was fast paced and intense but we’ve done great work. Let’s take 10 minutes to walk around the room looking at the questions posted on the newsprint that each pair generated. You might want to take notepaper and a pen with you. As you walk around think about the questions and consider, what changes might you make to questions? Are there any that you don’t think are necessary? Anything missing?After we review the questions on newsprint, we’ll discuss our observations briefly with a partner, and then we’ll have opportunities to revise the questions.
30 Refining questions Whole group (small groups if whole group is large): Edit, add, delete, and combine questions as necessaryWhat kinds of studies would result from the questions (descriptive, correlational, impact evaluation)?In pairs, discuss:What is important to you?What research topic and questions can provide a meaningful research agenda?Together we will review and refine the questions so that we have a good list from which we can identify our research questions and create a research agenda.For 15 minutes discuss:What did you think of the questions?What suggestions do you have for editing the questions?Are there any that are essentially the same and can be grouped together?Are there any that should be added? Deleted?[Facilitator or note-taker edits the questions, so that at the end of this segment you have a list of questions to use for canvassing.]For 5 minutes, turn to a partner and discuss what questions are important to the alliance’s research agenda and why.
31 Prioritizing research questions Prioritize the research questions in each priority topic areaFocus Four processBrainstormClarifyAdvocateCanvassOur list contains more questions than our alliance could realistically focus on so we will prioritize our research questions to form a research agenda. We will run through this process separately for each of the priority research topics for which we generated questions.We will again use a modified Focus Four approach.We have already brainstormed and clarified.Let’s take a few minutes for the advocating step. Let’s hear from you about questions that you think should be part of the research agenda.Let’s now canvass the group to identify the 3 to 5 questions for that topic area.
32 Preliminary research agenda Review identified priority topics and questionsAny surprises?How many top-rated questions should be on the agenda? Consider:Alliance goalTypes of researchSequenceShort- and long-term nature of questionsLet’s review the result of our canvassing.[Read out the questions in order of identified priority.]Any surprises?Lets think about how many questions we should include on our agenda per topic. Thoughts?Do we have a good mix of study types? What is the best sequence for this work?Let’s look now at the Research Agenda Template found on page 23 of the Workbook.
33 Research agenda template Turn to research agenda template on page 19Use template to organize research agendaDocument and summarize the main research topics and subtopics identified, along with research questionsLet’s use the template to summarize our work today.We can fill in the main topics and the subtopics that we prioritized.Then we can list the top 3–4 questions for each area that we just prioritized.[If there is not time to complete this template together at the workshop, the facilitator should show the template, and note he/she will complete it based on the workshop notes and will share it with participants in the near future.]
34 Next steps Summary document from workshop Iterative review and refinement of agendaAgenda as a “living document”Collaborative work to:Identify appropriate existing data or data collection for studiesIdentify existing research relevant to agenda priorities and questionsDevelop study designsConduct research or identify research supportWe’ve done a lot today. We have identified our priority research topics and the main questions we have for each of these topics; together these comprise a coherent research agenda that will guide our work. However, we should not see the agenda as static. We will want to revisit it over time, especially as new research and information becomes available.Now comes the hard but also fun part: developing a study or identifying research supports to carry out the studies indicated in our agenda.
35 Thank you For more information contact: [insert contact information or customize with any special closing information]