1. This work is supported by a National Science Foundation (NSF) collaboration between the Directorates for Education and Human Resources (EHR) and Geociences (GEO) under grant DUE - 1125331 Monica Bruckner, SERC Thursday, January 7, 2016 9am Pacific | 10am Mountain 11am Central | 12pm Eastern Getting Ready to Pilot (Setting Up and Collecting Data) Webinar

2. By the end of this presentation, you should be able to: 1.Know how to find more information as you need it: – web resources, – webteam liaison, – assessment consultant, – team leads 2.Know the timing for setting up your materials for testing in the classroom and when to collect data 3.Understand what data will be collected and why 4.Know how to effectively use your course status page

3. Defining the Audience Data collection varies by group: When I refer to author testers, I mean faculty who are on a multi-institutional authoring team that developed an InTeGrate course or module (or serving as a proxy for one of these authors) When I refer to non-author testers (NATs), I am referring to those who are testing materials originally designed by others (e.g. implementation programs, research team members).

4. Identify Course & Module Learning Goals Identify learning objectives for individual lessons Determine how to assess and measure student success on goals and objectives Design teaching resources and materials to match assessments Plan Instructional Strategies to implement teaching resources Test/Pilot Materials, Make changes

5. *All material author pilots must collect/provide all of these data. IP testers do not need to upload the summative assessments (but may need to provide to team lead, if directed).

6. Quick Checklist: Before Teaching Materials must pass rubric (author testers) or be approved by team lead Get IRB approval Fill out course set up form – Links to these forms (and other resources) are on the webinar event page: http://serc.carleton.edu/integrate/workshops/we binars/2015_2016/pilot.html#resources

7. Quick Checklist: First Week of Course Upload roster to course page Distribute consent forms for students to sign (signed) or a letter telling students about the project; give opt out mechanism (implied) Administer link to pre-course attitudinal survey (and check on completion progress) Administer pre-course GLE MC questions *Make sure all data is labeled with student ID number and names are redacted (except for consent forms)

8. Quick Checklist: During Course Administer and collect summative assessment (for course/module authors and research team) *Make sure all data is labeled with student ID number and names are redacted. Upload/send as soon as possible so we can start processing data.

9. Quick Checklist: Last Two Weeks of Course Re-upload roster to course page if it’s changed Administer link to post-course attitudinal survey (and check on completion progress) Administer post-course GLE MC questions Administer two post-course GLE essays Administer summative assessment if haven’t yet done so (for course/module authors and research team) Upload proxy for student engagement Let web support person know if you had students opt out of implied consent or not *Make sure all data is labeled with student ID number and names are redacted. Upload/send as soon as possible so we can start processing data.

10. Quick Checklist: After Course Complete faculty reflection. For course/module authors: Work on revision plan with team members, get it reviewed by team lead and/or assessment consultant, start making revisions after team lead/assessment consultant approves.

11. *All material author pilots must collect/provide all of these data. IP testers do not need to upload the summative assessments (but may need to provide to team lead, if directed).

12. Pre- & Post-course Geoscience Literacy Assessment (GLE 8 MC questions; 16 MC questions for research team members) – Given on ‘bubble sheets’ in a protected environment (e.g. in class); online version available if you can give it securely Two post-course short essay questions (interdisciplinary/global challenges & systems thinking) Gives an idea of the what geoscience literacy looks like across different departments and courses and course levels and helps to characterize the relationship between the different InTeGrate modules and courses on geoscience literacy Remember: This is project-level data – It’s okay if your materials don’t explicitly cover the GLE MC content. All materials that pass the rubric should be designed such that students have a general understanding of the essay content and have a well-thought out response.

13. Collected by course/module authors and research team Measure level of students’ understanding of learning related to the course/module overall goal Should be given in the same format for all author team members http://serc.carleton.edu/integrate/info_team_members/currdev/assessment.html#sum

14. Pre- & post-course motivation and career interest survey (related to project evaluation) Probes if students exposed to ITG materials have a shift in attitude or career preferences. Each course has a unique URL for these surveys, which is created when you fill out course set up form (these are given online) Also referred to as the ITG Attitudinal Instrument or ‘IAI’ Remember: Like the GLE, this is project-level data – It’s okay if your materials don’t specifically address/influence all the questions here.

15. Course Information Demographic data Faculty Reflection Surveys For non-author testers: Complete at end of course For author testers: Checkpoint 4 (after passing rubric) Notes while teaching (individual reporting page) After revisions

16. Conducted and set up through SERC using course set-up form

17. Classroom Pilot & Data Collection *IRB approval **Research team members, implementation programs, and other non-author testers should obtain IRB approval as soon as possible.**

18. Make contact with your IRB early Know that IRB procedures vary by institution Give the project a list of which students do not consent Carleton College has EXEMPT status You will need Carleton letter Consent form examples are posted http://serc.carleton.edu/integrate/info_team_members/currdev/documents/index.html Ask for help! Your team lead should be able to answer basic IRB questions; if they don’t know the answer, the SERC evaluation team can help. IRB: What you Need to Know

19. Give students option to opt out at beginning of term through one of two ways: Implied consent Signed consent – we need student name, ID, opt in/out checkbox, and signature IRB: What you need to know What about consent?

20. Data Collection Want to revisit the process? Instruction pages provide guidance. Linked from the webinar event page: http://serc.carleton.edu/integrate/workshops/webinars/2015_2016/pilot.html#resources

21. Providing your Data to the Project There are two ways to provide your data to the project: Scan and upload it to your course page (preferred) Mail a photocopy of the materials to the SERC office at Carleton College, including a cover sheet with a clear description of what they are (e.g. what course they relate to, what data are included). Packages should be sent with a tracking number regardless of shipment method (USPS, FedEx, etc.). Please forward this tracking number to MJ Davenport so that we know when to expect the package (mdavenport (at) carleton (dot) edu): Science Education Resource Center 200 Division Street, Suite 210 Northfield, MN 55057 Attn: MJ Davenport

22. Getting Started: Data Collection Step 1: Fill out the course set up form Submitting this form creates your course page, survey URLs, and provides the project personnel with info they need about your course

23. Getting Started: Data Collection Step 1: Fill out the course set up form Filling out the course set up page requires the following info: Your name and info about the class you’re collecting data in (course name, number of students, level, focus) Info about your role in the project (materials authors OR non-author testers, which includes IP teams and research teams) Course start/end date Estimated dates you plan to use InTeGrate materials in class Instructions for students to identify their ID number IRB info and consent type info When you plan to administer GLE, attitudinal survey, and summative assessments (if applicable) How you plan to measure student engagement – this is usually done through attendance or number of assignments turned in out of total possible **You can edit this information from your resulting course page if it changes!

24. After submitting the form, you’ll see this message (and get an email with the same info) Bookmark this page! (You can also find it on your acct page)

25. https://serc.carleton.edu/integrate/info_team_members/currdev/status_page.html The Course Status Page

26. Uploading your roster early helps us track completion rates for your surveys and can help ID unknown student IDs early. Can upload revised roster at any time (e.g. if students add or drop course) All student IDs are encrypted in our system, so student anonymity is protected

27. Paste in your roster and use the drop down menu to identify which students have/have not completed the attitudinal surveys. Useful tool for prompting students to complete these.

28. Student IDs are encrypted when they enter our system If a student turns in work that doesn’t match an ID on your roster, it’ll show up in this list. You can correct it IF AND ONLY IF you can ID that student (e.g. they put in an extra zero) Once corrected, we cannot make further changes to the ID

29. Give files descriptive names AND check box, if applicable Ensure names are redacted (except signed consent forms) and that all data is readable and labeled with student IDs Upload ungraded student work. If already graded, cover final grade if possible.

30. Key: Red dates – date has passed Green dates – we got what we needed “new” – file uploaded “accepted” – data ready for processing on our end “copied” – data copied and ready for review by assessment team Take home – much of this depends on our importing the data; it may not be updated immediately, but if several weeks have passed and you have a concern, let us know

32. Testing Process and Assessment What questions do you have about testing and data collection? If you have further questions, contact your team lead or webteam support person: Course/Module authors: Monica Bruckner mbruckne@carleton.edumbruckne@carleton.edu Research Team members: Kristin O’Connell koconnell@carleton.edukoconnell@carleton.edu Implementation Program participants: John McDaris jmcdaris@carleton.edujmcdaris@carleton.edu

33. Thanks for your time! We encourage you to use this time to fill out the course set up form if you haven’t yet done so. If you have already filled it out, you may want to explore your course page and upload your roster, if you have it.