1. Assessment: Measuring the Impact of our Programs on Students
Cathy Manduca and Ellen Iverson
SERC at Carleton College

2. Why Assess? “I don’t know what we are doing right”
“We only have 120 hours”

3. Performance Evaluation
Does the program address an important challenge?
What is the quality of the implementation?
What is the utility--are participants using it?
Is it being used in different situations?
Is it being shared by participants?
John McLaughlin

4. Why Assess? “I don’t know what we are doing right”
“We only have 120 hours”
“We will lose majors if we do X”
“Change is accomplished one funeral at a time”
“That would really help me with the dean”
“It must work because other institutions are doing it.”

5. Adoption of a Practice What results does it yield? How do you do it?
Where is it flexible?
How much does it cost?
What aspects of context influence use?
John McLaughlin

6. Powers of Observation Applied to Complex Systems

7. Experimental Design

8. Assessment and Geoscience Research
Experimental design:
What do I want to know? How can I obtain this information?
What do I think is going on? How can I test this idea?
Analysis:
Is the experimental apparatus working?
Is the experiment yielding the desired information?
How can I verify my results?
Interpretation:
How do I determine causality?
Is this the only interpretation?

9. Observing Complex Systems
Why are undergraduate research experiences valuable?
What do I think is going on?
What can I observe to test my theory?
How can I quantify my observations?

10. Geologic Strategies in Assessment
Multiple working hypotheses
Observations
Describing, classifying, coding
Inferring process and cause
Probing complex systems

11. Understanding your model through concept maps
Articulate and map change process
Go from your assumptions to desired goal
Present plausible rationale for how program works
Analyze likelihood of success
Clarify what should be measured when, how, and by whom
Depict the links between the resources, activities, or services and desired outcomes
Is a Critical thinking exercise to help you with your action plan. It lets you… goal-based, plausible, someone on one of the road checks said assessing is easy but if it fails what is plan B…this thinking helps you go through the what ifs and measurments when, how, by whom and then does it hold together

12. Internship logic model example Students gain from program:
Who then impact:
Which impact students’:
Employers perception of program
Success in finding a job in geoscience at graduation
Longevity at first job
Knowledge
Skills
Attitudes
Networks
Resources
Students participate in internship initiative
Other students’ perception of program
So I’d like to take you through a simplified example for adding a student internship component to a program. KSAs- knowledge of industry S: communication, ability to work with team, balance competing demands, A greater confidence, greater awareness of relevance of education

13. Are you implementing your model? Is your model correct?
Is what you are mapping what you think should happen?
What are the impacts?
Are the impacts caused by your planned actions?

14. Instruments to measure interventions
Qualitative – self report and artifacts
Open-ended survey questions
Embedded assessment
Artifacts such as reports or online discussion
Interviews and focus groups
Observations
Quantitative - counting
Data sets
Surveys
Counting – how many students, how many employers, how many internships, % partcipating and % satisfied (likert scale)
Open-ended survey or interview – end of internship survey student /employer
Embedded assessment – journal student keeps or could be online discussion artifacts or the assessment can be its own intervention by students giving oral report of internship back to intro geo class
Focus groups to get a sense of their expectations/structure ahead of time

15. Why Assess? “I don’t know what we are doing right”
“We only have 120 hours”
“We will lose majors if we do X”
“Change is accomplished one funeral at a time”
“That would really help me with the dean”
“It must work because other institutions are doing it.”
What is the purpose?
Who is the audience?
What will make a compelling argument?