1. Assessing Undergraduate Sustainability Knowledge Campus Wide:
Environmental and Social Sustainability Lab
Adam Zwickle - OSU
Tomas Koontz - OSU
Andrew Bodine - OSU
Mark Stewart – UMD
Nicole Horvath - UMD
Assessing Undergraduate Sustainability Knowledge Campus Wide:
From design to implementation and analysis

2. Overview
How we developed our Assessment of Sustainability Knowledge (ASK)
Why an ASK is important & how it can help
An aside on Knowledge and Literacy
Conducting an assessment
Thinking long term…

3. Developing an assessment
Built upon the “triple bottom line”, the “three legged stool”, the “3 p’s”
Environmental (planet)
Economic (prosperity)
Social (people)
Basic knowledge from each knowledge domain?
~OR~
Knowledge bridging the different spheres?
“knowledge domains”

4. Developing an assessment
Replicated questions used in the past
Coyle, “Environmental Literacy in America.”
Solicited topics and questions from experts
Held expert focus groups
Pilot tested among professors, graduate, and undergraduate students
Narrowed down to 30 questions

5. Developing an assessment
Distributed those 30 to OSU students
Used IRT to throw out 14
Added UMD’s 16
Distributed those to OSU and UMD students
Used IRT to throw out 2
Current ASK has 28 items:
ess.osu.edu

6. Conceptualizing sustainability knowledge
What is the most common cause of pollution of streams and rivers?
Surface water running off yards, city streets, paved, lots, and farm fields
Conceptualizing sustainability knowledge
Which of the following is the most commonly used definition of sustainable development?
Meeting the needs of the present without compromising the ability of future generations to meet their own needs
Social
Environmental
Sustainability
ADD EXAMPLE QUESTIONS
Many economists argue that electricity prices in the U.S. are too low because…
They do not reflect the costs of pollution from generating the electricity
Economic

7. How is this helpful?

8. Need for measuring knowledge
University goals-
More along the lines of:
“Become carbon neutral by 2050”
Less common:
“Create sustainably minded citizens of tomorrow”
Can help track improvement over time
By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.

9. Need for measuring knowledge
Serves as an evaluation for specific academic efforts:
Interdisciplinary programs
Sustainability majors/minors
STARS Credit:
ER 6 (STARS 2.0) – 3 points available
By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.

10. “Knowledge” vs. “Literacy”
Can be objectively measured
Can be used to evaluate academic programs
Literacy
= Knowledge
Knowledge + values, attitudes, and behaviors
Can be used to evaluate outreach efforts, campaigns
By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.

11. Measuring Behaviors, Values, and Attitudes
Make sure you are asking the right questions
Collaborate with sustainability departments to target specific behaviors (e.g., leaving lights on)
Include questions on behavioral barriers
Other behaviors that may get at the same concept
By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.

12. Measuring Behaviors, Values, and Attitudes
Collaborate with academic departments to develop a good survey design
No need to reinvent the wheel
Each survey could be a master’s thesis
By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.

13. Conducting an Assessment

14. Conducting an Assessment
Find your partners!
IRB approval
Required for publication
Exempt status
Registrar approval*
Student’s s, majors, and demographics
Survey software*
We use Qualtrics, but there are others
*If a large scale assessment is planned

15. Maximizing Response Rates
Maximizing response rates is important to reduce uncertainty about how well your completed sample matches the population of interest.
Dillman (2008) and others have long studied how to maximize response rates for surveys that were telephone, mailed, or in-person.
There is growing research on electronic survey response rates.

16. Research Methods
We compare response rates from five different survey implementations:
1: 2012 spring OSU (n=10,000)
2: 2013 spring OSU sample A (n=10,000)
3: 2013 spring OSU sample B (n=10,000)
4: 2013 spring OSU School of ENR (n=538)
5: 2013 spring UMD (n=10,000)
We tried different treatments and tracked survey responses with survey software (Survey Monkey and Qualtrics)

17. Key Variables Affecting Response Rates among College Students
Timing
When to send the invitation and reminders
Incentives
Questionnaire format
Long list of questions vs. more page clicks
text
Who it is from

18. Timing Matters Time of Semester: Time of Day: Reminders are critical
Last week of semester and into finals week
Middle of semester
Time of Day:
6:00 am
6:00 pm
Reminders are critical
big spike in completed surveys after each reminder, with a fast decay (see next slide)

19. Timing Matters Overall, you want students sitting at their computers…
but wanting to be distracted

20. Incentives
Some disagreement in research on best way to provide incentives
ahead of time (Dillman 2008)
randomly select winner of 1 big prize
give more/all respondents smaller prizes.
May impact the validity of the data

21. Survey Responses
UMD
n=1,556
Day
OSU
n=2,621
Day

22. text
How the invitation s are phrased affects response rates.
Besides making the invitation personal, clear, and as short as possible, prior research has found that who the invitation comes from matters.
An appeal from a trusted authority increases response rates.

23. Email text: Appeal from Authority
We compared an appeal from authority versus an appeal from a peer (student).
Two surveys, A and B, had an appeal from a higher authority (School Director or University VP) for first 3 contacts.
For the 4th contact kept the higher authority for A, but switched B to have to an appeal from a grad student

24. Email text: Appeal from Authority
Results:
= additional 538
= additional 348
over the next 65 hours

25. Questionnaire Format Trade-off: We analyzed respondent drop-outs spots
Long list of questions to scroll down
Shorter lists with a page click to get to the next page
We analyzed respondent drop-outs spots
Highest spots were just after clicking to the next page
We recommend finding a balance…

26. Non-Respondent Bias
Even after efforts to maximize response rates, are the non-respondents different from the respondents?
We conducted a non-respondent short survey (5 questions + some demographics).
Results indicate that non-respondents are slightly but significantly less knowledgeable about sustainability topics, but no difference in GPA or pro-environmental behaviors.

27. Planning ahead… For longitudinal studies: Write a multiple year IRB
Let registrar know this is a yearly survey
Partnering up with an academic department for help
Sustainability office
Academic departments with survey expertise (interdisciplinary social science, communications, sociology, political science, psychology)

28. Acknowledgements Funded by: & Natural Resources
The Ohio State Office of Sustainability
OSU’s School of Environment
& Natural Resources

29. Thank You! Questions? Environment and Social Sustainability Lab Email
ess.osu.edu
Contains:
This presentation
The 28 question ASK
Forthcoming article

31. Example Analysis Compare sub score by major Economic score
Aerospace engineering: 4.00
Animal science: 3.15
Finance: 3.02
English: 2.80
Accounting: 2.63
Best to worst

32. Phase 2
Combined our shortened set with a separate assessment developed at the University of Maryland
Administered survey at both campuses

33. Phase 2 - Analysis IRT to compare information richness of questions
across campuses
using local descriptives (“Fishermen on Lake Erie” vs. “Fishermen in the Chesapeake Bay”)
different ways of asking about the same topic

34. Survey Implementation
Best practices to maximize response rates:
Invitation script (from authority vs. student)
Page length vs. number of pages
Use of incentives
Time of academic calendar
Time of day, day of week
Reminders

35. Results Total Environmental Social Economic 73% Mean: 11.08/16
6 questions
5 questions
Mean: 4.39/6
Mean: 3.55/5
Mean: 3.03/5
SD: 1.48
SD: 1.23
SD: 1.27
73%
71%
61%
Average scores %
Total
16 questions
Mean: 11.08/16
SD: 3.21
69%

36. Example Analysis ANOVA F(3,1330)=8.09, p<.001 Sig differences
Display graphs
Bar graphs
Flag
“Out of 16”
ANOVA
F(3,1330)=8.09, p<.001

37. Moving Forward Adjust the instrument: Be clear about limitations:
Some answers change over time
Develop test bank of validated questions – try this on your campus and send us questions
Be clear about limitations:
Multiple choice format
Measures knowledge only
Publish
Phase 1 – International Journal of Sustainability in Higher Education, 2014
Discussion:
Weakness: some answers change over time, should aim for static knowledge concepts.
Give us your questions

38. Moving Forward Goals: Hopes:
Evaluate sustainability themed majors, programs, and courses
Serve as an indicator for comparison within institutions
Encourage friendly competition
Hopes:
Help solidify the concept of “sustainability”
Provide some credence for more abstract educational goals

39. Appendix
Phase Question Sustainability Literacy Assessment

40. Questions - Environmental
What is the most common cause of pollution of streams and rivers?
Dumping of garbage by cities
Surface water running off yards, city streets, paved lots, and farm fields
Litter near streams and rivers
Waste dumped by factories
Don't know
Ozone forms a protective layer in the earth's upper atmosphere. What does ozone protect us from?
Acid rain
Climate change
Sudden changes in temperature
Harmful UV rays

41. Questions - Environmental
What is the name of the primary federal agency that oversees environmental regulation?
Environmental Protection Agency (the EPA)
Department of Health, Environment, and Safety (the DHES)
National Environmental Agency (the NEA)
Federal Pollution Control Agency (the FPCA)
Don't know
What is the primary benefit of wetlands?
Promote flooding
Clean the water before it enters lakes, streams, rivers, or oceans
Keep the number of undesirable plants and animals low
Provide good sites for landfills

42. Questions - Environmental
Which of the following is an example of sustainable forest management?
Setting aside forests to be off limits to the public
Never harvesting more than what the forest produces in new 
growth
Producing lumber for nearby communities to build affordable housing
Putting the local communities in charge of forest resources
Don’t know
In the U.S., what do we currently do with the nuclear waste generated by nuclear power plants?
Use it as nuclear fuel
Sell it to other countries
Dump it in landfills
Store and monitor the waste
Don't know

43. Questions - Social
Which of the following is the most commonly used definition of sustainable development?
Creating a government welfare system that ensures universal 
access to education, health­care, and social services
Meeting the needs of the present without compromising the ability of future generations to meet their own needs
Setting aside resources for preservation, never to be used
Building a neighborhood that is both socio-demographically
 and economically diverse
Don’t know
The wealthiest 20% of people in the U.S. own approximately what percent of the nation’s privately held wealth?
20%
35%
50%
85%
Don't know

44. Questions - Social
Over the past 3 decades, what has happened to the difference between the wealth of the richest and poorest Americans?
The difference has increased
The difference has stayed about the same
The difference has decreased
Don’t know
Higher levels of education generally lead to...
Lower levels of voter turnout
Greater annual earnings
Larger family size
Higher self esteem

45. Questions - Social
Which of the following populations has the highest rate of growth?
North America
Europe
China
Africa
Don’t know

46. Questions - Economic
Many economists argue that electricity prices in the U.S. are too low because…
They do not reflect the costs of pollution from generating the 
electricity
Too many suppliers go out of business
Electric companies have a monopoly in their service area
Consumers spend only a small part of their income on energy
Don’t know
Which of the following countries has now passed the U.S. as the biggest emitter of the greenhouse gas carbon dioxide?
China
Sweden
Brazil
Japan

47. Questions - Economic
Which of the following is a leading cause of the depletion of fish stocks in the Atlantic Ocean?
Fishermen seeking to maximize their catch
Reduced fish fertility due to genetic hybridization
Ocean pollution
Global climate change
Don’t know
Which of the following is the most commonly used definition of economic sustainability?
Maximizing the share price of a company's stock
Long term profitability
When costs equal revenue
Continually expanding market share

48. Questions - Economic
Which of the following is the primary reason that gasoline prices have risen over the last several decades in the U.S.?
Growing percentage of gas stations owned by large 
corporations
Increasing oil discoveries overseas
Higher rates of state and federal gasoline tax
Increasing global demand for oil
Don’t know