1. Developing an Assessment of Sustainability Knowledge (ASK) for undergraduate students Adam Zwickle Tomas Koontz Andrew Bodine Kristina Slagle Environmental and Social Sustainability Lab

2. Overview  Need for a systematic way of measuring knowledge of sustainability concepts  Developing an assessment  Results  Where we go from here 2

3. Need for measuring knowledge  Lacking in the scientific community  “Sustainability” has been defined and measured a myriad of ways  Often focused merely on the environmental aspect  Commonly tangled with values, attitudes, and beliefs  Needed to operationalize abstract goals… 3

4. Need for measuring knowledge  University goals-  More along the lines of:  “Become carbon neutral by 2050”  Less common:  “Create sustainably minded citizens of tomorrow” 4

5. Developing an assessment  Built upon the “triple bottom line”, the “three legged stool”, the “3 p’s”  Environmental (planet)  Economic (prosperity)  Social (people)  Basic knowledge from each knowledge domain? ~OR~  Knowledge bridging the different spheres? 5

6. Developing an assessment  Replicated questions used in the past  Coyle, 2005. “Environmental Literacy in America.”  Solicited topics and questions from experts  Held expert focus groups  Pilot tested among professors, graduate, and undergraduate students  Narrowed down to 30 questions 6

7. Conceptualizing sustainability knowledge Social Economic Environmental Sustainability 7

8. Phase 1  Administered 30 question assessment  Through an online survey  Sent to over 10,000 currently enrolled undergraduates  14.3% response rate 8

9. Phase 1 - Analysis  Utilized Item Response Theory (IRT) To remove 14 questions  IRT-  Developed in the educational testing field  Used for standardized tests (GRE) 9

10. An example of IRT… Information Correct Incorrect 50/50 threshold “Over the past 3 decades, what has happened to the difference between the wealth of the richest and poorest Americans?” 10

11. Questions - Environmental  What is the most common cause of pollution of streams and rivers?  Ozone forms a protective layer in the earth's upper atmosphere. What does ozone protect us from?  What is the name of the primary federal agency that oversees environmental regulation?  What is the primary benefit of wetlands?  Which of the following is an example of sustainable forest management?  In the U.S., what do we currently do with the nuclear waste generated by nuclear power plants? 11

12. Questions - Social  Which of the following is the most commonly used definition of sustainable development?  The wealthiest 20% of people in the U.S. own approximately what percent of the nation’s privately held wealth?  Over the past 3 decades, what has happened to the difference between the wealth of the richest and poorest Americans?  Higher levels of education generally lead to...  Which of the following populations has the highest rate of growth? 12

13. Questions - Economic  Many economists argue that electricity prices in the U.S. are too low because…  Which of the following countries has now passed the U.S. as the biggest emitter of the greenhouse gas carbon dioxide?  Which of the following is a leading cause of the depletion of fish stocks in the Atlantic Ocean?  Which of the following is the most commonly used definition of economic sustainability?  Which of the following is the primary reason that gasoline prices have risen over the last several decades in the U.S.? 13

14. Results  Environmental 6 questions Mean: 4.39/6 SD: 1.48 73%  Social 5 questions Mean: 3.55/5 SD: 1.23 71%  Economic 5 questions Mean: 3.03/5 SD: 1.27 61%  Total 16 questions Mean: 11.08/16 SD: 3.21 69% 14

15. Example Analysis ANOVA F(3,1330)=8.09, p<.001 15

16. Example Analysis  Compare sub score by major  Economic score  Economics: 3.82  Aerospace engineering: 4.00  Animal science: 3.15  Finance: 3.02  English: 2.80  Accounting: 2.63 16

17. Phase 2  Combined our shortened set with a separate assessment developed at the University of Maryland  Administered survey at both campuses 17

18. Phase 2 - Analysis  IRT to compare information richness of questions  across campuses  using local descriptives (“Fishermen on Lake Erie” vs. “Fishermen in the Chesapeake Bay”)  different ways of asking about the same topic  Structural equation modeling  Compare question types (multiple answer?)  Knowledge structures (3 or 1 domain?) 18

19. Structural equation modeling Social Economic Environmental Sustainability 19

20. Survey Implementation  Best practices to maximize response rates:  Invitation script (from authority vs. student)  Page length vs. number of pages  Use of incentives  Time of academic calendar  Time of day, day of week  Reminders 20

21. Moving Forward  Adjust the instrument:  Some answers change over time  Develop test bank of validated questions – try this on your campus and send us questions  Be clear about limitations:  Multiple choice format  Measures knowledge only  Publish  Phase 1 – International Journal of Sustainability in Higher Education, 2014 21

22. Moving Forward  Goals:  Evaluate sustainability themed majors, programs, and courses  Serve as an indicator for comparison within institutions  Encourage friendly competition  Hopes:  Help solidify the concept of “sustainability”  Provide some credence for more abstract educational goals 22

23. Acknowledgements Funded by:  The Ohio State Office of Sustainability  http://sustainability.osu.edu/ http://sustainability.osu.edu/  OSU’s School of Environment & Natural Resources  http://senr.osu.edu/ http://senr.osu.edu/ 23

24. Thank You!  Environment and Social Sustainability Lab  www.ess.osu.edu www.ess.osu.edu  Contains:  This presentation  The 16 phase 1 questions with multiple choice answers  Email  Zwickle.1@osu.edu Zwickle.1@osu.edu  Koontz.31@osu.edu Koontz.31@osu.edu Questions? 24

25. Appendix  Phase 1 -16 Question Sustainability Literacy Assessment 25

26. Questions - Environmental  What is the most common cause of pollution of streams and rivers?  Dumping of garbage by cities  Surface water running off yards, city streets, paved lots, and farm fields  Litter near streams and rivers  Waste dumped by factories  Don't know  Ozone forms a protective layer in the earth's upper atmosphere. What does ozone protect us from?  Acid rain  Climate change  Sudden changes in temperature  Harmful UV rays  Don't know 26

27. Questions - Environmental  What is the name of the primary federal agency that oversees environmental regulation?  Environmental Protection Agency (the EPA)  Department of Health, Environment, and Safety (the DHES)  National Environmental Agency (the NEA)  Federal Pollution Control Agency (the FPCA)  Don't know  What is the primary benefit of wetlands?  Promote flooding  Clean the water before it enters lakes, streams, rivers, or oceans  Keep the number of undesirable plants and animals low  Provide good sites for landfills  Don't know 27

28.  Which of the following is an example of sustainable forest management?  Setting aside forests to be off limits to the public  Never harvesting more than what the forest produces in new growth  Producing lumber for nearby communities to build affordable housing  Putting the local communities in charge of forest resources  Don’t know  In the U.S., what do we currently do with the nuclear waste generated by nuclear power plants?  Use it as nuclear fuel  Sell it to other countries  Dump it in landfills  Store and monitor the waste  Don't know Questions - Environmental 28

29. Questions - Social  Which of the following is the most commonly used definition of sustainable development?  Creating a government welfare system that ensures universal access to education, health­care, and social services  Meeting the needs of the present without compromising the ability of future generations to meet their own needs  Setting aside resources for preservation, never to be used  Building a neighborhood that is both socio-demographically and economically diverse  Don’t know  The wealthiest 20% of people in the U.S. own approximately what percent of the nation’s privately held wealth?  20%  35%  50%  85%  Don't know 29

30. Questions - Social  Over the past 3 decades, what has happened to the difference between the wealth of the richest and poorest Americans?  The difference has increased  The difference has stayed about the same  The difference has decreased  Don’t know  Higher levels of education generally lead to...  Lower levels of voter turnout  Greater annual earnings  Larger family size  Higher self esteem  Don’t know 30

31. Questions - Social  Which of the following populations has the highest rate of growth?  North America  Europe  China  Africa  Don’t know 31

32. Questions - Economic  Many economists argue that electricity prices in the U.S. are too low because…  They do not reflect the costs of pollution from generating the electricity  Too many suppliers go out of business  Electric companies have a monopoly in their service area  Consumers spend only a small part of their income on energy  Don’t know  Which of the following countries has now passed the U.S. as the biggest emitter of the greenhouse gas carbon dioxide?  China  Sweden  Brazil  Japan  Don’t know 32

33. Questions - Economic  Which of the following is a leading cause of the depletion of fish stocks in the Atlantic Ocean?  Fishermen seeking to maximize their catch  Reduced fish fertility due to genetic hybridization  Ocean pollution  Global climate change  Don’t know  Which of the following is the most commonly used definition of economic sustainability?  Maximizing the share price of a company's stock  Long term profitability  When costs equal revenue  Continually expanding market share  Don’t know 33

34. Questions - Economic  Which of the following is the primary reason that gasoline prices have risen over the last several decades in the U.S.?  Growing percentage of gas stations owned by large corporations  Increasing oil discoveries overseas  Higher rates of state and federal gasoline tax  Increasing global demand for oil  Don’t know 34