The Evolution of Instruction Assessment - the UGA Perspective Diana Hartle Deb Raftus Amy Watts

What came before Half sheet handout to BIOL1107 (and other) classes SurveyMonkey for Academic Enhancement classes tinyurl.com/7kek7/ tinyurl.com/qguh6/ tinyurl.com/7kek7/ tinyurl.com/qguh6/

A dream born, a committee formed Charge from the University Librarian to assess instruction program. More than attendance numbers were desired. Committee formed with reference and other instruction librarians.

Our Quiz Who will be assessed? Students in introductory level classes across the curriculum.

How do we give the quiz? Rejected ideas Paper quizzes Collecting and correlating data Disseminating data Internet based quiz services Expense Reliability

WebCT is the winner! Student familiarity with interface Availability Technical and administrative support Data collection possibilities

What to ask? Learning outcomes were defined by: ACRL standards for Information Literacy Standard 2 – The information literate student accesses needed information effectively and efficiently. Outcomes include:  Identifies keywords, synonyms, and related terms for the information needed Local concerns

Skills / knowledge to be assessed Research strategies Moving from a question to a search strategy Narrowing/broadening topic Identifying keywords, synonyms Truncation and wildcards Boolean operators Choosing which database/resource to search Revising search based on results

Skills / knowledge to be assessed Locating full text of a journal article using a: Direct full-text link in a database Link to another database Electronic Journal Locator Library OPAC for print Using citation information and OPAC record to identify and locate specific item needed

Revising the quiz after Year 1 Merged Science/Humanities into one quiz Added questions for evaluative comments Did away with sequential elements to quiz (“Based on answer to question 2…”) Added graphics to questions Changed wording on questions

Results Snapshot: 1 st semester (Fall 2005) Humanities quiz: 199 students, Mean score 74.7% Science quiz: 437 students, Mean score 83.5%

Results from Science Quiz QuestionMeanDiscrimination #193.2%.50 #291.1%.49 #391.8%.49 #490.9%.49 #597.9%.57 #642.6%.37 #787.7%.53 #897.3%.65 #969.7%.42 #1071.5%.49

Results from Humanities Quiz QuestionMeanDiscrimination #182.5%.46 #271.8%.48 #362.1%.38 #481.1%.47 #595.6%.59 #651.0%.36 #797.1%.64 #895.6%.57 #945.1%.49 #1064.6%.36

Snapshot for Fall ,029 students took the quiz. Overall score was 81.5% correct. Spring students took the quiz 83.6% was the average score

Results from Quiz 2, Fall 2006 Graded questions QuestionMeanDiscrimination #194.3%0.48 #281.6%0.56 #392.6%0.52 #491.5%0.48 #543.3%-0.37 #671.7%0.48 #795.7%0.51

Results from Quiz 2, Spring 07 Graded Questions QuestionMeanDiscrimination #192.8%0.67 #284.3%0.61 #392.1%0.68 #491.0%0.71 #554.2%0.54 #676.6%0.58 #794.4%0.65

Results – evaluative questions Weeding responses – determining what counts as a “valid” response Coding responses – applying a standardized set of response possibilities to “valid” responses Crosscheck the coding If needed, combine categories

What did they learn…

Selected responses to question #9 I learned so much it was crazy. I didn't know the best way to use GALILEO. Now I know how to use it quickly and efficiently in order to get the results I want. I found out how to use Galileo, which I will now use a lot because of its credibility. I also did not know the availability of help by the librarians. That it is more important how you search than what you search. Being more specific with searches greatly affects how much time you have to put into a search.

Still confused about…

Selected responses to #10 Why we weren't told about this earlier. Ask me when I'm doing it on my own. I feel like I have it now, but that doesn't mean I can do it when I don't have someone leading me through it. I am overwhelmed with the system and am not sure how to find the article that I need. Not so much as confused, but a little overwhelmed. I like the fact that the instructor used an example to lead us through the search. I just think that it will take personal practice to really understand how to use the library. I think it would be helpful to have us work on something outside of class immediately to see if we have any questions and make sure we can do it on our own. I am not really confused about the search engines but overwhelmed because of the amount of information given to me in a little amount of time. Why I had to sit through an hour lecture... when only 30 minutes was necessary.

There must be 50 ways to spell truncation tranctuating truction truncationg trunkation trumcations trunkating troncation trunk trucation asterix trick asterick asterik asteric astricks asteriks astricts astericks asterics

We don’t know whether to laugh or to cry… That we actually have librarians! Well, first off, I finally learned where the library was. Nothing; this guy made it perfectly clear. He is amazing. He should get a raise and take over President Adams job. Why would there be a physical subscription to a journal but not an electronic when generally speaking the electronic is or should be cheaper... Nothing! It was brilliant BRILLIANT! Why everyone thinks Galileo is so great. I never have needed to use it. All the articles I need are available any time online through the UGA Libraries homepage. I reviewed library skills I learned in 9th grade.

Student perception of value of instruction session Do you think this library instruction session will help you with your research?

Future of quiz Possible additional questions Year of student Have they had instruction session before? Data Analysis Frequency of administration Migrate to new version of classroom management software

Thanks for coming! Don’t forget to take a bookmark. Your presenters may be contacted at: