Presentation is loading. Please wait.

Presentation is loading. Please wait.

Perception and Reality: Creating a Baseline for Longitudinal Study of Information Literacy Skills at the Outset of a High School Laptop Program Roberta.

Similar presentations


Presentation on theme: "Perception and Reality: Creating a Baseline for Longitudinal Study of Information Literacy Skills at the Outset of a High School Laptop Program Roberta."— Presentation transcript:

1 Perception and Reality: Creating a Baseline for Longitudinal Study of Information Literacy Skills at the Outset of a High School Laptop Program Roberta Jordan, Presenter Glickman Library, University of Southern Maine December 3, 2010

2 Whats the premise? PERCEPTION V. REALITY

3 Perception: I am able to use the terms AND and OR correctly in my searching.

4 Check one: _____ DO NOT UNDERSTAND _____ NEVER _____ SOMETIMES _____ FREQUENTLY _____ ALWAYS

5 62% of surveyed students expressed high confidence (i.e. answered frequently or always) in their ability to use Boolean operators in an information search.

6 Reality: Which of the following would retrieve the MOST results in an on-line search? a. Stars NOT planets b. Stars OR planets c. Stars AND planets d. Stars INSTEAD OF planets e. Dont know.

7 Just under 13% of students who took a follow-up information literacy skills test chose the correct answer (b) to the multiple choice question on Boolean operators.

8 Perception v. Reality = BASELINE The difference between the two numbers (62 versus 13 percent) constitutes a realistic baseline against which to measure gains in student achievement.

9 PROJECT BACKGROUND

10 My Interests Information Literacy/Skills Testing as a means to inform instruction Real-world application for capstone project for my M.L.S.

11 November 2009: 700 students in grades 9 through 12 at Morse High School received laptops for school and at-home use through the Maine Laptop Technology Initiative (MLTI). Approximately 65 staff members also received laptops the year before. Commitment by R.S.U. 1: $750,000+ over four years Goal: Be a successful participant in a pilot program in selected high schools; support attainment of Maine Learning Results, and help prepare students for a future economy that will rely heavily on technology and innovation. (Task Force on Maines Learning Technology Endowment, 2001, p. vi). An Opportunity

12

13

14

15

16 The Challenge -Document student skill levels at the program start; -Demonstrate increased student achievement attributable to laptop use (the holy grail of research efforts on 1:1 laptop use) to help justify the community investment.

17 Project Purpose Develop tools for collecting baseline data on information literacy skills of Morse High School students at the outset of 1:1 laptop initiative.

18 Why information literacy is a potentially good curricular area for demonstrating achievement: -a subject that integrates use of technology and other important academic skills (research, synthesis); -ample evidence suggesting student skills are lacking in this area; -there are widely accepted standards that guide instruction against which to measure improvement; -there are cost-effective ways to collect data.

19 Two Tools Were Developed Over the Course of the Project: 1.A Perceptions Survey -intended to provide a snapshot of student and teacher confidence in their ability to integrate technology, research, and information-seeking skills 2.An Information Literacy Skills Test -intended to measure how much students perceived skills matched up in reality.

20 PERCEPTIONS SURVEY Based on the work of Morrisette (2007) and Grimble and Williams (2004), the MHS perceptions survey asks students to express their confidence levels in five areas relating to information-seeking and technology. Point Access Skills Research Skills Database Skills Internet Skills Technology Skills

21 POINT ACCESS SKILLS DO NOT UNDERSTAND NEVERSOMETIMESFREQUENTLYALWAYS P1. I am able to access our high school library web site using a computer. P2. I feel competent using the on-line library catalog to search for books. P3. I know how to locate books on the shelves after looking them up on the on-line catalog. P4. I am able to locate and use the databases available on our high school library website using the blue sheet. P5. I am able to search for books that are in other Maine libraries using MaineCAT.

22 RESEARCH SKILLS DO NOT UNDERSTAND NEVERSOMETIMESFREQUENTLYALWAYS R1. I am able to develop questions focusing on my information needs. R2. I know what a keyword search is, and can think of keywords for a particular information need. R3. I am able to use several keyword terms to focus my information search. R4. I know how to cite a source in a bibliography and a footnote (or a parenthetical reference) using the pink sheets. R5. I understand what plagiarism is and follow copyright principles. R6. I am able to tell the difference between a primary and secondary source.

23 DATABASES DO NOT UNDERSTAND NEVERSOMETIMESFREQUENTLYALWAYS D1. I know when to use MARVEL for my research needs. D2. I am able to perform a successful search in Gales Student Resource Center Gold. D3. I am able to perform a successful search in Infotrac. D4. I am able to perform a successful search in SIRS Online. D5. I know what an advanced search is, and am able to perform one successfully. D6. I am able to use the terms AND and OR correctly in my searching.

24 THE INTERNET DO NOT UNDERSTAND NEVERSOMETIMESFREQUENTLYALWAYS I1. I generally use more than one search engine when doing research. I2. I know how to narrow a search when using search engines. I3. I know if a website is valid or might be biased. I4. I know when to use the Internet and when to use other information sources.

25 TECHNOLOGY BEGINNERINTERMEDIATEADVANCEDDONT KNOW HOW T1. I know how to create a Microsoft PowerPoint or Keynote presentation. T2. I know how to create a web page. T3. I know how to make a spreadsheet. T4. I know how to operate a video camera. T5. I know how to operate and upload pictures from a digital camera. T6. I know how to show a computer display through a digital projector (LCD). T7. I know how to create and insert special graphics, tables, and pictures in my documents.

26 ACTIVITY REGULARLYOFTENSOMETIMESNEVER E-MAIL FACEBOOK (or similar social networking, such as MySpace, Twitter) NEWS WEATHER DOWNLOADING MUSIC TO PLAY GAMES INFORMATION/RESEARCH USING SEARCH ENGINES INFORMATION/RESEARCH USING DATABASES SPORTS (including fantasy leagues and Heal point standings) SHOPPING VIDEO (YouTube, Hulu, DVDs, etc.) VIDEO CHAT ON-LINE BANKING

27 Student Participants: Total Number Completing Survey: 299 Male Respondents: 133 Female Respondents: 166 Totals by Class: Grade 9: 68 (36M, 32F) Grade 10: 84 (39M, 45F) Grade 11: 80 (30M, 50F) Grade 12: 67 (28M, 39F ) Faculty Participants: Total Number of Respondents: 23 Male Respondents: 10 Female Respondents: 13

28 STUDENT RESPONSES (out of 299) FACULTY RESPONSES (out of 23) POINT ACCESS SKILLSTotalPercentTotalPercent P1. I am able to access our high school library web site using a computer.14749.2% 1356.5% P2. I feel competent using the on-line library catalog to search for books.12341.1% 1460.9% P3. I know how to locate books on the shelves after looking them up on the on-line catalog.15953.2% 1773.9% P4. I am able to locate and use the databases on our high school library website using the blue sheet.12742.5% 1147.8% P5. I am able to search for books that are in other Maine libraries using MaineCAT. 8528.4% 1252.2% AGGREGATE64142.9% 6758.3% RESEARCH SKILLS R1. I am able to develop questions focusing on my information needs.19163.9%2191.3% R2. I know what a keyword search is, and can think of keywords for a particular information need.22775.9%2295.7% R3. I am able to use several keyword terms to focus my information search.21471.6%2191.3% R4. I know how to cite a source in a bibliography and a footnote (or a parenthetical reference) using the pink sheets.17558.5%1878.3% R5. I understand what plagiarism is and follow copyright principles.24682.3%2295.7% R6. I am able to tell the difference between a primary and secondary source.18260.9%2191.3% AGGREGATE123568.8%12590.6% Table 3. Frequency of Student and Faculty Survey Responses Frequently plus Always and Frequency of Student and Faculty Skill Assessments Intermediate plus Advanced

29 DATABASES D1. I know when to use MARVEL for my research needs.9130.4%1147.8% D2. I am able to perform a successful search in Student Resource Center Gold.10936.5%1043.5% D3. I am able to perform a successful search in Infotrac.6622.1%1043.5% D4. I am able to perform a successful search in SIRS Online.8428.1%1252.2% D5. I know what an advanced search is, and am able to perform one successfully.14849.5%1356.5% D6. I am able to use the terms AND and OR correctly in my searching.16555.2%1565.2% AGGREGATE66337.0%7151.4%

30 THE INTERNET I1. I generally use more than one search engine when doing research.13745.8%1460.9% I2. I know how to narrow a search when using search engines.18662.2%1773.9% I3. I know if a website is valid or might be biased.17458.2%1982.6% I4. I know when to use the Internet and when to use other information sources.20167.2%2191.3% AGGREGATE69858.4%7177.2%

31 TECHNOLOGY T1. I know how to create a Microsoft PowerPoint or Keynote presentation.24682.3%1773.9% T2. I know how to create a web page.11438.1%626.1% T3. I know how to make a spreadsheet.19866.2%1147.8% T4. I know how to operate a video camera.24782.6%1773.9% T5. I know how to operate and upload pictures from a digital camera.255`85.3%1878.3% T6. I know how to show a computer display through a digital projector (LCD).18963.2%1878.3% T7. I know how to create and insert special graphics, tables, and pictures in my documents21872.9%1356.5% AGGREGATE1,46770.1%10062.1%

32 RESPONSESSTUDENTS (299)FACULTY (23) (Frequently + Always) for all skill set questions (by category) Total Responses (by category) Percent of High Confidence Responses (by category) (Frequently + Always) for all skill set questions (by category) Total Responses (by category) Percent of High Confidence Responses (by category) SKILLS SETS POINT ACCESS (5 questions) 641149542.9%6711558.3% RESEARCH (6 questions) 1235179468.8%12513890.6% DATABASES (6 questions) 663179437.0%7113851.4% THE INTERNET (4 questions) 698119658.4%719277.2% TECHNOLOGY (7 skills rated) 1,467209370.1%10016162.1%

33 RANK (high to low confidence) STUDENT RANKING OF AGGREGATE HIGH- CONFIDENCE RESPONSES (by skill category) FACULTY RANKING OF AGGREGATE HIGH- CONFIDENCE RESPONSES (by skill category) 1TECHNOLOGY70.1%RESEARCH90.6% 2RESEARCH68.8%INTERNET77.2% 3INTERNET58.4%TECHNOLOGY62.1% 4POINT ACCESS42.9%POINT ACCESS58.3% 5DATABASES37.0%DATABASES51.4% Table 5. Comparison of Student and Faculty Aggregated High Confidence Responses (Ranked from highest to lowest.)

34 RANK (high to low confidence) STUDENT RANKING OF AGGREGATE HIGH- CONFIDENCE RESPONSES (by skill set and grade) Grade 9 Grade 10 Grade 11 Grade 12 1TECHNOLOGY (69.1%) TECHNOLOGY (78.1%) RESEARCH (72.1%) RESEARCH (74.9%) 2RESEARCH (58.1%) RESEARCH (72.0%) TECHNOLOGY (62.0%) TECHNOLOGY (72.9%) 3INTERNET (48.5%) INTERNET (58.0%) INTERNET (61.3%) INTERNET (69.0%) 4POINT ACCESS (39.1%) POINT ACCESS (50.0%) POINT ACCESS (37.5%) POINT ACCESS (46.6%) 5DATABASES (35.5%) DATABASES (44.2%) DATABASES (31.0%) DATABASES (36.3%) Table 6. Comparison of Aggregated High-Confidence Responses By Skill Category and Grade Level

35 Most frequently reported uses of the Internet: -The most frequently reported uses by students were e-mail, Facebook, search engines, and watching videos (YouTube, Hulu, DVDs.) -For faculty, e-mail, news, weather, search engines, and on-line banking. -Both faculty and students indicated more frequent use of search engines over databases when seeking information or doing research.

36 Developing the Skills Test A pool of 46 skills test questions was assembled from a variety of existing tests, including those used by the Wooster School Library, Carleton College, Bowdoin College, Wichita State University, TRAILS (Kent State University), and the University of Maryland, Baltimore County (UMBC).

37 Other Valuable Sources for Test Questions

38 Question Selection: -School technology committee members (including the school librarian) were asked to select 25 to 30 questions that: -were of an appropriate level of difficulty; -matched well with items from the perceptions survey; -represented a balance of questions relating to and testing ACRL or AASL information literacy standards. -School librarian, library aide assisted in final question selection (25 questions total). - Version one of the skills test was piloted with 37 students (using Survey Monkey); final test reduced to 20 questions.

39 Student Participant Profile Maximum Respondent Pool*: 302 students Grade Levels: Grade 9: 109 Grade 10: 91 Grade 11: 30 Grade 12: 72 Gender: Male: 138 Female: 163 No Response: 1 *A pilot survey of 26 questions was administered to 37 student respondents. Based on the feedback on the pilot test, the skills test was revised and shortened to twenty questions, which was administered in two separate surveys of ten questions each. The tests were administered on-line, through Survey Monkey. 302 students participated in the first part of the skills test; 202 students participated in the second part of the test. The demographic and background information (above) was taken from those who participated in the first part of the skills test.

40 Students were also asked about: Their Source of Knowledge of Library Resources: A course or project in which a teacher scheduled a class with a librarian: 198 Self-taught: 129 Sought help from librarian on my own: 57 Reference librarian at a public librarian: 47 None of the above: 23 Library Resources Used in the Last Six Months Electronic Library Catalog99 MARVEL 92 Student Resource Center Gold 76 SIRS Online 37 Infotrac 15 Skipped the question 100

41 Perception Versus Reality: Baseline Comparisons

42 POINT SOURCE SKILLS: PERCEPTION I feel competent using the on-line library catalog to search for books. 41.1% of students gave high-confidence responses on this statement.

43 The screen shot, above, is the search page for the Morse High Schools library catalog. If you were to hit the Search button using the terms shown, the list of search results would be: a. A list of magazine and journal articles about both Germany and World War II b. A list of different web sites about both Germany and World War II c. A list of databases with information about both Germany and World War II d. A list of books in the Morse High School library about both Germany and World War II. e. Dont know. POINT SOURCE SKILLS: REALITY

44 REALITY 37.4% (113) of students chose the correct response for this question. 26.5% (74) of students answered incorrectly. 15.9% (48) of students said that they did not know the answer. 22.2% (67) of students skipped the question. Comparison of Perception and Reality: 41.1% were confident about their ability to use the OPAC. 37.4% answered the corresponding skills test question correctly.

45 RESEARCH SKILLS: PERCEPTION I am able to tell the difference between a primary and secondary source. 60.9% of students gave high-confidence responses on this statement.

46 RESEARCH SKILLS: REALITY Which of these items relating to Walt Whitman is the best example of a primary source? a. A biography of Walt Whitman b. A recently published book of letters Whitman wrote to friends and family c. A Library of Congress collection of Whitmans personal notebooks d. A book of critical analysis about his poetry. e. Both (b) and (c).

47 REALITY: 36.1% (109) of students chose the correct response for this question. 28.8% (87) of students answered incorrectly. 11.6% (35) of students identified one of the two primary sources. 23.4% (71) of students skipped the question. Comparison of Perception and Reality: 60.9% were confident about their ability to identify primary sources. 36.1% answered the corresponding skills test question correctly.

48 DATABASE SKILLS: PERCEPTION I am able to perform a successful search in SIRS online. 28.1 % of students gave high-confidence responses on this statement.

49 DATABASE SKILLS: REALITY The Morse High School Library database that would be the best source of information for an in-class debate on the Iraq war is: a. Student Resource Center Gold b. Infotrac c. MARVEL d. SIRS online e. Dont know

50 REALITY 5.9% (12) of students chose the correct response for this question. 45.5% (92) of students answered incorrectly. 31.2% (63) of students said they did not know the correct answer. 17.3% (35) of students skipped the question. Comparison of Perception and Reality: 28.1% were confident about their ability to use SIRS online successfully in a search. 5.9% answered the corresponding skills test question correctly.

51 INTERNET SKILLS: PERCEPTION I know if a website is valid or might be biased. 58.2 % of students gave high-confidence responses on this statement.

52 If you are writing a paper on animal rights, and you use information from a web site produced by the organization People for the Ethical Treatment of Animals (PETA), which of the following considerations is important when you evaluate the information? a. Is there bias? b. How current is the information? c. Are the sources of factual information easy to verify? d. What related links are provided on the web site? e. All of the above. INTERNET SKILLS: REALITY

53 REALITY 54.3% (164) of students chose the correct response for this question. 24.2% (73) of students identified at least one aspect of bias, although they answered the question incorrectly. 21.5% (65) of students skipped the question. Comparison of Perception and Reality: 58.2% were confident about their ability to identify website bias. 54.3% answered the corresponding skills test question correctly.

54 TECHNOLOGY SKILLS: PERCEPTION I know how to make a spreadsheet. 58.2% of students gave high-confidence responses on this statement.

55 TECHNOLOGY SKILLS: REALITY ABCD 123 212 336 49 5 6=SUM(C1:C3) In the spreadsheet below, what is the result in cell C6? a. 80 b. 71 c. 36 d. 57 e. Dont know.

56 REALITY 17.2% (52) of students chose the correct response for this question. 22.2% (67) of students answered the question incorrectly. 38.1% (115) of students said they didnt know the correct answer. 22.5% (68) of students skipped the question. Comparison of Perception versus Reality: 66.2% were confident about their ability to make a spreadsheet. 17.2% answered the corresponding skills test question correctly.

57 SUMMARY TABLES: ALL DATA

58 PERCEPTIONS SURVEY DATA SKILLS TEST DATA CATEGORY followed by Question Number and Topic Aggregated High Confidence Responses for Category Question-Specific High-Confidence Response (when applicable) Correct Response Percentage Point Access 42.9% 1. url for library website 49.2%34.3% 2. OPAC screenshot 41.1%37.4% 3. Source location 25.7-53.5% 4. Source types 27.2-69.5% Research 68.8% 1. First step in library research 63.9%41.7% 2. Background research 63.9%31.7% 3. Keyword selection 75.9%31.2% 4. Citation identification 58.5%32.5-50.1% 5. Plagiarism 82.3%24.2-71.2% 6. Plagiarism 82.3%26.7% 7. Primary source identification 60.9%36.1%

59 Database 37.0% 1. MARVEL 40.6% 2. General database 31.2% 3. SIRS 28.1%5.9% 4. Database Advanced Search 49.5%55.3% 5. Narrowing search 55.2%44.3% Internet 58.4% 1. Search engine identification 27.2-69.5% 2. Boolean operators 62.2%12.9% 3. Website bias 58.2%54.3% 4. Website trustworthiness 58.2%59.3% 5. Website types 58.2%57.4-70.3% Technology 70.1% 1. Spreadsheet operations 66.2%17.2% High Confidence Responses % Correct (Perceptions Survey) (Skills Test) Aggregate Question Specific

60 QUESTIONS RAISED: 1. What confidence/skill levels are adequate, regardless of the gap between perception and reality? -What are goals for attainment? 2. How meaningful are the gaps? -Are there poor questions? (bad wording, poor correspondence) -Distinguishing true errors from errors that result from test fatigue or disinterest

61 Additional Observations and Issues: About the research process: 1.Students and faculty appeared more responsive to perceptions survey. Possible explanations: -perceptions survey was less threatening -timing of perceptions test versus skills test (at outset of program versus later in the school year) -survey fatigue (due to NEASC accreditation) -complaints that skills test was too long or too hard

62 2. Institutional issues/support affected the overall success of surveying. Such as: -technical issues (e.g. band width issues); -too short a time frame to develop and pilot the skills test (realities of graduate research) -lack of teacher commitment/teacher overload

63 3.Other institutions rely on perceptions evaluations or skills evaluations in longitudinal studies, but not necessarily both. For example: - Grimble and Williams repeated perceptions surveys longitudinally; - Bowdoin is administering only its skills survey longitudinally;

64 Instruction Opportunities: 1. Twenty-first century skills. The chance to provide students with skills for college and beyond (particularly skills relating to research and electronic resource access/use);

65 2. Evidence of achievement relating to 1:1 laptop use. The opportunity to improve (and to measure improvement in) skills that integrate technology and research, and thereby help justify the significant investment by R.S.U. 1 in the laptop program.

66 3. Professional development. The opportunity to improve high school faculty confidence/skills relating to the access and use of electronic resources available in the MHS and local public library.

67 Other Evidence of Progress.. One Year Later 1.Commitment by MHS to Continue Longitudinal Study 2.Staff Additions (Technology Integrator) 3. More Library-Based Projects and Instruction?

68

69

70 Morse High School Library Blog

71 Morrissette, R. (2007). What do they know? A strategy for assessing critical literacy. Knowledge Quest. 35, 15-17. Sources Cited Grimble, Bonnie J. & T.D. Williams. (2004). Students perceptions of their information literacy skills in the media center. Library Media Connection. 22:4, 26-28.


Download ppt "Perception and Reality: Creating a Baseline for Longitudinal Study of Information Literacy Skills at the Outset of a High School Laptop Program Roberta."

Similar presentations


Ads by Google