1. An Overview of the Higher Education ICT Literacy Assessment Ilene Rockman Gordon Smith California State University Irvin R. Katz Educational Testing Service http://www.ets.org/ictliteracy Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

2. 2 Importance to Higher Education Measures cognitive problem solving abilities within technological environment Clarifies and articulates ICT student learning outcomes Provides evidence for accreditation purposes Simulates academic, work, or personal queries Engages students through critical reading, reflecting, and taking action

3. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 3 How the Assessment Differs From Others Interactive, web, and performance-based Real-world storylines Higher order reading and thinking skills Uses digital technologies, communication tools, and networks to solve cognitive problems in an ethical and legal way

4. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 4 Student Feedback from Beta Tests Never taken test like this before Challenging; took it seriously Interface was easy to use Required thinking and technical skills Enjoyed real-world storylines Tasks reflect activities at school, work, or home

5. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 5 Why Measure ICT Literacy? ICT is changing the very nature and value of knowledge and information and impacts the way we live, learn, and work ICT literacy is not simply a mastery of technical skills, but is the application of technical skills in an information society. There is a lack of information about the ICT literacy of students, and debate about how best to address this issue in academic curriculum

6. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 6 What is ICT Literacy? Ability to use digital technology, communication tools, and/or networks to access, manage, integrate, evaluate, create, and communicate information ethically and legally in order to function in a knowledge society

7. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 7 Building on Prior Efforts ACRL Information Literacy Competency Standards for Higher Education ISTE National Educational Technology Standards Digital Transformation: A Framework for ICT Literacy (ETS, 2002)

8. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 8 ICT Literacy: a bridge between Technical Literacy Information Literacy Database Word Processing Presentation Can I bold a word? Can I open a database? AccessEvaluateUse Can you find information? Can you evaluate authority? Information and Communication Technology Literacy –Can I find information on the web? –Can I create a persuasive presentation?

9. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 9 Proficiency Model ICT Literac y Defin e Access Manag e Integrat e Evaluat e Create Communicate Cognitive Ethical Technical

10. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 10 The ICT Literacy Project Two separate products –ICT Literacy Large-Scale Assessment (2005) –ICT Literacy Individual Assessment (2006) Unique aspects of products: –Focus on cognitive skills rather than technical skills –Simulation-based tasks – Not multiple-choice –Generic software – Resistant to changes in technology –Scalable and sustainable

11. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 11 –What new courses should be offered? –Which courses should receive additional resources? –Accreditation: Have our investments in ICT education been paying off? –What courses should I take? –In what majors will I be successful? –Am I ready to enter the workforce or graduate school? Institutional comparisons across campuses or departments from year to year Individual decision-making Potential Uses for Higher Education

12. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 12 Basic Design Features Interactive tasks using simulated software Delivered via the web in a 2-hour test Tasks of varying lengths (4-, 15-, and 30-min) Automatic scoring, with multiple scorable elements per task Evidence-based design Valid and reliable

13. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 13 Recent Test Administration (January – April ’05) Completed: 4582 examinees across 31 campuses Both Four-year and Two-year institutions Overall Reactions: “Interesting & fun, but exhausting with a lot of reading” Library community feedback: –Not enough emphasis on print media –Too much emphasis on technology

14. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 14 Responding to the Feedback Investigate instances of “too much reading” –Identify problem tasks –Reduce scenario descriptions Increase number of scenarios that incorporate print media (e.g., on-line library catalog) Keep same level of technology –Assessing cognitive skills –Technology already minimal –Accurately reflects disparate views of information formats that exist in the disciplines

15. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 15 Sample Task: Comparing Information Short task: Four minute duration Assesses one ICT Literacy subproficiency –Integrate: The ability to interpret and represent information in an ICT context. This includes the ability to synthesize, summarize, compare, and contrast information from multiple sources.

16. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 16 Within the task Summarize information from three different types of sources Compare the information to reach a conclusion

17. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 17 In this task, examinees summarize information from a variety of sources and then draw conclusions from their summary. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

18. 18 Examinees are presented with an information need and three different types of information sources: Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

19. 19 Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 1. Webpage Examinees are presented with an information need and three different types of information sources:

20. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 20 2. Email Examinees are presented with an information need and three different types of information sources: Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

21. 21 3. Advertisement The three sources present information according to different conventions. A successful candidate must be able to locate the relevant information in each source. Examinees are presented with an information need and three different types of information sources: Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

22. 22 Deciding how to compare the sources involves identifying the requirements of the stated information need. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

23. 23 After filling in the table, examinees must interpret the summary to rank the three sources correctly. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

24. 24 Other Tasks Access: Simulated web or database searches Evaluate: Judge authority, currency, bias of search results Create/Communicate: Develop presentation slides based on accessed information

25. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 25 Web Search Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

26. 26 Database Search Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

27. 27 Database Results Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

28. 28 Create a Presentation Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

29. 29 Score Reports Test use guide and proficiency descriptions Interpretation guide Overall scores and four sub-scores –Define/Access, Evaluate, Manage/Integrate, Create/Communicate Institution-specific contrasts

30. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 30 Group Percentage of Students In Your School Percentage of Students Participating n=25,000n=112 Gender Male45 Female55 Race/Ethnicity Asian108 Black2022 Hispanic2018 Native American57 White45 100

31. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 31 Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

32. 32 Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

33. 33 Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

34. 34 Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

35. 35 Current Higher Education Partners The California State University Oklahoma State University Portland State University Purdue University University of California, Los Angeles University of Louisville University of Memphis University of Washington

36. Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service. 36 Summary Made for and by higher education –Original 7 institutions spent a total of about 1000 hours reviewing, planning, and designing Scenario-based –Simulations, not multiple choice Psychometrically sound Provide information for –Accreditation & institutional improvement –Individual certification Dynamic and responsive

37. Questions & Discussion Visit us at: http://www.calstate.edu/LS http://www.ets.org/ictliteracy Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.