1. 1 Further Development of a Digital Library Curriculum: Evaluation Approaches and New Tools Seungwon Yang, Uma Murthy, Seonho Kim and Edward A. Fox, Department of Computer Science, Virginia Tech, Barbara M. Wildemuth, Jeffrey Pomerantz and Sanghee Oh School of Information and Library Science, University of North Carolina – Chapel Hill ICADL 2007, Hanoi, Vietnam, Dec. 10 – Dec. 13, 2007

2. 2 Acknowledgements (Selected) Faculty / Staff / Others  Module evaluators, Advisory Board members, Digital Library Research Lab members, Susie Marion, Sidewalk team (David Maier, Lois Delcambre, Sudarshan Murthy) … Sponsor  US National Science Foundation Grants IIS-0535057 to VT IIS-0535060 to UNC-CH

3. 3 Overview Introduction DL curriculum framework Development lifecycle Future work Invitation

4. 4 Introduction Interdisciplinary project  VT CS & UNC-CH SILS Goal  DL lesson modules  courses  Community

5. 5 Introduction Team  VT: Dr. Edward A. Fox (PI), Seungwon Yang  UNC-CH: Dr. Barbara M. Wildemuth (PI), Dr. Jeffrey P. Pomerantz, Sanghee Oh Advisory Board  VT:10, UNC-CH:14, other:12 Term: Jan. 2006 - Dec. 2008

6. 6 Foundations - 5S Framework - S’sRole in DLExamples StreamsVarious types of contenttext, video, audio StructuresOrganize informationcatalog, metadata, hypertext SpacesStore and present informationinterfaces, storage, vector space ScenariosProvide servicessearching, browsing, recommending SocietiesForm a DL community of users and service managers service managers, teachers, learners Theoretical foundation Necessary condition to be a Minimal DL

7. 7 Foundations - Computing Curriculum 2001(ACM & IEEE-CS)- CS Body of Knowledge  Information Management (IM.14)  Digital Libraries DL topics and learning objectives Suggestions for CC2001 update

8. 8 Overview Introduction DL curriculum framework Development lifecycle Future work Invitation

9. 9 Current DL Module Framework  10 core modules  41 sub modules DL Curriculum Framework

10. 10 DL Module Framework ( 1-5 of 10)

11. 11 DL Module Framework ( 6-10 of 10)

12. 12 5-b: Application software Draft Module Example

13. 13 Overview Introduction DL curriculum framework Development lifecycle Future work Invitation

14. 14 Development Lifecycle Diagram (Selected) six draft modules Pilot test for formative evaluation

15. 15 Development Lifecycle Diagram

16. 16 (Selected) Six Draft Modules 1 st yellow box High-priority areas first  1-b: History of digital libraries and library automation  5-b: Application software  6-a: Information needs, relevance  6-b: Search strategy, information seeking behavior, user modeling  7-b: Reference services  9-c: DL evaluation, user studies

17. 17 Prioritize them Define the scope More consistent relationships International inputs in development  From Asia, Europe, Africa, Oceania,… Pilot Test for Module Evaluation - Feedback – Meeting during JCDL’07

18. 18 Overview Introduction DL curriculum framework Development lifecycle Future work Invitation

19. 19 Future work – Resource presentation in a module – Superimposed Information (SI) Technology  Sub-document level granularity presentation  “Marks”: references to selected regions within base (original) information

20. 20 SI Technology A) Mark used in a module; B) Mark highlighting the desired selection that describes DSpace in an article AB

21. 21 Future work – Understanding the community’s use of module – Visual User model Data Mining (VUDM) Tool

22. 22 VUDM Visualization of DL module usage trends for three weeks

23. 23 Future work – Plans for module distribution – Using wikis  Wikiversity: distribute modules, set up a DL community  Wikibooks: DL booklet Using Second Life (secondlife.com)  3D virtual environment  Supports collaborative, distance learning  Collaborator in SJSU: Dr. Lili Luo

24. 24 New Media Consortium Campus From http://b2e.nitle.org/media/Howard_beforethemob.jpg

25. 25 Overview Introduction DL curriculum framework Development lifecycle Future work Invitation

26. 26 Would you like to join us? Your contribution – two ways  Module development (e.g., based on your teaching experiences/classes)  Module evaluation Contacts for PIs  VT: Edward A. Fox (fox@vt.edu)fox@vt.edu  UNC-CH: Barbara M. Wildemuth (wildem@ils.unc.edu)wildem@ils.unc.edu

27. 27 More Information Project homepage: http://curric.dlib.vt.edu http://curric.dlib.vt.edu Module evaluation: http://curric.dlib.vt.edu/wiki http://curric.dlib.vt.edu/wiki Project diary: http://tuppence.dlib.vt.edu/blogs/index.php?blog=2 http://tuppence.dlib.vt.edu/blogs/index.php?blog=2

28. 28 Thank you! Questions and comments?

29. 29

30. 30 Foundations  5S framework  Advisory board  Computing Curriculum (CC) 2001

31. 31 Wikiversity

32. 32 Module Template (1/2) 1. Module name 2. Scope 3. Learning objectives 4. 5S characteristics of the module 5. Level of effort required 6. Relationships with other modules 7. Prerequisite knowledge required 8. Introductory remedial instruction

33. 33 9. Body of knowledge Topic 1 Lesson Topic 2... Topic 3... 10. Resources 11. Concept map 12. Exercises / learning activities 13. Evaluation of learning outcomes 14. Glossary 15. Additional useful links 16. Contributors Module Template (2/2)

34. 34 Foundations - Advisory Board From VTFrom UNCOther Institutions Steven Edwards, CS Roger Ehrich, CS Weiguo Fan, ACIS Steve Harrison, CS Gail McMillan, Library Chris North, CS Manuel Pérez-Quiñónez, CS Naren Ramakrishnan, CS Deborah Tatar, CS Layne Watson, CS Catherine Blake, SILS Laura Gasaway, Law School Jane Greenberg, SILS Stephanie Haas, SILS Brad Hemminger, SILS Thomas James, Dean, School of Education Paul Jones, Director, ibiblio; SILS & School of Journalism & Mass Communication Diane Kelly, SILS Gary Marchionini, SILS Montek Singh, CS Natasha Smith, Library Helen Tibbo, SILS Steve Weiss, CS Javed Mostafa, SILS Dan Atkins, U. Michigan Christine Borgman, UCLA Lillian Cassel, Villanova Michael Christel, CMU Raya Fidel, U. Washington Richard Furuta, Texas A&M University Elizabeth Liddy, Syracuse University Clifford Lynch, CNI Kurt Maly, ODU Tefko Saracevic, Rutgers Linda Smith, UIUC Ingeborg T. Sølvberg, NTNU (Norway)

35. 35 Pilot Test for Module Evaluation - Procedures - Project team  Provided an overview of our progress in developing modules  Presented the curriculum development lifecycle The participants  Worked in pairs to evaluate a module  Guided by an evaluation form All  discussed the evaluation process and how it should be implemented more widely

36. 36 Formative Evaluation Evaluate 5 sections + 2  Objectives  Body of knowledge  Readings  Learning activities  Level of effort and prerequisites  Overall structure  Additional comments

37. 37 Example: Guiding + Specific Questions Body of knowledge: Guiding question: Does the module address all areas of the topic that need to be addressed? Specific questions: Will the body of knowledge enable students to achieve the objectives? Are there any topics that you think are critical to add to the body of knowledge? Are there any topics that you would remove from the body of knowledge?

38. 38 Formative Evaluation Form (1 of 3) Objectives: Guiding question: Are the objectives appropriate for the topic? Specific questions: Are the objectives observable? Will students be able to achieve the objectives, given the content in the body of knowledge? Body of knowledge: Guiding question: Does the module address all areas of the topic that need to be addressed? Specific questions: Will the body of knowledge enable students to achieve the objectives? Are there any topics that you think are critical to add to the body of knowledge? Are there any topics that you would remove from the body of knowledge?

39. 39 Formative Evaluation Form (2 of 3) Readings: Guiding question: Are the readings the best and most appropriate for the topic? Specific questions: Are there any readings that you think are critical to add to the list? Are there any readings on the list that you would remove? Learning Activities: Guiding question: Are the activities appropriate for the topic? Specific questions: Will students be able to accomplish the activities, given the content in the body of knowledge? Will the activities enable students to achieve the objectives? Can you suggest any other learning activities that may be appropriate for this module?

40. 40 Formative Evaluation Form (3 of 3) Level of Effort and Prerequisites: Guiding question: Is it feasible to teach the module as it is currently constructed? Specific questions: Is the level of effort required in class appropriate to the scope of the body of knowledge? Prior to class? Is the prerequisite knowledge required sufficient for students to comprehend the body of knowledge?

41. 41 Current Status of Formative Evaluation Sending out emails Provided p.w. to evaluators who agreed to participate Expect to be done in Dec., 2007

42. 42 Future work – Resource presentation in a module – Superimposed Information (SI) Technology  Sub-document level granularity presentation  “Marks”: references to selected regions within base (original) information