1. European Schoolnet Learning Resource Exchange: A Developer Perspective Presented by David Massart Senior Software Architect, EUN Dec. 4, 2006 - Leuven, Belgium

2. European Schoolnet Outline Welcome Workshop Agenda Discovering resources in a hidden educational Web Metadata formats and query languages LRE architecture and services –Core services –SQI-based services –Harvesting services –DRM services Lab Exercises

3. European Schoolnet A Hidden Educational Web Most on line learning resources are unlocatable using text-based search engines (Repositories make it worse) Search engines results are difficult to assess by teachers and pupils Metadata help to retrieve and assess learning resources but metadata repositories are just another part of the “hidden web”

4. European Schoolnet Learning Resource Discovery Decentralized approach: Federated searching –A “source” sends a query to several “target” repositories –Each repository processes the query and returns results to the source Centralized approach: Metadata harvesting –A “harvester” builds a central index with metadata harvested in target “repositories” –A source queries the harvester index.

5. European Schoolnet Simple Query Interface (SQI) Standard API (CWA - 15454, Nov. 2005) For combining highly heterogeneous metadata repositories Neutral in terms of query languages and results format Supports synchronous and asynchronous query mode Based on a session management concept

6. European Schoolnet The SQI Application Program Interface

7. European Schoolnet SQI Methods Query Configuration –setQueryLanguage –setResultsFormat –setMaxQueryResults –setMaxDuration Synchronous Query Interface –setResultsSetSize –synchronousQuery –getTotalResultsCount Asynchronous Query Interface –asynchronousQuery –setSourceLocation –queryResultsListener Session Management –createSession –createAnonymousSession –destroySession

8. European Schoolnet Fault Mechanism Intentionally unsophisticated Simplicity rather than richness Only one fault: SQIFault –Thrown by all methods –Allows: Free-text messages Predefined error codes

9. European Schoolnet SQI Error Codes

10. European Schoolnet SQI Error Codes (cont.)

11. European Schoolnet The Open Archive Initiative Protocol for Metadata Harvesting Application-independent interoperability framework based on metadata harvesting Two classes of participants in the OAI-PMH framework: –Repositories support the OAI-PMH as a means of exposing metadata –Harvesters use metadata harvested via the OAI-PMH as a basis for building value-added services http://www.openarchives.org/OAI/openarchivespr otocol.html

12. European Schoolnet HTTP Embedding of OAI-PMH requests OAI-PMH requests are expressed as HTTP requests A typical implementation uses a standard Web server that is configured to dispatch OAI-PMH requests to the software handling these requests

13. European Schoolnet Item, Identifier, And Record Item: Conceptually a container that stores or dynamically generates metadata about a single resource in multiple formats Unique Identifier: Unambiguously identifies an item within a repository Record: Metadata expressed in a single format

14. European Schoolnet Protocol Requests GetRecord: Retrieves an individual metadata record from a repository Identify: Retrieve information about a repository ListIdentifiers: An abbreviated form of ListRecords, retrieving only identifiers rather than records ListMetadataFormats: Retrieves the metadata formats available from a repository ListRecords: Harvests records from a repository ListSets: Retrieves the set structure of a repository

15. European Schoolnet Error and Exception Conditions badArgument badResumptionToken badVerb cannotDisseminateFormat idDoesNotExist noRecordsMatch noMetadataFormats noSetHierarchy

16. European Schoolnet Metadata Formats and Query Languages Metadata / Result Format: –Information model (e.g., DC, LOM) –Binding (e.g., XSD) –Encoding (e.g., UTF-8) Query (Exchange) Language –Semantics –Syntax –Lexicon –Encoding

17. European Schoolnet MELT LOM Application Profile Information model of the LRE metadata Defines: – 4 mandatory elements (general.identifier, general.language, technical.location, rights.copyright and other restrictions) –14 recommended elements (e.g., title, description, keywords) –Optional elements –Controlled vocabularies

18. European Schoolnet XML Bindings Strict LOM Binding –http://ltsc.ieee.org/xsd/lomv1.0/lomStrict.xsdhttp://ltsc.ieee.org/xsd/lomv1.0/lomStrict.xsd –LRE LOM elements not compatible with strict LOM are ignored An MELT AP + binding are under construction Calendar –Since 2005: Strict LOM binding –By ???: Strict LOM binding + MELT LOM binding

19. European Schoolnet IEEE LOMv1.0 strict

20. European Schoolnet Strict LOM Results XML Header <strictLomResults xmlns=http://fire.eun.org/xsd/strictLomResults-1.0http://fire.eun.org/xsd/strictLomResults-1.0 xmlns:xsi=http://www.w3.org/2001/XMLSchema-instancehttp://www.w3.org/2001/XMLSchema-instance xsi:schemaLocation="http://fire.eun.org/xsd/strictLomResults-1.0 http://fire.eun.org/xsd/strictLomResults-1.0.xsdhttp://fire.eun.org/xsd/strictLomResults-1.0.xsd" >...

21. European Schoolnet Query Exchange Language Simple School Query Language (S 2 QL) A Query Language for Learning Repositories: PLQL –Under construction –http://ariadne.cs.kuleuven.be/lomi/index.php/Q ueryLanguageshttp://ariadne.cs.kuleuven.be/lomi/index.php/Q ueryLanguages Calendar –Since 2005: S 2 QL –In 2007: S 2 QL + PLQL

22. European Schoolnet S 2 QL

23. European Schoolnet S2QL Semantics Conjunction of elements (at least one is necessary): keywords AND languages AND age range Conjunction of keywords: keyword_1 AND keyword_2 AND keyword_3 Disjunction of languages: Language_1 OR language_2 OR language_3 From minimum age to maximum age

24. European Schoolnet LRE Query Context Additional information (optional) about the context of a query Helps to further specify the scope of a query

25. European Schoolnet Query Languages and Results Formats Are identified by the namespaces of their bindings S 2 QL: http://fire.eun.org/xsd/s2ql-2.0http://fire.eun.org/xsd/s2ql-2.0 Strict LOM resultsset: http://fire.eun.org/xsd/strictLomResults-1.0 http://fire.eun.org/xsd/strictLomResults-1.0 LOM resultsset: http://fire.eun.org/xsd/lomResults-1.0 http://fire.eun.org/xsd/lomResults-1.0 See also http://fire.eun.org/xsd/http://fire.eun.org/xsd/

26. European Schoolnet Learning Resource Exchange A federation of learning (content) management systems and learning resource repositories Based on a brokerage architecture (LIMBS) Federation = an infrastructure for discovering and exchanging learning resources, where each partner remains in control of learning resources and their metadata

27. European Schoolnet LIMBS and the LRE Architecture

28. European Schoolnet LRE Services LIMBS is organized as a set of independent services Services can be arbitrarily combined At the client side, each service corresponds to a pluggable module with a simplified (and, when possible, standard) interface

29. European Schoolnet LRE Services (cont.) Core services (connection + trust) Resource discovery –FIRE federated searching –Trusted federated searching –Harvesting (+ Agent Searching) Resource exchange (digital rights management) Semantic Interoperability

30. European Schoolnet Each Service Follows a common pattern Is based on a standard API (e.g., SQI, OAI) Is kept as simple as possible Is independent of the others Can be used either separately or with any combination of the others

31. European Schoolnet Service Interface

32. European Schoolnet Core Services Manage the connection with LIMBS Allow for: –Configuring a connection –Configuring security parameters –Turning a connection on/off –Monitoring a connection –Automatic reconnection Package: net.sourceforge.limbs.client.core

33. European Schoolnet Core Services Class Path Java Web Service Developer Pack 2.0 Spark-Core libraries: –spark-core.jar, sharedcomponents.jar, UTCTime.jar –AuthnManagerClient.jar, AuthnManagerWSClient.jar –imq.jar, jms.jar –commons-codec-1.3.jar

34. European Schoolnet Core Services API

35. European Schoolnet Turning A Connection On/Off

36. European Schoolnet Spark Identifier (SparkId) Uniquely identifies each system in the LRE Is part of the LRE resource identification scheme (metadata) Serves as a key alias (message signature)

37. European Schoolnet Brokerage System Instances LIMBS instances –Development –LRE test (default) –LRE production –Others A LIMBS instance is identified by –A URL: “http://stove.test.eun.org:8080” –A MQ prefix: “calibrate” –A LDAP suffix: “dc=eun,dc=org”

38. European Schoolnet Selecting A Brokerage System

39. European Schoolnet Trust

40. European Schoolnet Connection Status isBrokerageSystemUp() isSparkRegistered() getContract() isConnectionOn() isReallyConnected()

41. European Schoolnet Logs setVerbosity() –VERBOSE –MEDIUM –SILENT –MUTE setErrorMessageListener() –onErrorMessage( messageId, errorMessage )

42. European Schoolnet Additional Core Tools Package: net.sourceforge.limbs.client.util

43. European Schoolnet Discovery Services: Federated Searching Based on SQI (FireSQI) Package: net.sourceforge.limbs.client.sqi Classpath –Core Services Class Path –fireSQI-api.jar –spark-sqi.jar

44. European Schoolnet Federated Search LIMBS SPARK LO Repository SPARK LO Repository SPARK LO Repository

45. European Schoolnet Federated Search (Cont.) LIMBS SPARK LO Repository SPARK LO Repository SPARK LO Repository

46. European Schoolnet Demonstration (test instance) Who is connected ? http://stove.test.eun.org:8080/LimbsManager A simple query interface: http://eunbrux10a.eun.org:9080/LRE-Search/ A simple query watcher: http://eunbrux10a.eun.org:9080/SQIQueryW atcher/

47. European Schoolnet Direct WS Implementations of SQI Have Drawbacks

48. European Schoolnet Remote Facade, Data Transfer Object, and Publish & Subscribe

49. European Schoolnet Discovery (Federated Search): JMS versus Web Services Remote facade and data transfer object (“one” remote call only) Publish and subscribe (“one” remote call reaches many repositories) Asynchronous (don’t call us, we’ll call you) Ease of deployment (no web server is needed) Robustness and scalability

50. European Schoolnet The FireSQI API Java binding for a subset of SQI Composed of two interfaces... –FireSQISource to be implemented by query senders –FireSQITarget to be implemented by repositories... And an exception: SQIFault Package: org.eun.fire.sqi

51. European Schoolnet The FireSQI API (Cont.)

52. European Schoolnet Federated Search (revisited)

53. European Schoolnet SQI Service API Class SparkSQI –Singleton (as all the classes used as a service entry point) –Implements: ServiceManagement FireSQISource FireSQITarget Package: net.sourceforge.limbs.client.sqi

54. European Schoolnet SQI Service API (cont.)

55. European Schoolnet Additional Methods As a source: –setLocalSQISource(): To act as a source (i.e., receive results from the LRE) –setAutoMessageAccepted(): To receive its own queries As a target: –setLocalSQITarget(): To act as a repository (i.e., receive queries from the LRE) –setLocalAsynchronous(): Does the spark query the local repository in synchronous or asynchronous mode?

56. European Schoolnet Source Starts SQI Service

57. European Schoolnet Target Starts SQI Service

58. European Schoolnet Sending A Query

59. European Schoolnet Receiving A Query (Synch.)

60. European Schoolnet Receiving A Query (Asynch.)

61. European Schoolnet System Stops SQI Service

62. European Schoolnet FIRE Backward Compatibility Thanks to the class org.eun.fire.spark.Spark Spark = SparkCore + SparkSQI but: –No trust –No session monitoring Switching from FIRE to the new API: –Does not require a lot of effort –Is recommended

63. European Schoolnet FIRE Backward Compatibility (cont.)

64. European Schoolnet Harvesting Services Service Providers use harvested metadata as a basis for building value- added services (e.g., AGH agent search engine) LRE harvesting services use a Java binding (FireOAI) inspired of OAI-MHP In the future: The CEN-ISSS Learning Technology Workshop plans to develop a Simple Synchronization Interface (SSI)

65. European Schoolnet Harvesting Scenario

66. European Schoolnet The FireOAI API Java binding inspired of OAI-MPH (pull mechanism) Composed of two interfaces... –FireOAIHarvester to be implemented by harvesters –FireOAIRepository to be implemented by repositories... And an exception: FireOAIException Package: org.eun.fire.oai

67. European Schoolnet The FireOAI API (cont.)

68. European Schoolnet Harvesting Service API Class SparkOAI –Singleton (as all the classes used as a service entry point) –Implements: ServiceManagement SparkOAIRepository: Similar to the FireOAIRepository interface but methods are void (to be used by harvesters) Package: net.sourceforge.limbs.client.oai

69. European Schoolnet Harvesting Service API (cont.)

70. European Schoolnet Additional Methods As a harvester: –setFireOAIHarvester(): To act as a source (i.e., receive results from the LRE) –setAutoMessageAccepted(): To receive its own request As a repository: –setFireOAIRepository(): To act as a repository (i.e., receive requests from the LRE)

71. European Schoolnet Available Implementations AGH agent-based search engine LRE federated harvesting –Classpath: Core Services fireOAI-api.jar spark-oai.jar

72. European Schoolnet Harvester Starts Service

73. European Schoolnet Repository Starts Service

74. European Schoolnet Harvesting At Work (identify)

75. European Schoolnet XML Response Format Is defined by http://www.openarchives.org/OAI/2.0/OAI- PMH.xsd http://www.openarchives.org/OAI/2.0/OAI- PMH.xsd As described in http://www.openarchives.org/OAI/openarch ivesprotocol.html Strict LOM replaces Dublin Core as default metadata format

76. European Schoolnet Digital Rights Management (DRM) Services Planned in May 2007’s release

77. European Schoolnet Setting Up The Environment Installing Java libraries –JWSDP 2.0: http://java.sun.com/webservices/downloads/we bservicespack.html http://java.sun.com/webservices/downloads/we bservicespack.html –matchBox 2.0: https://sourceforge.net/projects/limbs https://sourceforge.net/projects/limbs Security aspects: Keystore and certificates

78. European Schoolnet Connecting To LRE import net.sourceforge.limbs.client.core.SparkCore; import net.sourceforge.limbs.client.utils.Logger; SparkCore sc = SparkCore.getInstance(); sc.setVerbosity(SparkCore.VERBOSE ); sc.setSecurityParameters( ”sparkId", ”password", "/path/to/keystore.jks", ”keystorePwd" ); sc.setMessageSigned( true ); sc.setUnsignedMessageAccepted( true );

79. European Schoolnet Connecting To LRE (cont.) sc.setLdapSuffix( "dc=eun,dc=org" ) ; sc.setLimbsUrl( "http://stove.test.eun.org:8080" ); sc.setMqPrefix("calibrate" ); sc.turnConnectionOn("sparkId", "password" ); Logger.getInstance().logMessage( Logger.INFO, sc.getContract() ); Thread.sleep( 60000 ); sc.turnConnectionOff();

80. European Schoolnet Starting SQI Services (source) import net.sourceforge.limbs.client.core.SparkCore; import net.sourceforge.limbs.client.sqi.SparkSQI; SparkSQI sqi = SparkSQI.getInstance(); SourceSQI mySource = new SourceSQI(); sqi.setLocalSQISource(mySource); sqi.startService(); sqi.setAutoMessageAccepted(true); String sessionId = sqi.createAnonymousSession();

81. European Schoolnet A S 2 QL Query String query = " \n" + "<s2ql xmlns=\"http://fire.eun.org/xsd/s2ql-2.0\" " + "xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\n" + " en \n " + " fr \n " + " test \n " + " mathematics \n " + " \n " + " 12 \n " + " 18 \n " + " \n " + " \n " ;

82. European Schoolnet Sending A Query sqi.setQueryLanguage(sessionId, "http://fire.eun.org/xsd/s2ql-2.0"); sqi.setResultsFormat(sessionId, "http://fire.eun.org/xsd/strictLomResults-1.0"); sqi.asynchronousQuery(sessionId, query, "my very first query"); Thread.sleep( 120000 ); sqi.destroySession(sessionId); sqi.stopService();

83. European Schoolnet Implementing FireSQISource import org.eun.fire.sqi.SQIFault; import org.eun.fire.sqi.FireSQISource; public class SourceSQI implements FireSQISource { public void queryResultsListener(String queryID, String queryResults) throws SQIFault { System.out.println("*****************************************"); System.out.println(queryResults); System.out.println("*****************************************"); System.out.println("queryId arrived = " + queryID); }

84. European Schoolnet Starting SQI Services (target) import net.sourceforge.limbs.client.core.SparkCore; import net.sourceforge.limbs.client.sqi.SparkSQI; TargetSQI mySQI = new TargetSQI() ; SparkCore.getInstance().setMessageSigned( false ) ; SparkCore.getInstance(). setVerbosity( SparkCore.VERBOSE ) ; SparkCore.getInstance(). turnConnectionOn("sparkId", "Password" ) ; SparkSQI.getInstance().setLocalAsynchronous( true ) ; SparkSQI.getInstance().setLocalSQITarget( mySQI ) ; SparkSQI.getInstance().startService() ; String sessionId = SparkSQI.getInstance().createAnonymousSession() ; Thread.sleep( 120000 );...

85. European Schoolnet Implementing FireSQITarget import org.eun.fire.sqi.SQIFault; import net.sourceforge.limbs.client.utils.Logger; import org.eun.fire.sqi.FireSQITarget; public class TargetSQI implements FireSQITarget, FireSQISource { FireSQISource source ; public void setQueryLanguage( String targetSessionID, String queryLanguageID ) throws SQIFault{ Logger.getInstance().logMessage( Logger.INFO, "query language set to : " + queryLanguageID ) ;} public void setResultsFormat( String targetSessionID, String resultsFormat ) throws SQIFault{ Logger.getInstance().logMessage( Logger.INFO, "result format set to : " + resultsFormat ) ;}

86. European Schoolnet FireSQITarget (Cont.) public String synchronousQuery( String targetSessionID, String queryStatement ) throws SQIFault{ Logger.getInstance().logMessage( Logger.INFO, "synchronous query : " + queryStatement ) ; return "results for " + queryStatement ;} public void setSourceLocation (String targetSessionID, FireSQISource sourceLocation) throws SQIFault{ Logger.getInstance().logMessage( Logger.INFO, "setSourceLocation" ) ; source = sourceLocation ;} public void asynchronousQuery( String targetSessionID, String queryStatement, String queryID ) throws SQIFault{ Logger.getInstance().logMessage( Logger.INFO, "asynchronous query (query id = " + queryID + "): " + queryStatement ) ; source.queryResultsListener( queryID, "query result" ) ;}

87. European Schoolnet FireSQITarget (Cont.) public String createAnonymousSession() throws SQIFault{ Logger.getInstance().logMessage( Logger.INFO, "Create Anonymous Session" ) ; return "my Anonymous Session ID" ;} public String createSession( String userID, String password ) throws SQIFault{ Logger.getInstance().logMessage( Logger.INFO, "Create Onymous Session" ) ; return "my Onymous Session ID" ;} public void destroySession( String sessionID ) throws SQIFault{ Logger.getInstance().logMessage( Logger.INFO, "Session " + sessionID + " destroyed" ) ;} }

88. European Schoolnet Starting OAI Service (repository) LocalRepository localRepository = new LocalRepository() ; SparkCore spc = SparkCore.getInstance() ; spc.setMessageSigned( false ) ; spc.turnConnectionOn("sparkId", ”password" ) ; SparkOAI oai = SparkOAI.getInstance() ; oai.setAutoMessageAccepted( true ) ; oai.setFireOAIRepository( localRepository ) ; oai.startService() ; Thread.sleep( 120000 ) ; oai.stopService() ; spc.turnConnectionOff() ;

89. European Schoolnet Implementing FireOAIRepository import org.eun.fire.oai.FireOAIRepository; import net.sourceforge.limbs.client.utils.Logger; import nsdl.oai.protocol.UTCdatetime; public class LocalRepository implements FireOAIRepository{ public String getRecord( String str, String str1 ){ return "";} public String identify(){ Logger.getInstance().logMessage( Logger.INFO, "identify request received" ) ; return "identify response";} public String listIdentifiers( UTCdatetime tCdatetime, UTCdatetime tCdatetime1, String str, String str2, String str3 ){ return ""; } public String listMetadataFormats( String str ) { return ""; } public String listRecords( UTCdatetime tCdatetime, UTCdatetime tCdatetime1, String str, String str2, String str3 ) { return ""; } public String listSets( String str ) { return ""; } }

90. European Schoolnet The Repository Deadline Sep. 30, 2006: Repositories are connected to the LRE –S 2 QL / UTF-8 –LRE app. profile / strict LOM binding / UTF-8 First pilots start in October

91. European Schoolnet Connecting A Repository Means Managing the connection with the brokerage system Implementing FireSQITarget on top of the repository: –Managing sessions –Setting the query parameters –Turning S 2 QL into a query language supported by the repository –Formatting metadata in strict LOM

92. European Schoolnet What’s Tricky ? 1.Session Management 2.Mapping the repository schema into strict LOM 3.Translating S 2 QL into a query language supported by the repository 4.Keep your “igniter” alive

93. European Schoolnet What’s Specific To A Repository? 1.Mapping the repository schema into strict LOM 2.Translating S 2 QL into a query language supported by the repository

94. European Schoolnet FireSQITarget setQueryLanguage() setResultsFormat() synchronousQuery() setSourceLocation () asynchronousQuery() createAnonymousSession() createSession() destroySession()

95. European Schoolnet Tools Connection Kit: Everything but a query method S 2 QL Query Translets: A S 2 QL parser generator Minor: A full-fledged LGPL repository –SQI (JMS & SOAP) –OAI-PMH (HTTP & JMS)

96. European Schoolnet Questions? Do you have a repository? Does your repository use an application server? Persistency? Does your repository use a database that allows for a Java connection (e.g., JDBC) ?

97. European Schoolnet Questions ? Thank You

98. European Schoolnet Contact Information: David Massart david.massart@eun.org Jean-Noel Colin European Schoolnet rue de Trèves 61 B-1040 Brussels, Belgium http://lre.eun.org/ http://limbs.sourceforge.net/ http://fire.eun.org/