Worldwide Lexicon Brian McConnell May, 2002

WWL – Brian McConnell Worldwide Lexicon Intro Automatic discovery of dictionary, semantic net and translation servers throughout the net Creates standard client/server interface for communicating with servers Creates distributed human computing grid (allows servers to poll idle users to enter data, score recent submissions) “GNUtella for language services”

WWL – Brian McConnell What WWL Does Creates a SOAP based interface for locating and communicating with language services Creates mechanism for discovering WWL servers on the fly Allows any application to talk to language servers with a few lines of code Allows existing dictionaries and MT systems to expose their data via WWL Creates something similar to except it taps idle users to contribute Creates a web services API for language services

WWL – Brian McConnell What WWL Does Not Do Does not create a global, centrally managed dictionary (WWL is a P2P network of dictionaries and language servers) WWL does not provide machine translation services (although WWL can be used to talk to existing MT servers) WWL does not compete with existing dictionaries or translation services. It makes existing systems more accessible to applications and their users. WWL does not specify details about how dictionary and MT server internal processes

WWL – Brian McConnell Some Example Applications Browser and text editor plug ins Extended dictionaries for machine translation systems Human assisted document translation client (polls users to enter data when they’re not Multilingual chat clients (poll WWL data sources as needed to assist with translations) Real-time translation (via Jabber or SMS) Teaching aids User supported dictionaries and translation memories

WWL – Brian McConnell Worldwide Lexicon Protocol Built upon the Simple Object Access Protocol Applications communicate via a small set of SOAP methods HTTP CGI interface also used for data entry and user peer review Goal: allow developers to locate and query any WWL data source with a few lines of code.

WWL – Brian McConnell Protocol Overview Three types of methods WWL server discovery and network status methods WWL client/server query methods Utility functions About a dozen methods overall

WWL – Brian McConnell System Overview Four basic types of nodes Supernodes (directory servers) WWL servers (dictionaries, MT servers, semantic nets) Gateways (allow non-WWL servers to present WWL front end) Client apps (plug ins, IM clients, etc)

WWL – Brian McConnell WWL Server Discovery Client app contacts a WWL supernode Invokes WWLFindServers() to fetch list of active servers and gateways that can process client’s request Supernode replies with a list of WWL servers, as well as information about each server’s capabilities WWL servers and gateways announce selves to supernodes at startup via WWLRegister() and WWLServerStatus() methods

WWL – Brian McConnell WWL Supernodes Track current status of WWL servers and their peers (servers send registration and status messages) Client apps use supernodes to locate WWL servers and gateways on the fly (e.g. locate Spanish-French full-text translation server) Supernodes also provide quality control (known WWL servers are listed first) Anyone can host a supernode (similar to GNUtella directory servers)

WWL – Brian McConnell WWL Gateways Translate WWL/SOAP method calls into other formats Can be used to talk to DICT dictionary servers Can be used to talk to proprietary systems Can do screen scraping (e.g. send query to web based MT server via CGI, scrape results from HTML response) Can even be used to cache and index static wordlists, and to make them appear to users as WWL data sources to any WWL client

WWL – Brian McConnell Client/Server Communication Three SOAP methods allow clients to submit queries to WWL servers via standard interface. WWL servers reply via SOAP, results are returned to client app in XML data structure WWL interface can co-exist with other interfaces (DICT, web/cgi, WAP, etc)

WWL – Brian McConnell Typical Client Session Contacts WWL supernode(s) to fetch list of active WWL servers according to language, services required Contacts top ranked WWL server to perform query (e.g. translate phrase from spanish to french) If query fails, contacts other WWL servers to perform query

WWL – Brian McConnell Application Development WWL defines a client/server interface Client and server apps can be developed and tested independently System is complex, but individual components are simple Perfect fit for open source development model

WWL – Brian McConnell Server Apps & Projects Updating existing dictionaries and machine translation servers for WWL and Building gateway servers that emulate WWL while talking to non-WWL servers (DICT, HTTP, etc) Document translation servers based on concept

WWL – Brian McConnell Client Applications Browser/text editor plug ins WWL chat clients Teaching aids

WWL – Brian McConnell Updating Existing Servers As simple as adding a few scripts to respond to SOAP calls (reply via SOAP versus HTML) SOAP/WWL interface co-exists with other front ends WWL server can be read-only, or can allow user data entry through Allows hundreds of existing dictionaries, encyclopedia and machine translation servers to participate in WWL with minimal effort

WWL – Brian McConnell Example: WWL Chat Client Listens to incoming and outgoing messages When user enables translation, IM client uses WWL to contact machine translation servers as needed When user enables dictionary features, IM client assists user in translating words and phrases when composing messages (ideal for users who know a language but are not fluent)

WWL – Brian McConnell Distributed human computing Users download small client program that polls WWL server(s) for jobs when user is not busy When WWL server has job, it instructs client to force browser to form/CGI user (data entry form is generated by WWL server) User enters requested information (definition, translation, score for other user’s submission) Each user does small amount of work, with large population system learns at rapid pace

WWL – Brian McConnell Quality Control Editorial oversight (WWL servers can require some or all user submissions to be reviewed by editors and trusted users via private CGI form) Randomized peer review (WWL server asks some users to score submissions from the Hybrid system that combines randomized peer review with editorial oversight (editors focus on submissions with ambiguous scores or from unknown users).

WWL – Brian McConnell Project Timeline WWL protocol spec is available at Work to develop first generation apps (supernodes, retrofit existing dictionary servers) is underway Work to develop client is in Looking for developers to contribute to project

WWL – Brian McConnell Development Priorities Stable supernode server Source libraries for use by existing dictionary and translation servers WWL gateway servers (to talk to non-WWL sites) Simple client apps (browser plug in, IM client that links to MT servers)

WWL – Brian McConnell