UMBC an Honors University in Maryland 1 Searching for Knowledge and Data on the Semantic Web Tim Finin University of Maryland, Baltimore County Joint work with Li Ding, Anupam Joshi, Yun Peng, Cynthia Parr, Pranam Kolari, Pavan Reddivari, Sandor Dornbush, Rong Pan, Akshay Java, Joel Sachs, Scott Cost and Vishal Doshi  This work was partially supported by DARPA contract F , NSF grants CCR and IIS and grants from IBM, Fujitsu and HP.

UMBC an Honors University in Maryland 2 This talk Motivation Semantic web 101 Swoogle Semantic Web search engine Use cases and applications State of the Semantic Web Conclusions

UMBC an Honors University in Maryland 3 Google has made us smarter

UMBC an Honors University in Maryland 4 But what about our agents? tell register Agents still have a very minimal understanding of text and images.

UMBC an Honors University in Maryland 5 This talk Motivation Semantic web 101 Swoogle Semantic Web search engine Use cases and applications State of the Semantic Web Conclusions

UMBC an Honors University in Maryland 6 XML helps “XML is Lisp's bastard nephew, with uglier syntax and no semantics. Yet XML is poised to enable the creation of a Web of data that dwarfs anything since the Library at Alexandria.” -- Philip Wadler, Et tu XML? The fall of the relational empire, VLDB, Rome, September 2001.

UMBC an Honors University in Maryland 7 “The Semantic Web will globalize KR*, just as the WWW globalize hypertext” -- Tim Berners-Lee Semantic Web adds semantics * Knowledge Representation

UMBC an Honors University in Maryland 8 Semantic Web 101 <rdf:RDF xmlns:rdf=" xmlns:foaf= xmlns:uni=http//ebiquity.umbc.edu/ontologies/uni/> Li Ding RDF/XML rdf:RDF tag namespaces  ontologies Semantic graph, URIs as nodes & links triples Li Ding foaf:name uni:Student rdf:type

UMBC an Honors University in Maryland 9 Where’s the semantics? URIs as common “rigid designators” Conventions let URIs denote things in the “real world” Namespaces + URIs give an unambiguous shared vocabulary RDF, RDFS and OWL have semantics defined using model theory and also axioms Ontologies allow agents to draw inferences –uni:Student is a subclass of foaf:Person –Every uni:Student uni:attends at least one uni:School –A foaf:Person with a uni:school is necessarily a uni:Student

UMBC an Honors University in Maryland 10 Much of the RDF data will come from databases, just like HTML content.

UMBC an Honors University in Maryland 11

UMBC an Honors University in Maryland 12 RDF/a RDF/a is a W3C proposal for embedding RDF in XHTML documents Jo Lambda's Home Page Hello. This is Jo Lambda 's home page. Work If you want to contact me at work, you can either me, or call <> foaf:name "Jo Lambda"^^rdf:XMLLiteral ; foaf:mbox ; foaf:phone " "^^rdf:XMLLiteral. An HTML Document with RDF embedded The triples in ntriple format.

UMBC an Honors University in Maryland 13 But what about our agents? A Google for knowledge on the Semantic Web is needed by software agents and programs Swoogle tell register

UMBC an Honors University in Maryland 14 This talk Motivation Semantic web 101 Swoogle Semantic Web search engine Use cases and applications State of the Semantic Web Conclusions

UMBC an Honors University in Maryland 15 Running since summer M RDF documents, 300M RDF triples, 10K ontologies

UMBC an Honors University in Maryland 16 Analysis Index Discovery IR Indexer Search Services Semantic Web metadata Web Service Web Server Candidate URLs Bounded Web Crawler Google Crawler SwoogleBot SWD Indexer Ranking document cache SWD classifier human machine htmlrdf/xml … the Web Semantic Web Information flowSwoogle‘s web interface Legends Swoogle Architecture

UMBC an Honors University in Maryland 17 A Hybrid Harvesting Framework Manual submission RDF crawlingBounded HTML crawlingMeta crawling Seeds MSeeds H Seeds R Swoogle Sample Dataset Inductive learner the Web Google API call crawl true would google

UMBC an Honors University in Maryland 18 Performance – Site Coverage SW06MAR - Basic statistics (Mar 31, 2006) – 1.3M SWDs from 157K websites – 268M triples – 61K SWOs including >10K in high quality –1.4M SWTs using 12K namespaces Significance –Compare with existing works ( DAML crawler, scutter ) –Compare SW06MAR with Google ’ s estimated SWDs SWDs per website Website

UMBC an Honors University in Maryland 19 Performance – crawlers’ contribution High SWD ratio: 42% URLs are confirmed as SWD Consistent growth rate: 3000 SWDs per day RDF crawler: best harvesting method HTML crawler: best accuracy Meta crawler: best in detecting websites # of documents

UMBC an Honors University in Maryland 20 This talk Motivation Semantic web 101 Swoogle Semantic Web search engine Use cases and applications State of the Semantic Web Conclusions

UMBC an Honors University in Maryland 21 Applications and use cases Supporting Semantic Web developers –Ontology designers, vocabulary discovery, who’s using my ontologies or data?, use analysis, errors,statistics, etc. Searching specialized collections –Spire: aggregating observations and data from biologists –InferenceWeb: searching over and enhancing proofs –SemNews: Text Meaning of news stories Supporting SW tools –Triple shop: finding data for SPARQL queries

UMBC an Honors University in Maryland 22

UMBC an Honors University in Maryland 23 By default, ontologies are ordered by their ‘popularity’, but they can also be ordered by recency or size. 80 ontologies were found that had these three terms Let’s look at this one

UMBC an Honors University in Maryland 24 Basic Metadata hasDateDiscoveredhasDateDiscovered: hasDatePinghasDatePing: hasPingStatehasPingState: PingModified typetype: SemanticWebDocument isEmbeddedisEmbedded: false hasGrammarhasGrammar: RDFXML hasParseStatehasParseState: ParseSuccess hasDateLastmodifiedhasDateLastmodified: hasDateCachehasDateCache: hasEncodinghasEncoding: ISO hasLengthhasLength: 18K hasCntTriplehasCntTriple: hasOntoRatiohasOntoRatio: 0.98 hasCntSwthasCntSwt: hasCntSwtDefhasCntSwtDef: hasCntInstancehasCntInstance: 8.00

UMBC an Honors University in Maryland 25

UMBC an Honors University in Maryland 26

UMBC an Honors University in Maryland 27 These are the namespaces this ontology uses. Clicking on one shows all of the documents using the namespace. All of this is available in RDF form for the agents among us.

UMBC an Honors University in Maryland 28 Here’s what the agent sees. Note the swoogle and wob (web of belief) ontologies.

UMBC an Honors University in Maryland 29 We can also search for terms (classes, properties) like terms for “person”.

UMBC an Honors University in Maryland 30 10K terms associated with “person”! Ordered by use. Let’s look at foaf:Person’s metadata

UMBC an Honors University in Maryland 31

UMBC an Honors University in Maryland 32

UMBC an Honors University in Maryland 33

UMBC an Honors University in Maryland 34

UMBC an Honors University in Maryland 35

UMBC an Honors University in Maryland 36

UMBC an Honors University in Maryland 37

UMBC an Honors University in Maryland 38 UMBC Triple Shop Online SPARQL RDF query processing based on HP’s Jena and Joseki with several interesting features Selectable level of inference over model Automatically finds SWDs for give queries using Swoogle backend database –Provide dataset creation wizard –Dataset can be stored on our server or downloaded –Tag, share and search over saved datasets

UMBC an Honors University in Maryland 39 Web-scale semantic web data access agent data access servicethe Web ask (“person”) Search vocabulary ask (“?x rdf:type foaf:Person”) inform (“foaf:Person”) Fetch docs Populate RDF database Query local RDF database inform (doc URLs) Search URIrefs in SW vocabulary Search URLs in SWD index Compose query Index RDF data

UMBC an Honors University in Maryland 40 Who knows Anupam Joshi? Show me their names, address and pictures

UMBC an Honors University in Maryland 41 The UMBC ebiquity site publishes lots of RDF data, including FOAF profiles

UMBC an Honors University in Maryland 42 No FROM clause! Constraints on where the data comes from

UMBC an Honors University in Maryland 43 PREFIX foaf: SELECT DISTINCT ?p2name ?p2mbox ?p2pix WHERE { ?p1 foaf:name "Anupam Joshi". ?p1 foaf:mbox ?p1mbox. ?p2 foaf:knows ?p3. ?p3 foaf:mbox ?p1mbox. ?p2 foaf:name ?p2name. ?p2 foaf:mbox ?p2mbox. OPTIONAL { ?p2 foaf:depiction ?p2pix }. } ORDER BY ?p2name

UMBC an Honors University in Maryland 44

UMBC an Honors University in Maryland 45 Swoogle found 292 RDF data files that appear relevant to answering our query

UMBC an Honors University in Maryland 46 Let’s save the dataset before we use it

UMBC an Honors University in Maryland 47

UMBC an Honors University in Maryland 48 And tag it so we and others can find it more easily.

UMBC an Honors University in Maryland 49 Here we are using it to get an answer to “Who knows Anupam Joshi”

UMBC an Honors University in Maryland 50 He has many friends!

UMBC an Honors University in Maryland 51

UMBC an Honors University in Maryland 52 This talk Motivation Semantic web 101 Swoogle Semantic Web search engine Use cases and applications State of the Semantic Web Conclusions

UMBC an Honors University in Maryland 53 Will it Scale? How? Here’s a rough estimate of the data in RDF documents on the semantic web based on Swoogle’s crawling System/dateTermsDocumentsIndividualsTriplesBytes Swoogle21.5x x10 5 7x10 6 5x10 7 7x10 9 Swoogle32x10 5 7x x x10 7 1x x10 6 5x10 7 5x10 9 5x x10 6 5x10 9 5x x10 13 We think Swoogle’s centralized approach can be made to work for the next few years if not longer.

UMBC an Honors University in Maryland 54 How much reasoning? SwoogleN (N<=3) does limited reasoning –It’s expensive –It’s not clear how much should be done More reasoning would benefit many use cases –e.g., type hierarchy Recognizing specialized metadata –E.g., that ontology A some maps terms from B to C

UMBC an Honors University in Maryland 55 This talk Motivation Semantic web 101 Swoogle Semantic Web search engine Use cases and applications State of the Semantic Web Conclusions

UMBC an Honors University in Maryland 56 Conclusion The web will contain the world’s knowledge in forms accessible to people and computers –We need better ways to discover, index, search and reason over SW knowledge SW search engines address different tasks than html search engines –So they require different techniques and APIs Swoogle like systems can help create consensus ontologies and foster best practices –Swoogle is for Semantic Web 1.0 –Semantic Web 2.0 will make different demands

UMBC an Honors University in Maryland 57 Annotated in OWL For more information