Presentation on theme: "Government Challenges With Big Data: A Semantic Web Strategy for Big Data Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic."— Presentation transcript:
1Government Challenges With Big Data: A Semantic Web Strategy for Big Data Dr. Brand NiemannDirector and Senior Enterprise Architect – Data ScientistSemantic CommunityAOL Government BloggerJanuary 24, 2013Link
2Outline Data Science Team Semantic Community 2013 Mission Statement Why We Are HereNIST Cloud Computing AND Big Data Forum and WorkshopSpotfire for Big Data Analytics and Data Science Analytics LibraryFrom the Year of Big Data to the Year of the Data Scientist Working With Big DataCross-Walk Table (in progress)The Practice of Data ScienceCurrent US Government Semantic Web StrategyInternational Linked Open Data StrategyOur Semantic Web Strategy for Big DataMy 5-Step Method To Get to 5-Stars With Open DataSystem of Systems ArchitectureData Federation in Spotfire15th SOA, Shared Services, and Big Data Analytics Conference (DRAFT)Comments: Semantic Medline, Noblis, Cray, and ORBIS TechnologiesQ & A
3Data Science TeamDr. Brand Niemann, Director and Senior Data Scientist, Semantic CommunityDr. Tom Rindflesch, Research Group Lead for Semantic Medline, National Library of MedicineDr. Victor Pollara, Senior Principal Scientist, NoblisDr. Eric Little, Director of Information Management, Orbis TechnologiesMark Guiton, Director, Government Relations, Cray Inc.Cray-YarcData announced semifinalists last month
4Semantic Community: Mission Statement for 2013 Help the Data Transparency Coalition help the 113th Congress with the re-introduction of the Data Act by Building the Federal Financial Information Network in the Cloud for the 113th Congress, January 4, Slides.Continue to work with Big Data Analytics (e.g. Recorded Future, Spotfire, etc.), Content Analytics and Knowledge Management (e.g. MindTouch), and Semantic Technologies (e.g. Be Informed, Semantic Insights, etc.) for data science and data journalism. Slides.Help start Open Government Data for Japan (and the US and Europe) with the Right Data (Statistical) with the Right People (Data Scientists) Working on the Right Business Problems (Return on Investment): January 21, Slides.Help the Federal Big Data Senior Steering Group with A Semantic Web Strategy for Big Data and to move From the Year of Big Data to the Year of the Data Scientist Working With Big Data, January 24, Slides.Help the ACT-IAC AMWG, C&T SIG, and ET-SIG with Big Data on Mobile Devices, Collaboration and Transformation, & Government Challenges With Big Data , January 16 and February 23, Slides.
5Why We Are Here Killer Semantic Web Application: Audit Function: George Strawn and Tom RindfleschData Architecture (5 Steps to Get to 5 Stars and Systems of Systems)Audit Function:Brand Niemann, Data Science Products at the US EPA and then for AOL Gov, etc.Todd Park, Data.gov and Communities (Health, Energy, Safety, and Education)Chris Greer, New Cloud AND Big Data Web SitesUsing Science (and Statistics) as Evidence in Public Policy:Kenneth Prewitt, NRC Committee Chair, and Former Census Bureau DirectorConnie Citro, NAS National Committee on Statistics Director (CNSTAT)Niall Brennan, Director of the Office of Information Products and Data Analytics at the Centers for Medicare and Medicaid Services, and the Institute of Medicine (IOM) Study on Geographic Variation in Health Care Spending and Promotion of High-value CareWorkforce Education:Professor Peter Fox, RPI, Individual and Group ProjectsProfessor Jeffrey Hammerbacher, UC Berkley and Cloudera Chief Scientist, Exploratory Data Analysis/Visualizations and ModelingProfessor Kirk Borne, GMU, Presentation to BDSSG Last SeptemberProfessor Michael Joseph, Big Data Pilot for First Year Engineering and Computer Science Students Last Semester
6NIST Cloud Computing AND Big Data Forum and Workshop Microscopes and Telescopes for Big Data, Professor Alexander SzalayMy Note: Spotfire 5 ApplicationScalable Parallel Interoperable Data Analytics Library (SPIDAL), Professor Geoffrey Fox Large scale data analytics on clouds Keynote in CloudDB '12 Proceedings of the Fourth International Workshop on Cloud Data Management Pages 21-24, October 29, 2012.Interoperability Between Clouds, Alan YoderMy Note: MindTouch and Spotfire 5 Application
7Spotfire for Big Data Analytics: Microscope NASA GCMD: Gateway to Big DataNSF Big Data Awards: Follow the WorkOSTP Harnessing The Power of Digital Data Report: Well-Defined URLsPCAST Designing a Digital Future Report: Interoperability InterfaceNITRD Dashboards: Live DemonstrationsFour Clicks: See, Sort/Search, Download, & Share (iPad)
8Data Science Analytics Library: Telescope & Library Live Links to Outside Data SourcesLive Information Links Between Analyticshttps://silverspotfire.tibco.com/us/library#/users/bniemann/Public
9From the Year of Big Data to the Year of the Data Scientist Working With Big Data Administration AnnouncementNSF Leads Federal EffortsTechAmerica Report Case StudiesMy Big Data at the Hill ReportMy Cross-Walk TableNASA Big Data Challenge Series:“How to make heterogeneous dataseem more homogeneous”.The Practice of Data Science
11The Practice of Data Science Josh Wills, Data writing on The Practice of Data Science said:Key trait of all data scientists. Understanding “that the heavy lifting of [data] cleanup and preparation isn’t something that gets in the way of solving the problem: it is the problem.” (DJ Patil)Inverse problems. Not every data scientist is a statistician, but all data scientists are interested in extracting information about complex systems from observed data, and so we can say that data science is related to the study of inverse problems. Real-world inverse problems are often ill-posed or ill-conditioned, which means that scientists need substantive expertise in the field in order to apply reasonable regularization conditions in order to solve the problem.Data sets that have a rich set of relationships between observations. We might think of this as a kind of Metcalfe’s Law for data sets, where the value of a data set increases nonlinearly with each additional observation. For example, a single web page doesn’t have very much value, but 128 billion web pages can be used to build a search engine.Open-source software tools with an emphasis on data visualization. One indicator that a research area is full of data scientists is an active community of open source developers.
12Current US Government Semantic Web Strategy Data.gov Advocates RDFa 1.1 Lite for Semantic Web Strategy.See Comment From Owen Ambur on Next Slide.I believe there is a better way to handle this that I showed the W3C eGov Special Interest Group on January 21st and have recommended for the reintroduction of the Data Act to the 113th Congress.Create a Semantic Index of Strong Relationships (SR) in RDF Format in a Spreadsheet.See next slide for example (spreadsheet and words)Integrate That With Other Spreadsheets and Relational Databases in An Interoperability Interface (e.g. Dashboard) That Can Searched.Essentially:Computer Scientists Use RD2RDF (James Hendler)Data Scientists Use SR2Excel2RDF (Brand Niemann)
13Comment From Owen Ambur OMB's official guidance to agencies on implementation of section 10 of the GPRA Modernization Act (GPRAMA) says they may use XML, JSON, spreadsheets or CSVs in order to meet the requirement to publish their strategic and performance plans and reports in machine-readable format... but not PDF or HTML -- at least not without "enhanced structural elements".I couldn't help but chuckle at how  is a PDF. I get your point however, which I think reinforces mine, that there is no US federal policy that prefers RDFa 1.1 over HTML Microdata for publishing metadata in HTML. RDFa Lite 1.1, W3C Recommendation, June 7, 2012, Manu Sporny, editor, see http://www.w3.org/TR/rdfa-lite/Source: Owen Ambur, December 18, 2012, W3C eGov Mailing List.
14International Linked Open Data Strategy: Linked Open Data Cloud Data My Question: Is it easy to add columns for who links to who?Answer: Not in a single table. SPARQL can't do cross-tabulation (Richard Cyganiak).
15International Linked Open Data: Comments to David Wood The Linked Open Data Cloud is not actually “linked data”.RDF at Data.gov is not linked data.The analytical and statistical communities view Data.gov and Linked Open Data as “IT projects”.Former Census Bureau Director Robert Groves.Conventional tools can do linked data and data integration.Spotfire Information Designer, Informatica, Information Builders, etc.
16International Linked Open Data: My EPA Green App Data App Example https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?EPAGreenAppsDataApp-Spotfire
17Our Semantic Web Strategy for Big Data: Previous Presentations MITRE: October 2, 2012GSA: November 29, 2012
18Our Semantic Web Strategy for Data: Simple Explanation One Table:Two ColumnsExample: Column 1: Section and Column 2: URLNote: A Column 3: Description could be in the URLExample: See Next SlideThree Columns:Example: Column 1: Subject, Column 2: Object, and Column 3: PredicateNote: This is the Semantic Web’s Linked Open Data Cloud as Linked Open Data for Network Analytics!Example: See Semantic MedlineFour Columns:Examples: Column 1: Subject, Column 2: Attribute, Column 3: From, and Column 4: To, or Column 1: City, Column 2: Country, Column 3: Longitude, and Column 4: LatitudeNote: This is the format for Spotfire’s Network Analytics Module developed for the CIA
19Our Semantic Web Strategy for Data: NASA Big Data Example
20Our Semantic Web Strategy for Data: Spotfire Network Analytics
21My 5-Step MethodSo what I like to do to illustrate (data science) and explain (data journalism) is the following (like a recipe):Put the Best Content into a Knowledge Base (e.g. MindTouch*)NASA Big DataPut the Knowledge Base into a Spreadsheet (Excel*)Linked Data to Subparts of the Knowledge BasePut the Spreadsheet into a Dashboard (Spotfire*)Data Integration and Interoperability InterfacePut the Dashboard into a Semantic Model (Excel*)Data Dictionaries and ModelsPut the Semantic Model into Dynamic Case Management (Be Informed*)Structured Process for Updating Data in the Dashboard* Examples of tools used.
22To Get to 5-Stars With Open Data DefinitionExample / Tool*Make your stuff available on the Web (whatever format) under an open licenseThis Story / MindTouchMake it available as structured data (e.g., Excel instead of image scan of a table)Spreadsheet / ExcelUse non-proprietary formats (e.g., CSV instead of Excel)Table / MindTouch and SpotfireUse URIs to identify things, so that people can point at your stuffTable of Contents / MindTouch and SpotfireLink your data to other data to provide context* Examples of tools used.Source of Star and Definition: http://www.w3.org/DesignIssues/LinkedData.html
23System of Systems Architecture Dynamic Case Management (e.g. Be Informed)Data Science Library (e.g. Spotfire)Data Science Products (e.g. Spotfire)SSemantic Index ofLinked Data(e.g. Excel)
24Data Federation in Spotfire: In-Memory and In-Database Data In-Memory DataWhen you are working with in-memory data tables (text files, Excel files, information links, etc.) you have access to all the functionality of Spotfire. You have the opportunity to use all columns as filters and perform any number of calculations. You can also use any of the tools within Spotfire to cluster data, calculate new columns, bin columns, make predictions etc. See Working With Large Data Volumes for some tips on how to improve the performance of an analysis with lots of data.In-Database DataWhen a connection to an external source is set up, all calculations are done using the external system and not with the Spotfire data engine. This will allow you to work with data volumes too large to fit into primary memory and take advantage of the power of the external system. When working with external data connections, you access only the current selection of data and all aggregations and calculations are made in-database (in-db).
25Data Federation in Spotfire: Database Connections, Information Links, & Analytics Library A database connection dialog is used to set up a connection to say a Teradata database, where you can analyze data from the database without bringing it into your analysis.An information link is a structured request for data which can be sent to the database. These specifications include one or more columns, and may include one or more filters.Stated in plain English, an information link could be: "Fetch the Name, Address and Phone_number for employees that pass the filter High_Income."Information links can also be used to limit what data to open in an analysis in a number of different ways.The library provides publishing capabilities for all of your analysis materials, so you can share data with your colleagues. The library can be used directly from Spotfire by anyone who has at least read privileges.
26Data Federation in Spotfire: Data Panel The Data panel is used to get an overview of the columns in all data tables, in-memory as well as in-database (in-db). When working with in-database data the Data panel is the starting point for configuring both visualizations and the filters panel, since no filters are created automatically for external data.https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?NASABigData-Spotfire
27Data Federation in Spotfire: Information Designer The Information Designer is a tool for setting up data sources and creating and opening information links. An information link is a database query specifying the columns to be loaded and any filters needed to narrow down the data table prior to creating visualizations in Spotfire. In Information Designer, information links are created from building blocks such as columns and filters using joins, calculations and aggregations.
2815th SOA, Shared Services, and Big Data Analytics Conference (DRAFT) Jason Bloomberg, President, ZapThink, a Dovel Technologies: New Book and Poster.The Agile Architecture Revolution: How Cloud Computing, REST-Based SOA, and Mobile Computing Are Changing Enterprise ITSteven Woodward, CEO and Founder of Cloud Perspectives, Member of the Shared Services Canada Advisory Committee, Leading Contributor to the NIST FedRamp, and Canadian Cloud Council Government CTO.Shared Services CanadaGeoffrey Fox, Professor of Informatics and Computing, Physics, Indiana UniversityLarge Scale Data Analytics on CloudsDr. Christopher L. Greer, Acting Senior Advisor for Cloud Computing and Associate Director, Program Implementation Office, ITL, NIST.NIST Cloud Computing AND Big Data Forum and WorkshopNITRD Interagency Big Data Senior Steering Group (BDSSG) Representative (s)What Does the Interagency Big Data Senior Steering Group Do?Peter Lyster, Program Director, Division of Biomedical Technology, Bioinformatics, and Computational Biology, National Institute of General Medical Sciences, National Institutes of Health, (NIH/NIGMS).NIH Chief Data Scientist.Adam Fuchs, Chief Technology Officer, SqrrlAccumulo for the NSA and Sqrrl for Open SourceWill Cukierski, Data Scientist, KaggleWhat is a Data Scientist?Big Data Use-Cases and PilotsData Science Teams working on Semantic Medline and Other Government Big Data Projects
29Comments: Semantic Medline, Noblis, Cray, and ORBIS Technologies Semantic Medline, Tom RindfleschNoblis, Victor PollaraCray, Mark GuitonORBIS Technologies, Eric Little
30Q & AContact Information:Brand NiemannTom RindfleschVictor PollaraMark GuitonEric Little