Presentation on theme: "Data Science With Spotfire for Opening Government Data for Innovators and Entrepreneurs Dr. Brand Niemann Director and Senior Enterprise Architect – Data."— Presentation transcript:
1 Data Science With Spotfire for Opening Government Data for Innovators and Entrepreneurs Dr. Brand NiemannDirector and Senior Enterprise Architect – Data ScientistSemantic CommunityAOL Government BloggerFebruary 18, 2012
6 My Silver Spotfire Data Science Library in the Cloud I will show you examplesof how I built these later.Federal Budget 2013in a day!Data Science Library in the Cloud
7 Federal Budget 2013 Dashboard PC Desktop Spotfire
8 Federal Budget 2013 Dashboard Silver Spotfire Web Player
9 The Value Proposition of Spotfire More to Do with Less?Take Control of Your Business DataVisualize Your Data - Drag and Drop Your SpreadsheetsCustomize Your Dashboards - Instantly Add New VisualizationShare Your Insights - Publish Your DashboardGet Trial of Silver SpotfireAgile Analysis:Fastest to Actionable InsightInsight Into the UnknownSelf-Service DiscoveryUniversal Analytics PlatformSource:Source: https://silverspotfire.tibco.com/us/home
10 The Value Proposition of Agile Analysis - Invert your "bath tub" with Spotfire Analytics Spotfire offers dimension-free data exploration, data mashups, predictive and event driven, contextual collaboration and enterprise class technology.Source: Jim Hawley, Spotfire Federal Government
12 The Value Proposition of Data Science We are interested in learning about Taxonomy and Enterprise Vocabulary for fundamental architectural elements to enable interoperability and provide consistent understanding of shared architecture information across the enterprise.Source: Walt Okon, Senior Architect Engineer, Enterprise Architecture & Standards, Department of Defense Chief Information Officer, October 4th, .Aneesh Chopra: Government’s Big Data Opportunity. “The Federal Government needs Data Science and Data Scientists!”Source: O’Reilly STRATA Conference New York, September 20, 2011.
13 What is Data Science?Data science enables the creation of data products.Data science is a holistic approach.The first step of any data analysis project is “data conditioning,” or getting the data into a state where it is usable.Statistics is the “grammar of data science.”Edward Tufte’s Visual Display of Quantitative Information is a foundational text for anyone practicing data science. He calls himself a data scientist!Data scientists are patient, inherently interdisciplinary, and can think outside the box.Some References:Data Science Graduate Class at RPI, Troy, NYData ScienceAOL Government
14 Data Science Architecture 1. Create an inventory of documents and data sets.2. Build that inventory in an Excel spreadsheet so it supports faceted search in a Spotfire dashboard.3. Provide a sample knowledgebase of each of the four types of documents (Word, PDF, PowerPoint, and Excel).4. Provide the multiple sample knowledgebases in a Spotfire dashboard so they can be seen, compared, merged, harmonized, sorted, searched, downloaded, and shared on mobile devices (e.g. iPad).5. Scale the previous architectural pattern with more content volume and types if necessary.
15 Knowledgebase What is a knowledgebase? How is a knowledgebase built? Knowledgebase = Model + InstancesModel = Vocabulary, Taxonomy, and Ontology/RulesInstances = Linked Data Semantically Linked to the ModelHow is a knowledgebase built?Model = Vocabulary – Glossary in MindTouchTaxonomy – Contents and Resources in MindTouchOntology/Rules in Be Informed 4Instances = Linked Data Semantically Linked to Model – MindTouch, Excel and Spotfire
16 The Knowledgebase in MindTouch MindTouch is often referred to as the “Swiss Army Knife” of collaboration tools! See MindTouch Web Site.So I make MindTouch look like a “Knowledge Hub” (e.g., on top of SharePoint Portal like the Army Corps of Engineers Knowledge Hub) and feature key documents and data sets.Relating one or more Spotfire dashboards to the key document and data sets points to the ability to track progress. It’s all about metrics!
17 MindTouch Social Knowledge Base – Social Help Center MindTouch provides exceptional, purpose-built social help desks and knowledge bases for some of the world’s largest and most respected technology and media brands. Our solutions layer social and collaborative capabilities over existing systems and deliver strategic value to our customers. Product help is strategic for user assistance teams, product and marketing teams, community managers, and product evangelists as they look to build engaged communities around their brands to increase top and bottom line revenues.
18 Army Corps of Engineers Knowledge Hub The Knowledge Hub is a dynamic online destination to feature products developed by the US Army Corps of Engineers as well as to engage end-users and others in innovative and intuitive interaction. Within the Knowledge Hub is a Navigation Community which provides a forum on which navigation personnel can discuss, share, learn, explore and search products, project and programs of concern them. One goal of the Hub is to be a web-based framework for enterprise decision support and tech transfer within the Corps of Engineers.POC: Marty Kittrell,Source:
19 MindTouch Knowledgebase AOL Government StorySpotfire DashboardResearch Notes (Metadata)Complete Budget DocumentAttachments (see next slide)Comments (see next slide)
21 Data Science is Part of My System of Systems Architecture Dynamic Case Management (e.g. Be Informed)Data Science Library (e.g. Spotfire)Data Science Products (e.g. Spotfire)SSemantic Index ofLinked Data (e.g. Excel)
22 Agile Methods: Questions on Our Minds What Should We Do with Enterprise Architecture?Be like a building architect that provides a blueprint with building specifications and a scale (able) model.How Should We Do That?With Be Informed, an internationally operating, independent software vendor that has been recognized recently by Gartner and Forrester.What is Be Structured?It is complimentary to various well-known development, compliance and architecture frameworks, including ITIL, Cobit, Prince II, RUP, TOGAF, Zachman, SCRUM, Cogniam, DEMO, and Pronto. Note: See my tutorials.
23 Working Within A Broader Context Begin with the End in Mind (see Next Slide):Open Innovator's ToolkitPresident Obama emphasizes a “bottom-up” philosophy that taps citizen expertise to make government smarter and more responsive to private sector demands. This philosophy of “open innovation” has already delivered tangible results in public and regulated sectors of the economy – areas like health IT, learning technologies, and smart grid – that are poised to deliver productivity growth and grow the jobs of the future. We have surfaced new or improved policy tools deployed by our government to achieve them. We’ve posted the Open Innovator’s Toolkit as a roster of 20 leading practices that an “open innovator” should consider when confronting any policy challenge – at any level of government. Our aspiration is to build upon this list, adding new tools and case studies to form an evidence base that will help to scale “open innovation” across the public sector.Follow 5 Easy Steps:1. Build an table of contents-like index of complex documents with well-defined web addresses in MindTouch.2. Build that index in an Excel spreadsheet so it supports faceted search in a Spotfire dashboard.3. Build a Spotfire knowledgebase with that Excel spreadsheet.4. Build multiple knowledgebases in a Spotfire dashboard so they can be seen, compared, merged, harmonized, sorted, searched, downloaded, and shared on mobile devices (e.g. iPad).5. Scale the previous architectural pattern with more content volume and types if necessary.
24 Open Government Initiative: Opening Data For Innovators and Entrepreneurs Our aspiration is to build upon this list, adding new tools and case studies to form an evidence base that will help to scale “open innovation” across the public sector.
25 Step 1. Build an table of contents-like index of complex documents with well-defined web addresses in MindTouch.
26 2. Build that index in an Excel spreadsheet so it supports faceted search in a Spotfire dashboard. Note: This MindTouch tablecopies directly to Excel in thenext slide.
27 2. Build that index in an Excel spreadsheet so it supports faceted search in a Spotfire dashboard.
28 3. Build a Spotfire knowledgebase with that Excel spreadsheet. PC Desktop Spotfire
29 3. Build a Spotfire knowledgebase with that Excel spreadsheet. Building Steps:1 - Drag and Drop Spreadsheet Onto Spotfire (see Scatter Plot automatically).2 - Add New Table to Display Spreadsheet Data (make any adjustments/corrections and Refresh Data).3 - Adjust Scatter Plot Axes, Color by, Shape by, Size by to produce desired display.4 - Add New Test Area, Rename Page as Dashboard and Add MindTouch and Excel with Web links to sources of metadata and data.5 - Insert Action Controls to Reset All Filters and Unmark Marked Rows.6 - Save Spotfire file to hard drive with desired name and then save to Library.7 - Test Web Player version and embed in MindTouch.
30 Building Step 7 - Test Web Player version and embed in MindTouch. Spotfire Web Player
31 Follow 5 Easy StepsStep 4. Build multiple knowledgebases in a Spotfire dashboard so they can be seen, compared, merged, harmonized, sorted, searched, downloaded, and shared on mobile devices (e.g. iPad).Another example: How To Simplify Benefits Website For Veterans (AOL Government, MindTouch, Excel, Spotfire, and PowerPoint Tutorial).Step 5. Scale the previous architectural pattern with more content volume and types if necessary.My Silver Spotfire Library in the Cloud!
33 New Features in Spotfire 4.0 Information At A Glance:Dynamic ValuesConditional IconsSparklinesGraphical Summary TableLook and Feel:All New Graphical ProfilePop-Over Filter PanelPop-Over LegendIndividual Control Over Axis Label VisibilityMore Control Over Legend Contents and PlacementFixed Size LayoutMix Filters and Controls on the PageNicer Looking TablesCombine Different Slices of Data on the Same PageToolbars and Information
34 New Features in Spotfire 4.0 (Continued) Navigation and Interaction:ActionsPage History NavigationEmbed Interactive ControlsBuilding Dashboards:Preserve Information When Switching VisualizationsChange All Fonts in One PlaceEasier Access to Toggling Visualization FeaturesMore Predefined Categorical Coloring SchemesManage Document Color SchemesBetter Defaults When Creating VisualizationsToggle Auto Column Additions OffAnalysis PreviewsControl Over Table Header Font
35 New Features in Spotfire 4.0 Collaboration:Share with TIBBRAdd TIBBR Discussions to the AnalysisEmbed Dashboards in Other Web PagesOther Enhancements:Export FooterStepped LinechartsAutomation Services 4.0
36 New Features in Spotfire 4.0 Information At A Glance:Dynamic Values:What – Dynamically display single values in text areas that responds to filtering and parameter changes.Why – Look at most important numbers first before diving into more details.My Note: See Next Slides.Conditional Icons:What – Dynamically calculated conditional icons that respond to filtering and parameter changes.Why – Indicate change, comparisons to target and highlight important events.Sparklines:What – Dynamically calculated sparklines that respond to filtering and parameter changes.Why – Show at a glance and when drilling in whether a metric is trending down, up or varies a lot.Graphical Summary Table:What – Dynamic values, conditional icons and sparklines in one compact table broken down by some category.Why – Visually show everything you need on a single screen.
37 New Features in Spotfire 4 New Features in Spotfire 4.0: Dynamic Values, Conditional Icons, Sparklines, and Graphical Summary TablePC Desktop Spotfire
38 New Features in Spotfire 4 New Features in Spotfire 4.0: Dynamic Values, Conditional Icons, Sparklines, and Graphical Summary TableFilter forDebt ServicePC Desktop Spotfire
39 New Features in Spotfire 4.0 Collaboration:Share with TIBBR:What – Right Click on any visualization or page and share the view in tibbr with a link back to the analysisWhy – Easy sharing of insights and findings.My Note: Tibbr host name has to be set by Administrator.Add TIBBR Discussions to the Analysis:What – Integrated tibbr discussions right in the analysis filtered to a particular subject.Why – Discuss insights and findings with colleagues directly in the analysis. Subscribe and get notified when someone posts a comment on an analysis you are interested in.My Note: See Next Slide.Embed Dashboards in Other Web Pages:What – One click access to HTML fragments that displays a Spotfire page that can be pasted directly into portals and other web pages.Why – Put a link to the analysis in your corporate blog or wiki. Integrate Spotfire analysis displays into SharePoint WebPart and other portals.My Note: I was already doing this!
40 New Features in Spotfire 4 New Features in Spotfire 4.0 Collaboration: Add TIBBR Discussions to the Analysis
41 New Features in Spotfire 4.0 Other Enhancements:Export Footer:What – Include a footer when exporting or printing pages.Why – Make it clear where the printout came from or indicate to the reader that the contents is confidential.My Note: See Next Slide.Stepped Linecharts:What – Draw stepped linecharts that only show a change in value at the exact point where the value changed.Why – Better representation of discrete data that avoids misleading the user by interpolating values in between data points.Automation Services 4.0:What – New task added to remap Information Services catalogs and schemas during an automated Library import.What – Allow for the automation of migrating a Spotfire Information Model from a test to production environment in instances when the test and production instances of the data source are in different database catalogs or schemas.My Note: See Slides That Follow.
42 New Features in Spotfire 4.0: Other Enhancements Export FooterStepped Linecharts
43 New Features in Spotfire 4.0: Other Enhancements: Stepped Linecharts PC Desktop Spotfire
44 New Features in Spotfire 4.0 Other Enhancements: Automation Services 4.0 TIBCO Spotfire Automation ServicesSelecting a "Set data source credentials" task in the job builder will now allow you to go back and select a different certificate if the first one selected is invalid.
45 New Features in Spotfire 4.0 Other Enhancements: Automation Services 4.0
46 New Features in Spotfire 4.0 Other Enhancements: Automation Services 4.0 My Note: Customize Spotfire Documentation in MindTouch