1. “Semantics” for Innovation in Visualization and Multimedia: Smarter Information Science ICSTI Workshop February 8, 2011, Redmond WA Peter Fox (RPI) pfox@cs.rpi.edupfox@cs.rpi.edu Tetherless World Constellation http://tw.rpi.eduhttp://tw.rpi.edu

2. Please buckle your seatbelt Working premise and the burden Opportunity – new means –Linked open data (LOD) –Open-source software (Field) Science conduct Semiotics of portrayal –Includes semantics Representing e.g. –Uncertainty, quality, bias Speculation 2Tetherless World Constellation

3. 3 Working premise Scientists – actually ANYONE - should be able to access and use a global, distributed knowledge base of scientific data that: appears to be integrated appears to be locally available But… data and information is obtained by multiple means (instruments, models, analysis) using various (often opaque) protocols, in differing vocabularies, using (sometimes unstated) assumptions, with inconsistent (or non-existent) meta-data. It may be inconsistent, incomplete, evolving, and distributed AND created in a form that facilitates generation, not use (except by accident) And … significant levels of semantic heterogeneity, large- scale data, complex data types, legacy systems, inflexible and unsustainable implementation technology… Uh-oh

4. Changing the equation “Changing the Equation for Scientific Data Visualization” – Fox and Hendler (Feb 11, 2011) Science (Perspectives), in press (embargoed, sorry) Three important points –Unlocked data (and it’s big, really, really…) –Visualization for the masses throughout the ‘life-cycle’ but scale-free (!) –Smarter data, smarter visualization

5. 5.. Data has Lots of Audiences From “Why EPO?”, a NASA internal report on science education, 2005 More Strategic Less Strategic Science too!

6. Fox Informatics and Semantics, © 2008 6 Shift the Burden from the User to the Provider – for Viz. too!

7. Too many diagrams

8. Visualizing Linked Open Data (logd.tw.rpi.edu)

9. Linked open data Simply put: data is in RDF and has a URI and/ or it’s behind a query-able ‘triple-store’ interface Raw data RDF Triple Store JSONGraphics ‘convert’‘load’ ‘query’‘render’

10. New means – artists to the rescue Digital artists, they needed good creative visual tools, art at the speed of creative thought, feeling, intuition, mental representation and they love programming And, RPI has EMPAC – Experimental Media and Performing Arts Center

11. From flat screen to black box - EMPAC

12. Field – rapid visualizing

13. What we are doing Field meets Linux! Linked data meets Field! –Feed the current LOD graphics into Field for manipulation –Then…. Unscrew the Google graphics Unscrew the JSON feed Query / consume raw RDF Visualizing at the speed of thought/ typing.. From the laptop to scale So this is where the semantics re-enter, especially for portrayal

14. Linked open data Field consumes JSON (webify it) Raw data RDF Triple Store JSONField ‘convert’‘load’ ‘query’‘render’

15. Linked open visualization Field queries ‘triple stores’ (semantic webify it) Raw data RDF Triple Store Field ‘convert’‘load’ ‘query’/‘render’

16. Linked open visualization Field queries ‘triple stores’ (semantic webify it) Raw data RDF Triple Store Field ‘dynamic’ ‘load’ ‘query’/‘render’ ‘access’

17. Science - Means of conduct 17

18. So what about abduction? No, not the criminal meaning… Is a method of logical inference introduced by Peirce which comes prior to induction and deduction for which the colloquial name is to have a "hunch". Abductive reasoning –starts when an inquirer considers of a set of seemingly unrelated facts, –armed with an intuition that they are somehow connected –oh, wait, good job for visualization!!! Leverage open world, semantics, too… on the web 18

19. Semiotics, also called semiotic studies or semiology, is the study of sign processes (semiosis), or signification and communication, signs and symbols, into three branches: –Syntactics: Relation of signs to each other in formal structures –Semantics: Relation between signs and the things to which they refer; their denotata –Pragmatics: Relation of signs to their impacts on those who use them 19 Information theory

20. Semiotic model 20

21. Semiotics of portrayal But we are talking about a digital world increasingly more than an analog one Beyond the separation of content from presentation –We have means for content (context and structure) semantics but pragmatics? Portrayal (not just ‘maps’ or ‘graphs’) –How – representation of content, context and structure, capture visualization provenance –Graphs, points, lines polygons, titles, axes, color, shade, dimensions, … and their relation to each other!

22. For science viz. We leave the untold things untold – big (really, really big) problem(s) – like: –Uncertainty –Quality –Bias –Need evidence An example?

23. 9/2/2015 MODIS Terra & Aqua vs. AIRS Cloud Top Pressure AIRS vs. MODIS AquaAIRS vs. MODIS Terra MODIS Aqua vs. MODIS Terra Correlation maps for Jan 1 – 16, 2008 Impact: Throw your hands up in the air and just walk away, silently…

24. Parameter AParameter B Difference alert Parameter Name :Aerosol Optical Depth at 550 nm Dataset:MYD08_D3.005MOD08_D3.005  Diff Data-Day definitionUTC (00:00-24:00Z) The same but…. Temporal resolutionDaily Spatial resolution1x1 degree Sensor:MODIS Platform:AquaTerra  Diff EQCT13:3010:30  Diff Day Time NodeAscendingDescending  Diff Pre-Giovanni Processes :ATBD-MOD-30 Giovanni Processes:Spatial subset Time average Spatial subset Time average MODIS Terra vs. MODIS Aqua AOD Correlation Included Overpass time Difference Known Issues: The difference of EQCT and Day Time Node, modulated by data-day definition, caused the included overpass time difference, which makes the artifact difference. See sample images: BUT WHY ARE WE SAYING THIS IN WORDS?

25. Abductive Information System? What would this look like in application tools? How to explore ‘hunches’ (hints)? If you consent that induction is fundamentally part of how an information system is developed, then how to allow for abduction before induction may be possible? Open world, integrative Design factors? Architecture factors? Library factors? Cognitive factors? 25

26. Speculation But back to big data and the need to turn the visualization ‘walls’ into exhibits, 4-dimensions – installations – i.e. not immersion but experience –Synesthesia – why only one sense? –Rapid

27. Speculation At scale – why? Stereo – why? Linked to the live data – minimal curation! Goal: restore abductive reasoning to the conduct of science for specialists and non-specialists And this has to be informatics-based not some ad-hoc techies making stuff up… Collaboration – wanna play?

28. So long and … pfox@cs.rpi.edu http://tw.rpi.edu http://openendedgroup.com http://logd.tw.rpi.edu http://empac.rpi.edu

29. Back shed

30. 20080602 Fox VSTO et al.30 Curation stages Need to be here

31. 31 Mind the Gap! There is/ was still a gap between science and the underlying infrastructure and technology that is available Cyberinfrastructure is the new research environment(s) that support advanced data acquisition, data storage, data management, data integration, data mining, data visualization and other computing and information processing services over the Internet.  Informatics - information science includes the science of (data and) information, the practice of information processing, and the engineering of information systems. Informatics studies the structure, behavior, and interactions of natural and artificial systems that store, process and communicate (data and) information. It also develops its own conceptual and theoretical foundations. Since computers, individuals and organizations all process information, informatics has computational, cognitive and social aspects, including study of the social impact of information technologies. Wikipedia.

32. Modern informatics enables a new scale-free** framework approach Use cases –requirements Stakeholders Distributed authority Access control Ontologies Maintaining Identity

33. Multi-tiered interoperability used by

34. Tetherless World Constellation tw.rpi.edu Themes Future Web Web Science Policy Social Xinformatics Data Science Semantic eScience Data Frameworks Semantic Foundations Knowledge Provenance Ontology Engineering Environments Inference, Trust Hendler Fox McGuinness Multiple depts/schools/programs ~ 35 (Post-doc, Staff, Grad, Ugrad)