IRI/LDEO Climate Data Library M.Benno Blumenthal, Michael Bell, John del Corral, Remi Cousin, and Haibo Liu International Research Institute for Climate and Society Columbia University http://iridl.ldeo.columbia.edu/

Overview multidimensional Specialized Data Tools Maproom Generalized Data Tools Data Viewer Data Language Dataset Variable ivar multidimensional So here is the IRI/LDEO Data Library shot at this, connecting the space of data and data manipulations. At the bottom we have the compute engine/data organization, which is what maps the data/manipulation space into URLs, i.e. the WWW. Built on top of that are some general data tools, i.e. they can be applied to any dataset and adapt accordingly. There is a data language, making it possible to specify sophisticated analyses. And there is a data viewer, making it possible to quickly graph data in a number of standard ways. And there are also more specialized tools, designed for particular audiences to view specific things. We have a Maproom (soon to be or already map rooms) which contains continuously updated views of aspects of the climate system, as well as specialized tools aimed at particular audiences that let a use extract views/data with a few clicks. There is a tradeoff here: the general tools are great, but require a user to navigate a vast set of datasets and a vast set of possible manipulations, which not everybody is up to. The specialized tools are a way to make sophisticated calculations easy to access. IRI Data Collection URL/URI for data, calculations, figs, etc

Data Flow based Analysis Results analysis data data The Data Library compute engine generates results by breaking down a needed calculation into a two-part data-flow, partial-execution network. Data variables are represented as buffers which contain multiple realizations, the set of all realizations spans the data variable. The buffers are connected by analysis filters, other filters also generate data files in various formats, sets of images, navigational pages, etc. The upper network processes the metadata, the lower network the numeric data. While the lower network is setup in parallel with the upper network, the realizations are not processed unless a message is received from the requested result that that particular realization is needed. Calculations are passed along the pipeline until they contribute to the requested result. Consequently results start transmitting as soon as the initial realizations are processed, quite possibly well before the later realizations have been accessed. This means that most of the processing time can be hidden behind the transmission time of the final result, depending on how streamable that final result might be. One can even include the reverse linkages, so that the final result is connected to the earlier results, enhancing the documentation of the results way beyond what is seen in a paper or presentation. This has the potential to greatly improve the reproducibility of the analysis, since anyone that the is really interested can ultimately see all the steps. Also, the analysis can easily be rerun, if, for example, the data is extended in time or otherwise modified. Data Data analysis

“geolocated by lat/lon” multidimensional “geolocation by IRI Data Collection Economics Public Health “geolocated by entity” Ocean/Atm “geolocated by lat/lon” multidimensional GIS “geolocation by vector object or projection metadata” spectral harmonics equal-area grids GRIB grid codes climate divisions IRI Data Collection Dataset Variable ivar multidimensional Data Cultures “Broadly Speaking” We started with Oceans/Atm – multidimensional geolocated by lat/lon – with exceptions that tend to get handled in non-standard ways. GRIB in some ways is the 600 lb gorilla, since it is very similar in style to the WCS standard in that metadata carries the geolocation, but, of course, it is a difficult if-not- impossible standard to completely code for. Economics/Public Health geolocation by entity, mostly tables GIS geolocation by vector object or by projection metadata -- mostly a 2D mindset in the tools, which makes time analysis of data difficult. IRI Data Collection – nested datasets, multidimensional variables with independent variables All variables have attributes which can affect the way the data is processed and/or displayed We use dimensions for a lot –lon,lat,height,time forecast time, lead time, eigenvalue number, member number, country, district, category Dataflow, delayed execution architecture, which means one can usefully define many dimensions even when one tends to only evaluate a few realizations at a time.

GRIB netCDF images binary Database Tables queries OpenDAP IRI Data Collection GRIB netCDF images binary spreadsheets shapefiles Database Tables queries Servers OpenDAP THREDDS images w/proj IRI Data Collection Dataset Variable ivar But, of course, those abstract data types are actual data files and data services in various formats and using various protocols.

descriptive and navigational pages IRI Data Collection GRIB netCDF images binary spreadsheets shapefiles Database Tables queries Servers OpenDAP THREDDS images w/proj IRI Data Collection Dataset Variable ivar Calculations “virtual variables” images graphics descriptive and navigational pages Having got all that data into the data library, we can process it in a uniform way. The data structure leads directly to calculations and “virtual variables”, i.e. many of the Data Library entries are actually calculations done on other entities, e.g. PressureLevel data zonal velocities computed from hybrid level divergence and vorticity, or sea surface temperature anomaly computed from sea surface temperature and sea surface temperature climatology User Interface – the data collection structure and metadata is used to generate a web interface to the data – provides navigation through the datasets, a viewer that slices and dices, many manipulations and calculations. We also generate output Data Files in many different formats, as well as tables of various kinds, many of which are useful to one kind of user or another, i.e. different data cultures have different preferred formats. Atm/Ocean like netcdf and straight binary, GIS prefers sets of images, Public Health prefers tables. We also act as a data server using OpenDAP and THREDDS, again mostly useful for Ocean/Atm We have perhaps implemented OpenGIS Web Map Server v1.3 – a bit of a mistake, since v1.3 is the next version rather than the currently widely used one. Time will correct this, with any luck. It is important to note that everything following from the structure and attributes in the IRI Data Collection, no additional configuration is done to control the conversions to different fromats or to serve the data with different protocols. Not all data can be served in all formats. OpenDAP/THREDDS is particularly important because it can express any dataset and/or any analysis, so that I can transfer calculations between servers. At least, it will once I code transmission of SimpleFeatures with OpenDAP. Clients OpenDAP THREDDS Data Files netcdf binary images Tables OpenGIS WMS/WCS

Data Page IRI General Data Tools While that is all there, that is not what the user sees (bad advertising for us, b ut life is easier for the user). This is a page representing a dataset, in this case weekly sst, ssta and error fields. It has links some analyses (as well as the full language interface – expert mode). It has some basic informatnion – note the last_modified and expiries times – as well as links to pages of virtual files in various commonly used formats (with coachs for commonly used programs).

Data Viewer IRI General Data Tools Here is the Data .viewer. As I said earlier, it can be applied to any dataset or variable, and what you see v aries according to the structure and metadata of the dataset. In this case, it is applied to a dataset, so one of the controls is which variable to look at. One can enter alternative dates (or a range of dates for an animation) change which dimensions are plotted/selected, many of the standard controls one might want in a data viewer. Click/drag to zoom got added a year ago, turned out to be more important that I thought, shame on me. Anyway, clearly it is simply creating the appropriate url for the graphic and letting the underlying compute engine generate the figure from the data. One feature that makes it very much a web interface, is the cut-and-paste link option on the bottom. It takes you to….

Cut and Paste IRI General Data Tools Which gives an image linked back to the viewer. Simple click-and-drag will add this to any document, creating a documented figure, i.e. the figure is linked to the viewer, which is linked to the analysis which is linked to the data …. A primative version of what we envisioned, to be sure, but the essential pieces are there. Now give this tool to a fairly persistent scientist or two, add some automatic menu generation, and you have maproom

IRI Map Room The Map Room is a specialized tool built on top of the more general Data Library tools. It uses particular sets of views of particular datasets to address the needs of particular sets of users.

Malaria Early Warning System IRI Map Room Malaria Early Warning System • Front page illustrates most recent dekadal rainfall estimates (FEWS RFE) • Administrative and epidemiological overlays available The Malaria Early Warning System is a more sophisticated example of the tools available in the IRI maproom. Again, this is a tool that uses preselected data to provide for a particular audience. The underlying data is updated every ten days, which means this interface updates accordingly. • Change dates to view different time periods • Click and drag box across map to zoom

MEWS Time Series Analyses IRI Map Room MEWS Time Series Analyses STEP 1: Select size of domain for analysis Administrative District OR Box – 11km, 33km, 55km, 111km STEP 2: Select location for which analysis will be created

MEWS Time Series Analyses IRI Map Room MEWS Time Series Analyses And time series for the particular district are present. Note that this is a lot more sophisticated than is apparent at first glance. Each time series is based on an on-the-fly analysis of the set of images presented in the map, averaged over the shape of the district chosen. So we have used data that depends on latitude, longitude, time and a GIS polygon representation of a particular district to generate a time series that is geolocated by spatial entity, tying together all three of the data realms mentioned in the earlier slide.

Climatic Suitability Interface IRI Map Room Climatic Suitability Interface • Front page illustrates number of months during the year that are suitable based on “normal” conditions • Same visual options as MEWS

Climatic Suitability: A Historical View IRI Map Room

Connection Between Resources IRI Map Room Links on the side menu allow users to interact between products

Connection Between Resources IRI Map Room Connection Between Resources

Going into the Future Continuing data efforts Expanding map rooms directed at particular user groups Semantic Web metadata/navigation/provenance http://iridl.ldeo.columbia.edu/ontologies/

Data Flow based Analysis with explicit semantics Results analysis data data Semantic Web One can even include the reverse linkages, so that the final result is connected to the earlier results, enchancing the documentation of the results way beyond what is seen in a paper or presentation. This has the potential to greatly improve the reproducibility of the analysis, since anyone that the is really interested can ultimately see all the steps. Also, the analysis can easily be rerun, if, for example, the data is extended in time or otherwise modified. Data Data analysis

Faceted Search http://iridl.ldeo.columbia.edu/ontologies/query2.pl?... Again as an example, here is a user interface built on that ontological structure

Metadata Browser http://iridl.ldeo.columbia.edu/ontologies/browse.pl Browses through the metadata the search interface is based on. Includes several alternative term-sets for the search interface, and a number of ontologies collected from the Internet.