Presentation on theme: "Brian Doty and Jennifer Adams"— Presentation transcript:
1Brian Doty and Jennifer Adams An Update on GrADSBrian and I are the only two members of the GrADS Team. Together we manage code design, implementation and testing, documentation, distribution, and user support, including the moderation of a very active forum.Brian Doty and Jennifer Adams
2GrADS CapabilitiesGrADS is an interactive tool for the analysis and visualization of geoscience dataOpen source and freely availableGrADS has 2 data models5-D gridded dataIn situ (station dataGrADS can read all the standard data formatsBinaryBUFRGRIB (versions 1 and 2)HDF (versions 4 and 5)NetCDF (versions 3 and 4)Missing data handled consistently throughoutExpression syntax is flexible, compact, recursiveProgrammable interface for scriptingFast I/O and graphicsHere is a quick overview of GrADS and its basic featuresFor each release, we provide source code and pre-compiled binariesExpression syntax allows for some very nimble data manipulationFully-functional in batch modeMost important, GrADS is FAST !!
320+ Years of GrADS Begin version 2 development Name chosen; First releaseRapid community adoptionBAMS CD-ROM distributionGDS; OPeNDAP integrationScripting language addedUser forum startedAMS Special AwardFirst code writtenVersion 2 alphaSurvived Y2K2.0.0 releasedGrADS is unusual in that it has been around for a very long time and it still a vibrant, relevant, useful piece of software. I joined the team in 2000, which is now just about half of its lifetime. Brian and I have developed a great working relationship as a collaborative team, and we’ve been able to get a lot of work done, especially in the past few years.19901995200020052010
4GrADS is Used Worldwide We know GrADS has an broad international user base, but we didn’t have any specific data to prove it. We looked back as far as our current records will allow and counted the number of downloads and mapped their IP addresses to a geographical location.The map shows locations of IP addresses that have downloaded GrADS from COLA over a 20 month period. We estimate a total of 77,500 downloads during that time.77,500 downloadsFebruary Present
5GrADS @ COLA How does COLA benefit from GrADS? Format neutralization facilitates COLA’s ”model agnostic” strategyCollaboration expedited by script-sharingEasy, format-independent metadata harvesting for COLA’s data management strategic planIn-house expertise for immediate help and code changesnew and complex data setsvery high resolution dataoptimization of scriptsbeautification of published figuresHow does GrADS benefit from COLA? Feedback from scientists (deliberate and accidental)Development needs are readily apparentPre-release testing of new featuresWe’ve established that GrADS is important to the community. Why is GrADS so important to COLA?GrADS and COLA have a symbiotic relationship.One of the most important ways COLA benefits from GrADS stems from its handling of all data formats, making their differences essentially invisible to the user.It facilitates COLA’s “model agnostic” philosophy because no single format is favored.<show quick example in next slide>
6Easy Multi-Model Intercomparison Error in July Mean Total Precipitation (in meters) ECMWF IFS - TRMMJapan NICAM - TRMMNOAA CFSv2 - TRMMNCAR CCSM4 - TRMMThis plot shows a comparison of the July mean precipitation from 4 different climate model runs compared to observations.Each of the 5 data sets-- is in a different data format (GRIB1, NetCDF4, GRIB2, Binary, and HDF4) and-- is on a different grid (8, 16, 25, 110, 130 km) and-- has different units.It takes just a few lines of code to normalize the units, do some averaging, interpolate to a common grid, and take the differences.
7GrADS @ COLA How does COLA benefit from GrADS? Format neutralization facilitates COLA’s ”model agnostic” strategyCollaboration expedited by script-sharingEasy, format-independent metadata harvesting for COLA’s data management strategic planIn-house expertise for immediate help and code changesnew and complex data setsvery high resolution dataoptimization of scriptsbeautification of published figuresHow does GrADS benefit from COLA? Feedback from scientists (deliberate and accidental)Development needs are readily apparentPre-release testing of new featuresCollaboration – everyone using same package makes it easy to exchange scripts, to communicate technically using a common language. COLA scientists are not required to use GrADS, but most people do anyway. Even if they arrive with expertise in other tools, they often come around to using GrADS.COLA has 360,000 GrADS descriptor files online – we harvest those descriptors to create a searchable indexed catalog of all our data holdings.Open source, in-house expertise for modifying code
8GrADS 2.0.0 Highlights Major changes to the core of GrADS: Added 5th data dimension (targeted to ensembles)Internal data handling in double precisionMore consistent missing data handlingVery stable: bugs fixed, memory leaks patchedNew data formatsGRIB2HDF5NetCDF4Early adopter of compression featuresPublic release of version (no more “alpha” or “beta” status) is around the corner."Very stable" is easy to say but takes a lot of workGrADS was an early adopter of the compression features in NetCDF4, worked closely with library developers as interface evolved.Really new features are in the GIS interface – the result of a collaboration with CPC under CTB program – intended to bridge the gap between climate data and GIS Tools.GIS InterfaceRead shapefilesOutput shapefiles, GeoTIFF, and KML
9The New Shaded Contouring Algorithm Here is a shaded contour plot with the component polygons outlined in blackNew algorithm had to meet a long list of criteria:(Shapefile Spec): polygons do not overlap, have no holes, and are filled on the right-hand-side of the path (Compatibility): polygons match line contours and legacy shaded contour algorithm(Performance): -- polygons are as large and small in number as is feasible-- the number of floating point operations is minimized (integer flags are used as much as possible) -- the new shaded contouring algorithm is a lot faster than the legacy shaded contouring
10Station Data Shapefile from GrADS in ArcMap (Dressed up with Topography, Political Boundaries, and City Names)An example of GrADS station data exported into ESRI’s ArcMap, an industry standard GIS tool. The topography and political boundary data are layers overlaid by ArcMap. Any GrADS expression (gridded or station) can be exported into ArcMap as points, lines or polygons.
11OLR and Precip Data from GrADS in Google Earth GrADS gridded fields (OLR and precip) exported into Google Earth, another widely used GIS tool.05Z 23May2009
12Climate Prediction Center’s GIS Portal An interactive, web-based system to display CPC products together with supplemental geographical dataThis is a screen shot from the Climate Prediction Center’s GIS Portal. All the data behind it were created by GrADS. Our project was one of the few that actually succeeded in bringing research to operations.
13What’s Next New options for rendering the graphics Use of Cairo library for X11, image, and hardcopy drawingOld methods still availableDesign of new data analysis capabilitiesEOFs, Sorting, etc.Internal/External operations on defined objects (defop)Work is underway to use the open-source Cairo library to significantly improve the quality of the graphicsWe are also working on the design of some new data analysis capabilities.EOFs and sorting are two specific examples that the users have been asking for, but we are thinking about a more general implementation, where users create a defined object in memory, and then perform operations on that object, using internal or external functions.
14Here’s an example of what the improved graphics will look like. -- Anti-aliasing for smooth-looking lines-- Basic font support for easy-to-ready contour labels, axis labels and figure annotationFor the web, you need low res images that load fast and clearly convey their content
15GrADS in the Cloud Using GrADS in the new cloud computing paradigm COLA has considerable expertise and experience in remote data analysisPioneered server-side analysis with GDSCurrently managing off-site computing at 3 remote centersWill mobile cloud-based computing change how we do our science work?We’ve seen the writing on the wall regarding the future of cloud-based computing services.We already have considerable expertise and experience in handling remote data,we pioneered server-side analysiswe are managing off-site computing and analysis at 3 centersBut we’re not sure how it’s going to affect our workflow in the coming years.Web-based interfaces to GrADS and underlying data may become essential.Brian has done a lot of thinking about this, and has developed a prototype of GrADS running as a ‘cloud service’.He’ll be giving a demo using an iPad during the poster session this afternoon.Live Poster SessionExperimental prototype of GrADS running as a "cloud" service