Presentation on theme: "Don Murray Unidata Program Center"— Presentation transcript:
1 Don Murray Unidata Program Center The Abstract Data Distribution Environment (ADDE) – A technical overviewDon MurrayUnidata Program Center
2 Outline What is the ADDE? Who uses ADDE? Technical details Datasets and supported data typesProtocol detailsOverview of ADDE requests and returned dataJava and ADDEADDE use in the IDVSummary analysis
3 What is the ADDE?Client/server data access model developed for McIDAS, but not limited to serving McIDAS dataIn use for nearly 10 years4 primary types of data objectsGRID, IMAGE, POINT, TEXTPrimary servers handle McIDAS format files (AREA, GRID, MD, LW)Secondary servers read non-McIDAS formats (GINI, NEXRAD Level III, MODIS, netCDF,…)Allows browsing and subsetting of datasetsADDE was originally developed as part of the McIDAS display and analysis package. It is based on a client/server data access model and allows McIDAS users to access data on local and remote servers.There are 4 basic types of data that ADDE defines – grid (gridded data), image (image data), point (observational data) and text (text bulletins). The design of ADDE is such that you could serve up a grid as an image, or an image as a grid.While the core servers distributed with McIDAS understand McIDAS-formatted files, the API allows the development of subservers to handle non-McIDAS data formats. These subservers adapt the data to the appropriate ADDE data structure which is then passed on to the client.Another important feature of ADDE is its browse and subset capabilities. You can probe a dataset to determine what its temporal and spatial characteristics as well as what parameters are available. And, you can subset the data before returning it to the client so if you had a 1km global Vis image, and you were only interested in a view over Albuquerque, you can pull out just that piece you are interested in.
4 Who uses ADDE? Unidata community McIDAS community Cooperating set of servers providing data to McIDAS and IDV usersMeteoforumMcIDAS communityInternational government agencies (e.g., Spain, ESA, Australia)US agencies (e.g., NESDIS, NTSB)Satellite researchers
5 ADDE Dataset definition A dataset is collection of one or more files with a common format. Each dataset has a name that consists of a group and a descriptor (usually written as group/descriptor).Each dataset is in a group:A group can have one or more data types in it. Examples:RTIMAGES – real time image data onlyBLIZZARD – data (images, grids, point data) for a case study or tutorialEach group has descriptors which define a set of data of the same type (i.e., image, grid, point or text).Examples:RTIMAGES/GE-IR – set of GOES-East IR images in the RTIMAGES groupRTIMAGES/MOLL-IR – Mollweide Composite IR images in RTIMAGESBLIZZARD/GRIDS – set of NGM grids for the BLIZZARD data groupGroups can be interrogated for a list of descriptors and data types it contains.
6 Currently Supported Data Formats ImageMcIDAS AREA, GINI, NEXRAD Level III radar, netCDF (output only), MODIS, AIRS, GVAR, POES, Level 1B, Meteosat (including MSG), NOWrad®, GMS and FY-nGridMcIDAS GRID, netCDF, GRIB (in development)PointMcIDAS MD, netCDF, HDF4TextPlain text, McIDAS-XCD observation text (e.g., raw METAR, RAOB) and bulletins (e.g., watches and warnings)
7 Protocol DetailsClient connects on a particular port (500, 503 or 112). With next version, only Port 112 will be used. Supports compression using compress or gzip.Handshake from client involves sending:Version info (make sure server supports this)ADDE version (1 for now)Pre-request information (for validation)Server IP and compression typeRequest Type (AGET, ADIR, etc)Request BlockServer IP and compression type (again)Client addressUser, project, password (for authentication)Request Type (again)Actual request for dataServer sends back data or error message
8 Validation/SecurityADDE supports 3 types of validation for a request to a server:IP filteringUsernameProject numberSecurity through obscurity – no mechanism for querying an ADDE server to find out what datasets are available.TCP wrappers can be used to limit access to portsSubservers can implement their own forms of validation/security (e.g., NEXRAD Level III server)
9 Request types Request Type Java equivalent Data Type Description ADIR imagedataimageimage header informationAGETimagedirimage header, navigation, calibration and data; data is returned line by line; commentsGDIRgriddirectorygridgrid header informationGGETgriddatagrid header and entire gridLWPRdatasetinfoinfodataset informationMDFHnot implementedpointpoint file header informationMDHDpoint header informationMDKSpointdatapoint header and dataOBTGobtexttextobservational weather textTXTGASCII text fileWTXGwxtexttextual weather information
10 RTIMAGES MOLL-IR 0 BAND=ALL X TRACE=0 AUX=YES VERSION=1 Anatomy of a RequestAn ADDE request is a text string containing positional parameters and some key=value pairs (just like a McIDAS command).A sample request for the latest Mollweide IR image:RTIMAGES MOLL-IR BAND=ALL X TRACE=0 AUX=YES VERSION=1Dataset DescriptorKey=value1 value2 … valuen(in this case, number of bands to request from image (ALL))Server debug flagADDE VersionDataset Group
11 Image data ADDE image data model supports multi-banded images 2 main types of requestsDirectory (ADIR) and data (AGET)Returned image object models McIDAS AREA format:Directory blockNavigation blockCalibration blockSupplemental (AUX) blockData blockComment blockImage data is quantitatively useless unless it is transformed into physical units (calibration) and oriented relative to time and physical space (navigation). In addition, it is often necessary to know when and how an image was collected and processed. The actual image, along with these ancillary data , is collectively called an image object . Each image object in McIDAS-X is composed of the following blocks of information:The directory block contains a list of ancillary information about the image, such as the number of lines and data points, the satellite ID, and the number of spectral bands.The data block contains the matrix of image data values.The line prefix block contains information about an image that may vary on a line-by-line basis, such as calibration or documentation information.The navigation block contains information for determining the location of data points in physical space. More information about navigation is presented in the section titled McIDAS navigation later in this chapter.The calibration block contains the information for converting image data from its internal (stored) units to more meaningful physical units, such as radiance or albedo. More information about calibration is presented in the section titled McIDAS calibration later in this chapter.The supplemental (or auxiliary) block contains additional information that is specific to a data type. For example, information specific to radar data is stored in this block. Also, the latitude/longitude grid for the LALO navigation is stored in this block.The comment block contains a variety of textual information, such as a list of commands run on the image object to-date.
12 Image Object Details Image Directory Data Block Navigation block contains a list of ancillary information about the image, such as the day and time, number of lines and data points, the satellite ID, and the number of spectral bands.Data Blockcontains the matrix of image data values. Multibanded image have values interleaved:Navigation blockcontains information for determining the location of data points in physical space. Client must have navigation module which uses this block to convert (line,element)<->(lat,lon) for geolocationCalibration blockcontains the information for converting image data from its internal (stored) units to more meaningful physical units, such as radiance or albedoAUX blockcontains additional information that is specific to a data type. For example, information specific to radar data is stored in this block. Also, the latitude/longitude grid for the LALO navigation is stored in this block.
13 AREA=FILE coordinates, ignore TV coordinates for now Image CoordinatesAREA=FILE coordinates, ignore TV coordinates for now
14 Image Data (con’t)An image directory (ADIR) request returns all images matching the requestAn image data (AGET) request returns only one image at a time.Request refinementsLocation can be specified by image, file or lat/lon coordinates.Single or multi-banded imagesDay and timeCalibration unitsSize (number of lines/elements)Resolution (line/element magnification)Relative position number (i.e. last N images)
15 Grid DataGrid data is two-dimensional data representing a parameter along an regularly spaced matrix (e.g., model output, objective analysis).2 main types of requestsDirectory (GDIR) and data (GGET)Grid object consists of:Grid HeaderData blockGridded data is two-dimensional data representing an atmospheric or oceanic parameter along an evenly spaced matrix. For the matrix to be useful, ancillary information about the grid must also be known. This ancillary information, along with the gridded data, is collectively called a grid object . Grid objects in McIDAS-X contain two blocks of information.The grid header contains a list of ancillary information about the grid, such as the parameters and units of the data in the grid, the level in the atmosphere or ocean the data represents, the grid navigation information, and the time.The data block contains the matrix of gridded data values.
16 Grid Object Details Grid Directory Data block contains a list of ancillary information about the grid, such as the parameters and units of the data in the grid, the level in the atmosphere or ocean the data represents, the grid navigation information, and the time.Client must have navigation module which uses the navigation info to convert (row, column)<->(lat,lon) for geolocationData blockcontains the matrix of gridded data values.
17 Grid Data (con’t)A grid directory (GDIR) request returns stream of grid directories matching requestA grid data (GGET) request returns stream of grid data objects matching requestRequest refinements:Parameters or Derived quantitiesLevelsModel run and/or valid day/timeOriginating centerMaximum number of grids to return
18 Point DataPoint data typically represents data occurring at irregularly spaced locations around the Earth (ex. Surface observations, upper air reports, lightning flashes)Main type of request is for data (MDKS)Point data object consists of 5 blocks:Parameter blockUnit blockScale blockForm blockData blockMcIDAS-X point data typically represents data occurring at irregularly spaced locations on the Earth. For this data to be useful, ancillary information about the data must also be known. This ancillary information, combined with the actual point data values, is collectively called the point object .Each point object in McIDAS-X contains five blocks of information.The parameter block contains a list of the parameter names in the point object returned by the server.The unit block contains a list of units for the parameters returned by the server.The scale block contains a list of scaling factors for the floating- point values returned by the server.The form block contains a list of the return forms for each of the parameters.The data block contains the actual data values returned by the server.
19 Point data object details The parameter block contains a list of the parameter names in the point object returned by the server.The unit block contains a list of units for the parameters returned by the server.The scale block contains a list of scaling factors for the floating- point values returned by the server.The form block contains a list of the return forms for each of the parameters.The data block contains the actual data values returned by the server.
20 MD File structureMeteorological Data (MD) file schema determines data layoutAn MD file is like a spreadsheet with each cell containing a predefined number of data values. Each cell contains data for a specific location at a given instant in time.
21 Point data (con’t) Request refinements Examples: List of parameters Maximum number of obs to return (default 1)The SELECT clause gives the user the ability to subset on any parameter in the dataset.Examples:SELECT='T[F] 40 50; ST WI, MI; TIME 12 13’ MAX=ALLSelects all parameters for all observations between 40 and 50 degrees Fahrenheit for stations in Wisconsin and Michigan between 12 and 13 UTCSELECT=‘ID KDEN’ PARM=T TD PRE MAX=ALLSelects all temperature, dewpoint and pressure values for all times in the dataset for Denver
22 Text Data There are 3 types of text data that are served up by ADDE: Flat files, ancillary data filesWeather bulletins such as forecasts, warnings, watchesobservational data, such as METAR or RAOB reports3 types of requests:TXTG ASCII text fileWTXG textual weather informationOBTG observational weather textReturned data has a header and the text
23 Java and ADDEClient interface developed and refined collaboratively by Unidata, SSEC and Australian Bureau of Meteorology (BoM)Provides ADDE data access for Java-based data analysis and display tools (e.g., IDV, Matlab)Uses specialized URLadde://server/request?keyword_1=value_1&keyword_2=value_2…keyword_n=value_n“adde” specifies protocol“request” specifies data type/actionkeyword/value pairs refine requestBundled with VisAD component libraryPackage edu.wisc.ssec.mcidas.adde contains core packagePackage visad.data.mcidas has classes for converting ADDE data objects into VisAD objects.The interface we’ve developed allows Java applications to access data from existing (and future) ADDE servers. Each of the groups has invested a lot of resources in setting up ADDE servers. But as we develop new tools, we still want access to the same datasets that our old tools use.We accomplished this by creating a specialized Java URLConnection class that defines the ADDE request.In this URL, adde defines the protocolrequest specifies the particular data type/action. This might define whether the user wants to browse and image data set and return some metadata, or return a particular portion of the dataset.Keyword/value pairs are used to refine the request. These might specify the geographic extent and size of the returned data, or the particular band of an image.
24 Java ADDE Use in Applications UnidataIntegrated Data Viewer (IDV)Access to satellite, Level III radar, METAR, synoptic, upper air and profiler dataSSECMcIDAS AREA to netCDF converter for AWIPS useJava client for browsing and copying real-time and archive imageryMODIS data explorationBoMDevelopment of subservers (Oracle/NEONS, radar)Java-based clientsAustralian Marine Forecast SystemTropical Cyclone Forecast
25 ADDE Use in IDV The IDV uses ADDE to access: Satellite and Level III radar imagerySurface (METAR, synop) dataUpper Air (RAOB) dataProfiler dataText dataADDE data objects are converted to VisAD data objectsNavigation of images done through AreaCoordinateSystem
27 Summary Analysis Strengths Weaknesses Supports many data formats, especially for imagerySubsetting capabilitiesDatasets can be queried for metadataServer freely available to research and education institutions through UnidataJava/C/FORTRAN client APIsJava client freely availablePoint data limitationsMD file limitations (4 character names/unit names, 400 parameter limit, scaled integers)Grid data limitations2D grids only, 4 character param/unit namesServer configuration (best done from McIDAS)Currently not separate from McIDASLimitationsWhen accessing point data, the MD (Meteorological Data) file structure has four limitations, which are explained below. These limitations pertain only to the MD file structure and are not limitations of the ADDE point object subsystem. Point servers for other formats, netCDF for example, use configuration files to map the parameter names in the file to McIDAS-compatible names.CellsEach cell is limited to 400 elements. For most observational data, this isn't problematic. The exception occurs with observations containing data at several levels of the atmosphere. For example, you can't store all the information for an upper air observation reporting values for eight different parameters at 50 levels of the atmosphere in one cell. Although the cell can accommodate the 400 values (8 x 50), it won't have enough space for the time and geographic location of the observation, which are also provided.Element names and unitsElement names and units are limited to four characters, which can be restricting when designating parameter units, especially derived parameters.Character string elementsCharacter string elements are limited to four characters, which can be limiting for any type of alphanumeric parameter, such as station ID or country. You can bypass this restriction by using several parameters strung together to represent strings. McIDAS-XCD uses this method when representing five-character IDs associated with ship reports.Numeric valuesNumeric values can be stored only as scaled integers.
28 Future Development Ideas Middleware to support metadata queries for available times and parameters (not just catalogs)Enhanced data choosers which take into account the semantics of the datasetsSupport more navigation modules in JavaEnhance netCDF point serverServe up GEMPAK gridsJava server