Components of a Data Analysis System Scientific Drivers in the Design of an Analysis System

Data Import Format –Either widely used/accepted, or –Can be converted easily from something widely used –User need not know the details of the format –Well documented (e.g., which flavor of latitude). Fast Access –Disk I/O speeds do not follow Moore’s law –Read speed is more important than write speed –Caching –File size is only important to keep access times low Content must represent the details of the data E2E - Full intent of the observer must be embedded

Data Export Format –Either widely used/accepted, or –Can be converted easily into something widely used –User need not know the details of the format –Well documented (e.g., which flavor of latitude). You can read what you write –Import format == Export format Fast Access –Disk I/O speeds do not follow Moore’s law –Read speed is more important than write speed Content must represent the details of the data E2E - Full intent of the observer must be embedded. Includes user annotation/comments

Data Base System Ability to work with more than one data set Data base for both export and import files Large data volumes –Access using scan numbers is no longer sufficient –Require the ability to select subsets of data via sophisticated data-base queries –Moderate number of columns in data base index –‘Index’ to data kept in memory to speed data access –File summaries at various levels of detail Various levels of ‘granularity” Calibrated and raw data E2E - User can add annotation/comments Security – Only the observer can access data

Data Archive Write speed more important than read speed. File size is very important Cannot anticipate types of user queries –Large number of columns in data base index –Very sophisticated/fast RDBMS Storage need not be a widely used data format –Format can be very different from that used by analysis system. Export format should be a widely used data format

Interactive On-Line Data Analysis The ability to access data ASAP –Import file updates automatically as observations proceed (real-time “filler”). –Index to file updates automatically –Updates happen per ‘integration’ (spectral-line) or per N seconds (continuum) –Minimum integration time ~ few times the minimum time of real-time “filler” –Analysis system automatically is aware of updated index. –Read-protect online/filled data? User should be able to ‘see’ the data within an ‘integration’ of when it was taken (or N seconds).

User Interface Command line –Familiar syntax better than a good syntax –Procedural with byte-wise compiling (performance) –History, min-match or command completion –Useful error messages –Interruptible –Error trapping and exception handling –Ability to “Undo”

User Interface GUI’s best for: –Interacting with data visualizations –Filling in forms data base queries options for data pipelines –Browsing for data files –Defining E2E data flow (ala labview)

Imaging Tools Visualization –Shouldn’t try to recreate those things already available in another package – export instead. Data Flagging – Pick a system that works Graphics –Traditional capabilities (zoom in/out, scroll, print, save, …) –Data volume requires great performance, smart libraries (screen resolution << # data pts) –Interactive feedback (e.g., defining baseline regions). Publishable plots or export into something else? –Default plot style –Ability to tweak everything (label formats; char sizes; add, remove, move annotation; tick mark size; major/minor ticks, full box; grid; multiple X and Y axes, …..)

Analysis Algorithms Algorithms well documented Study what exists in other packages. Robustness very important but so is speed –Provide less robust but faster alternatives Developers should not force an algorithm on users Developers should provide ‘defaults’ only Building blocks better than a do-all algorithm. Ability to use and modify ‘header’ information as well as data. E2E – do-alls are built out of the same building blocks.

Documentation On-line and hardcopy –Tutorials/Quick Guides –Cookbook Based on observing types –Reference Manuals Full, gory details Data Formats Algorithms –Searchable by keywords Quick, interactive command help from within the system. Never release until these are in place

User Support/Feedback A familiar system minimizes staff support Easily accessed, on-line “help desk” and “Suggestion” box Automatic generation of “bug” reports Observers of observers

Marketing A familiar system already has a market Don’t be another cereal on the supermarket shelf Workshops are better than papers Create a User Community Responsive feedback from developers Independent Beta testers Reputation & first experiences are everything

User Community User Forums Newsletters Accept User Contributions/Additions –Sourceforge-like system –NRAO-seal-of-approval NRAO Moderator

Real-Time Data Display To guarantee data quality –Product is not stored (except for hardcopy) –Sequential processing -- different from E2E/Data pipeline –Fast is more important than accurate –Few bells and whistles -- must avoid the RTD black hole –A simple display for all observation types more important than sophisticated displays for a few data types Display happens within an ‘integration’ of when data were taken – tied to real time filler GUI based – underlying language is unimportant Output understandable by an operator

Real Time Data Analysis Pointing/Focus/Tipping/… are different from RTD –Results should be stored (Data Base) –Results are used by the control system (pointing/focus) or by subsequent analysis (tipping) –Accuracy is as important as speed –More bells, whistles, user-options –Sequential processing (non E2E/data pipeline) –Only a few observation types are handled Analysis happens within an ‘integration’ of when data were taken GUI based – underlying language is unimportant Output understandable by an operator

IDL Work Package SDFITS –Interim solution for data import/export –Class/IDL specific; soon Aips++/Aips/UniPOPS? –MD/BDFITS next generation (keywords, incompleteness of contents, versatility, …) IDL – Tom Bania –Uses UniPOPS as a ‘model’ – familiar to many –Very good reproduction –Bania-centric – needs to be generalized

IDL Work Package Glen Langston –Assess whether IDL will meet performance, extensibility, usability, … goals. –Generalization to other observing types. –Real-Time data access and display –Developed on top of and in parallel with Tom’s work (so, implementations have diverged) –Works well for Glen’s own experiments

IDL Work Package Institutionalize what Tom and Glen have done –Code management –Code review –Combine Tom and Glen’s branch –Generalize code –Provide ways for Tom and Glen to contribute within the same revision-control branch. Develop ‘Institutionalized’ code –Improve performance, usability, maintenance –Add/Replace I/O components with better CS methods.

Calibration Work Package User-tunable algorithms –Options for the ‘real-time filler’ – sequential –Options for E2E pipeline – non-sequential –Options for interactive data reduction Default algorithms for all observing cases Extensible as new algorithms are developed User-defined/tweaked algorithms Robust and not-so-robust algorithms

Calibration Work Package Opacity/atmosphere model Output units Efficiencies –Source size –Telescope model Tsys(f) estimates Differencing schemes Non-linearities/template fitting/….