Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 A Data Management System (DMS) for Air Quality Management Agencies Prepared by: Mark Stoelting Bay Area Air Quality Management District San Francisco,

Similar presentations


Presentation on theme: "1 A Data Management System (DMS) for Air Quality Management Agencies Prepared by: Mark Stoelting Bay Area Air Quality Management District San Francisco,"— Presentation transcript:

1 1 A Data Management System (DMS) for Air Quality Management Agencies Prepared by: Mark Stoelting Bay Area Air Quality Management District San Francisco, CA and Eric Gray Sonoma Technology, Inc. Petaluma, CA Presented at: U.S. EPA 2008 National Air Quality Conference Portland, OR April 6-9, 2008

2 2 DMS: Why, What, How, Who Why build a DMS?Why build a DMS? What is it?What is it? How does DMS work?How does DMS work? What are the DMS capabilities?What are the DMS capabilities? Who is involved?Who is involved? Who paid for DMS?Who paid for DMS? Where is it going?Where is it going?

3 3 Why the DMS? New technology: instruments, communications, data accessNew technology: instruments, communications, data access Real-time data accessReal-time data access Improve data quality (QC checks)Improve data quality (QC checks) Graphical data reviewing toolsGraphical data reviewing tools Minimize/eliminate need for paperMinimize/eliminate need for paper Track instruments and performanceTrack instruments and performance Shared need by all monitoring agenciesShared need by all monitoring agencies

4 4 What is DMS? Data Management SystemData Management System Database to ingest, process, export dataDatabase to ingest, process, export data ArchiveArchive Programmable database with optional featuresProgrammable database with optional features Process sub-hourly data (1 minute)Process sub-hourly data (1 minute) Collect instrument metadataCollect instrument metadata Track instruments and performance (OpCodes)Track instruments and performance (OpCodes) Automatic Quality ChecksAutomatic Quality Checks Export (AirNow, AQS)Export (AirNow, AQS) Notification ( , process controls)Notification ( , process controls)

5 5 Simple Data Structure DATA SITE PARAMETER INSTRUMENT {OpCode} Site,Parameter,Date_Time,OpCode,Data,[alpha_data] 7011,CO_chamt,10/05/ :16,0, , (optional, assigned in database) DATE _TIME

6 6 Sites, Parameters, Instruments Establish Monitoring Sites Create Parameters Create Instruments

7 7 Assign Parameters to Instruments and Instruments to Sites Assign Parameters to Instruments Assign Instruments to Sites

8 8 Automated Quality Checks/Export Raw Data Ambient Air Quality Data Range checks Instrument checks Site checks Network (off-site) checks AIRNow District Web Page Exports filtered by QC code

9 9 Automatic Quality Checks Assign Automatic Quality Control Checks to Parameters

10 10 Data Input Process Ingests data files generated by datalogger or data formatting softwareIngests data files generated by datalogger or data formatting software Duplicate input data is automatically rejectedDuplicate input data is automatically rejected Automated Quality Checks establish initial QC codeAutomated Quality Checks establish initial QC code Min/max, zero drift, sticking, rate-of-change, compareMin/max, zero drift, sticking, rate-of-change, compare QC conditions can activate process/alarmQC conditions can activate process/alarm Electronic chain-of-custody createdElectronic chain-of-custody created Sub-hourly data aggregated to 1-hr averagesSub-hourly data aggregated to 1-hr averages Automated data export to AIRNow and Air District web siteAutomated data export to AIRNow and Air District web site

11 11 DMS Status Station Reporting Status Event Log QC Log

12 12 Manual Data Review Time-series graphical editing and QC tools for staff data reviewTime-series graphical editing and QC tools for staff data review DMS records all reviews and manual adjustments to QC status and data values (electronic chain-of-custody)DMS records all reviews and manual adjustments to QC status and data values (electronic chain-of-custody) Tracks manual data reviewer statusTracks manual data reviewer status Lock final data prior to exportLock final data prior to export Exports final data for regulatory submission to AQS or other agenciesExports final data for regulatory submission to AQS or other agencies

13 13 Time Series Edit/QC Screen

14 14 Scatter Plot Analysis

15 15 Other DMS Features Data access security (user rights management)Data access security (user rights management) Maintains instrument equipment databaseMaintains instrument equipment database Supports electronic instrument and site logsSupports electronic instrument and site logs Daily instrument performance reports (using OpCodes)Daily instrument performance reports (using OpCodes)

16 16 DMS Software Requirements DMS host computer:DMS host computer: Windows Server 2000/2003 or XP/VistaWindows Server 2000/2003 or XP/Vista Microsoft SQL Server 2005 (all editions)Microsoft SQL Server 2005 (all editions) DMS Client Software:DMS Client Software:.NET 2.0 Framework.NET 2.0 Framework Windows 2000, XP, VistaWindows 2000, XP, Vista Future tools may require Java and ERSI arcGISFuture tools may require Java and ERSI arcGIS

17 17 DMS Hardware DMS Server: Minimum server requirements – Pentium 4 laptopMinimum server requirements – Pentium 4 laptop Must support Microsoft WindowsMust support Microsoft Windows Hardware determined by processing needsHardware determined by processing needs Laptop processes hundreds of thousands of records per hour and store millionsLaptop processes hundreds of thousands of records per hour and store millions Server can process millions of records per hour and store billionsServer can process millions of records per hour and store billions DMS Client: Pentium III class PC or betterPentium III class PC or better Broadband connection to DMS ServerBroadband connection to DMS Server

18 18 DMS Performance Benchmarks AIRNow DMC DMS will be running in August on a Quad Core Intel processor with 8 GB of RAM. The operating system will be Windows Server bit EditionAIRNow DMC DMS will be running in August on a Quad Core Intel processor with 8 GB of RAM. The operating system will be Windows Server bit Edition Currently, AIRNow DMC processes 12 million records a day for over 5000 sitesCurrently, AIRNow DMC processes 12 million records a day for over 5000 sites The DMS system can process 22 million records per hour or ½ billion a day (historical data import)The DMS system can process 22 million records per hour or ½ billion a day (historical data import) The Bay Area DMS system will process 1 million data records per dayThe Bay Area DMS system will process 1 million data records per day

19 19 DMS Storage Benchmarks The database will grow by 100 MB for every one million recordsThe database will grow by 100 MB for every one million records One billion records can be stored in 100 GB dataOne billion records can be stored in 100 GB data DMS query performance has been tested with ½ billion data recordsDMS query performance has been tested with ½ billion data records AIRNow DMC will test performance for over a billion records by fallAIRNow DMC will test performance for over a billion records by fall

20 20 DMS Can Be … An instrument monitoring systemAn instrument monitoring system A data analysis systemA data analysis system A data Quality Check and Review systemA data Quality Check and Review system A data export tool (to AirNow, AQS)A data export tool (to AirNow, AQS) A data warehouse for historical dataA data warehouse for historical data

21 21 Agency Deployment DMS developed using AIRNow technology in 2004DMS developed using AIRNow technology in 2004 DMS to replace AIRNow technology by fall 2008DMS to replace AIRNow technology by fall 2008 DMS in use by BAAQMD, SCAQMD, and CARB by end of 2008DMS in use by BAAQMD, SCAQMD, and CARB by end of 2008 AIRNow International will be based on DMS for 2009AIRNow International will be based on DMS for 2009 Stable DMS distribution package available for agency use by early 2009Stable DMS distribution package available for agency use by early 2009 Looking for 3 or 4 additional agency development partners in 2008Looking for 3 or 4 additional agency development partners in 2008

22 22 Development and Funding DMS was development and funded by agencies with an interest in the resultDMS was development and funded by agencies with an interest in the result Contributors include:Contributors include: Bay Area AQMD (BAAQMD)Bay Area AQMD (BAAQMD) EPA Region IX, 103 GrantEPA Region IX, 103 Grant EPA National Environmental Information Exchange Network (NEIEN) Challenge GrantEPA National Environmental Information Exchange Network (NEIEN) Challenge Grant South Coast AQMD (SCAQMD)South Coast AQMD (SCAQMD) EPA (AirNow and AirNow International)EPA (AirNow and AirNow International) California Air Resources Board (CARB) soonCalifornia Air Resources Board (CARB) soon New DMS features must be available to all agenciesNew DMS features must be available to all agencies New features must not reduce DMS functionality to other agenciesNew features must not reduce DMS functionality to other agencies

23 23 Future DMS Development Enhanced view/edit featuresEnhanced view/edit features Viewing tools - fingerprint plotsViewing tools - fingerprint plots Statistical tools - box plots, reportsStatistical tools - box plots, reports Support for toxics, PM filter data, Special Purpose MonitoringSupport for toxics, PM filter data, Special Purpose Monitoring Expanded reporting capabilitiesExpanded reporting capabilities Instrument Control ChartsInstrument Control Charts GIS mapping capability, web services, and NEIEN XML data exchange nodeGIS mapping capability, web services, and NEIEN XML data exchange node

24 24 DMS Summary Simple, scalable, cost-effectiveSimple, scalable, cost-effective Fast, secure, and efficient operationFast, secure, and efficient operation Comprehensive quality control toolsComprehensive quality control tools Applicable to a wide variety of dataApplicable to a wide variety of data Free to public agencies under a share- ware use agreementFree to public agencies under a share- ware use agreement

25 25 Contact Information Mark Stoelting Bay Area Air Quality Management District (415) Eric Gray Sonoma Technology, Inc. (707)


Download ppt "1 A Data Management System (DMS) for Air Quality Management Agencies Prepared by: Mark Stoelting Bay Area Air Quality Management District San Francisco,"

Similar presentations


Ads by Google