Batch Process Overview

Batch Processes 2 What Do You Mean By Batch Processing? Allows for Mass Entry of Data All Processing of the Data Happens Without Your Interaction The System Will Tell You When the Job Has Completed and Provide You With Reports at the End of Each Process

Batch Processes 3 What Can Be Input by Batch? CAN Batch LoadCANNOT Batch Load Site information (all)Comments (free format text describing Special Information for Sites, Monitors, Audits, and Raw Data Values) Monitor information (all)Technical systems audits Raw dataExc. Event definitions Precision & bias dataEPA Concurrence (Regional user only) Blanks dataPrimary Monitor Period designation for PM2.5

Batch Processes 4 CDX Production Tables – Production Status Legend AQS Batch Data Input Flowchart Errors Edit Error Detail Post End Correct Raw Edit/Load Summary Submit Correct Data Load File Stats CR Post Staging Tables Production Tables - Preproduction Status Y N Y N Stat Eval Scan Report Raw Data Inventory Batch Report Online R = Relational Status S = Stat Analysis Maintain Raw Y N

Batch Processes 5 What’s the Big Picture? Collect Samples Format the Data Send the Data so AQS Can See it (CDX) Load the Data (Includes Error Detection) Correct Any Errors & Reprocess Analyze and Approve Sample Data Make Available to Everyone Happens Outside of AQS Raw Data Only

Batch Processes 6 Where Do I Start?

Batch Processes 7 Where To Go From Here Files that You have Transferred to CDX List of All Batch Processes run for Screening Group Available Processes Available Reports

Batch Processes 8 Batch Load: Use CDX to transfer files to AQS 1. Start AQS, Login and select Screening Group. 2. Choose Batch Menu option 3. Use CDX button 3 7

Batch Processes 9 Batch Load: Use CDX to transfer data from their PC to server (cont.) 4. Enter user name, CDX password and Login

Batch Processes 10 Batch Load: Use CDX to transfer data from their PC to server (cont.) 5. Select AQS File Transfer

6. Click on Select and Specify files to collect from your computer Batch Processes 11 Batch Load: Use CDX to transfer data from their PC to server (cont.)

Batch Processes 12 Batch Load: Use CDX to transfer data from their PC to server (cont.) 7. Click on Send to deliver all files to AQS. Close browser window.

Batch Processes 13 Batch Load: Use CDX to transfer data from their PC to server (cont.) 8. After receiving notice of delivery, click on Refresh CDX to see the newly loaded files.

Batch Processes 14 File is on the server - We are finished with CDX Now, how to load the file(s) to AQS? CDX Production Tables – Production Status Legend AQS Batch Data Input Flowchart Errors Edit Error Detail Post End Correct Raw Edit/Load Summary Submit Correct Data Load File Stats CR Post Staging Tables Production Tables - Preproduction Status Y N Y N Stat Eval Scan Report Raw Data Inventory Batch Report Online R = Relational Status S = Stat Analysis Maintain Raw Y N

Batch Processes 15 CDX Production Tables – Production Status Legend AQS Batch Data Input Flowchart Errors Edit Error Detail Post End Correct Raw Edit/Load Summary Submit Correct Data Load File Stats CR Post Staging Tables Production Tables - Preproduction Status Y N Y N Stat Eval Scan Report Raw Data Inventory Batch Report Online R = Relational Status S = Stat Analysis Maintain Raw Y N Batch Load: Use Batch Load to Install Data 1. Select file (highlight) to load in CDX block 2. Choose “Load File” 3. Wait for or Refresh in session block for ERROR/COMPLETE status

Batch Processes 16 CDX Production Tables – Production Status Legend AQS Batch Data Input Flowchart Errors Edit Error Detail Post End Correct Raw Edit/Load Summary Submit Correct Data Load File Stats CR Post Staging Tables Production Tables - Preproduction Status Y N Y N Stat Eval Scan Report Raw Data Inventory Batch Report Online R = Relational Status S = Stat Analysis Maintain Raw Y N Batch Load: Loading Raw Data Possible Results of Load - COMPLETED load - Load with ERROR No data in Staging tables, Summary reports successes; next step Stat/CR Data in Staging tables, Summary reports successes and failures; next step fix errors

Batch Processes 17 CDX Production Tables – Production Status Legend Errors End Raw Edit/Load Summary Load File Staging Tables N Batch Report Online N Batch Load: Loading other than Raw Data Possible Results of Load - COMPLETED load - Load with ERROR No data in Staging tables, Summary reports successes; End Data in Staging tables, Summary reports successes and failures; next step fix errors

Batch Processes 18 Recap Steps: No Raw Data and No Errors CDX  Load No Raw Data and Errors CDX  Load  Correct  Submit Correct Data Raw Data and No Errors CDX  Load  Stat/CR  Post Raw Data and Errors CDX  Load  Correct  Submit Correct Data  Stat/CR  Post Status = R Status = S Status = P

Batch Processes 19 Caveats: A batch load job can end with a status of “WARNING” The only way to see the warning message is to look at the job log. Example Warnings: Urbanized Area Code replaced by Geospatial Lookup

Batch Processes 20 Put it into Practice: Exercise 2.2

What is changing? Submit a file without logging into AQS New CDX website (ENSC) & new CDX user ID Files submitted to ENSC will be automatically processed by AQS Stat/CR is now part of Load New AQS batch form Processing done by file, not by screening group Reports are auto-generated Reports are consolidated Batch Processes 21

“Old” vs “New” process AQS User AQS CDX replacement 1 st Q 2012 AQS Autoloader User Decides Where to Stop EN Node CDX Web Old process New process Stage LoadPost Upload Via CDX LoadStatCRPost Batch Processes 22

AQS Batch form (“new”) Batch Processes 23

ENSC website Exchange Network Services Center (ENSC) The CDX replacement EN Username = your address Batch Processes 24

ENSC -- file upload form required optional required for XML Stage Load Post Yes No File to Upload address AQS User ID Screening Group Stop on Error: required optional Final Processing Step XML Schema File Type XML Flat Card One file only; Must be zipped Default Final Processing Step is Load. Default Stop on Error is Yes. Batch Processes 25