Presentation is loading. Please wait.

Presentation is loading. Please wait.

Untapped Power of Data Validation

Similar presentations


Presentation on theme: "Untapped Power of Data Validation"— Presentation transcript:

1 Untapped Power of Data Validation
Texas Commission on Environmental Quality Environmental Trade Fair and Conference (ETFC) Austin Convention Center May 16-17, 2017 Erin E. Rodgers – Environmental Standards, Inc.

2 What, When, Why Data Validation? Data Validation Case Studies
Agenda What, When, Why Data Validation? Data Validation Case Studies Forensic Data Review Case Study Conclusions 2

3 Why Data Validation? There are many common misconceptions regarding environmental laboratory data. For example, do you believe any of the following statements? The laboratory is accredited; therefore, my data must be correct, accurate, and precise. The laboratory follows the published analytical method; therefore, my data must be acceptable. Only the laboratory impacts the quality of the analytical data. 3

4 Why Data Validation? Many costly and important remedial decisions are made based on environmental analytical data. The quality of the analytical data will directly impact the overall project decisions that are made on the basis of the data. Project data drive remediation and remediation costs. 4

5 When Perform Data Validation?
Potential for litigation? Potential for high level of scrutiny? CERCLA or Superfund Development / Redevelopment of Potentially Contaminated Site Regulatory Driven Issues Identified or Questions Arise During Limited Data Review 5

6 When Perform Data Validation?
Limited data validation is routinely performed End data user desire to understand the usability or acceptability of the data Can be automated Full Data Validation is performed usually due to one of the reasons listed on the prior slide Most of the issues that could cause substantial delays, liability, and cost impacts are identified during full validation Always performed manually 6

7 What is Data Validation?
Several levels or stages of validation Limited data validation “Checklist” Validation Reviewing QC to determine appropriate qualification and use of the data Stage 2A - Summary form review of batch QC analyses Stage 2B - Summary form review of batch QC analyses and calibrations/instrumental analyses Full data validation Comprehensive, critical review of data Stage 4 – Includes everything in Stage 2 plus a critical review of raw data and instrumental outputs to qualitatively and quantitatively confirm the reported results 7

8 Typical Data Package Limited Data Package Full Data Package
Summary Forms Only Sample Results Batch QC Results Method Blank Blank Spike Matrix Spike Field/Laboratory Duplicate Summary Forms & Raw Data Sample Results Batch QC Results Method Blank Blank Spike Matrix Spike Field/Laboratory Duplicate Calibrations Tunes, Initial, Continuing Checks, Run logs Instrumental QC Internal Standards, Interference Checks, Serial Dilutions, Dual-Column Precision Extraction, Digestion, and/or Preparation logs 8

9 What is Data Validation?
Process of critically examining laboratory data Includes review of: Verification of Correctness Compliance Usability In this presentation, we are going to discuss issues identified during Stage 4 or full validation review Full validation is the process of reviewing data for correctness, compliance and usability Lower levels or stages of validation typically only encompass the usability aspect of validation. Validation is often performed to “check the box” only encompasses a usability assessment wherein “flags” are appended to the analytical data. In this presentation I will detail issues that were identified during the correctness or verification assessment and compliance review aspects of validation that saved the client time, money, and potentially additional parameters being added to the COC list for a particular site 9

10 What is Data Validation?
Correctness Assessment All samples collected/requested were analyzed and reported All analytes that were analyzed/requested were reported Correct methods, MDLs/RLs, units, basis Confirm qualitative identification of target compounds Recalculate reported concentrations from raw data AKA – “Verifying” the analytical data 10

11 What is Data Validation?
Compliance Assessment Compliance with analytical method Compliance with project control documents Quality Assurance Project Plan (QAPP) Sampling and Analysis Plan (SAP) Work Plan Project Specifications Compliance with permit requirements Compliance with regulatory requirements 11

12 What is Data Validation?
Usability Assessment Review of QC, calibrations, holding times, and sample receipt Determines appropriate “flagging” of data Blank Qualification – U or B Rejection of Data – R Estimating Data – J Information to data user on the potential limitations and appropriate use and potential limitations of the analytical data All stages of validation include a usability review. Data may be qualified for a number of reasons during validation. This presentation will focus on the correctness and compliance assessments performed during Stage 4 or full validation, rather than detailing qualifications applied during a usability assessment. 12

13 DV Case Study #1 Transcription Errors – Most Common Errors
Impact Reported Results Incorrect Dry-Weight Reported Percent solids reported as 7% Should have been reported as 75% Results were reported on a dry-weight basis Concentration of metals in soil samples decreased 10-fold. Samples were soil samples that all had % solids values in the 70 to 80% range. One result stuck out as having very low % solids at less than 10% solids. This did not seem realistic knowing the matrix of the samples. Review of the percent solids raw data showed that the % solids should have been reported as 75% Analyte Initial Results (7% solids) Revised Results (75% solids) Benzo(a)pyrene 0.054 0.0051 Antimony 14 1.32 Arsenic 4.62 0.437 Lead 840 79.3 13

14 Issue #1 Continued Transcription Errors - Incorrect Initial Weight
Manual transfer of initial sample weights to the laboratory information management system (LIMS) for final concentration LIMS preparation log reported initial weight as g Handwritten preparation log reported initial weight as g Metals concentrations increased 1.5 fold Transcription errors become a real issue when manual transfer of values occur throughout several areas of the laboratory. Some labs or some areas of a lab have direct transfer of balances to LIMS. Other labs or lab sections (wet chem lab) must manually record the sample weight and transfer this weight into the LIMS system manually. Transcription errors are often seen with manual transcription. units = mg/kg Reported concentration Recalculated concentration Reported MDL Recalculated MDL Reported RL Recalculated RL antimony 0.352 0.521 0.163 0.241 0.678 1.00 arsenic 0.477 0.705 0.339 0.502 lead 9.64 14.3 0.23 0.341 0.542 0.80 14

15 of metals in soil samples decreased.
Issue #1 Continued Reporting Errors – Dry vs. Wet Weight Reporting Samples were air-dried and sieved upon receipt Should have been reported on an as-received basis (100% solids) All results were reported on dry-weight basis, using the percent solids from the received sample (prior to air-drying and sieving) Not included on this slide but another example of a reporting error that I’ve run across was where all metals results in all soil samples were corrected for percent moisture, rather than percent solids. The resulting data were revised and re-reported and the concentration of metals in the samples decreased 2 to 4-fold. Concentration of metals in soil samples decreased. Analyte Original Result Revised Result Dry-Weight Corrected Air-dried Aluminum 3560 mg/kg 2980 mg/kg Antimony 17.8 mg/kg 15 mg/kg Arsenic 63 mg/kg 52.8 mg/kg Barium 18.6 mg/kg 15.6 mg/kg 15

16 Resolution Laboratory was requested to review the observed discrepancies Data were revised and re-reported with the corrected concentrations One instance resulted in significantly lower reported concentrations One instance resulted in slightly lower reported concentrations One instance resulted in slightly higher reported concentrations Issues observed during quantitative review – recalculation of results from raw data 16

17 DV Case Study #2 Misidentified PCB Aroclor
Qualitative identification issue Review of PCB pattern and ratio of peaks in sample versus PCB standard Overlapping/similar peaks present in both Aroclors Submitted a blind performance evaluation (PE) sample to the laboratory with environmental PCB samples to verify PCB reporting Identified by qualitative review of the chromatogram 17

18 Resolution Critical review of sample raw data/chromatograms and PE sample qualitatively identified and confirmed the correct Aroclor pattern present in the samples Laboratory revised and reissued data with correct Aroclor reported prior to data submission Saved client by reporting the correct detected Aroclor Reporting incorrect Aroclor detection may have added an additional COC to the parameter list as the incorrect Aroclor was not known to be used at the site historically 18

19 DV Case Study #3 False positive result reported – Improper Integration
Incorrect integration of the analyte by the instrument Identified during qualitative review of chromatogram Before After 19

20 Resolution Laboratory was requested to review integration and revise data. Laboratory originally reported positive result. Revised result was ND < RL. Saved the client by not reporting a false positive result which may have expanded the COC list to include the analyte. 20

21 DV Case Study #4 False positives – Method Limitation
Positive results for hexavalent chromium were reported for samples, but the matrix spike recoveries were 0%, indicating a reducing sample matrix For ultraviolet-visible spectrophotometry (UV/VIS), dark/turbid extracts often produce a signal that may be reported as false positive results Technical expertise – 0% MS recoveries coupled with dark extracts 7196A UV/VIS method Issue was noted when the sample digestions included notes stating that the extracts were very dark in color and turbid Knowing that dark extracts can lead to color interferences the data reviewer noted that the MS/MSD recoveries were 0% for the soil sample. The LCS recoveries were spot-on. 0% MS/MSD recoveries for hex chrom often tells you that the samples are reducing in nature and hexavalent chromium cannot exsist (is quickly reduced to trivalent chromium). 21

22 Resolution Samples reanalyzed by a different method
Ion chromatography physically removes extract interferences Ion chromatography confirmed that hexavalent chromium was not present in the samples 7199 IC method Physically removes extract interferences by absorbing organics and suspended solids through the use of a guard column Anion exchange column separates the remaining constituents in the extracts This analytical method and separation technology yields better qualitative identifications, with less need for extract dilution, resulting in more accurate and precise results In this specific example, a study was also performed to determine whether the soil samples were reducing in nature as hexavalent chromium cannot exist in nature under reducing environmental conditions. In order to determine whether the samples were reducing, pH an ORP were performed. Another line of evidence used in this example, was to request and analyze total chromium which wasn’t previously being done. We were able to show that the total chromium results were less than the hexavalent chromium results providing another line of evidence that the color interferences were leading to false positive results. 22

23 DV Case Study #5 Method Non-compliance Led to Potential NOV
Identified during compliance evaluation Single BOD sample way outside normal results Logistics and timing issue prevented additional samples from being collected Agency started process for permit violation Review of raw data noted BOD dilution water blank reading was 2.2 mg/L Method requires BOD < 0.2 mg/L in dilution water blanks Elevated blank reading resulted in inaccurate and unreliable quantitation of BOD results Method 5210B Section 6 QC Checks C – Dilution Water quality check. Serves as a check on quality of unseeded dilution water and cleanliness of incubation bottles. The DO update in 5 d MUST not be more than 0.2 mg/L before making seed corrections. If the dilution blank exceeds 0.2 mg/L, discard all data for tests using this dilution water. 23

24 Resolution Laboratory requested to review and verified the analytical error. Laboratory formally rescinded analytical results. Laboratory contacted the reporting agencies on client’s behalf. NOV was rescinded. Data were non-compliant with analytical method. 24

25 DV Case Study #6 Insufficient field sample preservation.
Identified during compliance evaluation TOC method requires samples to be preserved to pH < 2. Received with pH values ranging from 4 – 6. Sample preservation was method non-compliant. Method does not have an allowance for laboratory preservation like metals analysis. 25

26 Resolution Current data set was qualified during validation.
Quick study was performed in the field to determine sufficient amount of preservative needed to properly preserve the samples to pH < 2. Training provided to field staff so future samples were properly preserved and method compliant. 26

27 DV Case Study #7 Multiple Major Method Non-Compliances – Metals Data
Improper calibration sequences Failure to analyze initial calibration blank (ICB) Failure to analyze continuing calibration verification (CCV) and continuing calibration blank (CCB) at method-specified frequency Failure to digest and analyze QC samples at method-specified frequency Failing QC analyses Failure to analyze several method-required QC analyses 27

28 Resolution Laboratory Case Narratives indicated that all metals results should be rejected due to the many compounding issues. This assessment was confirmed by validators when reviewing the packages. Retained metals samples were pulled from the laboratory and sent to another laboratory for analysis. Saved by the long holding time for metals! 28

29 DV Case Study #8 Blank contamination for same analytes
Identified during usability assessment Laboratory water contained several analytes above MDL. Laboratory supplied deionized water for field & equipment blanks. Same water used for laboratory method blanks. Consistent levels observed for several analytes in the various blanks. Similar levels in blanks and samples would have led to many results qualified for blank contamination. 29

30 Resolution Laboratory requested to investigate the consistent low-level positive results in blanks. Analysis of the laboratory DI water indicated background levels of several constituents. Water source run through an additional polishing process to remove background contaminants. Constituents observed in DI water were not used to qualify sample results. Active data management enabled the data validation chemists to quickly identify the issue and observe the consistent level of constituents reported in the various blanks. 30

31 Conclusions Common data issues are not often identified during a cursory review. Oftentimes validation is performed to “check the box.” Lower levels of validation are typically performed to determine the usability of the result. Full, comprehensive validation includes review for correctness and compliance. Most issues were identified during the correctness/verification or compliance review. 31

32 Forensics Case Study Natural gas condensate released onto residential properties Review of SVOC analysis full scan and PAH data to determine when cleanup activities successfully removed source material Evaluation of remaining PAH concentrations to determine petrogenic versus pyrogenic source 32

33 Forensics Case Study SVOC chromatogram – impacted soil sample
Large upward hump characteristic of source material 33

34 Forensics Case Study SVOC chromatogram – same location after excavation and removal of impacted soil Large upward hump not present – source material removed 34

35 Forensics Case Study SVOC analysis indicated source material was successfully remediated. PAH constituents remained – present in confirmation/after excavation samples. PAHs were present in most samples, but without a discernable pattern. i.e., PAHs were not necessarily higher closer to the source of the release; some of the highest PAH concentration was far away, outside of the reasonable mist deposition area. 35

36 Forensics Case Study Selected a class of high molecular weight compounds to review. These compounds are known to be products of incomplete combustion. If we could show that these compounds were not present in the source material, but present in the samples, then we could show that the presence of these compounds was not due to the released material and were, instead, “background” or present due to other environmental factors. 36

37 Forensics Case Study Sample collected outside of the mist deposition area. Some of the highest PAH results reported. “Background” Soil Pattern of high mw doublets and singlets are present here, outside the mist deposition (below). Characteristic of background prior to release 37

38 Forensics Case Study Source oil sample
High molecular weight compounds not observed Pattern of high mw doublets and singlets not present in source material 38

39 Forensics Case Study Visually impacted soil sample
Same sample shown under SVOC analysis Pattern of high mw doublets and singlets not present 39

40 Forensics Case Study Same sample after excavation/cleanup
Source confirmed removed by SVOC analysis Pattern of high mw doublets and singlets are present here. 40

41 Forensics Case Study - Conclusions
SVOC full scan chromatograms showed that the excavation of the impacted soil was successful in removing the source material. PAH SIM chromatograms developed by forensics team showed that petrogenic PAHs of concern were not present in the source material. PAH SIM analysis showed that petrogenic PAHs were ubiquitously distributed and not associated with source material. 41

42 “Setting the Standards for Innovative Environmental Solutions”
Thank You “Setting the Standards for Innovative Environmental Solutions” Environmental Standards, Inc. 1140 Valley Forge Road


Download ppt "Untapped Power of Data Validation"

Similar presentations


Ads by Google