1. Unemployment Insurance Integrity Performance Measures
A Discussion of Proposed Integrity Measures
ETA
Office of Unemployment Insurance

3. RHONDA COWIE Team Lead US DOL – ETA Division of Performance Management
National Office

4. DANIEL SOMMERS ANDY SPISAK Statistician Statistician US DOL – ETA
Division of Performance Management
National Office
ANDY SPISAK
Statistician
US DOL – ETA
Division of Performance Management
National Office

5. Outline Background/Motivation Measure 1 Measure 2 Summary & Feedback
Combined Benefit Accuracy Measurement (BAM)- Benefit Payment Control (BPC) Fraud Measure
Measure 2
BPC Fraud and Non-Fraud Measure
Summary & Feedback

6. Background/Motivation
In 2011 ETA established the Benefit Year Earnings (BYE) Performance Measure (UIPL No ). This measure supported states as they addressed the largest reason for overpayments both on a state and national level.
For example, in Performance Year 2014 – 2015, there were an estimated $1.08 billion in BYE overpayments, representing 30 percent of overpayments.
For the CY 2014 performance period the acceptable level of performance (ALP) was a 25% reduction from a state’s
CY 2010 to 2012 baseline BYE rate . UIPL 34-11, change 1 communicated ETA’s intent to replace this measure.

7. Background/Motivation
In 2014, Mathematica Policy Research (MPR) conducted a Methodology Study to assess various revisions to the sampling and rate estimation methodology, and operation of the BAM survey. Based on consultations with States that were part of the study, MPR recommended the Department explore the establishment of a new measure to emphasize the importance of detecting and reducing fraud overpayments. To begin this process, ETA developed the proposed measures, which are outlined below for discussion.

8. Measure 1: Combined BAM, BPC Fraud Measure
Definition and Calculation
Ratio of Fraud Overpayments Established by BPC to BAM-estimated Fraud Overpayments, expressed as percentage:
𝐹 𝑐𝑜𝑚𝑏𝑖𝑛𝑒𝑑 = 𝐹 𝑏𝑝𝑐 𝐹 𝑏𝑎𝑚 ×100
Similar construction as the current Overpayment Detection Measure (BPC overpayments / BAM operational rate X 100)
Data Sources: ETA 227 and BAM

9. Measure 1: Combined BAM, BPC Fraud Measure
This measure utilizes the data collected through the BAM and BPC programs. The following are two criteria for this measure:
A single performance target of 50%
Based on historical data trends that suggest 50% would be the most appropriate target.
A three-year performance period
Variability of the BPC/BAM detection ratio is significantly lower for a three year performance period, compared to a one year period
Reduces sampling errors for the BAM component
Reduces variation due to operational issues such as the reprogramming of state staff

10. Measure 1: Combined BAM, BPC Fraud Measure
Pros:
Uses BAM-estimated fraud as a benchmark for BPC fraud established
BPC fraud overpayments established are based on administrative data and is not subject to sampling error
Encourages state BPC units to use all available tools such as cross-matches, NDNH/SDNH, wage record, and SIDES in order to improve the efficiency of their operations and increase detection ratio.

11. Measure 1: Combined BAM, BPC Fraud Measure
Cons:
Considerable variation in the data, even with a three-year performance period, which may increase difficulty of meeting and maintaining performance targets
States vary with regards to definitions of fraud
May be more advantageous to states having a narrower fraud definition
The BAM data are subject to sampling error, which raises an operational issue in evaluating state performance.
Potential for non-sampling error, such as BPC’s application of filters to cross-matches, or miscoding in BAM.

12. Measure 2: BPC Average Overpayment Measure
Definition and Calculation:
This measure utilizes the data collected through the BPC program, derived from the ETA 227 report.
It measures an average overpayment (OP) for both fraud and nonfraud cases across root causes and is represented in the following formula:
𝐴𝑣𝑔 𝑂𝑃 𝑏𝑝𝑐 = 𝐹𝑟𝑎𝑢𝑑+𝑁𝑜𝑛𝑓𝑟𝑎𝑢𝑑 𝑂𝑃 𝐹𝑟𝑎𝑢𝑑+𝑁𝑜𝑛𝑓𝑟𝑎𝑢𝑑 𝐶𝑎𝑠𝑒𝑠

13. Operational Issues
1. Should a single measure combining fraud and non-fraud overpayments be established or should there be separate fraud and non-fraud measures?
Based on historical data, a combination fraud and non-fraud measure is advantageous to most states because the impact of varying fraud definitions among states is greatly reduced.
States with large differences between their fraud and non-fraud overpayment rates might have difficulty meeting separate fraud and non-fraud measures. A combined measure balances one rate against the other, possibly leading to more favorable performance for this measure.  Therefore, establishing a combined fraud and non-fraud measure appears to be the better alternative.

14. Operational Issues
2. How should changes in the average amount overpaid be adjusted in order to take into account increases in the WBA?
Option 1: Adjust the average amount overpaid by an adjustment factor that represents the inflation (increase) of the WBA.
Option 2: Adjust the average amount overpaid by expressing it as the average number of UI weeks overpaid: (Avg. $ OP / Avg. $ Paid).

15. Operational Issues
Option 1: Adjust the average amount overpaid by an adjustment factor that represents the inflation (increase) of the WBA.
Example:
If the average amount paid increases from $300 in to $315 in a 5% increase - the adjustment factor is 1.05 ($315 / $300).
If the average amount overpaid increases from $800 in 2014 to $875 in 2015, the adjusted average overpayment in 2015 is $ ($875 / 1.05).

16. Operational Issues
Option 2: Adjust the average amount overpaid by expressing it as the average number of UI weeks overpaid: (Avg. $ OP / Avg. $ Paid).
From the preceding example:
The average number of UI weeks overpaid in is: $800 / $300 = 2.67.
The average number of UI weeks overpaid in is: $875 / $315 = 2.78.

17. Measure 2: BPC Average Overpayment Measure
Pros:
Provides incentive for states to pursue overpayments more aggressively (through increased use of cross- matches such as NDNH, wage record, and SIDES) to detect overpayments sooner and therefore reduce average overpayments.
States strive to detect and pursue all overpayments, versus examining only fraud.
Uses average overpayment established as opposed to BPC fraud rate (BPC fraud rate is consistently low and difficult to demonstrate improvement)
Single percentage reduction target from each state’s baseline average can be set.

18. Measure 2: BPC Average Overpayment Measure
Cons:
A measure that includes all overpayments represents a shift in the Department’s focus from targeting BYE overpayments.
Potential for non-sampling error, such as erroneous reporting on the 227 report and other state operational issues, such as integrity staff reprogramming.
If “UI Weeks overpaid” is used, there is a potential challenge linking a fractional reduction in the number of weeks paid to the improper payment rate since WBA’s vary greatly across states.

19. SUMMARY Measure comparison: Measure 1 (BPC/BAM measure)
Pulling strength from both programs for a composite measure
Non-sampling error is controllable
Similar to the current Overpayment Detection Measure
Measure 2 (BPC only)
Gives full picture of overpayments
Non-sampling error controllable

20. Feedback:
Do these measures address the concerns you raised during the Study?
What, if anything would you want to change in these new measures and why?

21. Subri Raman Rhonda Cowie Team Lead Division Chief
Division of Performance Management
Division Chief
Division of Performance Management