Session #65 Draft Results of Quality Assurance Program Data from Award Year David Rhodes and Anne Tuccillo

2 Goals Share draft results of program- wide analysis of data Illustrate additional ways to analyze ISIR data

3 Background Quality Assurance (QA) Program Participants granted regulatory flexibility to decide which ISIR data they verify ISIR Analysis Tool

4 The Analysis 146 Quality Assurance Program institutions Each school drew a random sample of at least 350 applicants Each school verified the ISIR information for the entire random sample of applicants We analyzed 68,077 records

5 Previous data collections Population ISIR Records subject to school verification Random sample of all applicants Number of schools

6 See a pattern? Alternating focus –Random sample of all applicants –Institutionally verified applicants This year ( ) –All QA Program institutions –ISIRS subject to institutional verification

7 Is analysis over time appropriate? For individual schools, yes with care For program-wide analyses the answer “should” be no, but…..

8 When Looking Across Years Keep in Mind: The two different populations Changes to institutional verification criteria between years Other differences between award years Percents and averages not counts

9 Why we “shouldn’t” QA schools supplying data differ slightly from year to year (146 ≠140 ≠133) Unmeasured differences in institutional verification across years

10 Key areas of analysis Description of population Critical ISIR fields How changes affect aid eligibility Improper payments in the Pell Grant program

11 Characteristics of Aid Applicants at QA Program Schools

12 Dependent and independent record counts over time

13 Dependent and independent percentages over time

14 Percentage of dependent applicants with changes to the most commonly changed ISIR Fields:

15 Percentage of independent applicants with changes to the most commonly changed ISIR Fields:

16 Changes to critical ISIR fields among dependent students over time

17 Changes to critical fields among independent students over time

18 Dependent records: percent of ISIR fields experiencing an EFC change

19 Independent records: Percent of ISIR fields experiencing an EFC change

20 Comparison of changes to EFC among dependent students with a change to the indicated ISIR field

21 Comparison of changes to EFC among independent students with a change to the indicated ISIR field

22 Percentage of dependent records with a change to the indicated ISIR field and a change to a Pell Grant:

23 Percentage of independent records with a change to the indicated ISIR field and a change to a Pell Grant:

24 Comparison of change to a Pell Grant among dependent students with a change to the indicated field

25 Comparison of change to Pell Grants among independent students with a change to the indicated field

26 Average EFC change among dependent records with a change to the indicated ISIR field:

27 Average EFC change among independent records with a change to the indicated ISIR field:

28 Average EFC change among dependent students with a change to the indicated field over time

29 Average EFC change among independent students with a change to the indicated field over time

30 Average Pell change for dependent records with a change to the indicated ISIR field:

31 Average Pell change for independent records with a change to the indicated ISIR field:

32 Average Pell Grant change for dependent records with a change to the indicated field over time

33 Average Pell Grant change for independent records with a change to the indicated field over time

34 Potential improper payments in Pell Grants prior to verification at QA Schools:

35 Improper Payments in Pell Grants before and after school verification

36 Improper Payments in Pell Grants before and after CPS verification

37 Lessons Learned While verification is crucial in some cases, many ISIR records do not experience a meaningful change from their initial transaction value QA school verification procedures target larger EFC and Pell changes QA school verification reduces improper payment risks in Pell

38 What Do the Findings Mean for Schools and FSA? Given that relatively few ISIR records experience a change that affects eligibility for need-based aid, verification efforts should strive to focus on the records that matter.

39 Two Ways to Focus Look for the records that matter Look for the records that don’t

40 Suggested Additional Analysis Look for ways to reduce the number of school verified records that experience no or only trivial changes to aid eligibility after verification

41 Additional Analysis ISIR Analysis Tool reports Exporting data from the Tool

42 ISIR Analysis Tool Key Filters –EFC change -400 –EFC change > 400 –EFC change < -400 –Institutional verification criteria

43 ISIR Analysis Tool Key Reports –Sample summary –Field Increment (EFC or AGI) Drill down to refine understanding

44 New Analytic “Recipe” See handout

45 Exporting Data See handout

Institutional Profiles Comparison of Program- wide and school specific data

Institutional Profile Example of a Institutional Profile

48 Profile Differs from Year to Year provided data of improper payments PREVENTED by institutional verification provides data on estimated levels of POTENTIAL improper payments in a schools applicant population

49 Contact Information We appreciate your feedback and comments. We can be reached at: