1. Using a Web-Based Reporting Application to Evaluate an Immunization Registry Software Transition
Letty Cherry Kreger, MHA
California Immunization Registry (CAIR)
Inland Empire Region

2. Outline Background Evaluation Objectives Preparing for the Evaluation
PSR Reports Used in Evaluation Process and Results
Barriers
Next Steps

3. Background
The Provider Status Report (PSR) was developed by the LA-OC Region of CAIR
Web-based application
Contains several queries used to assess CAIR use and quality of data
It became available to all CAIR regions in 2009
Next logical step – use PSR to evaluate the transition from VaxTrack to CAIR software

4. Evaluation Objectives
How many providers were using the inventory feature?
Did the inventory feature make an impact on provider use of the registry?
If providers did not use inventory, did they enter any immunizations?
If they did not enter any immunizations, how long had it been since their last log-in to system?
What are their barriers to using CAIR, if any?
How well are the high use providers adjusting to CAIR?

5. Preparing for the Evaluation
Determine which PSR reports to use
Inventory Transactions
Records without Inventory Transactions
Log-in Activity
Identify providers to evaluate
Run baseline report
Compare to historical data
Contact providers
Monitor and run follow-up report

6. PSR Reports – Inventory Transactions
Identifies the number of shots given and deleted from inventory during the specified time frame

7. PSR Reports – Records without Inventory Transactions
Identifies records containing only transcribed immunization history. Active patients only.
May be a sign that provider is not using or struggling with Inventory

8. PSR Reports – Login Activities
Identifies login information for users (days since last login, total logins
Is provider at least looking up records?
Does this provider have an EMR?

9. Selecting Providers to Evaluate
All “high use” providers under VaxTrack
Random Selection
Small sample size (n = 34)
Checked records in Provider Management Database (PMDB)
Offices Closed = 1
Offices Trained = 29
Offices Not Trained = 4

10. How Many Providers Used CAIR?
Evaluation Indicator
07/13/2009 to 12/08/2009
07/13/2009 to 04/12/2010
Sample Size 34
# Providers Trained 29 31
# Using Inventory 18
# Entering Transcribed Immunizations 6 4
The number of providers using Inventory remained consistent The number of providers entering transcribed data decreased by 33%
Time Period – July 13 to December 8
56% providers using inventory
74% inventory and/or transcribed immunizations

11. Findings about CAIR Use
The clinics using the inventory function in December continued using the inventory function.
Two clinics entering transcribed immunizations stopped entering immunizations
One provider lost primary user, other uses an EMR

12. How Many Providers Logged Into CAIR?
Log In Frequency
07/13/2009 to 12/08/2009
07/13/2009 to 04/12/2010
Within Last 7 days 19 21
Within Last 8 – 30 days 4 2
Within last days 1
With last days 5
> 180 days ago 3
Never
Untrained providers (4) not included 23 providers consistently logging into registry

13. Findings about Log-in Frequency and Use of CAIR
All provider sites using Inventory logged into CAIR within the past week (most within the past 24 hours).
Sites entering transcribed vaccines logged into the registry within two weeks.
Providers who were not entering data into CAIR did not log in to check children’s immunizations.
McGinness – transcribed some immunizations shortly after go live, then dropped off (EMR)

14. Barriers to CAIR Use Excludes providers who had not been trained
Inventory Use:
50% = Time 50% = Training
CAIR Use:
75% = Time 25% = Staff Turn-Over

15. Did the Transition Affect the Quantity of Immunizations Entered?
Quantity of Data Entry
07/13/2009 to 12/08/2009
07/13/2009 to 04/12/2010
New Immunizations Entered Same Time Period Last Year ( )
22,014
40,837
Current New Immunizations Entered
12,514
29,653
Percent Change
-43%
-27%

16. Possible Explanations to the Change in Volume
Learning curve and additional need for training
Lack of retention-based follow-up by staff
No deployment visits
No follow-up site visits
Lack of feedback to provider office (reports)
Lack of staff time to enter data
DeMola – one high use user who entered most of the records was not working as much
Staff time – audits, increased patient load

17. Next Steps Resume/continue with regular retention activities
Quarterly phone calls to assess registry use and training needs
Offer retraining as needed
Deployment visits with any site receiving additional training
Timely and regular feedback will help increase registry adoption

18. Next Steps, continued Follow-up Visits to all transitioned providers
Introduce Provider Status Reports to office manager
Discuss other CAIR reports and benefits
Implement action plan to improve CAIR use
Continue to monitor providers’ registry use

19. Lessons Learned
Make sure retention activities fit into the transition plan.
Be prepared to give feedback to providers early and often.
Teach providers how to measure their own success.

20. Acknowledgements
California Immunization Registry (CAIR) State Level Staff CAIR - Los Angeles-Orange County Region Staff CAIR - Inland Empire Staff

21. For questions or more information:
Letty Cherry Kreger, MHA
Regional Manager, CAIR Inland Empire