Consistency and Repeatability in Analytics (Using M2)

FOR OFFICIAL USE ONLY 2 Consistency and Repeatability in Analytics 1) Context: The practice of analytics requires discipline to ensure results are repeatable, consistent and reliable. 2) Purpose: This presentation will describe best practices for reporting 3) Outcome: After attending this session, participants will meet the objectives described on the next slide.

Objectives Attendees can: 1. List best practices for preparing reports in M2 2. Utilize the M2 data dictionary. 3. Retrieve a copy of the M2 Data Dictionary 4. Describe an analysts reponsibilities with respect to data protection. 5. Describe a good M2 deliverable product. 6. Track work on repeated projects for anomalies 7. Keep proper documentation associated with the use of M2. 3

Motivation Reliable analysts usually become very valuable to an organization. Especially true when analysts use ad-hoc systems to prepare studies. Ad-hoc offers maximum flexibility for question answering But there is always the risk that the user will not properly retrieve/handle data. This presentation offers tips on the optimal use of M2, to achieve reliability, consistency, and repeatability. FOR OFFICIAL USE ONLY 4

Preparing M2 Reports FOR OFFICIAL USE ONLY 5

Restricted vs. Unrestricted M2 There are two options for M2 access: M2 Restricted: Contains direct identifiers for beneficiaries (i.e. name, SSN) Contains rank of sponsor Most users do not need the restricted M2. Appropriate for care managers or other people who need to know who the person is on a record in M2 Can only retrieve <10K rows of data. M2: Encrypted person identifiers (mostly), grouped rank only Considered safer, but still contains protected health information. Can only retrieve up to 500K rows of data at a time. 6 Use the Unrestricted M2 when you can!!!!!

Report Writing Check to see if there is a corporate report that meets your needs. Handbook can be obtained from M2 Infoview FOR OFFICIAL USE ONLY 7

Report Writing If you find a candidate report, read the documentation carefully before using the report. [Corporate Reports session on Tuesday at the Symposium] If there is no available corporate report Use available resources M2 Data Dictionary WISDOM notes DCO Refresher Sessions Navy Newsletters Navy DCOs Reports previously done FOR OFFICIAL USE ONLY 8

M2 Data Dictionary 9 FOR OFFICIAL USE ONLY

M2 Data Dictionary M2 Data Dictionary is published once per month Or even easier, just Google M2 Data Dictionary! Excel workbook Tabs for each type of data Additional Reference tabs Hyperlinked for easy navigation May have to click enable content for navigation to work. 10 FOR OFFICIAL USE ONLY Use the M2 Data Dictionary when you write reports!!!!!

M2 Data Dictionary M2 Alerts are not all that useful. 11

It says population, so this must be something about counting people?? What does 0005 mean? What about age group A or Beneficiary Category DA? 12 FOR OFFICIAL USE ONLY

M2 Data Dictionary Click on M2 Data Dictionary and open it. Click on the Table of Contents tab. Each file is listed with a hyperlink to the page that describes the file. Click on Population Summary. Data elements are listed in the same order as in M2. Review the header row. Review the rows for Age Group Code and Beneficiary Category. Review the row for Catchment Area ID. Click on the DMISID link. 13 FOR OFFICIAL USE ONLY

14 To use the DD: Click on the title that represents the data file you are looking at In this case, its population summary.

FOR OFFICIAL USE ONLY 15 Header Information Above. Detailed data descriptions in each row below

Using the DD FOR OFFICIAL USE ONLY 16 Description of valid values for age group code

FOR OFFICIAL USE ONLY 17 Using the DD Description of valid values for Beneficiary Category

FOR OFFICIAL USE ONLY 18 Using the DD No values are listed for Catchment Area ID But if you click on the link…..

FOR OFFICIAL USE ONLY 19 Using the DD DMISIDs that represent the area being reported on, and associated names

Notice how the values you see in this report line up with what you just saw in the dictionary? 2,759 Active Duty Family members ages 0- 4 live within 40 miles of Bassett ACH in Alaska 20 FOR OFFICIAL USE ONLY

M2 Data Dictionary As with Catchment Area DMISID, when a variable in M2 has too many valid values to list, either: Use the hyperlinks to click to an appendix with information, OR Use reference tables within M2 There are a few variables that do not have valid value lists in the DD or in an M2 reference table. Might have to resort to web searches or other means in these cases Sometimes the data dictionary tells you not to use a variable. Back to table of contents and click on Purchased Care Non-Institutional Detail Look at the comment for Number of Visits, Raw DD contains all variables, restricted or not, you just wont be able to see restricted variables if you only have regular access. 21 FOR OFFICIAL USE ONLY

22 Using the DD After reading this note, users should not retrieve this element and called what they report a visit. Very important to always check the DD.

M2 Data Dictionary The M2 Data Dictionary is an excellent resource to use, but it is not always correct. 23 FOR OFFICIAL USE ONLY This MTF Service Area definition is incorrect. MTF Service Areas apply to inpatient and ambulatory clinics. Has been reported and will be corrected. Be careful! Really evaluate your data!

Other Resources WISDOM CD: Given to WISDOM Attendees Detailed descriptions of data in M2 and how to use it. Reports and more reports Step by step instructions for Business Objects. WISDOM instructors contact information. WISDOM DCOs: Conducted monthly, for all M2 users. Materials are posted on Infoview (like the corporate reports) Navy Analytics Newsletters (also in Infoview) Navy DCOs (coming soon) FOR OFFICIAL USE ONLY 24

Previous Reports Many questions are similar Similar cohorts Active Duty? Prime? Diabetics? Etc… Most users save their reports so they can be referred to again. Can be cumbersome to find things though Its also a very good idea to keep a running list of criteria that you use to address questions for easy reference. FOR OFFICIAL USE ONLY 25

Example of a Running List of Query Criteria Data NeedRule PrimeACV Group = Prime PlusACV Group = Plus TFL<65MERHCF Flag = U TFL 65+MERHCF Flag = T Medicare EligiblesMedicare Code is not N Unit MembersUIC matches patten Navy Afloat Sponsor Svc Agg = V, use bencat to sift out dependents Deployed MembersOCO Deployed Flag = Y Guard/ReserveBencat (GRD, IDG) Active Duty w/o guard/resBencat = ACT Unenrolled ADACV = M Network EnrolleesEnrollment Site Service = M FOR OFFICIAL USE ONLY 26

27 FOR OFFICIAL USE ONLY Data Status Table Run the Data Status table to check for freshness of data and any comments that may be applicable.

Report Checking Once your report is written, make liberal use of slice and dice to check your work. Check each variable to ensure that the content is what you expect. Check counts of data for reasonability. Look at monthly data to be sure you understand trends and such. Double check calculations by doing crosstabs and by doing hand calculations where appropriate. FOR OFFICIAL USE ONLY 28

Report Checking If you are running the same report regularly, keep track of how the data changes over time Use that knowledge to know what to expect. Example of same report being run once per month, with counts by month. Keeping adding data w/ each new run. FOR OFFICIAL USE ONLY 29 Month11/10/201112/10/20111/10/2012 Oct Nov Dec 14

Report Checking Be careful of row limits! 30 FOR OFFICIAL USE ONLY Check to make sure you dont get Partial Results Your data will be wrong if you use them! Tiny little error message at the bottom of the screen!

Report Checking Row limits can cause analytic problems: 31 FOR OFFICIAL USE ONLY If you need data that exceeds the row limit: If you have both restricted and unrestricted access, see if your query can be written if the unrestricted universe. This will give you 490K extra rows to work with. If you still need more than your row limit allows there are not many good choices. Some users run their queries in pieces (i.e. month at a time) and then piece results back together. If you do this, be sure you are authorized. (i.e. both universes are PHI, be careful what you do!). Also, do very carefully, because you can introduce more human error this way. Make a checklist of all of the pieces you need, check off that you got them, and run summary queries to confirm you put things together properly.

Data Protection 32 FOR OFFICIAL USE ONLY

Row Limits M2 obviously contains protected health information (PHI) and personally identifiable information (PII) PII: Generally defined by the Privacy Act PHI: Defined by HIPAA. Currently both the M2 and M2 Restricted Universes contain PII and PHI There are many rules and regulations regarding the use of PII and PHI 33 FOR OFFICIAL USE ONLY

Data Protection When using M2 data, what do you need to know? Which data fields are considered PII or PHI? Which data fields are sensitive? When you use these types of data, how do you protect yourself and the beneficiaries in the data? 34 FOR OFFICIAL USE ONLY

PHI or PII? PII Defined by the Privacy Act Such as names, Social Security Numbers, etc.. Small cell (<=30) data fields (i.e. age 90+, or Alaskan Indian Race). DEERS Person ID (EDIPN): Has recently been classified as PII by the TMA Privacy Office. DEERS Person ID is available in both the M2 and M2 Restricted Universes. Must be handled as you would a social security number. DEERS Person ID will be encrypted in regular M2 in a future update. The DEERS Person ID of the user is visible on the screen in M2. Take particular caution if you are a beneficiary and an M2 User! 35 FOR OFFICIAL USE ONLY

Sensitive Data Some data are not covered under laws but are deemed sensitive Some examples include: Procurement sensitive data: Such as Ingredient cost and dispensing fee in purchased care pharmacy data Proprietary Data: Such as CPT Codes and Descriptions. Restricted use: Such as cause of death 36 FOR OFFICIAL USE ONLY

PII or PHI? PHI Defined by HIPAA. HIPAA is much stricter than the Privacy Act. More than just PII. HIPAA seeks to further restrict deduction of identities, in addition to just restricting access to direct identifiers. Stiffer fines for breaches (25K per HIPAA, 1K per Privacy Act). HIPAA Limited Data Sets are a type of PHI but less strict than Full PHI. Most research is done with HIPAA Limited Data Sets (LDS). Lower hurdle to gain access (Privacy Office). Still subject to all the rules and regulations of PHI, though. 37 FOR OFFICIAL USE ONLY

Specifically Covered Elements Names Any geographic designation lower than state level (zip codes, catchment area, PRISM area, etc.). Any geography < 20K people. Any date related to a person below the level of year (FM, Service Date, Date of Birth, Age (if >89) Telephone Numbers, Fax Numbers, Addresses Social Security Numbers Medical Record Number (FMP/SSN) Health Plan Beneficiary Number (EDIPN, DEERS Beneficiary ID) Any other unique identifying number (Other HIPAA elements are not generally available in MHS systems) The PHI bar is set very high by HIPAA 38 FOR OFFICIAL USE ONLY

HIPAA HIPAA Limited Data Sets (LDS) are PHI, but dates (except DOB) and geographic designations are allowed. Absolutely essential elements in health research and analytics Especially dates, where the temporal relationship between events must be understood. M2 is a PHI system. Even the non-restricted version of M2 does not qualify as a limited data set. 39 FOR OFFICIAL USE ONLY

HIPAA Under HIPAA, a citizen has the right to information about who has accessed their PHI. Implemented as the Chain of Custody requirement. Must document use/release of PHI. Done with a HIPAA Log. After writing a report, also consider whether the data use needs to be tracked. Must have data use agreements in place to release external to the organization. HIPAA breaches are very serious matters. Reporting of breaches are expected immediately. The MHS has the biggest HIPAA breach in history, with CHCS data in San Antonio. Has resulted in a billion dollar lawsuit. 40 FOR OFFICIAL USE ONLY

OUCH! FOR OFFICIAL USE ONLY 41

Sample HIPAA Log Date of File Disposition Date Fed Ex Number Destruction Witnessed By Delivery Mechanism Date Data Retrieved User Company/Organization Task Order Project Name External Client/POC Received From External Source File? Sent To External Source? Date Delivered External Agency DUA # File Names and Locations Protected Info Tracking is required by HIPAA for all PHI handling, not just M2 Well-suited for a spreadsheet 42 FOR OFFICIAL USE ONLY

Best Practices Have all users in an organization use the same template for a HIPAA log. Assign someone to be the HIPAA Log Coordinator at your location. Have all users submit the log to the Coordinator on a routine basis for consolidation. Review with DUA custodian or supervisor routinely to ensure data are being properly used and documented. 43 FOR OFFICIAL USE ONLY

44 Physically secure your data Locked cabinets, locked doors, locked file cabinets Implement technological safeguards Encryption Mark copies FOUO, etc Protect passwords/log-on information Protect terminal while logged in (lock your workstation) Do not leave CAC card unattended Best Practices FOR OFFICIAL USE ONLY

45 Dont release (or re-release) PHI data w/o authorization Check DUA if a contractor or consider need to know if internal government. Dont publish PHI or PII. Limit reports to retrieve the minimum necessary information. Destroy data when you are done with it (cross-cut shred, overwrite and delete, or deGauss) Deliver data securely Secure FTP, Encrypted Drives Best Practices FOR OFFICIAL USE ONLY

M2 Deliverable Products FOR OFFICIAL USE ONLY 46

M2 Deliverable Products Since M2 allows ad-hoc querying, there is always a risk that reports will not yield the results you want! Ad-hoc can be your best friend or your worst enemy! A good M2 deliverable product will contain detailed documentation so that recipients of data understand what they got. Most users document in spreadsheets, with multiple tabs if needed. Screen shot of query panel(s). Screen shots of additional math/manipulation or language to describe it. List of filters used with an indication of why. Snapshot of the data status table and other metadata. 47 FOR OFFICIAL USE ONLY

Screen Shot Data Elements Retrieved Filters Applied Using the DMIS ID Table 48 FOR OFFICIAL USE ONLY

Screen Shot of Additional Calculations Created a variable called % Diff 49 FOR OFFICIAL USE ONLY

Picture of a slice and dice panel. Similar to Excel Pivot Tables Enables users to reshape/filter data within the M2 application. 50 FOR OFFICIAL USE ONLY

Documenting Filters Filters or conditions can be applied in many places in M2. Its good practice to document what filters were applied and most importantly, what the filters were intended to do. Example Filter Grid Enables the recipient to understand what was pulled and why, and to check for mistakes. FiltersWhy ACV Group = 'Prime' Only Prime Enrollees MEPRS1 Code = 'B' Only Ambulatory Care Provider Specialty not between 910 – 999 No unlisted specialties Beneficiary Category in list 'ACT', 'GRD' Only Active Duty and Active Guard/Reserve 51 FOR OFFICIAL USE ONLY

Metadata Metadata: Data about the data! Data Status table is a good example. Metadata tab would include: Picture of data status table Query name and location When the query was run How long it took Etc.. 52 FOR OFFICIAL USE ONLY

53 FOR OFFICIAL USE ONLY

Example Deliverable What are the top MS-DRGs in Navy MTFs in FY 2011? How many dispositions and bed days? What was the average length of stay for each of these MS-DRGs? What was the case mix? Lets review an example deliverable for this question. 54 FOR OFFICIAL USE ONLY

Final Tab FOR OFFICIAL USE ONLY 55

Query Panel Tab FOR OFFICIAL USE ONLY 56

Variables Tab FOR OFFICIAL USE ONLY 57

Metadata Tab FOR OFFICIAL USE ONLY 58

Metadata Tab FOR OFFICIAL USE ONLY 59 Available Date FY UpdatedObject Class Reported As Of Date Reported Date Range 1/23/2012FY 2011 Direct Care Inpatient1/10/2012Oct 04 - Jan 12

60 FOR OFFICIAL USE ONLY