Role Prediction Using Electronic Medical Record System Audits Wen Zhang 1, Carl Gunter 3, David Liebovitz 4, Jian Tian 1, Bradley Malin 1,2 1 Dept. of Electrical Engineering & Computer Science, Vanderbilt University 2 Dept. of Biomedical Informatics, Vanderbilt University 3 Dept. of Computer Science, University of Illinois at Urbana Champaign 4 Dept. of Medicine, Northwestern University 1

Misuse of EMR Systems is Real Medical center employees misuse medical record systems to breach privacy WhenWhereWho 2007Palisades Medical CenterGeorge Clooney 2011UCLAVarious Celebrities 2 HIPAA Security Rule  Access to EMRs should be limited The problem is not limited to celebrity snooping But how?

Challenges to Security in EMRs Basic security principle: –Least privilege –Separation of duty Access control technologies have been around since the 1970’s Information systems often provide role-based access control (RBAC) capability [1] –Privileges mapped roles –Users mapped to privileges Roles are hard to define, so EMR systems often provide broad access rights 3 [1] R.Sandhu, E.Coyne, H.Feinstein and C.Youman. IEEE computer

In “Rare” Cases – Break the Glass A user may not sufficient access rights to perform job This model allows users to temporarily escalate privilege Access is logged and reviewed by administrator May require user to specify “reason” for access 4

Rare Cases? Central Norway Health Region enabled break the glass 53,000 of 99,000 patients (54.5%)  broken glass 5,000 of 12,000 users (42.7%)  broke the glass Over 295,000 logged breakage events in one month Role UsersInvoked Glass Breaks in Past Month Nurse563336% Doctor292752% Health Secretary187652% Physiotherapist38256% Psychologist19458% 5 [3] L. Røstad and N. Øystein. Proceedings of the 2 nd International Conference on Availability, Reliability and Security (ARES)

Idea! Refine Access Control Based on Behavior Experience-based Access Management (EBAM) Combine static knowledge (RBAC) with actual actions (access logs) and organizational knowledge for feedback control 6 RBAC EMR Access Logs Medical Center Knowledge Experience- Based Access Management [2] [2] C.Gunter, D.Liebovitz, B.Malin. IEEE Security and Privacy Magazine

Use audit logs to predict if a user is associated with a role Goals: –Determine if expert-defined job titles are reasonable –Provide administrators with a better idea of how to refine roles The Role Prediction Problem for EBAM Doctor Nurse Role Classifier Biller …. 7 Access Reason Medical Service Location of Patient

UserPatientTimeServiceUser Position (Role)ReasonLocation u1u1 p1p1 8/4/10OBSTETRICSNMH Physician Office - CPOEAttending Phys/ProvWard A u2u2 p2p2 12/14/10OBSTETRICSNMH Physician - CPOEPatient CareWard A u 23 p3p3 12/14/10PEDIATRICSUnit Secretary 2Unit Secretary OrdersWard B Evaluation with Cerner EMR of Northwestern Memorial Hospital Represent users as vectors Statistics 8 UsersRolesReasonsServicesLocations Example audit logs

To assist in role management, we worked with organization experts to build a hierarchy (specialized to Northwestern) Optimization Tradeoff: Goal 1: Accuracy (should increase as we step up in hierarchy) Goal 2: Separation of Duty (will increase as we step down) Leveraging Role Hierarchies Employee Doctor Specific Clinician Dietitian Junior Dietitian Senior Dietitian PhysicianNurse … … … … … … General (62 roles) Conceptual (5 roles) Specific (140 roles) 9

Basis of a “Role-Up” Algorithm General idea: Audit roles at different levels of the hierarchy 1.Score each role in conceptual position & general position 2.Select role with the highest score & generalize its children 3.Repeat 1 & 2 until a threshold score is reached 10 Allow administrators to balance between the prediction accuracy and separation of duties (number of roles)

Balanced Scoring Function R measures the extent to which specificity could be kept by the node A measures the extent to which predictablity could be achieved by the node 11

Employee Doctor Specific Clinician Dietary Junior Dietician Senior Dietician PhysicianNurse Nurse 1Nurse 2 Physician 2 Physician α = 0.5, Threshold =

Employee Doctor Specific Clinician Dietary Junior Dietician Senior Dietician PhysicianNurse Nurse 1Nurse α = 0.5, Threshold = 0.4

Employee Doctor Specific Clinician DietaryNurse Nurse 1 Nurse 2 After one iteration, the role set is {Doctor, Nurse 1, Nurse 2, Dietary} 14 α = 0.5, Threshold = 0.4

Training & Testing at the Same Level of the Role Hierarchy Employee Specific Clinician Nurse Nurse 1 15 Conceptual General Specific 82.38% 52.45% 51.34% Accuracy Level

Distribution of Accuracy Over the Role Hierarchy 16

RankRoleAccuracyUsers 1 (tie)AP-Technologist100%54 1 (tie)ED Assistant100%26 1 (tie)ED NMH Physician-CPOE100%43 1 (tie)NMH Resident/Fellow ID Clinic-CPOE100%10 1 (tie)Patient Care Staff Nurse – Lactation100%14 17 Most Predictable Roles

Least Predictable Roles RankRoleAccuracyUsers 140Patient Care Staff Nurse7.6% Rehab OT14.3%28 138Transfer20.0%20 137View Only PC 321.4%14 136Patient Care Staff Nurse (Pilot)22.1%217 18

Number of Users in the Role Can Influence Accuracy 19

Case Study: Most Likely Mispredictions for Patient Care Staff Nurse Predicted RolePrediction Patient Care Staff Nurse - Lactation19.6% View Only PC 114.3% Radiology – Nurse14.0% Patient Care Staff Nurse (Pilot)10.4% SN-RN/Customer Service5.8% 20

Original RolePredicted RoleProbability Rehab OTRehab PT85.7% Patient Care Staff Nurse - Agency Patient Care Staff Nurse - Lactation 75.0% Rehab PTRehab OT60.0% View Only PC 3 Patient Care Staff Nurse - Lactation 50.0% Medical Records - Scanner Medical Records47.4% 21 Most Likely Mispredictions

Parameter Bias Trades Between Accuracy and Separation of Duty Biased toward Accuracy: number of roles is small (27) accuracy is highest (63%) 22  0.1… Number of Roles Recommended 27…6064 Accuracy of Role Predictions 63.3%…51.8%51.3% Biased toward Specificity: number of roles is high (60) accuracy is lower (52%)

Conclusion and Future Plans 23 EHR audit logs can be analyzed to determine if the users’ behaviors are consistent with their designated job titles Role hierarchies enable automatic discovery of appropriate levels of role management Plan to expand Role-“up” to allow for Role-“down” and Role-“over” Need to evaluate Role-up with real hospital administrators, to assess its usability and acceptance of results

Acknowledgements National Science Foundation –CCF –CNS National Library of Medicine –R01-LM Office of the National Coordinator for HIT –SHARPS (sharps.org) 24

Questions? 25