How to Solve BigData Security Puzzle?

Two Reasons for Security in BigData 1 Hadoop Contains Sensitive Data As Hadoop adoption grows so too has the types of data organizations look to store. Often the data is proprietary or personal and it must be protected. In this context, Hadoop is governed by the same security requirements as any data center platform. Hadoop is subject to Compliance adherence Organizations are often subject to comply with regulations such as HIPAA, PHI, PCI, DSS, FISAM that require protection of personal information. Adherence to other Corporate security policies. 2

BigData Security: Key Aspects Centralized Security Administration Kerberos API security Authentication Who am I/prove it? Authorization What can I do? Fine grain access control Centralized audit reporting Audit What did I do? Wire encryption in Hadoop Native and partner encryption Data Protection Can data be encrypted at rest and over the wire?

Authentication: Kerberos Provides Strong Authentication Prevents impersonation on unauthorized account Supports token delegation model Works with existing directory services Basis for Authorization

API Security Pattern Hadoop Cluster Application Tier App A App N App B App C Data Ingest ETL Admin/ Operators Bastian Node SSH RPC Call Falcon Oozie Scoop Flume Data Operator Business User JDBC/ODBC REST/HTTPS API Gateway

API Security …. Why we need it? Challenges/Limitations Extend reach of Hadoop APIs to Anyone on Any device Enterprise authentication Apply Enterprise capabilities to All REST APIs – IdM Integration, SSO, OAuth, SAML Avoid exposing Cluster port, hostnames to all users Challenges/Limitations Not suitable for heavy data ingestion activities Supports specific services WebHDFS (HDFS) Templeton (HCatalog) Stargate (HBase) Oozie Hive/JDBC Yarn RM Storm

Authorization and Audit Fine grain access control HDFS – Folder, File Hive – Database, Table, Column HBase – Table, Column Family, Column Storm, Knox and more Audit Extensive user access auditing in HDFS, Hive and Hbase etc. IP Address Resource type/ resource Timestamp Access granted or denied Flexibility in defining policies Monitoring through Auditing

Authorization Apache Ranger Delivers a ‘single pane of glass’ for the security administrator Centralizes administration of security policy Ensures consistent coverage across the entire Hadoop stack

Audit

Data Protection in Hadoop must be applied at three different layers in Apache Hadoop Storage: encrypt data while it is at rest Direct data flows “into” and “out of” 3rd party encryption tools and/or rely upon hardware specific techniques (i.e. drive-level encryption). Transmission: encrypt data as it is in motion Native Apache Hadoop 2.0 provides wire encryption. Upon Access: apply restrictions when accessed Direct data flows “into” and “out of” 3rd party encryption tools.

Data Protection: Data at Rest Encryption Data converted to binary Ciphertext using mathematical algorithm. Can be one-way (Hash) or reversible (Symmetric/Asymmetric). Tokenization Real data is replaced with randomly generated characters of same data type.

Data Protection: Transmission WebHDFS Provides read/write access to HDFS Optionally enable HTTPS Authenticated using SPNEGO (Kerberos for HTTP) filter RPC Communications between NNs, DNs, etc. and Clients SASL based wire encryption JDBC/ODBC SSL based wire encryption Shuffle Mapper to Reducer over HTTP(S) with SSL

Data Protection: Upon Access Hadoop Ecosystem KMS Encryption server/agent Ingestion stream Hadoop Component UDFs (e.g. HIVE) Encrypt/ Decrypt Reporting HDFS File/Directory encryption – HDFS Column level encryption – Hadoop Ecosystem components or ingestion framework Data tokenization – Encryption server/agent

Infrastructure Security A physically zoned off Data lake using Firewall with network segmentation Data lake cluster nodes running on specific ports open & IP tables turned on SSSD or Centrify for users and groups OS Hardening Security scans to shut down non-essentials services on individual hosts. Up-to date patching levels applied to the server.

Enterprise-wide Authorization Policy Management AD is identity store with existing user and group provisioning process as prescribed at an enterprise level. Active Directory Group & User Group & user sync Enterprise Policy authorization Software Integration API Hadoop Policy Server DBMS Enterprise Policy authorization software is the enterprise level repository for defining policies across all applications managed under this umbrella for a seamless authorization experience based on role and group participations in AD. Policy store Hadoop Component Hadoop Ecosystem

Application Access Security Pattern LDAP Sync users/groups from LDAP Policy Manager SSL SSL SSL 2.Knox Authenticates user/pass 5. Ranger AuthZ HDFS HiveServer 2 1.Original request w/user id/password Gateway Security 6.Hive creates map reduce using NN ST 4.Knox calls as proxy user SASL A B C SSL SSL Client gets query result Use Hive ST, submit query O/JDBC Client Hive gets Namenode (NN) service ticket At rest data encryption option at HDFS level 3.Knox gets service ticket for Hive KDC

Lessons Learned .. Start Early Think Enterprise Engage Experts Collaborate and Socialize Manage Vulnerabilities http://www.apache.org/security/committers.html Monitor, Report and Audit