An Authenticated Big Data Journey

An Authenticated Big Data Journey
September 27th 2017 Toni LeTempt – Senior Technical Expert Matt Bolte – Technical Expert © Wal-Mart Stores, Inc. All Rights Reserved Any reference in this presentation to any specific commercial product, process, or service, or the use of any trade, firm, or corporation name is for information and convenience only and is not an endorsement, favor, or recommendation by Wal-Mart Stores, Inc.

Agenda Introduction to Walmart Labs Retail
Hadoop Challenge at Walmart Labs Overview of Hadoop Security Timeline of Hadoop product to platform maturation Q&A © Wal-Mart Stores, Inc. All Rights Reserved

a division of Wal-Mart Stores, Inc.(WMT) Fortune 1
Who we are… a division of Wal-Mart Stores, Inc.(WMT) Fortune 1 260M customers weekly 11,504 stores, under 59 banners in 28 countries eCommerce websites in 11 countries Walmart Labs employs more than 4,000 worldwide Development centers in USA, Brazil, Dublin, and India What we do… Help people save money and live better Additional information about Walmart can be found on social media: © Wal-Mart Stores, Inc. All Rights Reserved

Hadoop at Walmart 2010: 2017: 1 Cluster +20 Clusters 12 Servers
Products 2010 2017 Sales Marketing Back Office Inventory IOT © Wal-Mart Stores, Inc. All Rights Reserved

Hadoop Challenge @ Walmart Labs
Balancing Accessibility and Security Kerberos Dog - Hadoop Elephant – “Beyond the 3 Vs: Where Is Big Data Now?” By Alex Woodie © Wal-Mart Stores, Inc. All Rights Reserved

In the beginning… How do we begin to transition Hadoop from application ownership to infrastructure ownership? How do we manage an ever-growing user base? How do we manage hundreds of users on hundreds of servers in multiple clusters? Increase in differing user needs and application complexity Hadoop to be managed as a platform. Hadoop growth results in multiple clusters. Introduction of Centrify management. 2012 CDH introduced and managed by application team (1 cluster/ a dozen servers) 2010 Hadoop growth in environment © Wal-Mart Stores, Inc. All Rights Reserved

Authentication and Authorization
Authentication comes from the Greek word (αὐθεντικός) authentikos, "real, genuine”) Kerberos is used to validate you are who you say you are! Authorization comes from the Latin word Auctor (master or leader). Centrify is used to verify what you are allowed to do in the cluster. © Wal-Mart Stores, Inc. All Rights Reserved

Kerberos Kerberos /ˈkərbərɒs/ is an authentication protocol that works on the basis of 'tickets' to allow nodes communicating over a non-secure network to prove their identity to one another in a secure manner. The protocol was named after the character Kerberos (or Cerberus) from Greek mythology, the ferocious three-headed guard dog of Hades (hellhound) Kerberos Image - © Wal-Mart Stores, Inc. All Rights Reserved

Kerberos In a Hadoop Cluster without authentication a user may change their identity to other users and even super users such as HDFS. Look up ‘insecure hadoop filesystems’ to read more about the danger. Kerberos Image - © Wal-Mart Stores, Inc. All Rights Reserved

Key Distribution Center
Kerberos and Hadoop Key Distribution Center Hadoop_<CLUSTER1>.REALM.COM CLUSTER1 HADOOP NODE End User © Wal-Mart Stores, Inc. All Rights Reserved

Kerberos and Hadoop <kdc_path>kdc.conf [kdcdefaults] kdc_ports = 88,750 kdc_tcp_ports = 88,750 supported_enctypes = aes128-cts-hmac-sha1-96:normal aes256-cts-hmac-sha1-96:normal permitted_enctypes = aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha1-96 [realms] HADOOP_<CLUSTER1>.REALM.COM = { acl_file = /<kdc_path>/kadm5.acl dict_file = /<dictionary_path>/words admin_keytab = /kdc_path/kadm5.keytab max_renewable_life = 7d supported_enctypes = aes128-cts-hmac-sha1-96:normal aes256-cts-hmac-sha1-96:normal permitted_enctypes = aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha1-96 } Key Distribution Center Hadoop_<CLUSTER1>.REALM.COM CLUSTER1 HADOOP NODE End User © Wal-Mart Stores, Inc. All Rights Reserved

Kerberos and Hadoop KRB5_CONFIG=<nonstandard_path>/krb5.conf [libdefaults] udp_preference_limit = 1 default_realm = HADOOP_CLUSTER1.REALM.COM ticket_lifetime = 24h renew_lifetime = 7d forwardable = true default_tkt_enctypes = aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha1-96 default_tgs_enctypes = aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha1-96 permitted_enctypes = aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha1-96 [realms] HADOOP_CLUSTER1.REALM.COM = { kdc = <server_name>:88 admin_server = <server_name>:749 default_domain = hadoop_cluster1.realm.com } [logging] kdc = SYSLOG:INFO admin_server = FILE=/<logpath>/kadm5.log Key Distribution Center Hadoop_<CLUSTER1>.REALM.COM CLUSTER1 HADOOP NODE End User © Wal-Mart Stores, Inc. All Rights Reserved

Kerberos in Operation Key Distribution Center
Client Key Distribution Center Authentication Server (AS) Ticket Granting Server (TGS) 1. AS Request 2. AS Reply 3. TGS Request 4. TGS Reply Hadoop Cluster1 5. AP Request 6. AP Reply © Wal-Mart Stores, Inc. All Rights Reserved

Hadoop Centrify Zones Universal Zone Hadoop Cluster
UID management here Hadoop Cluster Top of Zone Role Assignments Hadoop Cluster A Hadoop Cluster B Computer Role Hadoop A Role Assignments (Ad Groups for Login) Hadoop ..... Computer Role Hadoop B Role Assignments (Ad Groups for Login) Hadoop ..... Cluster C Cluster D Cluster E © Wal-Mart Stores, Inc. All Rights Reserved

Access To Hadoop Clusters @ Walmart Labs
Access to Walmart Labs Hadoop clusters is controlled via a combination of Centrify/Active Directory and Kerberos. A user opens a request to add their ID to the Hadoop Cluster's login Centrify group Once added, the ID should resolve on all nodes in the cluster Users will then be able to access cluster edge nodes, BUT NOT ACCESS HADOOP. An automated process then adds the user to local KDC and generates a user keytab © Wal-Mart Stores, Inc. All Rights Reserved

Authentication To Hadoop Clusters @ Walmart Labs
To access Hadoop the user will have to kinit to the cluster. This process is essentially synonymous with connecting with a password. kinit –kt $USER.keytab This kerberos ticket will allow access to ONLY that Hadoop cluster. However, they still cannot access data if they have not been added to the correct AD groups. © Wal-Mart Stores, Inc. All Rights Reserved

Authorization @ Walmart Labs
User/Group permissions at the HDFS level Ex: /hive/warehouse/marketbasket.db/salestable owned by user1:group1 with permissions 750 (User id RWX, Group RX, Universal no access). drwxr-x--- - user1:group :13 /hive/warehouse/marketbasket.db/salestable Users in group1 R&X the <salestable> but only user1 update. Both user1 and group1 provisioned via Centrify/AD. All user ids and groups provisioned via Centrify/AD. The only local ids are cluster system ids(HDFS, Hive, Yarn, etc ...) Application teams request their own ad groups and manage their own HDFS object permissions. HDFS default permissions are configured so objects are not automatically created with Universal Read but application teams can and do at times over ride this setting. © Wal-Mart Stores, Inc. All Rights Reserved

Manual setup of Kerberos.
Migrated off CDH. Manual setup of Kerberos. First Cross Realm Trust to the Homeoffice KDCs. How do we continue to make cluster access easier for users? Security request 2FA for Hue. Need for Disaster Recovery. Implemented Hadoop user automated provisioning process using Centrify/AD and the local Hadoop Clusters’ KDCs. 2015 Migrate to Pivotal Hadoop. Migrate users to keytabs. Many to Many Cross Realm Trusts ODBC connections from workstations and servers Increase in differing user needs and application complexity 2014 2012 Hadoop brought into engineering area - managed as a platform. Multiple Hadoop clusters Centrify Introduced and tied to AD to manage cluster access. 2010 CDH introduced and managed by application teams Hadoop growth in environment © Wal-Mart Stores, Inc. All Rights Reserved

The User Experience of Windows MIT Kerberos.
Users had to follow a series of complicated steps to connect their workstations to our kerberized clusters: If you are connecting from a Windows based machine make sure you have followed the steps detailed in: Setting Up Windows MIT Kerberos for ODBC Connections to Hive. Download and Install Windows MIT Kerberos Release 4 64-Bit. From the browser go to the URL download site> Click Request Software and search for kerberos. Select the correct software and proceed to checkout. Once installed replace the contents of the C:\ProgramData\MIT\Kerberos5\krb5.ini with the contents of the <Nonstandard Path>krb5.conf file from the cluster you want to connect to. Edit the krb5.ini to add this default_ccache_name line to the [libdefaults] section. The directory you specify must be one you have permissions to update. default_ccache_name=c:\temp\krb5ccache FTP your keytab (or the keytab of the user that will be connecting to Hive) to the windows machine that Windows MIT Kerberos has been installed on. Make sure the keytab is saved in a secure directory that is not accessible to the public. kinit from the windows based machine: Using a command line window, verify you have a valid credential: klist -l Once you have kinit’d and have a valid ticket Download an ODBC Hive Driver. The current recommendation is the HortonWorks ODBC Driver. From the browser go to the enterprise download page. Click Request Software and search for Hortonworks Select the HortonworksODBC64 driver and checkout. Configure the connection as shown (image removed for privacy) Test connection Only after completing all this could a user connect to Hadoop from their own machine. © Wal-Mart Stores, Inc. All Rights Reserved

AD/Centrify environment
CROSS_REALM TRUST WITH KERBEROS User and Group Identify always resolved to Enterprise AD Default /etc/krb5.conf always resolves to Enterprise realm Enterprise AD/Centrify environment Additional values in local krb5.conf: Realm defined for other cluster and HDFS NNs identified each cluster. Non-standard krb5.conf controls cluster KDCs. /(local secure directory)/krb5.conf Cluster 1 Cluster 2 Tickets in local KDC: Tickets in local KDC: To enable cross-realm trust, each cluster must know the identify of the other cluster and also have a ticket (krbtgt) allowing authentication © Wal-Mart Stores, Inc. All Rights Reserved

Removed weak encryption from all clusters.
2017 Removed weak encryption from all clusters. Begin Elimination of local KDC Implemented Apache Ranger to provide even more granular access control in our Secure Data Lake clusters. Migration from Pivotal to HortonWorks How to get the Hadoop team out of the business of managing user authentication? More Business Users querying Hadoop via ODBC and Kerberos is still a difficulty for these connections, what can be done about it? Security requirement to remove weak encryptions. KDC Single Point of Failure. 2016 Implement new Centrify zone management models for all clusters Migrate clusters to HortonWorks Data Platform Simplified business user access. Plan moving user authentication to HO realm. Hue 2FA implemented 2015 Increase in differing user needs and application complexity Implemented Hadoop user automated provisioning process using Centrify/AD and the local Hadoop Clusters’ KDCs. 2014 Migrate to Pivotal Hadoop. Migrate users to keytabs. Many to Many Cross Realm Trusts ODBC connections from workstations and servers. 2012 Hadoop to engineering area – multiple clusters – Introduced Centrify. 2010 CDH introduced and managed by application teams Hadoop growth in environment © Wal-Mart Stores, Inc. All Rights Reserved

Kerberos Encryption Problem:
Weak Encryption Types from Original Hadoop 2010 Build Update supported encryption types to only include aes128-cts-hmac-sha1-96 and aes256-cts-hmac-sha1-96 in 1 quarter Change Encryption Types minimum impact Could not break the many to many cross realm trusts defined. Could not take full environment outage (All Dev/Prod) © Wal-Mart Stores, Inc. All Rights Reserved

Kerberos Encryption Solution:
Added permitted_enctypes to the krb5.conf and kdc.conf to include the weaker encryptions until the encryption change was applied to all of the clusters. Changed these parameters to the stronger encryption types. Then dropped and recreated all of the non-krbtgt principals and keytabs. default_tkt_enctypes – krb5.conf default_tgs_enctypes – krb5.conf supported_enctypes – kdc.conf Rolled encryption change one cluster at a time. © Wal-Mart Stores, Inc. All Rights Reserved

Walmart Labs Hadoop Security Pain Points
Kerberos – User Experience. Problems integrating third party products. Cluster Management. Centrify/Identity management – User IDs resolve inconsistently . UIDs suddenly and unexpectedly changing. Users added to groups but still not resolving in the cluster. © Wal-Mart Stores, Inc. All Rights Reserved

Elimination of weak encryption.
Elimination of local KDCs to reduce complexity and duplication of identity management Implemented Apache Ranger In Summary 2017 Migrate to HortonWorks Data Platform Continue integration with AD and Centrify to improve user access and easy of use for Hadoop 2FA Hue Access 2016 2015 Implemented Hadoop user automated provisioning process using Centrify/AD and the local Hadoop Clusters’ KDCs. Increase in differing user needs and application complexity 2014 Migrate to Pivotal Hadoop. Migrate users to keytabs. Many to Many Cross Realm Trusts ODBC connections from servers and workstations 2012 Hadoop to engineering area – multiple clusters – Introduced Centrify. 2010 CDH introduced and managed by application teams Hadoop growth in environment © Wal-Mart Stores, Inc. All Rights Reserved

Our End Goal Positive Hadoop Experience for both End Users and Admins!
© Wal-Mart Stores, Inc. All Rights Reserved

Our Recommendations Avoid Multiple migrations in short periods of time
Use standard security implementations (KDC/conf files) Level-set expectations of non-technical users © Wal-Mart Stores, Inc. All Rights Reserved

Work Cited Slide 5: Kerberos Dog - Hadoop Elephant – “Beyond the 3 Vs: Where Is Big Data Now?” By Alex Woodie , Slide 7: Authentication Authorization Kerberos Image - Centrify Image - Slide 8: Slide 9: Kerberos Image - Slide 13: Text in Notes from: and Slide 14: Slide 20: Nervous User Image - Conquering Nerves - Slide 24: “Lightbulbs of Fire”, Slide 27: Happy Users Image - “CLIPART FOR KIDS IMAGE #45”, Slide 34: “Kerberos Protocol”, Slide 35,36: “Kerberos Encryption” - © Wal-Mart Stores, Inc. All Rights Reserved

Key Kerberos Terms KDC – Key Distribution Center
Krb5.conf – Default Kerberos configuration File. @Walmart Labs krb5.conf is NOT currently in the standard install path of /etc. Nonstandard install has caused numerous issues Third party products often expect (hard coded) for the krb5.conf to be in /etc Administration Maintenance Principal – A unique identity to which Kerberos assigns tickets. (userid) For Hadoop we use our own naming standard. Hold over from our “legacy 2010” environment. Keytab – A file containing the principal and it’s encrypted password. Kinit - obtains and caches an initial ticket-granting ticket for principal Klist – Lists the Kerberos principals and tickets held in cache or the keys held in a keytab file. KRB Cache File – Location of the default Kerberos 5 credentials cache Kdestroy – Utility that destroys the user’s active kerberos authorization tickets Kadmin – Command line interface Kdb5_util – utility used by admin to perform maintenance on the KDC database © Wal-Mart Stores, Inc. All Rights Reserved

KDC The client authenticates itself to the Authentication Server (AS) which forwards the username to a key distribution center (KDC). The KDC issues a ticket- granting ticket (TGT), which is time stamped, encrypts it using the user's password and returns the encrypted result to the user's workstation. This is done infrequently, typically at user logon; the TGT expires at some point, though may be transparently renewed by the user's session manager while they are logged in. When the client needs to communicate with another node ("principal" in Kerberos parlance) the client sends the TGT to the ticket-granting service (TGS), which usually shares the same host as the KDC. After verifying the TGT is valid and the user is permitted to access the requested service, the TGS issues a ticket and session keys, which are returned to the client. The client then sends the ticket to the service server (SS) along with its service request. Reference diagram on Slide 11 for visual © Wal-Mart Stores, Inc. All Rights Reserved

Kerberos Encryption Kerberos can use a variety of cipher algorithms to protect data. A Kerberos encryption type (also known as an enctype) is a specific combination of a cipher algorithm with an integrity algorithm to provide both confidentiality and integrity to data.” Clients make two types of requests (KDC-REQ) to the KDC: AS-REQs and TGS-REQs. The client uses the AS-REQ to obtain initial tickets (typically a Ticket-Granting Ticket (TGT)), and uses the TGS-REQ to obtain service tickets. The KDC uses three different keys when issuing a ticket to a client: The long-term key of the service: the KDC uses this to encrypt the actual service ticket. The KDC only uses the first long-term key in the most recent kvno for this purpose. The session key: the KDC randomly chooses this key and places one copy inside the ticket and the other copy inside the encrypted part of the reply. The reply-encrypting key: the KDC uses this to encrypt the reply it sends to the client. For AS replies, this is a long-term key of the client principal. For TGS replies, this is either the session key of the authenticating ticket, or a subsession key. Each of these keys is of a specific enctype. Each request type allows the client to submit a list of enctypes that it is willing to accept. For the AS-REQ, this list affects both the session key selection and the reply-encrypting key selection. For the TGS-REQ, this list only affects the session key selection. - © Wal-Mart Stores, Inc. All Rights Reserved

default_tkt_enctypes controls the default set of enctypes that the Kerberos client library requests when making an AS-REQ. Do not set this unless required for specific backward compatibility purposes; stale values of this setting can prevent clients from taking advantage of new stronger enctypes when the libraries are upgraded. (From Slide 11 Step 1) default_tgs_enctypes controls the default set of enctypes that the Kerberos client library requests when making a TGS-REQ. Do not set this unless required for specific backward compatibility purposes; stale values of this setting can prevent clients from taking advantage of new stronger enctypes when the libraries are upgraded. . (From Slide 11 Step 3) supported_enctypes controls the default set of enctype-salttype pairs that kadmind will use for generating long-term keys, either randomly or from passwords permitted_enctypes controls the set of enctypes that a service will accept as session keys. © Wal-Mart Stores, Inc. All Rights Reserved

Bringing Hadoop to Walmart
2010 – CDH introduced to Walmart (by an application team). Decisions made for Kerberos due to existing enterprise environment. Existing AD environment (early Centrify installations at Walmart didn’t support AD environments and Hadoop Kerberos design) Result: separate KDC for Hadoop cluster – AD still running against its own KDC backend. NOTE: we only use Kerberos for authentication to HDFS layers, we still depend on Linux security and AD for authorization. © Wal-Mart Stores, Inc. All Rights Reserved

Growing Hadoop and Security
– Growing Hadoop environment results in multiple Hadoop clusters, each with its own KDC. Start of using Centrify to manage access to the clusters, but is rudimentary implementation. Results in several pain points Create user identities in Kerberos, and assign a password to the user. – problem because we then had to provide the password to the user. Hadoop admin team responsible for user passwords for authentication Admin issues - Users forgot passwords and had to have them reset. CDH managing Kerberos in background helped to hide some issues that we would experience when we moved away from Cloudera. Service principle management was all contained within CDH and the Admin team didn’t have to do that. Cross-realm trust was all self-contained within CDH CDH also managed auth_to_local values in *-site.xml files for us. © Wal-Mart Stores, Inc. All Rights Reserved

2014 – Implement Pivotal Hadoop and have to begin managing detailed pieces of Kerberos configurations. We realized how much CDH had been doing in the background for us. We had to start managing Kerberos outside of the Pivotal software in order to gain stability and consistent operation. Had to learn the hard way to be very specific with host values and service keytabs. (ex. Spnego containing all servers vs just local host) Migrate from using passwords for users to generating keytabs for users. Allows Hadoop Admin team to get out of password management This solved our password involvement but did open us to centrify identity issues. (ownership of the files changing) Begin setting up trust between Hadoop KDC realms and Homeoffice realms. Uncovered performance issues and configuration management issues between our own Kerberos implementation and the larger AD Kerberos implementation (separate configuration files, etc as mentioned above) Worked with users to connect ODBC/JDBC connections to Hadoop (pre-knox) © Wal-Mart Stores, Inc. All Rights Reserved

Begin designing new model of Centrify layout to include nested groups.
2015 – Begin designing new model of Centrify layout to include nested groups. Implement fully automated user add scripts to speed up user access. Tied access to specific AD groups – and use Centrify to tie those groups to our KDC. One overall ‘zone’ and each cluster being a computer role. Authority granted to AD group and AD group tied to the computer roles. Query AD group periodically to identify new users, add those users to the KDC, generate keytabs for the new users, and deploy the keytabs to the edge nodes. Tying to AD groups also provided ability to revoke users who are no longer valid members of the group. Security states that two factor authentication is required for Hue Access. © Wal-Mart Stores, Inc. All Rights Reserved

Implement new Centrify zone management models for all clusters
2016 – Implement new Centrify zone management models for all clusters Help ‘hide’ the backend structures from the end users. Worked with application teams to shield business from infrastructure models. Simplify what the business users need to know to use the platform. We tell the application teams what AD group to use for cluster access. The application team can then manage their own AD groups and associations in Centrify to grant access to the necessary clusters. Begin planning to relocate individual authentication into the home office central AD realm. Automated Windows MIT Kerberos package for user – selectable from our Enterprise download page Implemented Knox to allow near seamless connection of ODBC/JDBC connections to local KDC realms. Implemented 2FA for Hue © Wal-Mart Stores, Inc. All Rights Reserved

Provide user with “frictionless” access to Hadoop.
2017 – Had to remove weak encryption definitions from all of the Hadoop clusters. This required a complete recreate of all Kerberos Principals and Keytabs. Still had to maintain connectivity between clusters while implementing the encryption changes in a rolling outage method. I.E. Changed a Development cluster but it still had to be able to communicate to the Prod. Implemented Apache Ranger to provide even more granular access control in our Secure Data Lake clusters. Local KDC High Availability. Fully Integrate Hadoop into the Home Office Realm. Why? Automated authorization and authentication managed from a single source Elimination of local KDCs to reduce complexity and duplication of identity management while providing a seamless High Availability for the authentication process. Provide user with “frictionless” access to Hadoop. Why aren’t we done yet? Requires resources from multiple teams with differing objectives and deliverables. So in the interim we implemented local KDC High Availability. This is a temporary setup until we can fully integrate Hadoop to the Homeoffce AD Realm. © Wal-Mart Stores, Inc. All Rights Reserved

An Authenticated Big Data Journey

Similar presentations

Presentation on theme: "An Authenticated Big Data Journey"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

An Authenticated Big Data Journey

Similar presentations

Presentation on theme: "An Authenticated Big Data Journey"— Presentation transcript:

Similar presentations

About project

Feedback