Using Shibboleth as an SSO University at Buffalo Joel Murphy jmurphy@buffalo.edu Copyright Joel W Murphy, 2006. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

overview Id mgmt @ UB UB deployment Timeline Deployment considerations Certificate management LDAP Load Balancer Web Farm and Service Providers Webmail proposed configuration Cosign Shibboleth Identity Provider Load Testing Challenges

Id Mgmt @ UB “UBIT Name” branding All University affiliated persons (faculty, staff, students, etc.) single username/password, many directories, many applications usernames and passwords and groups synced across directories One user namespace for campus People retain same username after returning to Univ Unix groups primary mechanism for authorization (authz) and entitlements Applications responsible for authorization

Id Mgmt @ UB “Accounts Management” oracle database and application core to Id Mgmt Home grown Perl, C, oracle SQL Downstream Directories DCE – authentication, authorization, dir svcs LDAP – dir svcs, authn via Kerberos, authz Kerberos – authn, enables encrypted authenticated communication channels Active Directory X.509 Certificates

Password management password replication across directories Password changes work in AD when logged in via Kerberos cross realm trust Web form available for password changing

Long road to SSO Pre-1997 – Unix passwords/NIS – just Unix services 1997 - DCE – first success at consolidation of Id mgmt and password database, multiple platforms 1998 – DCE/DFS deployed as enterprise filesystem 2003 – Kerberos/Active Directory/LDAP/Shibboleth/Cosign/Cisco CSS Summer 2005 – production Oracle RAC database Fall 2005 – First production Shibboleth Services – Course Management, Download service Fall 2005 - LDAP Campus Portal and administrative apps (was DCE) Summer 2006 – Network Appliance deployment for DFS replacement for enterprise filesystem (CIFS/NFS) Summer 2006 – Business Continuity site deployment and machine move Fall 2006 – Shibbolize authn portions of Campus web server Fall 2006 – Shibbolize Campus Portal and apps (from LDAP) Fall 2006 or Spring 2007 – Shibbolize Web Mail (IMP or @mail)

Id Mgmt/SSO Outliers Enterprise Active Directory “exception” accounts (eg. OU administrators) – not in Kerberos/LDAP Science and Engineering NIS passwd (migrating to enterprise Kerberos for auth) Various Active Directory Forests outside Enterprise AD Various applications with limited technical resources or need for Enterprise integration

Id Mgmt challenges Authorization not always done or works by “accident” (user not in /etc/passwd) Application owners don’t always know who should be authorized – typically request a group DCE still core for to IdM Authoritative for groups and data for deactivated users RPCs use heavily, replacing with SOAP/SSL CIO level requests Better authorization for network access Online password reset Requests for LDAP access for outsourced apps Hard to get resources for not-so-low hanging fruit

Some Shibboleth/SSO and Web Farm Goals/Considerations Survive network split (multiple campuses) Don’t preclude logoff Ability Load balance commodity hardware Detect and Isolate failures - Transparent failover of Identity Providers and Service Providers (multiple machines, same configuration) Capacity/Responsiveness – Shibboleth is Enterprise SSO Commitments made to sell Shibboleth solution

How we got there Internal certificate signing service LDAP environment for our attribute data (eduPerson, UBEduPerson) Cisco CSS (Content Switch) load balancer Multi-purpose Oracle RAC database (“Real Application cluster” - active/active) Hitachi SAN Cosign SSO Shibboleth Lots of glue Lots of help and contributions from UB and I2 community

Did we make it? Survive network split (multiple campuses) No – multiple load balancers, but can’t migrate IPs across subnets, opted against DNS delegation Business continuity site not on one of the campuses (no people) Don’t preclude logoff Supported in Cosign, limited Shib 2.0 support? Detect and Isolate failures - Ability Load balance commodity hardware Large farm of Sun 280R Solaris servers for admin applications Farm of 8 Dell 1750 Linux servers for shibboleth/cosign 4 Sun 280R Solaris servers for LDAP 4 Sun V120 Solaris servers for Kerberos Transparent failover of Identity Providers and Service Providers Yes, except for failure mid-transaction Capacity/Responsiveness – Shibboleth is Enterprise SSO Yes, our auth systems are very under capacity Adding 2 additional LDAP servers (new/bad applications with unpredictable performance profiles)

Physical Layout

Shibboleth IDP Setup InCommon member Apache 1.3/Tomcat connector/Tomcat Handle Service authenticated by Cosign, trusted by Shib Still at version 1.2 IDP (origin) Hindsight: put IDP source tree in CVS Web Farm setup requires Shib Handle configured as “CryptoHandleGenerator” instead of shared memory All Attributes currently resolved via LDAP (from Person objects) Non-web apps may have similar needs Entitlements are currently mapped from group memberships and stored in LDAP Once configured, IDP changes very infrequently Attribute release is negotiated when configuring new targets Meta-data for Federations (sites and trusts) changes periodically We continue to deploy all configuration in one “war” file. Log files are somewhat large and unwieldy No deterministic policies, data custodians authorize attribute release

Application Web Farm/Shibboleth SP Setup Apache 1.3 Caching apache proxy in front of application apache Shibboleth integrated into application apache Apache proxy/cache runs SSL Machines load balanced with Cisco CSS Currently assume UB IdP for applications Standard attribute release and acceptance Multiple services (virtual hosts) on one machine Haven’t needed to adjust ARP for Virtual Hosts One certificate and ARP shared by all web farm machines Modified MySQL cache to be Shibboleth Oracle shar cache Sticky connections on CSS makes unnecessary Was one source of early stability problems

Application Web Farm/Shibboleth SP Issues Canonical Service names Forcing Canonical host names (UseCanonicalName) makes IDP configuration of AssertionConsumerServiceURL simple. Prevents dreaded error message from Handle Service (unknown ACS URL…) Makes it very difficult for developers or sysadmins to poke a specific machine (use hosts file) Alternative – list every possible hostname/short name in meta-data? Web cache Cookie based authentication doesn’t prevent caching of data where HTTP BasicAuth does automatically Need to set header in response of authenticated pages: “Cache-Control: private" Web proxy (URL rewrite) We needed to set checkAddress=“false” in shibboleth XML ShibURLScheme, ServerName and Port needed for apache Shibboleth needs to know how to write a “return address” for redirects Multiple virtual hosts, odd port mapping schemes allow us to run proxy and server on same hosts

Shibboleth SP Issues Provide a “UB” shibboleth distribution and instructions Build out of date, XML config, metadata and instructions mostly valid Where to put certs What startup files are needed What should be monitored What settings to tweak in a web farm environment We generally recommend rebuilding apache module for local apache Unnecessary? New versions of shib from I2 come as packages and are significantly simpler than the past

Blackboard/Course Management Shibboleth just used for SSO (authentication) Nightly data processing feeds Blackboard system data from institutional systems Creates users object in Blackboard Manages course registrations

Download Service Shibboleth used as SSO Group membership used for authorization Authorization based on licensing Microsoft software download

Campus Portal Shibboleth to be used as SSO Current phase is to transition from LDAP auth (was DCE) Nightly processing generates data about people authorized portal in portal application

Webmail Proposal Cyrus IMAP email Would like to link Campus Portal/Webmail/Course mangement Multiple possibilities Integrate with cosign directly and use Kerberized IMAP session Shibbolize webmail and use secure fabricated password for auth Trusted webmail Pass Kerberos ticket via Shibboleth Attribute??? Upcoming Shibboleth solution for n-tier apps??? Concerns/Issues Desire to abstract out Cosign Fabricated password must not be reproducible or usable via normal IMAP connections Fabricated password requires maintaining a password file Kerberizing IMAP possibly more Webmail customization than tinkering with password

Webmail Proposal Current working proposal is to use shib with fabricated passwords

Shibboleth Authorization Our group management is inflexible (DCE still authoritative) Working to remove DCE, would like Grouper (groups of groups?) Business rules don’t always match single group membership Needed to match sets of groups Require membership in active and some staff group: ShibRequireAll on require memberof ~ ^…staff$ require memberof ~ ^active$ Require membership in active or some staff group: ShibRequireAll off Shib1.3b adds XML access control, nested requirements https://authdev.it.ohio-state.edu/twiki/bin/view/Shibboleth/XMLAccessControl Using mod_setenvif/mod_access does not work with Shibboleth environment variables Shibboleth module doesn’t have a chance to generate env variables before mod_access and order is Apache hard coded, not dependant on module order

Cosign Setup SSO just for Shibboleth Handle Service Cosign supports logoff (hidden) Cosign can pass Kerberos tickets and do certificate based auth SSO’s at the time of our deployment did not support logon server redundancy. Cosign writes state information to disk, extended this to write state to Oracle. Oracle transactions through an external server (cookie_server)

Cosign Issues Custom code makes upgrading difficult Backported fixes from newer cosigns Bugs in our oracle interface Oracle RAC configured Active/Active, but updates not synchronous Needed max_commit_propagation_delay=0 on Oracle RAC instances Newer cosign versions have async replication of state Could loose some state in logon server failure Move state files from Oracle RAC to HA Netapp NFS share?

Cosign diagram

LDAP Environment 4 load balanced Sun ONE 5.2 Directory Servers One giant directory ou=People, dc=buffalo, dc=edu posixAccount Person OrganizationalPerson InetOrgPerson eduPerson UBEduPerson ou=Groups, dc=buffalo, dc=edu posixGroup groupOfUniqueNames Added institutional IDs and UBEPmemberOf to UBEduPerson memberOf on Person makes shib Attribute release simple and fast Future deployments should use eduMember schema http://middleware.internet2.edu/dir/docs/internet2-mace-dir-ldap-group-membership-200507.html

LDAP Environment (cont) Hourly processing updates LDAP with changes from Id mgmt database (Perl Net::LDAP) Oracle layout One row per principal One column per attribute, multi-valued attributes glued with a separator Goal to co-locate non-institutional data in directory Available via LDAP or via Shibboleth Allow administrators to manage source oracle table or write changes to LDAP All Access management policy at the root of the directory granting to groups, some based on attribute value (FERPA) Avoids confusion by avoiding object acls or deny acls Multi-master replication – supports application writes with load balancing No passwords – simple binds proxied to Kerberos with a modified version of preAuth plugin by Mike Gettes Forces simple binds over SSL

LDAP Issues Large groups don’t perform well, use memberOf Can’t “browse” the directory – administrative limits Last modified time DS 5.1 didn’t support multi-master replication Introduced memory leak in Kerberos pre-auth plugin when porting to DS 5.2 Replication of schemas created redundant definitions (fixed?) Replication failures with large directory and lots of changes Fixes for bugs 6242420 and 6283717 included in DS 5.2 patch 4 Turned multi-master back on to support LDAP applications needing to write data

Internal Certificate Management Manages certs for test servers, shibboleth SP’s and VPN access Openssl on Id mgmt system requests via email Daily job for monitoring expirations No audit trail and little metadata associated with certs Not particularly safe for multiple administrators Tightening access/clarifying procedures Plan on moving to OpenCA

Load Balancer Setup Began with Cisco Local Directors for email system load balancing (SMTP/IMAP/POP) Current - Two Cisco CSS 11503, active/standby Load balanced services on private network behind load balancers, using normal UB address space Servers see IP address of Clients NAT, health monitoring, sticky connections, load balancing Standard architecture for all UB load balanced services (LDAP, Shibboleth, Cosign, web farm, etc.) Ability to directly connect to a specific machine in a pool

Load Balancer Issues Services see the IP of client connecting Can easily roll-in/deploy and test changes with no downtime Network management of private net Everything on one subnet Allows migrating IPs between CSS Extended subnet to Bus. Cont. site, but not other campus sites Danger of private net connecting to public Load Balanced services talking to other load balanced services Must NAT both source and destination host Sysadmin Management Too many services, anyone can take everything down Shared serial console (HTTP disabled for security/auditing concerns) Many ports on many machines Secure web tool in development Monitoring services, removing from pool CSS can monitor HTTP results for testing server health Notification for down servers (hooked into Big Brother)

Load Balancer example

Troubleshooting Check our Big Brother console for alerts Check Load balancer – what’s active? Application Service provider Check access_log/error_log Check Shibboleth shar.log and shire.log Is shar (shibd) running? Can we connect to Shib AA with openssl s_client?

Troubleshooting (cont) Identity Provider Are apache/cosignd running and accepting connections? Is cookie_server running? Check all apache logs Check tomcat bootstrap log – did shib start? Check shib application logs

Load Testing Webload (Radview) JMeter Perl LWP SAR (Linux/Solaris) Infrastructure survives intense load testing Shibboleth initial logon “costs” lessen with longer sessions (perhaps less in some cases?)

Challenges Transitioning into production Too much change at one time Project team not all of support team Too much change at one time Service maturity takes time Lots of layers Lots of machines and services Troubleshooting complex Errors don’t go travel upstream well Multiple support teams Help Desk support complex, especially for Fed apps