1. Using Shibboleth as an SSO
University at Buffalo
Joel Murphy
Copyright Joel W Murphy, This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2. overview Id mgmt @ UB UB deployment Timeline Deployment considerations
Certificate management
LDAP
Load Balancer
Web Farm and Service Providers
Webmail proposed configuration
Cosign
Shibboleth Identity Provider
Load Testing
Challenges

3. Id Mgmt @ UB “UBIT Name” branding
All University affiliated persons (faculty, staff, students, etc.)
single username/password, many directories, many applications
usernames and passwords and groups synced across directories
One user namespace for campus
People retain same username after returning to Univ
Unix groups primary mechanism for authorization (authz) and entitlements
Applications responsible for authorization

4. Id UB
“Accounts Management” oracle database and application core to Id Mgmt
Home grown Perl, C, oracle SQL
Downstream Directories
DCE – authentication, authorization, dir svcs
LDAP – dir svcs, authn via Kerberos, authz
Kerberos – authn, enables encrypted authenticated communication channels
Active Directory
X.509 Certificates

5. Password management password replication across directories
Password changes work in AD when logged in via Kerberos cross realm trust
Web form available for password changing

6. Long road to SSO Pre-1997 – Unix passwords/NIS – just Unix services
DCE – first success at consolidation of Id mgmt and password database, multiple platforms
1998 – DCE/DFS deployed as enterprise filesystem
2003 – Kerberos/Active Directory/LDAP/Shibboleth/Cosign/Cisco CSS
Summer 2005 – production Oracle RAC database
Fall 2005 – First production Shibboleth Services – Course Management, Download service
Fall LDAP Campus Portal and administrative apps (was DCE)
Summer 2006 – Network Appliance deployment for DFS replacement for enterprise filesystem (CIFS/NFS)
Summer 2006 – Business Continuity site deployment and machine move
Fall 2006 – Shibbolize authn portions of Campus web server
Fall 2006 – Shibbolize Campus Portal and apps (from LDAP)
Fall 2006 or Spring 2007 – Shibbolize Web Mail (IMP

7. Id Mgmt/SSO Outliers
Enterprise Active Directory “exception” accounts (eg. OU administrators) – not in Kerberos/LDAP
Science and Engineering NIS passwd (migrating to enterprise Kerberos for auth)
Various Active Directory Forests outside Enterprise AD
Various applications with limited technical resources or need for Enterprise integration

8. Id Mgmt challenges
Authorization not always done or works by “accident” (user not in /etc/passwd)
Application owners don’t always know who should be authorized – typically request a group
DCE still core for to IdM
Authoritative for groups and data for deactivated users
RPCs use heavily, replacing with SOAP/SSL
CIO level requests
Better authorization for network access
Online password reset
Requests for LDAP access for outsourced apps
Hard to get resources for not-so-low hanging fruit

9. Some Shibboleth/SSO and Web Farm Goals/Considerations
Survive network split (multiple campuses)
Don’t preclude logoff
Ability Load balance commodity hardware
Detect and Isolate failures - Transparent failover of Identity Providers and Service Providers (multiple machines, same configuration)
Capacity/Responsiveness – Shibboleth is Enterprise SSO
Commitments made to sell Shibboleth solution

10. How we got there Internal certificate signing service
LDAP environment for our attribute data (eduPerson, UBEduPerson)
Cisco CSS (Content Switch) load balancer
Multi-purpose Oracle RAC database (“Real Application cluster” - active/active)
Hitachi SAN
Cosign SSO
Shibboleth
Lots of glue
Lots of help and contributions from UB and I2 community

11. Did we make it? Survive network split (multiple campuses)
No – multiple load balancers, but can’t migrate IPs across subnets, opted against DNS delegation
Business continuity site not on one of the campuses (no people)
Don’t preclude logoff
Supported in Cosign, limited Shib 2.0 support?
Detect and Isolate failures - Ability Load balance commodity hardware
Large farm of Sun 280R Solaris servers for admin applications
Farm of 8 Dell 1750 Linux servers for shibboleth/cosign
4 Sun 280R Solaris servers for LDAP
4 Sun V120 Solaris servers for Kerberos
Transparent failover of Identity Providers and Service Providers
Yes, except for failure mid-transaction
Capacity/Responsiveness – Shibboleth is Enterprise SSO
Yes, our auth systems are very under capacity
Adding 2 additional LDAP servers (new/bad applications with unpredictable performance profiles)

13. Physical Layout

14. Shibboleth IDP Setup InCommon member
Apache 1.3/Tomcat connector/Tomcat
Handle Service authenticated by Cosign, trusted by Shib
Still at version 1.2 IDP (origin)
Hindsight: put IDP source tree in CVS
Web Farm setup requires Shib Handle configured as “CryptoHandleGenerator” instead of shared memory
All Attributes currently resolved via LDAP (from Person objects)
Non-web apps may have similar needs
Entitlements are currently mapped from group memberships and stored in LDAP
Once configured, IDP changes very infrequently
Attribute release is negotiated when configuring new targets
Meta-data for Federations (sites and trusts) changes periodically
We continue to deploy all configuration in one “war” file.
Log files are somewhat large and unwieldy
No deterministic policies, data custodians authorize attribute release

15. Application Web Farm/Shibboleth SP Setup
Apache 1.3
Caching apache proxy in front of application apache
Shibboleth integrated into application apache
Apache proxy/cache runs SSL
Machines load balanced with Cisco CSS
Currently assume UB IdP for applications
Standard attribute release and acceptance
Multiple services (virtual hosts) on one machine
Haven’t needed to adjust ARP for Virtual Hosts
One certificate and ARP shared by all web farm machines
Modified MySQL cache to be Shibboleth Oracle shar cache
Sticky connections on CSS makes unnecessary
Was one source of early stability problems

16. Application Web Farm/Shibboleth SP Issues
Canonical Service names
Forcing Canonical host names (UseCanonicalName) makes IDP configuration of AssertionConsumerServiceURL simple.
Prevents dreaded error message from Handle Service (unknown ACS URL…)
Makes it very difficult for developers or sysadmins to poke a specific machine (use hosts file)
Alternative – list every possible hostname/short name in meta-data?
Web cache
Cookie based authentication doesn’t prevent caching of data where HTTP BasicAuth does automatically
Need to set header in response of authenticated pages: “Cache-Control: private"
Web proxy (URL rewrite)
We needed to set checkAddress=“false” in shibboleth XML
ShibURLScheme, ServerName and Port needed for apache
Shibboleth needs to know how to write a “return address” for redirects
Multiple virtual hosts, odd port mapping schemes allow us to run proxy and server on same hosts

18. Shibboleth SP Issues
Provide a “UB” shibboleth distribution and instructions
Build out of date, XML config, metadata and instructions mostly valid
Where to put certs
What startup files are needed
What should be monitored
What settings to tweak in a web farm environment
We generally recommend rebuilding apache module for local apache
Unnecessary?
New versions of shib from I2 come as packages and are significantly simpler than the past

19. Blackboard/Course Management
Shibboleth just used for SSO (authentication)
Nightly data processing feeds Blackboard system data from institutional systems
Creates users object in Blackboard
Manages course registrations

20. Download Service Shibboleth used as SSO
Group membership used for authorization
Authorization based on licensing
Microsoft software download

21. Campus Portal Shibboleth to be used as SSO
Current phase is to transition from LDAP auth (was DCE)
Nightly processing generates data about people authorized portal in portal application

22. Webmail Proposal Cyrus IMAP email
Would like to link Campus Portal/Webmail/Course mangement
Multiple possibilities
Integrate with cosign directly and use Kerberized IMAP session
Shibbolize webmail and use secure fabricated password for auth
Trusted webmail
Pass Kerberos ticket via Shibboleth Attribute???
Upcoming Shibboleth solution for n-tier apps???
Concerns/Issues
Desire to abstract out Cosign
Fabricated password must not be reproducible or usable via normal IMAP connections
Fabricated password requires maintaining a password file
Kerberizing IMAP possibly more Webmail customization than tinkering with password

23. Webmail Proposal
Current working proposal is to use shib with fabricated passwords

24. Shibboleth Authorization
Our group management is inflexible (DCE still authoritative)
Working to remove DCE, would like Grouper (groups of groups?)
Business rules don’t always match single group membership
Needed to match sets of groups
Require membership in active and some staff group:
ShibRequireAll on
require memberof ~ ^…staff$
require memberof ~ ^active$
Require membership in active or some staff group:
ShibRequireAll off
Shib1.3b adds XML access control, nested requirements
Using mod_setenvif/mod_access does not work with Shibboleth environment variables
Shibboleth module doesn’t have a chance to generate env variables before mod_access and order is Apache hard coded, not dependant on module order

25. Cosign Setup SSO just for Shibboleth Handle Service
Cosign supports logoff (hidden)
Cosign can pass Kerberos tickets and do certificate based auth
SSO’s at the time of our deployment did not support logon server redundancy.
Cosign writes state information to disk, extended this to write state to Oracle.
Oracle transactions through an external server (cookie_server)

26. Cosign Issues Custom code makes upgrading difficult
Backported fixes from newer cosigns
Bugs in our oracle interface
Oracle RAC configured Active/Active, but updates not synchronous
Needed max_commit_propagation_delay=0 on Oracle RAC instances
Newer cosign versions have async replication of state
Could loose some state in logon server failure
Move state files from Oracle RAC to HA Netapp NFS share?

27. Cosign diagram

28. LDAP Environment 4 load balanced Sun ONE 5.2 Directory Servers
One giant directory
ou=People, dc=buffalo, dc=edu
posixAccount
Person
OrganizationalPerson
InetOrgPerson
eduPerson
UBEduPerson
ou=Groups, dc=buffalo, dc=edu
posixGroup
groupOfUniqueNames
Added institutional IDs and UBEPmemberOf to UBEduPerson
memberOf on Person makes shib Attribute release simple and fast
Future deployments should use eduMember schema

29. LDAP Environment (cont)
Hourly processing updates LDAP with changes from Id mgmt database (Perl Net::LDAP)
Oracle layout
One row per principal
One column per attribute, multi-valued attributes glued with a separator
Goal to co-locate non-institutional data in directory
Available via LDAP or via Shibboleth
Allow administrators to manage source oracle table or write changes to LDAP
All Access management policy at the root of the directory granting to groups, some based on attribute value (FERPA)
Avoids confusion by avoiding object acls or deny acls
Multi-master replication – supports application writes with load balancing
No passwords – simple binds proxied to Kerberos with a modified version of preAuth plugin by Mike Gettes
Forces simple binds over SSL

30. LDAP Issues Large groups don’t perform well, use memberOf
Can’t “browse” the directory – administrative limits
Last modified time
DS 5.1 didn’t support multi-master replication
Introduced memory leak in Kerberos pre-auth plugin when porting to DS 5.2
Replication of schemas created redundant definitions (fixed?)
Replication failures with large directory and lots of changes
Fixes for bugs and included in DS 5.2 patch 4
Turned multi-master back on to support LDAP applications needing to write data

31. Internal Certificate Management
Manages certs for test servers, shibboleth SP’s and VPN access
Openssl on Id mgmt system
requests via
Daily job for monitoring expirations
No audit trail and little metadata associated with certs
Not particularly safe for multiple administrators
Tightening access/clarifying procedures
Plan on moving to OpenCA

32. Load Balancer Setup
Began with Cisco Local Directors for system load balancing (SMTP/IMAP/POP)
Current - Two Cisco CSS 11503, active/standby
Load balanced services on private network behind load balancers, using normal UB address space
Servers see IP address of Clients
NAT, health monitoring, sticky connections, load balancing
Standard architecture for all UB load balanced services (LDAP, Shibboleth, Cosign, web farm, etc.)
Ability to directly connect to a specific machine in a pool

33. Load Balancer Issues Services see the IP of client connecting
Can easily roll-in/deploy and test changes with no downtime
Network management of private net
Everything on one subnet
Allows migrating IPs between CSS
Extended subnet to Bus. Cont. site, but not other campus sites
Danger of private net connecting to public
Load Balanced services talking to other load balanced services
Must NAT both source and destination host
Sysadmin Management
Too many services, anyone can take everything down
Shared serial console (HTTP disabled for security/auditing concerns)
Many ports on many machines
Secure web tool in development
Monitoring services, removing from pool
CSS can monitor HTTP results for testing server health
Notification for down servers (hooked into Big Brother)

34. Load Balancer example

35. Troubleshooting Check our Big Brother console for alerts
Check Load balancer – what’s active?
Application Service provider
Check access_log/error_log
Check Shibboleth shar.log and shire.log
Is shar (shibd) running?
Can we connect to Shib AA with openssl s_client?

36. Troubleshooting (cont)
Identity Provider
Are apache/cosignd running and accepting connections?
Is cookie_server running?
Check all apache logs
Check tomcat bootstrap log – did shib start?
Check shib application logs

37. Load Testing Webload (Radview) JMeter Perl LWP SAR (Linux/Solaris)
Infrastructure survives intense load testing
Shibboleth initial logon “costs” lessen with longer sessions (perhaps less in some cases?)

38. Challenges Transitioning into production Too much change at one time
Project team not all of support team
Too much change at one time
Service maturity takes time
Lots of layers
Lots of machines and services
Troubleshooting complex
Errors don’t go travel upstream well
Multiple support teams
Help Desk support complex, especially for Fed apps