1. Administering PennGroups Chris Hyzer ISC/ASTT August 1, 2011 8/24/2015ISC1

2. Administering PennGroups Install –DDL –API –UI –WS Source control Upgrade Maintenance Roadmap 8/24/2015ISC2

3. Administering PennGroups Demo server Internet2 has a Grouper Demo ServerGrouper Demo Server Address is: https://grouperdemo.internet2.edu/https://grouperdemo.internet2.edu/ Host various versions of Grouper Show features (e.g. permissions, external users, syncing between groupers) Allow users or potential users to kick the tires (not for production obviously)

4. Administering PennGroups Demo server for this training 9 accounts setup /home/test1, /home/test2, etc Each participant gets a temporary account and pass (ask Chris) The pass will work for SSH, Apache (UI, WS), mysql Each account has a mysql database, and user with access to that database

5. Administering PennGroups Demo server training tomcats Except for development, you should only have one application per Tomcat –One application wont take all memory and kill tomcat –You can restart after deploys without affecting other apps Note: for development, you can use either basic authentication –Or link cosign/shib tomcat on test cluster to dev tomcat

6. Administering PennGroups Demo server training tomcats (continued) Install dir: [test6@i2midev1 tomcat]$ pwd /home/test6/tomcat [test6@i2midev1 tomcat]$ ls conf logs temp webapps work TOMCAT_HOME, CATALINA_HOME elsewhere [test6@i2midev1 tomcat]$ more \ /etc/init.d/tomcat_test6 … export TOMCAT_HOME="/opt/tomcat6base" export CATALINA_HOME="/opt/tomcat6base"

7. Administering PennGroups Demo server training tomcats (continued) Note you have your own Java symlink (for ps) [test6@i2midev1 tomcat]$ more /etc/init.d/tomcat_test6 … export JAVA_HOME="/opt/javas/java_test6" Start/stop tomcat, lets run these commands (for your user) [test6@i2midev1 ~]$ tomcat start [test6@i2midev1 ~]$ ps -ef | grep java_test6 [test6@i2midev1 ~]$ netstat -pan | grep 5231 (note the port numbers) [test6@i2midev1 ~]$ jstack 5231 | less [test6@i2midev1 ~]$ tomcat stop [test6@i2midev1 ~]$ ps -ef | grep java_test6

8. Administering PennGroups Demo server training tomcats (continued) To set this up: $ sudo useradd -g i2mi test1 $ sudo passwd test1 - create user in DB, database assign permissions with sql yog mysql> create database grouper; mysql> create user 'grouper'@'localhost' identified by 'somesecret'; mysql> grant all on grouper.* to 'grouper'@'localhost'; $ sudo htpasswd /etc/httpd/conf.d/users.pass test1 $ sudo htpasswd /etc/httpd/conf.d/users.pass test1a $ cp -R /opt/tomcats/tomcat6bullet /home/test6/tomcat ports start at 9010, edit the ~/tomcat/conf/server.xml, set the three ports: http: 90x0, jk: 90x1, shutdown: 90x2 [root@i2midev1 init.d]# cp tomcat_k tomcat_test1 [root@i2midev1 init.d]# chkconfig --add tomcat_test1 [appadmin@i2midev1 javas]$ ln -s../java6 java_test1 [test1@i2midev1 test1]$ mkdir bin [test1@i2midev1 bin]$ ln -s /etc/init.d/tomcat_test1 tomcat

9. Administering PennGroups Demo server training tomcats (continued) Put this in /etc/profile JAVA_HOME=/opt/java6 export JAVA_HOME ANT_HOME=/opt/ant export ANT_HOME #note: maven3 is needed for grouper 2.0 M2_HOME=/opt/maven export M2_HOME M2=$M2_HOME/bin export M2 PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$M2:$PATH:$HOME/bin export PATH

10. Administering PennGroups Demo server training tomcats (continued) Put this in /var/www/html test1 Grouper UI Grouper WS Put this in /etc/httpd/conf.d/proxy_ajp.conf ProxyPass /test1_grouper/ ajp://localhost:9011/test1_grouper/ ProxyPass /test1_grouperWs/ ajp://localhost:9011/test1_grouperWs/ Bounce apache [root@i2midev1 init.d]# /sbin/service httpd configtest Syntax OK [root@i2midev1 init.d]# /sbin/service httpd graceful [root@i2midev1 init.d]#

11. Administering PennGroups Install Grouper API (First step in training) Download the API (1.6.3, latest stable) $ mkdir 1.6.3 $ cd 1.6.3 $ pwd /home/test1/1.6.3 $ wget http://www.internet2.edu/grouper/release/1.6.3/grouper.apiBinary-1.6.3.tar.gz $ tar xzvf grouper.apiBinary-1.6.3.tar.gz $ cd grouper.apiBinary-1.6.3/conf/ $ emacs grouper.properties

12. Administering PennGroups Install Grouper API (continued) Note, Powerpoint might mess-up chars (e.g. dashes)… might need to type them in if there is a problem, instead of copy/paste Note, a backslash: \ means the next line is the same line Note, all commands are in the "notes" section of the presentation, copy/paste from there

13. Administering PennGroups Emacs cheatsheet Note: Feel free to use your favorite editor obviously $ emacs somefile.whatever Note: might need to do CTRL-backspace instead of backspace Save: CTRL-x CTRL-s Exit: CTRL-x CTRL-c Find: CTRL-s Find (wrap): CTRL-s, put in criteria, CTRL-s Find backwards: CTRL-r Stop command: CTRL-g Replace: ESC-x query-replace (then “y” to replace each) Cut rest of line: CTRL-k Paste: CTRL-y Put in background: CTRL-z BASH get back: fg

14. Administering PennGroups Configure grouper.properties $ emacs grouper.properties groups.wheel.use = true configuration.autocreate.system.groups = true configuration.autocreate.group.name.0 = etc:webServiceClientUsers configuration.autocreate.group.description.0 = users allowed in WS configuration.autocreate.group.subjects.0 = GrouperSystem,testX,testXa configuration.autocreate.group.name.1 = etc:sysadmingroup configuration.autocreate.group.description.1 = sys admin users configuration.autocreate.group.subjects.1 = testX grouperIncludeExclude.use = true grouperIncludeExclude.requireGroups.use = true

15. Administering PennGroups Configure grouper.hibernate.properties $ emacs grouper.hibernate.properties hibernate.dialect = org.hibernate.dialect.MySQL5Dialect hibernate.connection.driver_class = com.mysql.jdbc.Driver hibernate.connection.url = jdbc:mysql://localhost:3306/testX hibernate.connection.username = testX hibernate.connection.password = *******************

16. Administering PennGroups Configure grouper-loader.properties $ emacs grouper-loader.properties loader.autoadd.typesAttributes = true

17. Administering PennGroups Init DB and startup GSH $ mysql -utest1 -p test1 mysql> show tables; Empty set (0.00 sec) mysql> exit Bye $ cd /home/testX/1.6.3/grouper.apiBinary-1.6.3/bin/ $./gsh.sh -registry $./gsh.sh -registry -check -runscript $./gsh.sh -registry -check $ mysql -utestX -p testX mysql> show tables; 76 rows in set (0.00 sec) mysql> exit Bye

18. Administering PennGroups Quickstart and subjects $ wget -O quickstart.xml http://anonsvn.internet2.edu/cgi-\ bin/viewvc.cgi/i2mi/tags/GROUPER_1_6_3/grouper-qs-\ builder/quickstart.xml?view=co $ wget -O subjects.sql http://anonsvn.internet2.edu/cgi-\ bin/viewvc.cgi/i2mi/tags/GROUPER_1_6_3/grouper-qs-\ builder/subjects.sql?view=co $./gsh.sh -registry -runsqlfile subjects.sql $./gsh.sh -xmlimportold GrouperSystem quickstart.xml

19. Administering PennGroups MySQL GUI Optional (if you have SQLYog free gui, or whatever) Port forward over SSH, remote port 3306, local port whatever (3302?) Connect to localhost 3302 In either case, lets open a GSH window, a mysql window, and a linux window

20. Administering PennGroups Add your users $./gsh.sh (note: testX was automatically inserted and into wheel and WS) gsh 0% grouperSession = GrouperSession.startRootSession(); gsh 1% addSubject("testX", "person", "John Smith"); gsh 2% addSubject("testXa", "person", "Johna Smitha"); gsh 3% addMember("etc:webServiceClientUsers", "testXa"); gsh 4% exit Review other GSH commandsGSH commands Note: GrouperSession is a ThreadLocal ActAs for the API

21. Administering PennGroups Add your users (continued) $ mysql -utestX -p testX INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\ VALUES ('testX', 'loginid', 'testX', 'testX'); INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\ VALUES ('testX', 'name', 'John Smith', 'john smith'); INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\ VALUES ('testX', 'description', 'John Smith', 'john smith'); INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\ VALUES ('testXa', 'loginid', 'testXa', 'testXa'); INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\ VALUES ('testXa', 'name', 'Johna Smitha', 'johna smitha'); INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\ VALUES ('testXa', 'description', 'Johna Smitha', 'johna smitha'); COMMIT; exit;

22. Administering PennGroups Get the UI $ cd ~/1.6.3 $ wget http://www.internet2.edu/grouper/release/1.6.3/grouper.ui-\ 1.6.3.tar.gz $ tar xzvf grouper.ui-1.6.3.tar.gz $ cd grouper.ui-1.6.3/ $ cp build.properties.template build.properties $ emacs build.properties grouper.folder=../grouper.apiBinary-1.6.3 should.copy.context.xml.to.metainf=false webapp.name=testX_grouper default.webapp.folder=/home/testX/tomcat/webapps/${webapp.name}

23. Administering PennGroups Get the UI (continued) $ emacs webapp/WEB-INF/web.core.xml From the bottom, remove security-contraint,login-config,security-role Note, if you are running on tomcat locally, leave that in, and edit tomcat-users.xml $ ant default $ tomcat restart Go to: https://grouperdemo.internet2.edu and click on your UI: https://grouperdemo.internet2.edu/testX_grouper/

24. Administering PennGroups UI authentication In this case, I added this to the httpd.conf or include: AuthType Basic AuthName "By Invitation Only" AuthUserFile /etc/httpd/conf.d/users.pass Require valid-user Bounce apache: [root@i2midev1 conf.d]# /sbin/service httpd configtest Syntax OK [root@i2midev1 conf.d]# /sbin/service httpd graceful Authentication is pluggable, can write a servlet filter to put in remote user Works with Shib, CAS, Cosign, Basic (web server), Basic (app server), any web server plugin, out of the box Note: if UI is internal users, but using external users via invites, might map multiple URLs to multiple authn schemes (like Penn or demo server)

25. Administering PennGroups Create objects Act as admin in upper right. Create a root folder named "test". (system and friendly) Inside that folder (aka stem), create a group called "testGroup", but uncheck allow all to "read" and "view". (system and friendly name is testGroup) Add some members to testGroup: babl, babr, babu

26. Administering PennGroups Get the WS $ cd ~/1.6.3 $ wget http://www.internet2.edu/grouper/release/1.6.3/grouper.ws-\ 1.6.3.tar.gz $ tar xzvf grouper.ws-1.6.3.tar.gz $ cd grouper.ws-1.6.3/grouper-ws $ emacs build.properties grouper.dir=../../grouper.apiBinary-1.6.3 webapp.name=testX_grouperWs $ emacs conf/grouper-ws.properties ws.client.user.group.name = etc:webServiceClientUsers $ emacs webapp/WEB-INF/web.xml From the bottom, remove security-contraint,login-config,security-role Note, if you are running on tomcat locally, leave that in, and edit tomcat-users.xml

27. Administering PennGroups Get the WS (continued) $ ant dist $ cp -R build/dist/testX_grouperWs ~/tomcat/webapps $ tomcat restart Go to: https://grouperdemo.internet2.edu and click on your WS: https://grouperdemo.internet2.edu/testX_grouperWs/servicesRest Should get error though there is a URL to get members…

28. Administering PennGroups WS authentication In this case, it’s the same as UI: AuthType Basic AuthName "By Invitation Only" AuthUserFile /etc/httpd/conf.d/users.pass Require valid-user Bounce apache: [root@i2midev1 conf.d]# /sbin/service httpd configtest Syntax OK [root@i2midev1 conf.d]# /sbin/service httpd graceful Authentication is pluggable, or can write a servlet filter to put in remote user Works with Kerberos, SOAP WS-security, Basic (web server), Basic (app server), any web server plugin, out of the box

29. Administering PennGroups Loader example Lets make a table with subjectIds in it mysql> CREATE OR REPLACE VIEW loader_employee AS \ (SELECT subjectId AS subject_id FROM testX.subject\ WHERE subjectId LIKE 'b%'); Make a folder for community under root Add a group called employee under folder "community" Edit the employee group, select type "grouperLoader" (make sure acting as admin in upper right of UI)

30. Administering PennGroups Loader example (continued) Edit attributes on the group: grouperLoaderDbName: grouper NOTE: configure other DB connections in grouper- loader.properties NOTE: every minute just for testing… grouperLoaderQuartzCron: 0 * * * * ? grouperLoaderQuery: select subject_id subject_id from \ loader_employee grouperLoaderScheduleType: CRON grouperLoaderType: SQL_SIMPLE

31. Administering PennGroups Loader example (continued) Run manually just to see it work: gsh 0% grouperSession = GrouperSession.startRootSession(); gsh 1% loaderGroup = GroupFinder.findByName(grouperSession, \ "community:employee"); gsh 2% loaderRunOneJob(loaderGroup); loader ran successfully, inserted 12 memberships, deleted 0 memberships, total membership count: 12 gsh 3%

32. Administering PennGroups Loader example (continued) Run continuously: %./gsh.sh -loader Change the view: mysql> CREATE OR REPLACE VIEW loader_employee AS \ (SELECT subjectId AS subject_id FROM SUBJECT \ WHERE subjectId LIKE 'b%o' or subjectId like 'el%'); At the top of the minute, check the memberships, and there should be some deleted, and some added

33. Administering PennGroups Loader details at Penn In the fastGrouperProdDaemon web application, we run the loader jobs in a FAST daemon All loader jobs are based on views, to ease maintenance Then we don’t have a command line application to monitor etc. The grouper_loader_log table has an entry for every daemon run Generally the only problem we have is Warehouse jobs for people without PennId’s, the people can’t be found, and it’s an error –The warehouse will assign a fake pennId which starts with 0 –Change the loader view to have where clause PENN_ID not like '0%'

34. Administering PennGroups Loader include/exclude example Create a group Read/update should not be granted to everyone Use addIncludeExclude type Look in folder, there will be 5 groups created with that type. Open the system of record, and lets make that the loader group Create this view in the DB: mysql> CREATE OR REPLACE VIEW loader_student AS \ (SELECT subjectId AS subject_id FROM SUBJECT WHERE \ subjectId LIKE 'fi%');

35. Administering PennGroups Loader include/exclude example (continued) Never edit the loader group, unless you expect it to get overwritten Add fico to the excludes group Add bapo to the includes group Look at the overall group Generally the privileges are: Assign READ on all to admins Assign UPDATE on include/exclude groups to admins Assign READ to service principal of app for overall group or other people who need to use the group

36. Administering PennGroups Customize UI text % cd ~/tomcat/webapps/testX_grouper/WEB-INF/classes/resources % mkdir custom % cd custom % touch media.properties % emacs nav.properties find.browse.here=Current folder is: % emacs init.properties default.module=grouper default.locale=en_US % cd ~/tomcat/webapps/testX_grouper/WEB-INF/classes/resources % emacs init.properties default.module=custom default.locale=en_US Bounce tomcat: % tomcat restart

37. Administering PennGroups Customize UI text (continued) Should see:

38. Administering PennGroups Customize lite UI for an application % cd ~/tomcat/webapps/testX_grouper/WEB-INF/classes % mkdir membershipLiteName % cd membershipLiteName % touch testName.properties % cd ~/tomcat/webapps/testX_grouper/WEB-\ INF/classes/resources % emacs custom/nav.properties Add line: membershipLiteName.testName.simpleMembershipUpdate.updateTitle = PTO\ admins Bounce tomcat: % tomcat restart https://grouperdemo.internet2.edu/testX_grouper/grouper\ Ui/appHtml/grouper.html?operation=SimpleMembership\ Update.init&groupName=apps:pto:ptoAdmins_systemOfRecord\ &membershipLiteName=testName

39. Administering PennGroups Customize lite UI for an application (continued)

40. Administering PennGroups Get the Grouper Client $ cd ~/1.6.3 $ wget \ http://www.internet2.edu/grouper/release/1.6.3/ \ grouper.clientBinary-1.6.3.tar.gz $ tar xzvf grouper.clientBinary-1.6.3.tar.gz $ cd grouper.clientBinary-1.6.3 $ emacs grouper.client.properties grouperClient.webService.url = \ https://grouperdemo.internet2.edu/testX_grouperWs/servicesRest grouperClient.webService.login = testX grouperClient.webService.password = **************

41. Administering PennGroups Get the Grouper Client (continued) Get usage: $ java -jar grouperClient.jar $ java -jar grouperClient.jar --operation=getMembersWs \ --groupNames=test:testGroup Customize the output: $ java -jar grouperClient.jar --operation=getMembersWs \ --groupNames=test:testGroup \ --outputTemplate='${wsSubject.id}$newline$'

42. Administering PennGroups Try from your local machine (win, mac, etc) NOTE: you need java6+ > cd c:\temp (or translate for mac or whatever) Download and unzip: http://www.internet2.edu/grouper/release/1.6.3/grouper.clientBinary -1.6.3.tar.gz > cd grouper.clientBinary-1.6.3 Edit grouper.client.properties grouperClient.webService.url = \ https://grouperdemo.internet2.edu/testX_grouperWs/servicesRest grouperClient.webService.login = testX grouperClient.webService.password = ************** Get usage: > java -jar grouperClient.jar > java -jar grouperClient.jar --operation=getMembersWs \ --groupNames=test:testGroup

43. Administering PennGroups Grouper deployment control at Penn See documentdocument The Grouper team hopes to have a maven version of this some time soon Everything in Penn’s CVS, external encrypted passwords, separated out filters, customizations Localdev, Dev, Test, Prod managed for custom application, WS, UI, client Checkout pennGrouper project from CVS and look at customizations

44. Administering PennGroups Grouper upgrades at Penn See document of 1.6 to 1.7 upgradedocument I believe we skipped 1.5, though we generally upgrade to new versions when available since we need features Generally we want everything to be up as much as possible in readonly mode Switch to readonly mode for WS/UI, stop updates on LDAP Get counts of important groups for sanity test Upgrade the DB (generally most time consuming part, if have to edit millions of rows) Upgrade the WS/UI, test everything Confirm counts of important groups

45. Administering PennGroups Grouper maintenance at Penn Process forms from DA, add kerberos principals Add new loader jobs Add new folders and delegate to new clients Consult on design of how applications use Grouper Look at errors emailed from Grouper log4j Run GSH scripts

46. Administering PennGroups Grouper sample GSH script See this documentationdocumentation E.g. will need to do this to delete old course groups

47. Administering PennGroups Daily report Email sent everyday to give a pulse of Grouper Need to deal with unresolvable subjects

48. Administering PennGroups Monitoring grouperWs has a status servlet hooked up to nagios /grouperWs/status?diagnosticType=all

49. Administering PennGroups Roadmap Hopefully uses for central permissions –E.g. warehouse permissions –E.g. PennCommunity Direct permissions Always available read-only web services Shibboleth entitlement group membership integration PennCommunity Direct getPerson WS secure attributes

50. Administering PennGroups Questions? 8/24/2015ISC50