1. Java Web Service Servers and Clients in Internet2 Grouper
February 2009
Chris Hyzer
University of Pennsylvania IT
Internet2

2. Purpose of this presentation
Show how Internet2 Grouper developed web services
Grouper is open source, feel free to borrow or suggest improvements
Discuss the successes and areas for improvement
Mention planned enhancements
4/13/2017
University of Pennsylvania

3. University of Pennsylvania
Contents
Introduction to Internet2 Grouper
Introduction to web services
Architecture of REST/SOAP web service
SOAP web services with Axis2 (servers and clients)
REST web services with xstream (servers and clients)
Client
Documenting web services
Bonus material
Security
Axis serving Rest
Testing
4/13/2017
University of Pennsylvania

4. Introduction to Internet2 Grouper
4/13/2017
University of Pennsylvania

5. University of Pennsylvania
Internet2 Grouper
Open source group management
Internet2 has been working on group management for 8 years
Generally used in educational institutions, but could be anywhere
Funded by Internet2
4/13/2017
University of Pennsylvania

6. Why central group management with Grouper?
Instead of apps managing own groups
Reuse group lists
Central place to see which groups a person is in
Central auditing of group and membership actions
Central management of authorization
Security:
Who can view/edit groups and memberships
Opt-in/Opt-out
Delegate authority
Automatic or manual membership management
Composite groups for group match: and / or / minus
Groups of groups
4/13/2017
University of Pennsylvania

7. University of Pennsylvania
Grouper architecture
Note: the one-way arrow doesn’t mean the traffic originates from one side or the other. E.g. no ldap reads originate from the ldap server.
4/13/2017
University of Pennsylvania

8. Introduction to WEB SERVICES
4/13/2017
University of Pennsylvania

9. University of Pennsylvania
User web request
Person using browser makes a request to a server
Person (user) views the results in browser, and types and or clicks to continue
4/13/2017
University of Pennsylvania

10. University of Pennsylvania
Web service request
Program makes a request to a web application
Program parses the output
Note, it doesn’t have to be a server which makes a web service request, it could be any program, e.g. ajax.
4/13/2017
University of Pennsylvania

11. Overlap of web request and web service?
Ajax for example
Can be kicked off by a user click
Can update the screen similar to a web application
However, Ajax is making the request and parsing the response, it is a web service
If it doesn’t parse the output, and just puts the resultant HTML into the browser DOM, then not a web service
Web service screen scraping a web application
A program can “screen scrape” a web application
Beware of changes in the HTML!
This is not a web service
Is a browser an application making requests, are all user requests web services?
4/13/2017
University of Pennsylvania

12. University of Pennsylvania
Why web services
http(s) is a well understood protocol by programming languages and programmers
Ports 80/443 might already be available in firewall rules
Http is text based (easy to debug)
Http is not programming language specific, so the server technology can be different than the client (e.g. ajax)
Webpages are either XML, XHTML, or XML-like (e.g. HTML)
Most programming languages have XML libraries
Note: web services do not have to be XML, though generally the are
Development and production environments might be similar (or same) to existing web applications
Penn generally communicates between systems with WS
4/13/2017
University of Pennsylvania

13. University of Pennsylvania
SOAP web services
Simple Object Access Protocol
Specifies how web service messages are exchanged
W3C standard
Must use XML and XML schema for data
Messages have XML envelopes, headers, body, exception handling
Web Service Description Language (WSDL) describes the SOAP messages in a programmatic way (XML)
Many features (security, error handling, caching, resource discovery, etc)
Many programming languages generate SOAP
Not considered light-weight
4/13/2017
University of Pennsylvania

14. University of Pennsylvania
Example SOAP request
<?xml version='1.0' encoding='UTF-8'?> <soapenv:Envelope xmlns:soapenv=" <soapenv:Body> <ns1:addMemberLite xmlns:ns1=" <ns1:groupName>aStem:aGroup</ns1:groupName> <ns1:subjectIdentifier>mchyzer</ns1:subjectIdentifier> </ns1:addMemberLite> </soapenv:Body> </soapenv:Envelope>
4/13/2017
University of Pennsylvania

15. University of Pennsylvania
REST web services
Representational State Transfer
Two definitions:
(strict or RESTful): Protocol that specifies how HTTP (perhaps) and XML are used for web services
(non-strict): Any web service that does not have the overhead of SOAP. Aka Remote Procedure Call (RPC)
4/13/2017
University of Pennsylvania

16. University of Pennsylvania
RESTful web services
The web services are organized like static web resources
URL’s represent resources, not operations
HTTP methods indicate the operations. Generally: GET, POST (update), PUT (insert), DELETE. Can use more, or custom
Messages can be HTML so that systems or browsers can consume them
4/13/2017
University of Pennsylvania

17. Example REST web service
PUT /grouperWs/servicesRest/xhtml/v1_4_000/groups/aStem%3AaGroup/members/mchyzer
This means add this member to the group
Note, there can be body here, though in this case it isn’t needed
4/13/2017
University of Pennsylvania

18. HTTP / XML / RPC / Hybrid web services
Each service makes its own standards
URL can be:
Resource:
Operation:
Generally just use POST or GET as the HTTP method
The XML document sent can be:
Complete object representation: <group><name>myGroup</name><desc>MyGroup</desc></group>
Operational: <groupChangeName><groupId>123</groupId> <newName>myGroup2</newName></groupChangeName>
4/13/2017
University of Pennsylvania

19. University of Pennsylvania
GROUPER WEB SERVICES
4/13/2017
University of Pennsylvania

20. University of Pennsylvania
Grouper web services
Generally web services are programmed to host a service
Grouper is software, so its WS are programmed so institutions can download and host services
E.g. Grouper is app server and database server agnostic
Requirements
Dozen operations
SOAP and REST (as close to RESTful as possible)
SOAP and REST should deploy in one webapp
Simple operations (Lite), and batched operations
Pluggable authentication
Documented well
Versioned (generally Grouper has bi-annual releases)
4/13/2017
University of Pennsylvania

21. Grouper web service operations
findGroups
findStems
stemSave
stemDelete
memberChangeSubject
getGrouperPrivileges
assignGrouperPrivileges
addMember
deleteMember
getMembers
hasMember
getGroups
groupSave
groupDelete
4/13/2017
University of Pennsylvania

22. University of Pennsylvania
Lite vs batched
One batched operation has less overhead than many smaller operations (performance test for yourself to validate)
Benchmarks
Add group (inserts), requires many queries
100 batches of 1 add groups take 19.8 seconds
10 batches of 10 add groups take 12.4 seconds
5 batches of 20 add groups take 12.0 seconds
Has member, lightweight, readonly
100 batches of 1 hasMember checks take 8.3 seconds
10 batches of 10 checks take 1.9 seconds
5 batches of 20 checks take 1.6 seconds
Benchmarks notes
Completely run on developer PC, local mysql
E.g. WsSampleHasMemberRest100
4/13/2017
University of Pennsylvania

23. University of Pennsylvania
Object model
Assume web service data are simple POJOs (Plain Old Java Objects)
Use only:
Beans
Arrays (of beans or simple types)
Simple types: String, Integer
Note:
Don’t use Collections, enums, dates, timestamps, booleans
4/13/2017
University of Pennsylvania

24. Object model (continued)
What makes up the data of a group?
Here is a simple group
4/13/2017
University of Pennsylvania

25. Object model (continued)
What makes up the data of a group?
Here is a more complex group
4/13/2017
University of Pennsylvania

26. Object model (continued)
4/13/2017
University of Pennsylvania

27. Object model (continued)
Request object model
4/13/2017
University of Pennsylvania

28. Object model (continued)
Response object model
4/13/2017
University of Pennsylvania

29. Object model (continued)
Metadata object model
Response metadata is one per response
Result metadata is one per Lite response, or one per each line item in batch
Success: T|F
Result code: many enums
Codes also in HTTP headers
4/13/2017
University of Pennsylvania

30. Object model (continued)
Operations should be idempotent if possible
If they are sent twice, generally it is ok
Delete member mchyzer from group etc:sysAdminGroup
Idempotent
Delete the first member of group etc:sysAdminGroup
NOT idempotent
4/13/2017
University of Pennsylvania 30 30

31. SOAP web services with Axis2
4/13/2017
University of Pennsylvania

32. University of Pennsylvania
Axis architecture
Axis can do many things, but how Grouper uses Axis is:
Servlet to accept web requests and call Grouper’s business logic
Generates WSDL from business logic
Generates sample client library from WSDL
4/13/2017
University of Pennsylvania

33. Business logic for Axis
Create a class (GrouperService)
Contains only instance methods of business logic
Each method takes all fields of the input bean, and returns the output bean
Each bean is only simple pojo (uses Javabean properties)
Note the Lite methods only take scalars are inputs
4/13/2017
University of Pennsylvania

34. Business logic for Axis (continued)
4/13/2017
University of Pennsylvania

35. Business logic for Axis (continued)
GrouperService isn’t great for Javadoc since enums are strings
Delegate to GrouperServiceLogic
Decode booleans, dates, enums, etc
4/13/2017
University of Pennsylvania

36. University of Pennsylvania
Axis generate WSDL
Generate WSDL from POJOs and GrouperService class
<target name="java2wsdl" description="convert the java to a wsdl">
<touch file="${generated.client.project.dir}/GrouperService.wsdl" />
<delete file="${generated.client.project.dir}/GrouperService.wsdl" />
<java classname="org.apache.ws.java2wsdl.Java2WSDL" fork="true">
<classpath refid="ws.class.path" />
<arg value="-o" /><arg value="${generated.client.project.dir}" />
<arg value="-of" /><arg value="GrouperService.wsdl" />
<arg value="-cn" />
<arg value="edu.internet2.middleware.grouper.ws.soap.GrouperService" />
<arg value="-stn" />
<arg value=" />
<arg value="-l" />
<arg value=" />
</target>
4/13/2017
University of Pennsylvania

37. Axis generate WSDL (continued)
Generate WSDL from POJOs and GrouperService class
C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws>ant java2wsdl
Buildfile: build.xml
java2wsdl:
[delete] Deleting: C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws-java
-generated-client\GrouperService.wsdl
BUILD SUCCESSFUL
Total time: 9 seconds
C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws>
4/13/2017
University of Pennsylvania

38. Axis generate WSDL (continued)
Result
4/13/2017
University of Pennsylvania

39. Axis generate WSDL (continued)
Result – 2000 lines of SOAP definition
This might be idealistic, but Grouper is trying not to touch this WSDL file, and only use the one generated by Axis.
Note this is in CVS so we can diff revisions, and it is versioned (e.g. get the WSDL from a particular Grouper branch or version).
Features in WSDL which are not in Axis generated WSDL are not being used, e.g. enums.
4/13/2017
University of Pennsylvania

40. University of Pennsylvania
Axis generate client
Ant script to generate SOAP client from WSDL (any WSDL)
<target name="wsdl2java" description="convert the wsdl to a java client">
<delete><fileset dir="${generated.client.project.dir}">
<include name=“…” /></fileset></delete>
<java classname="org.apache.axis2.wsdl.WSDL2Java" fork="true">
<classpath refid="ws.class.path" />
<arg value="-uri" />
<arg file="${generated.client.project.dir}/GrouperService.wsdl" />
<arg value="-t" /><arg value="-u" /><arg value="-p" />
<arg value="edu.internet2.middleware.grouper.webservicesClient" />
<arg value="-o" /><arg value="${generated.client.project.dir}" />
</java>
</target>
Note: we used to have one large class (with inner classes), but now we do separate classes so we don’t have a class several megs large), this is the –u option
4/13/2017
University of Pennsylvania

41. Axis generate client (continued)
Run ant script to generate client
C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws>ant wsdl2java
Buildfile: build.xml
wsdl2java:
[java] Retrieving document at 'C:\dev_inst\eclipse\workspace\grouper_v1_4\g
rouper-ws-java-generated-client\GrouperService.wsdl'.
BUILD SUCCESSFUL
Total time: 9 seconds
C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws>
4/13/2017
University of Pennsylvania

42. Axis generate client (continued)
Result: 100 classes, ~5megs of source
4/13/2017
University of Pennsylvania

43. University of Pennsylvania
Axis archive
Axis needs an AAR file of logic, create via ant to WEB-INF/services/GrouperService.aar
<target name="generate-aar" depends="compile">
<property name="webservice.folder" value="${basedir}/webservices" />
<delete dir="${webservice.folder}/classes" />
<copy toDir="${webservice.folder}/classes" failonerror="false">
<fileset dir="${build.dir.grouper-ws}">
<include name="edu/internet2/middleware/grouper/ws/**/*.class" />
</fileset></copy>
<jar destfile="${basedir}/webapp/WEB-INF/services/GrouperService.aar">
<fileset excludes="edu/internet2/middleware/grouper/ws/**/*Test.class" dir="${webservice.folder}/classes" />
<fileset dir="webservices/GrouperService.aar" />
</jar>
</target>
4/13/2017
University of Pennsylvania

44. University of Pennsylvania
Axis configuration
Configure the web.xml for Axis
<servlet> <servlet-name>AxisServlet</servlet-name>
<display-name>Apache-Axis Servlet</display-name>
<servlet-class>
edu.internet2.middleware.grouper.ws.GrouperServiceAxisServlet
</servlet-class>
<load-on-startup>1</load-on-startup>
<!-- hint that this is the wssec servlet -->
<!-- init-param>
<param-name>wssec</param-name>
<param-value>true</param-value>
</init-param -->
</servlet>
<servlet-mapping>
<servlet-name>AxisServlet</servlet-name>
<url-pattern>/services/*</url-pattern>
</servlet-mapping>
4/13/2017
University of Pennsylvania

45. Axis configuration (continued)
Boilerplate WEB-INF/conf/axis2.xml
WEB-INF/modules/*.mar
WEB-INF/modules/modules.list
WEB-INF/services/*.aar (including GrouperService.aar)
WEB-INF/service/services.list
WEB-INF/lib/ (50 axis jars)
4/13/2017
University of Pennsylvania

46. University of Pennsylvania
Axis Example
See video (make sure you have the Xvid codec):
4/13/2017
University of Pennsylvania

47. REST web services with xstream
4/13/2017
University of Pennsylvania

48. University of Pennsylvania
Xstream
Easy to use Java object to XML processor
In this example I alias the class names so they arent so long
public class XstreamPocGroup {
public XstreamPocGroup(String theName, XstreamPocMember[] theMembers) {
this.name = theName;
this.members = theMembers; }
private String name;
private XstreamPocMember[] members;
public String getName() { ...
4/13/2017
University of Pennsylvania

49. University of Pennsylvania
Xstream (continued)
This is the child bean
public class XstreamPocMember {
public XstreamPocMember(String theName, String theDescription) {
this.name = theName;
this.description = theDescription; }
private String name;
private String description;
4/13/2017
University of Pennsylvania

50. University of Pennsylvania
Xstream (continued)
public static void main(String[] args) { XstreamPocGroup group = new XstreamPocGroup("myGroup", new XstreamPocMember[]{ new XstreamPocMember("John", "John Smith - Employee"), new XstreamPocMember("Mary", "Mary Johnson - Student")}); XStream xStream = new XStream(new XppDriver()); xStream.alias("XstreamPocGroup", XstreamPocGroup.class); xStream.alias("XstreamPocMember", XstreamPocMember.class); StringWriter stringWriter = new StringWriter(); xStream.marshal(group, new CompactWriter(stringWriter)); String xml = stringWriter.toString(); System.out.println(GrouperUtil.indent(xml, true)); group = (XstreamPocGroup)xStream.fromXML(xml); System.out.println(group.getName() + ", number of members:" + group.getMembers().length); }
4/13/2017
University of Pennsylvania

51. University of Pennsylvania
Xstream (continued)
<XstreamPocGroup>
<name>myGroup</name>
<members>
<XstreamPocMember>
<name>John</name>
<description>John Smith - Employee</description>
</XstreamPocMember>
<name>Mary</name>
<description>Mary Johnson - Student</description>
</members>
</XstreamPocGroup>
myGroup, number of members: 2
4/13/2017
University of Pennsylvania

52. University of Pennsylvania
Xstream JSON
You can control things with annotations
@XStreamOmitField
private Group group = null;
You can convert Java object to JSON
XStream xStream = new XStream(new JettisonMappedXmlDriver());
{"XstreamPocGroup": {"name":"myGroup","members": {"XstreamPocMember":[
{"name":"John","description":"John Smith - Employee"},
{"name":"Mary","description":"Mary Johnson - Student"}
]}} }
myGroup, number of members: 2
4/13/2017
University of Pennsylvania

53. University of Pennsylvania
XHTML output
Grouper had a requirement to have XHTML output
Rest people like XHTML, it can turn a web service into a browsable application
I have some reservations about that (what are the clients? Browsers, or screen scraping applications? Can you change the output?)
I rolled my own bean->XHTML converter
Based on XmlStreamWriter for output
Based on JDOM for input
4/13/2017
University of Pennsylvania

54. XHTML output (continued)
WsXhtmlOutputConverter wsXhtmlOutputConverter = new WsXhtmlOutputConverter(true, null); StringWriter stringWriter = new StringWriter(); wsXhtmlOutputConverter.writeBean(group, stringWriter); String xhtml = stringWriter.toString(); System.out.println(GrouperUtil.indent(xhtml, true)); WsXhtmlInputConverter wsXhtmlInputConverter = new WsXhtmlInputConverter(); wsXhtmlInputConverter.addAlias("XstreamPocGroup", XstreamPocGroup.class); wsXhtmlInputConverter.addAlias("XstreamPocMember", XstreamPocMember.class); group = (XstreamPocGroup)wsXhtmlInputConverter.parseXhtmlString(xhtml); System.out.println(group.getName() + ", number of members: " + group.getMembers().length);
4/13/2017
University of Pennsylvania

55. XHTML output (continued)
<?xml version='1.0' encoding='iso '?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" " <html xmlns=" xml:lang="en" lang="en"> <head> <title>XstreamPocGroup</title> </head> <body> <div title="XstreamPocGroup"> <ul class="members"> <li title="XstreamPocMember"> <p class="name">John</p> <p class="description">John Smith - Employee</p> </li>
4/13/2017
University of Pennsylvania

56. XHTML output (continued)
<li title="XstreamPocMember">
<p class="name">Mary</p>
<p class="description">Mary Johnson - Student</p>
</li>
</ul>
<p class="name">myGroup</p>
</div>
</body>
</html>
myGroup, number of members: 2
Note: Im not sure why this would be useful…
4/13/2017
University of Pennsylvania

57. University of Pennsylvania
HTTP input
Similar to Axis input, REST (Lite) should accept HTTP params (URL params or in HTTP body)
PUT /grouperWs/servicesRest/v1_4_000/groups /aStem%3AaGroup/members/ HTTP/1.1
Connection: close
Authorization: Basic xxxxxxxxxxxxxxxxx==
User-Agent: Jakarta Commons-HttpClient/3.1
Host: localhost:8092
Content-Length: 72
wsLiteObjectType=WsRestAddMemberLiteRequest &actAsSubjectId=GrouperSystem
4/13/2017
University of Pennsylvania

58. Indenting of XML, XHTML, JSON
Couldn’t find a 3rd party indenter, rolled my own
System.out.println(GrouperUtil.indent(xhtml, true));
System.out.println(GrouperUtil.indent(json, true));
System.out.println(GrouperUtil.indent(xml, true));
This is valuable when showing examples
4/13/2017
University of Pennsylvania

59. Make the operations more Restful
URLs need to represent hierarchical resources:
/servicesRest/v1_4_000/groups/aStem%3AaGroup/members
Servlet, client version, top level “folder”, name of item, inner “folder”
The client version is in there since that is the first part of versioning
GET that URL would retrieve all the members of that group
POST would update the list (send all new members)
PUT would create a new list
DELETE would delete all the members
…/groups/aStem%3AaGroup/members/mchyzer
Drilling down further
GET could see if mchyzer is a member
PUT would add mchyzer to group
etc
4/13/2017
University of Pennsylvania

60. University of Pennsylvania
Problems with Rest
Some things are programmatic, not resource based (e.g. a fixBadMemberships diagnostic tool)
The input params could be complicated, e.g.
Get the membership list of a group
Get only immediate members
Act as a different user (proxy)
Get the student list, not the faculty list
There might be many different ways to ID a resource
E.g. name or UUID
Composite ID’s of resources are not convenient
/groups/aStem:aGroup/members/sourceId/someSource/subjectId/1234
4/13/2017
University of Pennsylvania

61. Problems with Rest (continued)
The only HTTP methods which take a body are POST and PUT
How can you delete something with XML constraints (actAs proxy, certain list type, include details on result) if a DELETE method does not take an XML body???
Rest is supposed to “model the web”
Which web? Static? When is the last time you deleted a static resource with an HTTP DELETE method?
The web is more and more dynamic, so Rest/HTTP/XML/hybrid might be more similar for developers
4/13/2017
University of Pennsylvania

62. University of Pennsylvania
Grouper Rest
URLs are resource based
HTTP methods are honored (GET/POST/PUT/DELETE)
If there is a body (POST/PUT), then the object type sent will trump the HTTP method (can DELETE with metadata)
HTTP status codes are sent
Though they really shouldn’t be read, they don’t mean much, 404 could be a success
HTTP headers on the response
boolean to determine if the operation is a success or failure
Enum based text status code which is specific to the operation
4/13/2017
University of Pennsylvania

63. Grouper Rest (continued)
The same operations are exposed as SOAP
The XML document can specify the data, or the URL/HTTP method
Could add member like this:
PUT /grouperWs/servicesRest/xhtml/v1_4_000 /groups/aStem%3AaGroup/members/ HTTP/1.1
Or you could add a member with a POST and a body of the right object type:
POST /grouperWs/servicesRest/v1_4_001/groups /aStem%3AaGroup/members HTTP/1.1
<WsRestAddMemberLiteRequest><subjectId> </subjectId> <actAsSubjectId>GrouperSystem</actAsSubjectId> </WsRestAddMemberLiteRequest>
4/13/2017
University of Pennsylvania 63 63

64. Grouper Rest Architecture
4/13/2017
University of Pennsylvania

65. Grouper Rest (continued)
Client can use same beans as server (not true with Axis)
Enums for content types
Use Jakarta HTTP client for communication (Axis uses this too)
Show movie of Rest client (make sure you have the Xvid codec):
4/13/2017
University of Pennsylvania

66. WHICH WEB SERVICE ARCHITECTURE SHOULD I USE?
4/13/2017
University of Pennsylvania 66

67. University of Pennsylvania
When SOAP
More complex operations (describe in WSDL)
Clients can handle SOAP
Client code generation
WS-* security (e.g. kerberos ticket authentication)
4/13/2017
University of Pennsylvania 67

68. University of Pennsylvania
When Restful
Simple operations
Non batched
Simple resources
Nice to not have composite identifiers
Operations with little or no metadata (e.g. actAs)
Clients are known to handle Rest HTTP methods
Perhaps not for Ajax (limitation might be HTTP response code and HTTP methods)
4/13/2017
University of Pennsylvania 68

69. When Rest / HTTP / XML hybrid / POX
When supporting “Rest” and SOAP
Since SOAP is not resource based
Disparate clients
Need to send a body of metadata with a GET or DELETE
4/13/2017
University of Pennsylvania 69

70. University of Pennsylvania
Client
4/13/2017
University of Pennsylvania

71. University of Pennsylvania
GrouperClient
Architecture
4/13/2017
University of Pennsylvania

72. University of Pennsylvania
GrouperClient
Its nice to give a packaged client with web services
Writing HTTP/XML does not make a quick start
Anyone can use it command line
Can use as Java library
Can make custom XML samples
Debugging tools
More samples
Rest only (since easiest to version, most lightweight… doesn’t have 50 Axis jars!)
Since command line, its one jar
Can be used along side other jars
4/13/2017
University of Pennsylvania

73. GrouperClient (continued)
Very simple POJOs
Refactor 3rd party libs
Xstream
HttpClient
Commons-logging
Jexl: expression language
morphString
So they don’t conflict
Want only one jar
4/13/2017
University of Pennsylvania

74. GrouperClient (continued)
Example:
C:\gc>java -jar grouperClient.jar --operation=getGroupsWs
--subjectIds=
SubjectIndex 0: success: T: code: SUCCESS: subject: : groupIndex: 0: aStem:aGroup
SubjectIndex 0: success: T: code: SUCCESS: subject: : groupIndex: 1: etc:webServiceClientUsers
SubjectIndex 0: success: T: code: SUCCESS: subject: : groupIndex: 2: etc:sysadmingroup
Show movie (make sure you have the Xvid codec):
4/13/2017
University of Pennsylvania

75. GrouperClient (continued)
Java API
No longer need to deal with httpClient or authentication
Error handling built in
public static void main(String[] args) {
WsGetGroupsResults wsGetGroupsResults = new GcGetGroups()
.addSubjectLookup(new WsSubjectLookup(" ", null, null)).execute();
WsGetGroupsResult wsGroupsResult = wsGetGroupsResults.getResults()[0];
for (WsGroup wsGroup : wsGroupsResult.getWsGroups()) {
System.out.println(wsGroup.getName()); } }
aStem:aGroup
etc:webServiceClientUsers
etc:sysadmingroup
4/13/2017
University of Pennsylvania

76. GrouperClient – external encrypted passwords
Set encrypt key in grouper.client.properties
encrypt.key = fnh453hfbdw
Encrypt the password:
C:\gc>java -jar grouperClient.jar --operation=encryptPassword
Type the string to encrypt (note: pasting might echo it back):
Encrypted password: mpAdW53ekchSGAX3vq1UiQ==
C:\gc>
Put this in a file, refer to file in grouper.client.properties
grouperClient.webService.password = c:/gc/ws.pass
(more) sanitized config files for or source control
Perhaps auditing requirement forbidding clear text passwords
4/13/2017
University of Pennsylvania

77. GrouperClient – JEXL output templates
If using command line utility in prod, will screenscrape STDOUT
C:\gc>java -jar grouperClient.jar --operation=getGroupsWs
--subjectIds=
SubjectIndex 0: success: T: code: SUCCESS: subject: : groupIndex: 0: aStem:aGroup
To give flexibility, and ability to change defaults, template
--subjectIds= –outputTemplate
="${groupIndex+1} groupName: ${wsGroup.name}$newline$"
1 groupName: aStem:aGroup
2 groupName: etc:webServiceClientUsers
3 groupName: etc:sysadmingroup
4/13/2017
University of Pennsylvania

78. Grouper client traffic capture
To help troubleshoot or create samples
grouperClient.logging.webService.documentDir = c:/gc/xmls
grouperClient.logging.webService.indent = true
Organize files so they are easy to archive or delete
4/13/2017
University of Pennsylvania

79. Grouper client traffic capture (continued)
4/13/2017
University of Pennsylvania

80. DOCUMENTING WEB SERVICES: AUTOMATIC SAMPLES
4/13/2017
University of Pennsylvania

81. University of Pennsylvania
Automatic samples
Samples are tedious to maintain
Too many combinations:
Rest and Soap
Lite and Batched
Within Rest: XML, XHTML, JSON, HTTP params
12 operations
Program automatically generates 150+ samples
4/13/2017
University of Pennsylvania

82. Automatic samples (continued)
The sample generator starts up a Java TCP listener/proxy, tunnels traffic through that, captures the request and response to the server, and creates documents
4/13/2017
University of Pennsylvania

83. University of Pennsylvania
Automatic samples
See movie (make sure you have the Xvid codec):
4/13/2017
University of Pennsylvania

84. University of Pennsylvania
References and links
cvs login
cvs export -r GROUPER_1_4_BRANCH grouper-ws
4/13/2017
University of Pennsylvania 84

85. University of Pennsylvania
Questions?
4/13/2017
University of Pennsylvania

86. University of Pennsylvania
BONUS MATERIAL: MISC
4/13/2017
University of Pennsylvania 86

87. Xstream words of warning
The default JSON outputted is awful
An array of size 1 is the same as a field
A string with an number value is the same as a number
If you use this, find settings so that it makes sense
Xstream by default works by fields, not Javabean properties
This has surfaced from mismatches between Xstream and Axis (which goes by Javabean properties)
There is a setting to enable this, it is on my TODO list
You can override some settings in Xstream to ignore extraneous XML fields (for backwards compatibility)
Fields are lower case, Classes are uppercase (as aliased), and a user reported difficulty with Xpath
4/13/2017
University of Pennsylvania 87 87

88. Grouper Rest (continued)
GrouperRestServlet
<servlet>
<servlet-name>RestServlet</servlet-name>
<display-name>WS REST Servlet</display-name>
<servlet-class>
edu.internet2.middleware.grouper.ws.rest.GrouperRestServlet
</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<url-pattern>/servicesRest/*</url-pattern>
</servlet-mapping>
4/13/2017
University of Pennsylvania 88 88

89. JEXL output templates (continued)
Easy to setup, get jakarta commons jexl.jar
I have a utility method to substitute, something like this:
JexlContext jc = JexlHelper.createContext();
for (String key: variableMap.keySet()) {
jc.getVars().put(key, variableMap.get(key)); }
Pattern pattern = Pattern.compile("\\$\\{(.*?)\\}");
Matcher matcher = pattern.matcher(stringToParse);
StringBuilder result = new StringBuilder();
while(matcher.find()) {
result.append(stringToParse.substring(index,matcher.start()));
String script = matcher.group(1);
Expression e = ExpressionFactory.createExpression(script);
Object o = e.evaluate(jc);
result.append(o);
index = matcher.end(); }
4/13/2017
University of Pennsylvania 89 89

90. BONUS MATERIAL: GROUPER OBJECT MODEL
4/13/2017
University of Pennsylvania 90

91. Grouper data model (simplified)
Oversimplified data model
Stems are the folders, which are hierarchical. Non-top-level stems refer to their parent stem
A group is in a stem. It has many attributes (e.g. name), which are fields.
Composite groups link up 3 groups and a type (union, intersection, minus).
Memberships are a combination of group, member, and field (field is the list type)
Members are an internal representation of externally managed people (or groups).
4/13/2017
University of Pennsylvania 91 91

92. Object model (continued)
How to represent “Group” in a web service pojo?
Where do we need “Group” in web services?
Input examples:
Add member to a group
Save a group
Delete a group
See if a member is in a group
Find a group
Output examples:
List groups for member
Find group
Save a group?
Delete a group?
4/13/2017
University of Pennsylvania 92

93. Object model (continued)
Break “Group” into two cases
Lookup
Object representation
Lookup (allow multiple ways)
Lookup by UUID
Lookup by name
There are several lookups in Grouper WS:
Stem lookup
Subject lookup
Group lookup
How to handle object representation?
4/13/2017
University of Pennsylvania 93

94. Object model (continued)
Each request has its own object
One for Lite request
Note: each Lite request has only scalars
One for Batch request
Each response has its own object
One for Lite response
One for Batch response
One for each line item of a batch response
4/13/2017
University of Pennsylvania 94 94

95. Object model (continued)
Option 1: minimal, make another request for more info
For list groups for member, send back the group UUID and name
If more info needed make another request for more info
Option 2: send it all
Just send everything about a group on all requests
Option 3: pick and choose
When an operation returns groups, tell the server what to return
Option 4: two levels to decide
Generally send back basic data
If includeGroupDetail in a request, send it all
Note: for saveGroup, same object goes back and forth, including an optional groupLookup on request.
We chose Option 4 to minimize the number of requests if more info is needed.
If there is any operation done on an object, return that object.
Option 3 is not very restful.
4/13/2017
University of Pennsylvania 95 95

96. Lite vs batched (continued)
Original vision was Lite operations would require only scalars
Do not need XML in request
Only use HTTP params
Response would still have a body
Originally we used Axis REST
It didn’t satisfy our requirements, so we moved to custom REST
4/13/2017
University of Pennsylvania 96

97. BONUS MATERIAL: AXIS SERVING REST
4/13/2017
University of Pennsylvania 97

98. University of Pennsylvania
Axis serving Rest
Axis can serve “Rest” services (loose-Rest)
Basically it is the same service without the SOAP envelope
Can also pass params as HTTP params
Set this in axis2.xml: <parameter name="disableREST" locked="true">false</parameter>
Set this option in the client: options.setProperty( Constants.Configuration.ENABLE_REST, Constants.VALUE_TRUE);
Show movie (make sure you have the Xvid codec):
4/13/2017
University of Pennsylvania 98 98

99. Axis serving Rest (continued)
Not clear how to customize the XML, and it is not great looking XML (not what you would create by hand)
Not real Rest, since the HTTP method is still GET or POST, not PUT or DELETE
Still resembles RPC calls, not resource centric calls
There is a bad bug with Axis SOAP and REST where if you skip params it marshals them in the wrong order. Hopefully this will be fixed soon.
Grouper abandoned Axis Rest in favor of custom Rest
4/13/2017
University of Pennsylvania 99 99

100. BONUS MATERIAL: WEB SERVICE SECURITY
4/13/2017
University of Pennsylvania
100

101. University of Pennsylvania
Authentication
Easiest way to go is HTTP basic authentication
Assumes using SSL (since only Base64 encoded)
Send this with Commons Http client
httpClient.getParams().setAuthenticationPreemptive(true);
Credentials defaultcreds = new UsernamePasswordCredentials(RestClientSettings.USER,
RestClientSettings.PASS);
//e.g. localhost and 8093
httpClient.getState()
.setCredentials(new AuthScope(RestClientSettings.HOST, RestClientSettings.PORT), defaultcreds);
4/13/2017
University of Pennsylvania
101

102. Authentication (continued)
Results in this HTTP
PUT /grouperWs/servicesRest/v1_4_000/groups/aStem%3AaGroup/members/ HTTP/1.1
Connection: close
Authorization: Basic SDF423SFD423xxxx==
User-Agent: Jakarta Commons-HttpClient/3.1
Host: localhost:8092
Content-Length: 72
4/13/2017
University of Pennsylvania
102

103. Authentication (continued)
Handle this on the server
Solution should work with Axis or Rest
Make a servlet filter for Soap and Rest:
<filter><filter-name>Grouper service filter</filter-name>
<filter-class>edu.internet2.middleware.grouper.ws.GrouperServiceJ2ee</filter-class></filter>
<filter-mapping>
<filter-name>Grouper service filter</filter-name>
<url-pattern>/services/*</url-pattern>
</filter-mapping>
<url-pattern>/servicesRest/*</url-pattern>
4/13/2017
University of Pennsylvania
103

104. Authentication (continued)
Keep threadlocals of request and response
threadLocalRequest.set((HttpServletRequest) request);
threadLocalResponse.set((HttpServletResponse) response);
threadLocalRequestStartMillis.set(System.currentTimeMillis());
try {
filterChain.doFilter(request, response);
} finally {
threadLocalRequest.remove();
threadLocalResponse.remove();
threadLocalRequestStartMillis.remove(); }
4/13/2017
University of Pennsylvania
104

105. Authentication (continued)
Utility method, called from business logic, to authenticate
String authenticationClassName =
GrouperWsConfig.getPropertyString(
GrouperWsConfig.WS_SECURITY_NON_RAMPART_AUTHENTICATION_CLASS,
WsGrouperDefaultAuthentication.class.getName());
Class<? extends WsCustomAuthentication> theClass =
GrouperUtil.forName(authenticationClassName);
WsCustomAuthentication wsAuthentication = GrouperUtil.newInstance(theClass);
userIdLoggedIn = wsAuthentication retrieveLoggedInSubjectId(retrieveHttpServletRequest());
// cant be blank!
if (StringUtils.isBlank(userIdLoggedIn)) {
throw new WsInvalidQueryException("No user is logged in"); }
4/13/2017
University of Pennsylvania
105

106. Authentication – container authN
Two built in authentication methods, first is container auth
public String retrieveLoggedInSubjectId(HttpServletRequest httpServletRequest)
throws RuntimeException {
// use this to be the user connected, or the user act-as
String userIdLoggedIn = GrouperServiceJ2ee.retrieveUserPrincipalNameFromRequest();
return userIdLoggedIn; }
4/13/2017
University of Pennsylvania
106

107. Authentication – container authN
Configure in web.xml
<security-constraint><web-resource-collection>
<web-resource-name>Web services</web-resource-name>
<url-pattern>/servicesRest/*</url-pattern>
</web-resource-collection><auth-constraint>
<role-name>grouper_user</role-name>
</auth-constraint></security-constraint>
<login-config>
<auth-method>BASIC</auth-method>
<realm-name>Grouper Application</realm-name>
</login-config>
<security-role>
<description>Web service</description>
</security-role>
4/13/2017
University of Pennsylvania
107

108. Authentication – container authN (continued)
Configure in tomcat-users.xml (servlet container specific)
<?xml version='1.0' encoding='utf-8'?>
<tomcat-users>
<role rolename="grouper_user"/>
<user username="mchyzer" password="mchyzer1"
roles="grouper_user"/>
</tomcat-users>
4/13/2017
University of Pennsylvania
108

109. Authentication – Kerberos
Alternative builtin authentication: Kerberos
String authHeader = request.getHeader("Authorization");
//if no header, we cant go to kerberos
if (StringUtils.isBlank(authHeader)) {
LOG.error("No authorization header in HTTP");
return null; }
Matcher matcher = regexPattern.matcher(authHeader);
String authHeaderBase64Part = null;
if (matcher.matches()) {
authHeaderBase64Part = matcher.group(1);
if (StringUtils.isBlank(authHeaderBase64Part)) {
LOG.error("Cant find base64 part in auth header");
return null; }
4/13/2017
University of Pennsylvania
109

110. Authentication – Kerberos (continued)
//unencrypt this byte[] base64Bytes = authHeaderBase64Part.getBytes(); byte[] unencodedBytes = Base64.decodeBase64(base64Bytes); String unencodedString = new String(unencodedBytes); //split based on user/pass String user = GrouperUtil.prefixOrSuffix(unencodedString, ":", true); String pass = GrouperUtil.prefixOrSuffix(unencodedString, ":", false); if (authenticateKerberos(user, pass)) { return user; }
4/13/2017
University of Pennsylvania
110

111. Authentication – Pluggable
Deployers can use their own authentication
# to provide custom authentication (instead of the default
# httpServletRequest.getUserPrincipal()
# for non-Rampart authentication. Class must implement the
# interface:
# edu.internet2.middleware.grouper.ws.security
# .WsCustomAuthentication
# class must be fully qualified. e.g.
# edu.school.whatever.MyAuthenticator
# blank means use default:
# .WsGrouperDefaultAuthentication
# kerberos: edu.internet2.middleware.grouper.ws.security
# .WsGrouperKerberosAuthentication
ws.security.non-rampart.authentication.class =
4/13/2017
University of Pennsylvania
111

112. Authentication – Pluggable
Implement this interface
public interface WsCustomAuthentication {
/**
* retrieve the current username (subjectId) from the request
* object.
httpServletRequest
the logged in username (subjectId)
WsInvalidQueryException if there is a problem */
public String retrieveLoggedInSubjectId(HttpServletRequest httpServletRequest)
throws WsInvalidQueryException; }
4/13/2017
University of Pennsylvania
112

113. Authentication – Caching
It is a TODO to provide the option for the an HTTP basic authN to cache a hashed version of the authentication header (and translation to subject ID)
Hash this so a Base64 version is not held in memory
Do not want to load down the authentication service (e.g. kerberos) if the web service is under high load
4/13/2017
University of Pennsylvania
113

114. Authentication – Ws- Security
Contributed by Sanjay Vivek at Newcastle University
Axis has a module called Rampart which implements WS –security standards
Important if the service requires multiple hops or proxying
Useful if not using SSL
Only for Soap, not for Rest
Useful for authenticating without a user/pass (e.g. Kerberos ticket or x.509)
Cannot run rampart along with HTTP basic authn (need multiple webapps) in Axis2
4/13/2017
University of Pennsylvania
114

115. Authentication – Ws- Security (continued)
Set a param in the web.xml
<servlet>
<servlet-name>AxisServlet</servlet-name>
<display-name>Apache-Axis Servlet</display-name>
<servlet-class>edu.internet2.middleware.grouper.ws.GrouperServiceAxisServlet</servlet-class>
<load-on-startup>1</load-on-startup>
<init-param>
<param-name>wssec</param-name>
<param-value>true</param-value>
</init-param>
</servlet>
4/13/2017
University of Pennsylvania
115

116. Authentication – Ws- Security (continued)
Configure axis2 GrouperServiceWssec.aar/services.xml …
<module ref="rampart" />
<wsp:Policy>
<sp:HttpsToken RequireClientCertificate="false"/>
</wsp:Policy>
<ramp:RampartConfig
xmlns:ramp="
<ramp:passwordCallbackClass>edu.internet2.middleware.grouper.ws.security.RampartHandlerServer</ramp:passwordCallbackClass>
</ramp:RampartConfig>
4/13/2017
University of Pennsylvania
116

117. Authentication – Ws- Security (continued)
Default services.xml requires implementation of interface
public interface GrouperWssecAuthentication {
/**
* <pre>
* authenticate the user, and find the subject and return.
* See GrouperWssecSample for an example
* </pre>
wsPasswordCallback
true if callback type is supported, false if not
IOException if there is a problem or if user is
* not authenticated correctly */
public boolean authenticate(WSPasswordCallback wsPasswordCallback) throws IOException; }
4/13/2017
University of Pennsylvania
117

118. Authentication – Ws- Security (continued)
Then you also need to configure this on the client
There is an example in RampartSampleGetGroupsLite.java
Includes service.xml
And RampartPwHandlerClient callback for credentials
4/13/2017
University of Pennsylvania
118

119. University of Pennsylvania
Authorization
Grouper has built in authorization about who is allowed to do what
Who can add to which stems
Who can view memberships
Who can view that groups exist
Who can edit group attributes
Another layer is which users are allowed to use web services
Since Grouper is all about groups, use a group
4/13/2017
University of Pennsylvania
119

120. Authorization (continued)
Setting in config file lists a group that users must be in to use WS (if blank allow all)
4/13/2017
University of Pennsylvania
120

121. Authorization (continued)
If not in group, log it and alert the caller (should make option to suppress this)
HTTP/ Internal Server Error
X-Grouper-resultCode: EXCEPTION
X-Grouper-success: F
<?xml version='1.0' encoding='iso '?>
, params: null, java.lang.RuntimeException: User is not authorized
at edu.internet2…
Caused by: edu.internet2… GroupNotFoundException: Cannot find group with name: 'etc:webServiceClientUsers'
Note: it is a security best practice not to give too much information to clients, so they don’t know what the next hurdle is to hack the application. However, the administrators running the service must know errors (logs), and if clients can get information which does not help them hack in, but which reduces support time, then it is good. This should able to be suppressed so that deployers can decide.
4/13/2017
University of Pennsylvania
121
121

122. Authorization (continued)
Given that callers must be in group, make it easy to setup
This setting in grouper.properties will auto-create groups and auto-populate for ease of startup and testing
4/13/2017
University of Pennsylvania
122
122

123. Authorization (continued)
If there is an auto-created group on startup, log it
:28:03,327: [main] WARN GrouperCheckConfig.checkGroup(130) - cannot find group from config: grouper.properties key configuration.autocreate.group.name.1: etc:webServiceClientUsers
:28:04,952: [main] WARN GrouperCheckConfig.checkGroup(149) - auto-created grouper.properties key configuration.autocreate.group.name.1: etc:webServiceClientUsers
:28:05,015: [main] WARN GrouperCheckConfig.checkGroups(469) - auto-added subject mchyzer to group: etc:webServiceClientUsers
4/13/2017
University of Pennsylvania
123
123

124. Authorization – actAs proxy
Sometimes you need run as a different user than logged in as
E.g. to run things as the “system” [root] user
Or if there is an app users are using, and you call the web service as that user
Or if an admin user needs to proxy another user in the office (maybe someone on vacation)
All calls have an optional actAs input
<WsRestAddMemberRequest>
<actAsSubjectLookup>
<subjectId>GrouperSystem</subjectId>
</actAsSubjectLookup>
</WsRestAddMemberRequest>
4/13/2017
University of Pennsylvania
124
124

125. Authorization – actAs proxy (continued)
Any user can actAs themselves (why? Due to examples  )
GrouperSystem [root] can act as anyone
You can configure groups of users who can actAs anyone
(in grouper-ws.properties)
ws.act.as.group = etc:webServiceActAsGroup
Or you can specify groups who can act as users in another group (to clamp down a bit)
ws.act.as.group = orgs:admins123 :::: orgs:users123
In this case users in orgs:admins123 group can only actAs any user in orgs:users123
4/13/2017
University of Pennsylvania
125
125

126. Authorization – actAs proxy (continued)
Important to audit both the logged in user and actAs user
4/13/2017
University of Pennsylvania
126
126

127. Authorization – two factor (TODO)
Factor 1: What you know (password)
Factor 2: Where you are (assumes SSL)
4/13/2017
University of Pennsylvania
127
127

128. BONUS MATERIAL: DEVELOPMENT ENVIRONMENTS
4/13/2017
University of Pennsylvania
128

129. University of Pennsylvania
TCP/IP monitor
Need to have a way to see HTTP traffic to/from web service
Network level proxy works in non-SSL only
Axis has a swing tcp monitor
Ethereal
Axis has log settings to log the traffic from client (not server?)
Eclipse has built-in TCP/IP monitor (my choice)
Note: this has more uses than just web services
See video (make sure you have the Xvid codec):
4/13/2017
University of Pennsylvania
129

130. BONUS MATERIAL: JAVADOC IN CVS
4/13/2017
University of Pennsylvania
130

131. University of Pennsylvania
Javadoc in CVS
Ant script to make javadoc CVSweb ready
4/13/2017
University of Pennsylvania
131
131

132. Javadoc in CVS (continued)
4/13/2017
University of Pennsylvania
132
132

133. Javadoc in CVS (continued)
4/13/2017
University of Pennsylvania
133
133

134. BONUS MATERIALS: WEB SERVICE TESTING
4/13/2017
University of Pennsylvania
134

135. University of Pennsylvania
Testing
There are lots of places to test web services
Best place is at the client
See movie (make sure you have the Xvid codec):
Most bugs found in Grouper web services could have been avoided with more testing
Web services are easier to automatically test than web applications
4/13/2017
University of Pennsylvania
135
135