Presentation is loading. Please wait.

Presentation is loading. Please wait.

Java Web Service Servers and Clients in Internet2 Grouper

Similar presentations

Presentation on theme: "Java Web Service Servers and Clients in Internet2 Grouper"— Presentation transcript:

1 Java Web Service Servers and Clients in Internet2 Grouper
February 2009 Chris Hyzer University of Pennsylvania IT Internet2

2 Purpose of this presentation
Show how Internet2 Grouper developed web services Grouper is open source, feel free to borrow or suggest improvements Discuss the successes and areas for improvement Mention planned enhancements 4/13/2017 University of Pennsylvania

3 University of Pennsylvania
Contents Introduction to Internet2 Grouper Introduction to web services Architecture of REST/SOAP web service SOAP web services with Axis2 (servers and clients) REST web services with xstream (servers and clients) Client Documenting web services Bonus material Security Axis serving Rest Testing 4/13/2017 University of Pennsylvania

4 Introduction to Internet2 Grouper
4/13/2017 University of Pennsylvania

5 University of Pennsylvania
Internet2 Grouper Open source group management Internet2 has been working on group management for 8 years Generally used in educational institutions, but could be anywhere Funded by Internet2 4/13/2017 University of Pennsylvania

6 Why central group management with Grouper?
Instead of apps managing own groups Reuse group lists Central place to see which groups a person is in Central auditing of group and membership actions Central management of authorization Security: Who can view/edit groups and memberships Opt-in/Opt-out Delegate authority Automatic or manual membership management Composite groups for group match: and / or / minus Groups of groups 4/13/2017 University of Pennsylvania

7 University of Pennsylvania
Grouper architecture Note: the one-way arrow doesn’t mean the traffic originates from one side or the other. E.g. no ldap reads originate from the ldap server. 4/13/2017 University of Pennsylvania

8 Introduction to WEB SERVICES
4/13/2017 University of Pennsylvania

9 University of Pennsylvania
User web request Person using browser makes a request to a server Person (user) views the results in browser, and types and or clicks to continue 4/13/2017 University of Pennsylvania

10 University of Pennsylvania
Web service request Program makes a request to a web application Program parses the output Note, it doesn’t have to be a server which makes a web service request, it could be any program, e.g. ajax. 4/13/2017 University of Pennsylvania

11 Overlap of web request and web service?
Ajax for example Can be kicked off by a user click Can update the screen similar to a web application However, Ajax is making the request and parsing the response, it is a web service If it doesn’t parse the output, and just puts the resultant HTML into the browser DOM, then not a web service Web service screen scraping a web application A program can “screen scrape” a web application Beware of changes in the HTML! This is not a web service Is a browser an application making requests, are all user requests web services? 4/13/2017 University of Pennsylvania

12 University of Pennsylvania
Why web services http(s) is a well understood protocol by programming languages and programmers Ports 80/443 might already be available in firewall rules Http is text based (easy to debug) Http is not programming language specific, so the server technology can be different than the client (e.g. ajax) Webpages are either XML, XHTML, or XML-like (e.g. HTML) Most programming languages have XML libraries Note: web services do not have to be XML, though generally the are Development and production environments might be similar (or same) to existing web applications Penn generally communicates between systems with WS 4/13/2017 University of Pennsylvania

13 University of Pennsylvania
SOAP web services Simple Object Access Protocol Specifies how web service messages are exchanged W3C standard Must use XML and XML schema for data Messages have XML envelopes, headers, body, exception handling Web Service Description Language (WSDL) describes the SOAP messages in a programmatic way (XML) Many features (security, error handling, caching, resource discovery, etc) Many programming languages generate SOAP Not considered light-weight 4/13/2017 University of Pennsylvania

14 University of Pennsylvania
Example SOAP request <?xml version='1.0' encoding='UTF-8'?> <soapenv:Envelope xmlns:soapenv=""> <soapenv:Body> <ns1:addMemberLite xmlns:ns1=""> <ns1:groupName>aStem:aGroup</ns1:groupName> <ns1:subjectIdentifier>mchyzer</ns1:subjectIdentifier> </ns1:addMemberLite> </soapenv:Body> </soapenv:Envelope> 4/13/2017 University of Pennsylvania

15 University of Pennsylvania
REST web services Representational State Transfer Two definitions: (strict or RESTful): Protocol that specifies how HTTP (perhaps) and XML are used for web services (non-strict): Any web service that does not have the overhead of SOAP. Aka Remote Procedure Call (RPC) 4/13/2017 University of Pennsylvania

16 University of Pennsylvania
RESTful web services The web services are organized like static web resources URL’s represent resources, not operations HTTP methods indicate the operations. Generally: GET, POST (update), PUT (insert), DELETE. Can use more, or custom Messages can be HTML so that systems or browsers can consume them 4/13/2017 University of Pennsylvania

17 Example REST web service
PUT /grouperWs/servicesRest/xhtml/v1_4_000/groups/aStem%3AaGroup/members/mchyzer This means add this member to the group Note, there can be body here, though in this case it isn’t needed 4/13/2017 University of Pennsylvania

18 HTTP / XML / RPC / Hybrid web services
Each service makes its own standards URL can be: Resource: Operation: Generally just use POST or GET as the HTTP method The XML document sent can be: Complete object representation: <group><name>myGroup</name><desc>MyGroup</desc></group> Operational: <groupChangeName><groupId>123</groupId> <newName>myGroup2</newName></groupChangeName> 4/13/2017 University of Pennsylvania

19 University of Pennsylvania
GROUPER WEB SERVICES 4/13/2017 University of Pennsylvania

20 University of Pennsylvania
Grouper web services Generally web services are programmed to host a service Grouper is software, so its WS are programmed so institutions can download and host services E.g. Grouper is app server and database server agnostic Requirements Dozen operations SOAP and REST (as close to RESTful as possible) SOAP and REST should deploy in one webapp Simple operations (Lite), and batched operations Pluggable authentication Documented well Versioned (generally Grouper has bi-annual releases) 4/13/2017 University of Pennsylvania

21 Grouper web service operations
findGroups findStems stemSave stemDelete memberChangeSubject getGrouperPrivileges assignGrouperPrivileges addMember deleteMember getMembers hasMember getGroups groupSave groupDelete 4/13/2017 University of Pennsylvania

22 University of Pennsylvania
Lite vs batched One batched operation has less overhead than many smaller operations (performance test for yourself to validate) Benchmarks Add group (inserts), requires many queries 100 batches of 1 add groups take 19.8 seconds 10 batches of 10 add groups take 12.4 seconds 5 batches of 20 add groups take 12.0 seconds Has member, lightweight, readonly 100 batches of 1 hasMember checks take 8.3 seconds 10 batches of 10 checks take 1.9 seconds 5 batches of 20 checks take 1.6 seconds Benchmarks notes Completely run on developer PC, local mysql E.g. WsSampleHasMemberRest100 4/13/2017 University of Pennsylvania

23 University of Pennsylvania
Object model Assume web service data are simple POJOs (Plain Old Java Objects) Use only: Beans Arrays (of beans or simple types) Simple types: String, Integer Note: Don’t use Collections, enums, dates, timestamps, booleans 4/13/2017 University of Pennsylvania

24 Object model (continued)
What makes up the data of a group? Here is a simple group 4/13/2017 University of Pennsylvania

25 Object model (continued)
What makes up the data of a group? Here is a more complex group 4/13/2017 University of Pennsylvania

26 Object model (continued)
4/13/2017 University of Pennsylvania

27 Object model (continued)
Request object model 4/13/2017 University of Pennsylvania

28 Object model (continued)
Response object model 4/13/2017 University of Pennsylvania

29 Object model (continued)
Metadata object model Response metadata is one per response Result metadata is one per Lite response, or one per each line item in batch Success: T|F Result code: many enums Codes also in HTTP headers 4/13/2017 University of Pennsylvania

30 Object model (continued)
Operations should be idempotent if possible If they are sent twice, generally it is ok Delete member mchyzer from group etc:sysAdminGroup Idempotent Delete the first member of group etc:sysAdminGroup NOT idempotent 4/13/2017 University of Pennsylvania 30 30

31 SOAP web services with Axis2
4/13/2017 University of Pennsylvania

32 University of Pennsylvania
Axis architecture Axis can do many things, but how Grouper uses Axis is: Servlet to accept web requests and call Grouper’s business logic Generates WSDL from business logic Generates sample client library from WSDL 4/13/2017 University of Pennsylvania

33 Business logic for Axis
Create a class (GrouperService) Contains only instance methods of business logic Each method takes all fields of the input bean, and returns the output bean Each bean is only simple pojo (uses Javabean properties) Note the Lite methods only take scalars are inputs 4/13/2017 University of Pennsylvania

34 Business logic for Axis (continued)
4/13/2017 University of Pennsylvania

35 Business logic for Axis (continued)
GrouperService isn’t great for Javadoc since enums are strings Delegate to GrouperServiceLogic Decode booleans, dates, enums, etc 4/13/2017 University of Pennsylvania

36 University of Pennsylvania
Axis generate WSDL Generate WSDL from POJOs and GrouperService class <target name="java2wsdl" description="convert the java to a wsdl"> <touch file="${generated.client.project.dir}/GrouperService.wsdl" /> <delete file="${generated.client.project.dir}/GrouperService.wsdl" /> <java classname="" fork="true"> <classpath refid="ws.class.path" /> <arg value="-o" /><arg value="${generated.client.project.dir}" /> <arg value="-of" /><arg value="GrouperService.wsdl" /> <arg value="-cn" /> <arg value="" /> <arg value="-stn" /> <arg value="" /> <arg value="-l" /> <arg value="http://localhost:8090/grouper-ws/services/GrouperService" /> </target> 4/13/2017 University of Pennsylvania

37 Axis generate WSDL (continued)
Generate WSDL from POJOs and GrouperService class C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws>ant java2wsdl Buildfile: build.xml java2wsdl: [delete] Deleting: C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws-java -generated-client\GrouperService.wsdl BUILD SUCCESSFUL Total time: 9 seconds C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws> 4/13/2017 University of Pennsylvania

38 Axis generate WSDL (continued)
Result 4/13/2017 University of Pennsylvania

39 Axis generate WSDL (continued)
Result – 2000 lines of SOAP definition This might be idealistic, but Grouper is trying not to touch this WSDL file, and only use the one generated by Axis. Note this is in CVS so we can diff revisions, and it is versioned (e.g. get the WSDL from a particular Grouper branch or version). Features in WSDL which are not in Axis generated WSDL are not being used, e.g. enums. 4/13/2017 University of Pennsylvania

40 University of Pennsylvania
Axis generate client Ant script to generate SOAP client from WSDL (any WSDL) <target name="wsdl2java" description="convert the wsdl to a java client"> <delete><fileset dir="${generated.client.project.dir}"> <include name=“…” /></fileset></delete> <java classname="org.apache.axis2.wsdl.WSDL2Java" fork="true"> <classpath refid="ws.class.path" /> <arg value="-uri" /> <arg file="${generated.client.project.dir}/GrouperService.wsdl" /> <arg value="-t" /><arg value="-u" /><arg value="-p" /> <arg value="edu.internet2.middleware.grouper.webservicesClient" /> <arg value="-o" /><arg value="${generated.client.project.dir}" /> </java> </target> Note: we used to have one large class (with inner classes), but now we do separate classes so we don’t have a class several megs large), this is the –u option 4/13/2017 University of Pennsylvania

41 Axis generate client (continued)
Run ant script to generate client C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws>ant wsdl2java Buildfile: build.xml wsdl2java: [java] Retrieving document at 'C:\dev_inst\eclipse\workspace\grouper_v1_4\g rouper-ws-java-generated-client\GrouperService.wsdl'. BUILD SUCCESSFUL Total time: 9 seconds C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws> 4/13/2017 University of Pennsylvania

42 Axis generate client (continued)
Result: 100 classes, ~5megs of source 4/13/2017 University of Pennsylvania

43 University of Pennsylvania
Axis archive Axis needs an AAR file of logic, create via ant to WEB-INF/services/GrouperService.aar <target name="generate-aar" depends="compile"> <property name="webservice.folder" value="${basedir}/webservices" /> <delete dir="${webservice.folder}/classes" /> <copy toDir="${webservice.folder}/classes" failonerror="false"> <fileset dir="${build.dir.grouper-ws}"> <include name="edu/internet2/middleware/grouper/ws/**/*.class" /> </fileset></copy> <jar destfile="${basedir}/webapp/WEB-INF/services/GrouperService.aar"> <fileset excludes="edu/internet2/middleware/grouper/ws/**/*Test.class" dir="${webservice.folder}/classes" /> <fileset dir="webservices/GrouperService.aar" /> </jar> </target> 4/13/2017 University of Pennsylvania

44 University of Pennsylvania
Axis configuration Configure the web.xml for Axis <servlet> <servlet-name>AxisServlet</servlet-name> <display-name>Apache-Axis Servlet</display-name> <servlet-class> </servlet-class> <load-on-startup>1</load-on-startup> <!-- hint that this is the wssec servlet --> <!-- init-param> <param-name>wssec</param-name> <param-value>true</param-value> </init-param --> </servlet> <servlet-mapping> <servlet-name>AxisServlet</servlet-name> <url-pattern>/services/*</url-pattern> </servlet-mapping> 4/13/2017 University of Pennsylvania

45 Axis configuration (continued)
Boilerplate WEB-INF/conf/axis2.xml WEB-INF/modules/*.mar WEB-INF/modules/modules.list WEB-INF/services/*.aar (including GrouperService.aar) WEB-INF/service/services.list WEB-INF/lib/ (50 axis jars) 4/13/2017 University of Pennsylvania

46 University of Pennsylvania
Axis Example See video (make sure you have the Xvid codec): /soapExample.avi 4/13/2017 University of Pennsylvania

47 REST web services with xstream
4/13/2017 University of Pennsylvania

48 University of Pennsylvania
Xstream Easy to use Java object to XML processor In this example I alias the class names so they arent so long public class XstreamPocGroup { public XstreamPocGroup(String theName, XstreamPocMember[] theMembers) { = theName; this.members = theMembers; } private String name; private XstreamPocMember[] members; public String getName() { ... 4/13/2017 University of Pennsylvania

49 University of Pennsylvania
Xstream (continued) This is the child bean public class XstreamPocMember { public XstreamPocMember(String theName, String theDescription) { = theName; this.description = theDescription; } private String name; private String description; 4/13/2017 University of Pennsylvania

50 University of Pennsylvania
Xstream (continued) public static void main(String[] args) { XstreamPocGroup group = new XstreamPocGroup("myGroup", new XstreamPocMember[]{ new XstreamPocMember("John", "John Smith - Employee"), new XstreamPocMember("Mary", "Mary Johnson - Student")}); XStream xStream = new XStream(new XppDriver()); xStream.alias("XstreamPocGroup", XstreamPocGroup.class); xStream.alias("XstreamPocMember", XstreamPocMember.class); StringWriter stringWriter = new StringWriter(); xStream.marshal(group, new CompactWriter(stringWriter)); String xml = stringWriter.toString(); System.out.println(GrouperUtil.indent(xml, true)); group = (XstreamPocGroup)xStream.fromXML(xml); System.out.println(group.getName() + ", number of members:" + group.getMembers().length); } 4/13/2017 University of Pennsylvania

51 University of Pennsylvania
Xstream (continued) <XstreamPocGroup> <name>myGroup</name> <members> <XstreamPocMember> <name>John</name> <description>John Smith - Employee</description> </XstreamPocMember> <name>Mary</name> <description>Mary Johnson - Student</description> </members> </XstreamPocGroup> myGroup, number of members: 2 4/13/2017 University of Pennsylvania

52 University of Pennsylvania
Xstream JSON You can control things with annotations @XStreamOmitField private Group group = null; You can convert Java object to JSON XStream xStream = new XStream(new JettisonMappedXmlDriver()); {"XstreamPocGroup": {"name":"myGroup","members": {"XstreamPocMember":[ {"name":"John","description":"John Smith - Employee"}, {"name":"Mary","description":"Mary Johnson - Student"} ]}} } myGroup, number of members: 2 4/13/2017 University of Pennsylvania

53 University of Pennsylvania
XHTML output Grouper had a requirement to have XHTML output Rest people like XHTML, it can turn a web service into a browsable application I have some reservations about that (what are the clients? Browsers, or screen scraping applications? Can you change the output?) I rolled my own bean->XHTML converter Based on XmlStreamWriter for output Based on JDOM for input 4/13/2017 University of Pennsylvania

54 XHTML output (continued)
WsXhtmlOutputConverter wsXhtmlOutputConverter = new WsXhtmlOutputConverter(true, null); StringWriter stringWriter = new StringWriter(); wsXhtmlOutputConverter.writeBean(group, stringWriter); String xhtml = stringWriter.toString(); System.out.println(GrouperUtil.indent(xhtml, true)); WsXhtmlInputConverter wsXhtmlInputConverter = new WsXhtmlInputConverter(); wsXhtmlInputConverter.addAlias("XstreamPocGroup", XstreamPocGroup.class); wsXhtmlInputConverter.addAlias("XstreamPocMember", XstreamPocMember.class); group = (XstreamPocGroup)wsXhtmlInputConverter.parseXhtmlString(xhtml); System.out.println(group.getName() + ", number of members: " + group.getMembers().length); 4/13/2017 University of Pennsylvania

55 XHTML output (continued)
<?xml version='1.0' encoding='iso '?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" ""> <html xmlns="" xml:lang="en" lang="en"> <head> <title>XstreamPocGroup</title> </head> <body> <div title="XstreamPocGroup"> <ul class="members"> <li title="XstreamPocMember"> <p class="name">John</p> <p class="description">John Smith - Employee</p> </li> 4/13/2017 University of Pennsylvania

56 XHTML output (continued)
<li title="XstreamPocMember"> <p class="name">Mary</p> <p class="description">Mary Johnson - Student</p> </li> </ul> <p class="name">myGroup</p> </div> </body> </html> myGroup, number of members: 2 Note: Im not sure why this would be useful… 4/13/2017 University of Pennsylvania

57 University of Pennsylvania
HTTP input Similar to Axis input, REST (Lite) should accept HTTP params (URL params or in HTTP body) PUT /grouperWs/servicesRest/v1_4_000/groups /aStem%3AaGroup/members/ HTTP/1.1 Connection: close Authorization: Basic xxxxxxxxxxxxxxxxx== User-Agent: Jakarta Commons-HttpClient/3.1 Host: localhost:8092 Content-Length: 72 wsLiteObjectType=WsRestAddMemberLiteRequest &actAsSubjectId=GrouperSystem 4/13/2017 University of Pennsylvania

58 Indenting of XML, XHTML, JSON
Couldn’t find a 3rd party indenter, rolled my own System.out.println(GrouperUtil.indent(xhtml, true)); System.out.println(GrouperUtil.indent(json, true)); System.out.println(GrouperUtil.indent(xml, true)); This is valuable when showing examples 4/13/2017 University of Pennsylvania

59 Make the operations more Restful
URLs need to represent hierarchical resources: /servicesRest/v1_4_000/groups/aStem%3AaGroup/members Servlet, client version, top level “folder”, name of item, inner “folder” The client version is in there since that is the first part of versioning GET that URL would retrieve all the members of that group POST would update the list (send all new members) PUT would create a new list DELETE would delete all the members …/groups/aStem%3AaGroup/members/mchyzer Drilling down further GET could see if mchyzer is a member PUT would add mchyzer to group etc 4/13/2017 University of Pennsylvania

60 University of Pennsylvania
Problems with Rest Some things are programmatic, not resource based (e.g. a fixBadMemberships diagnostic tool) The input params could be complicated, e.g. Get the membership list of a group Get only immediate members Act as a different user (proxy) Get the student list, not the faculty list There might be many different ways to ID a resource E.g. name or UUID Composite ID’s of resources are not convenient /groups/aStem:aGroup/members/sourceId/someSource/subjectId/1234 4/13/2017 University of Pennsylvania

61 Problems with Rest (continued)
The only HTTP methods which take a body are POST and PUT How can you delete something with XML constraints (actAs proxy, certain list type, include details on result) if a DELETE method does not take an XML body??? Rest is supposed to “model the web” Which web? Static? When is the last time you deleted a static resource with an HTTP DELETE method? The web is more and more dynamic, so Rest/HTTP/XML/hybrid might be more similar for developers 4/13/2017 University of Pennsylvania

62 University of Pennsylvania
Grouper Rest URLs are resource based HTTP methods are honored (GET/POST/PUT/DELETE) If there is a body (POST/PUT), then the object type sent will trump the HTTP method (can DELETE with metadata) HTTP status codes are sent Though they really shouldn’t be read, they don’t mean much, 404 could be a success HTTP headers on the response boolean to determine if the operation is a success or failure Enum based text status code which is specific to the operation 4/13/2017 University of Pennsylvania

63 Grouper Rest (continued)
The same operations are exposed as SOAP The XML document can specify the data, or the URL/HTTP method Could add member like this: PUT /grouperWs/servicesRest/xhtml/v1_4_000 /groups/aStem%3AaGroup/members/ HTTP/1.1 Or you could add a member with a POST and a body of the right object type: POST /grouperWs/servicesRest/v1_4_001/groups /aStem%3AaGroup/members HTTP/1.1 <WsRestAddMemberLiteRequest><subjectId> </subjectId> <actAsSubjectId>GrouperSystem</actAsSubjectId> </WsRestAddMemberLiteRequest> 4/13/2017 University of Pennsylvania 63 63

64 Grouper Rest Architecture
4/13/2017 University of Pennsylvania

65 Grouper Rest (continued)
Client can use same beans as server (not true with Axis) Enums for content types Use Jakarta HTTP client for communication (Axis uses this too) Show movie of Rest client (make sure you have the Xvid codec): /restDemo.avi 4/13/2017 University of Pennsylvania

4/13/2017 University of Pennsylvania 66

67 University of Pennsylvania
When SOAP More complex operations (describe in WSDL) Clients can handle SOAP Client code generation WS-* security (e.g. kerberos ticket authentication) 4/13/2017 University of Pennsylvania 67

68 University of Pennsylvania
When Restful Simple operations Non batched Simple resources Nice to not have composite identifiers Operations with little or no metadata (e.g. actAs) Clients are known to handle Rest HTTP methods Perhaps not for Ajax (limitation might be HTTP response code and HTTP methods) 4/13/2017 University of Pennsylvania 68

69 When Rest / HTTP / XML hybrid / POX
When supporting “Rest” and SOAP Since SOAP is not resource based Disparate clients Need to send a body of metadata with a GET or DELETE 4/13/2017 University of Pennsylvania 69

70 University of Pennsylvania
Client 4/13/2017 University of Pennsylvania

71 University of Pennsylvania
GrouperClient Architecture 4/13/2017 University of Pennsylvania

72 University of Pennsylvania
GrouperClient Its nice to give a packaged client with web services Writing HTTP/XML does not make a quick start Anyone can use it command line Can use as Java library Can make custom XML samples Debugging tools More samples Rest only (since easiest to version, most lightweight… doesn’t have 50 Axis jars!) Since command line, its one jar Can be used along side other jars 4/13/2017 University of Pennsylvania

73 GrouperClient (continued)
Very simple POJOs Refactor 3rd party libs Xstream HttpClient Commons-logging Jexl: expression language morphString So they don’t conflict Want only one jar 4/13/2017 University of Pennsylvania

74 GrouperClient (continued)
Example: C:\gc>java -jar grouperClient.jar --operation=getGroupsWs --subjectIds= SubjectIndex 0: success: T: code: SUCCESS: subject: : groupIndex: 0: aStem:aGroup SubjectIndex 0: success: T: code: SUCCESS: subject: : groupIndex: 1: etc:webServiceClientUsers SubjectIndex 0: success: T: code: SUCCESS: subject: : groupIndex: 2: etc:sysadmingroup Show movie (make sure you have the Xvid codec): /grouperClient.avi 4/13/2017 University of Pennsylvania

75 GrouperClient (continued)
Java API No longer need to deal with httpClient or authentication Error handling built in public static void main(String[] args) { WsGetGroupsResults wsGetGroupsResults = new GcGetGroups() .addSubjectLookup(new WsSubjectLookup(" ", null, null)).execute(); WsGetGroupsResult wsGroupsResult = wsGetGroupsResults.getResults()[0]; for (WsGroup wsGroup : wsGroupsResult.getWsGroups()) { System.out.println(wsGroup.getName()); } } aStem:aGroup etc:webServiceClientUsers etc:sysadmingroup 4/13/2017 University of Pennsylvania

76 GrouperClient – external encrypted passwords
Set encrypt key in encrypt.key = fnh453hfbdw Encrypt the password: C:\gc>java -jar grouperClient.jar --operation=encryptPassword Type the string to encrypt (note: pasting might echo it back): Encrypted password: mpAdW53ekchSGAX3vq1UiQ== C:\gc> Put this in a file, refer to file in grouperClient.webService.password = c:/gc/ws.pass (more) sanitized config files for or source control Perhaps auditing requirement forbidding clear text passwords 4/13/2017 University of Pennsylvania

77 GrouperClient – JEXL output templates
If using command line utility in prod, will screenscrape STDOUT C:\gc>java -jar grouperClient.jar --operation=getGroupsWs --subjectIds= SubjectIndex 0: success: T: code: SUCCESS: subject: : groupIndex: 0: aStem:aGroup To give flexibility, and ability to change defaults, template --subjectIds= –outputTemplate ="${groupIndex+1} groupName: ${}$newline$" 1 groupName: aStem:aGroup 2 groupName: etc:webServiceClientUsers 3 groupName: etc:sysadmingroup 4/13/2017 University of Pennsylvania

78 Grouper client traffic capture
To help troubleshoot or create samples grouperClient.logging.webService.documentDir = c:/gc/xmls grouperClient.logging.webService.indent = true Organize files so they are easy to archive or delete 4/13/2017 University of Pennsylvania

79 Grouper client traffic capture (continued)
4/13/2017 University of Pennsylvania

4/13/2017 University of Pennsylvania

81 University of Pennsylvania
Automatic samples Samples are tedious to maintain Too many combinations: Rest and Soap Lite and Batched Within Rest: XML, XHTML, JSON, HTTP params 12 operations Program automatically generates 150+ samples 4/13/2017 University of Pennsylvania

82 Automatic samples (continued)
The sample generator starts up a Java TCP listener/proxy, tunnels traffic through that, captures the request and response to the server, and creates documents 4/13/2017 University of Pennsylvania

83 University of Pennsylvania
Automatic samples See movie (make sure you have the Xvid codec): /sampleCapture.avi 4/13/2017 University of Pennsylvania

84 University of Pennsylvania
References and links cvs login cvs export -r GROUPER_1_4_BRANCH grouper-ws 4/13/2017 University of Pennsylvania 84

85 University of Pennsylvania
Questions? 4/13/2017 University of Pennsylvania

86 University of Pennsylvania
BONUS MATERIAL: MISC 4/13/2017 University of Pennsylvania 86

87 Xstream words of warning
The default JSON outputted is awful An array of size 1 is the same as a field A string with an number value is the same as a number If you use this, find settings so that it makes sense Xstream by default works by fields, not Javabean properties This has surfaced from mismatches between Xstream and Axis (which goes by Javabean properties) There is a setting to enable this, it is on my TODO list You can override some settings in Xstream to ignore extraneous XML fields (for backwards compatibility) Fields are lower case, Classes are uppercase (as aliased), and a user reported difficulty with Xpath 4/13/2017 University of Pennsylvania 87 87

88 Grouper Rest (continued)
GrouperRestServlet <servlet> <servlet-name>RestServlet</servlet-name> <display-name>WS REST Servlet</display-name> <servlet-class> </servlet-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <url-pattern>/servicesRest/*</url-pattern> </servlet-mapping> 4/13/2017 University of Pennsylvania 88 88

89 JEXL output templates (continued)
Easy to setup, get jakarta commons jexl.jar I have a utility method to substitute, something like this: JexlContext jc = JexlHelper.createContext(); for (String key: variableMap.keySet()) { jc.getVars().put(key, variableMap.get(key)); } Pattern pattern = Pattern.compile("\\$\\{(.*?)\\}"); Matcher matcher = pattern.matcher(stringToParse); StringBuilder result = new StringBuilder(); while(matcher.find()) { result.append(stringToParse.substring(index,matcher.start())); String script =; Expression e = ExpressionFactory.createExpression(script); Object o = e.evaluate(jc); result.append(o); index = matcher.end(); } 4/13/2017 University of Pennsylvania 89 89

4/13/2017 University of Pennsylvania 90

91 Grouper data model (simplified)
Oversimplified data model Stems are the folders, which are hierarchical. Non-top-level stems refer to their parent stem A group is in a stem. It has many attributes (e.g. name), which are fields. Composite groups link up 3 groups and a type (union, intersection, minus). Memberships are a combination of group, member, and field (field is the list type) Members are an internal representation of externally managed people (or groups). 4/13/2017 University of Pennsylvania 91 91

92 Object model (continued)
How to represent “Group” in a web service pojo? Where do we need “Group” in web services? Input examples: Add member to a group Save a group Delete a group See if a member is in a group Find a group Output examples: List groups for member Find group Save a group? Delete a group? 4/13/2017 University of Pennsylvania 92

93 Object model (continued)
Break “Group” into two cases Lookup Object representation Lookup (allow multiple ways) Lookup by UUID Lookup by name There are several lookups in Grouper WS: Stem lookup Subject lookup Group lookup How to handle object representation? 4/13/2017 University of Pennsylvania 93

94 Object model (continued)
Each request has its own object One for Lite request Note: each Lite request has only scalars One for Batch request Each response has its own object One for Lite response One for Batch response One for each line item of a batch response 4/13/2017 University of Pennsylvania 94 94

95 Object model (continued)
Option 1: minimal, make another request for more info For list groups for member, send back the group UUID and name If more info needed make another request for more info Option 2: send it all Just send everything about a group on all requests Option 3: pick and choose When an operation returns groups, tell the server what to return Option 4: two levels to decide Generally send back basic data If includeGroupDetail in a request, send it all Note: for saveGroup, same object goes back and forth, including an optional groupLookup on request. We chose Option 4 to minimize the number of requests if more info is needed. If there is any operation done on an object, return that object. Option 3 is not very restful. 4/13/2017 University of Pennsylvania 95 95

96 Lite vs batched (continued)
Original vision was Lite operations would require only scalars Do not need XML in request Only use HTTP params Response would still have a body Originally we used Axis REST It didn’t satisfy our requirements, so we moved to custom REST 4/13/2017 University of Pennsylvania 96

4/13/2017 University of Pennsylvania 97

98 University of Pennsylvania
Axis serving Rest Axis can serve “Rest” services (loose-Rest) Basically it is the same service without the SOAP envelope Can also pass params as HTTP params Set this in axis2.xml: <parameter name="disableREST" locked="true">false</parameter> Set this option in the client: options.setProperty( Constants.Configuration.ENABLE_REST, Constants.VALUE_TRUE); Show movie (make sure you have the Xvid codec): /axisRestExample.avi 4/13/2017 University of Pennsylvania 98 98

99 Axis serving Rest (continued)
Not clear how to customize the XML, and it is not great looking XML (not what you would create by hand) Not real Rest, since the HTTP method is still GET or POST, not PUT or DELETE Still resembles RPC calls, not resource centric calls There is a bad bug with Axis SOAP and REST where if you skip params it marshals them in the wrong order. Hopefully this will be fixed soon. Grouper abandoned Axis Rest in favor of custom Rest 4/13/2017 University of Pennsylvania 99 99

4/13/2017 University of Pennsylvania 100

101 University of Pennsylvania
Authentication Easiest way to go is HTTP basic authentication Assumes using SSL (since only Base64 encoded) Send this with Commons Http client httpClient.getParams().setAuthenticationPreemptive(true); Credentials defaultcreds = new UsernamePasswordCredentials(RestClientSettings.USER, RestClientSettings.PASS); //e.g. localhost and 8093 httpClient.getState() .setCredentials(new AuthScope(RestClientSettings.HOST, RestClientSettings.PORT), defaultcreds); 4/13/2017 University of Pennsylvania 101

102 Authentication (continued)
Results in this HTTP PUT /grouperWs/servicesRest/v1_4_000/groups/aStem%3AaGroup/members/ HTTP/1.1 Connection: close Authorization: Basic SDF423SFD423xxxx== User-Agent: Jakarta Commons-HttpClient/3.1 Host: localhost:8092 Content-Length: 72 4/13/2017 University of Pennsylvania 102

103 Authentication (continued)
Handle this on the server Solution should work with Axis or Rest Make a servlet filter for Soap and Rest: <filter><filter-name>Grouper service filter</filter-name> <filter-class></filter-class></filter> <filter-mapping> <filter-name>Grouper service filter</filter-name> <url-pattern>/services/*</url-pattern> </filter-mapping> <url-pattern>/servicesRest/*</url-pattern> 4/13/2017 University of Pennsylvania 103

104 Authentication (continued)
Keep threadlocals of request and response threadLocalRequest.set((HttpServletRequest) request); threadLocalResponse.set((HttpServletResponse) response); threadLocalRequestStartMillis.set(System.currentTimeMillis()); try { filterChain.doFilter(request, response); } finally { threadLocalRequest.remove(); threadLocalResponse.remove(); threadLocalRequestStartMillis.remove(); } 4/13/2017 University of Pennsylvania 104

105 Authentication (continued)
Utility method, called from business logic, to authenticate String authenticationClassName = GrouperWsConfig.getPropertyString( GrouperWsConfig.WS_SECURITY_NON_RAMPART_AUTHENTICATION_CLASS, WsGrouperDefaultAuthentication.class.getName()); Class<? extends WsCustomAuthentication> theClass = GrouperUtil.forName(authenticationClassName); WsCustomAuthentication wsAuthentication = GrouperUtil.newInstance(theClass); userIdLoggedIn = wsAuthentication retrieveLoggedInSubjectId(retrieveHttpServletRequest()); // cant be blank! if (StringUtils.isBlank(userIdLoggedIn)) { throw new WsInvalidQueryException("No user is logged in"); } 4/13/2017 University of Pennsylvania 105

106 Authentication – container authN
Two built in authentication methods, first is container auth public String retrieveLoggedInSubjectId(HttpServletRequest httpServletRequest) throws RuntimeException { // use this to be the user connected, or the user act-as String userIdLoggedIn = GrouperServiceJ2ee.retrieveUserPrincipalNameFromRequest(); return userIdLoggedIn; } 4/13/2017 University of Pennsylvania 106

107 Authentication – container authN
Configure in web.xml <security-constraint><web-resource-collection> <web-resource-name>Web services</web-resource-name> <url-pattern>/servicesRest/*</url-pattern> </web-resource-collection><auth-constraint> <role-name>grouper_user</role-name> </auth-constraint></security-constraint> <login-config> <auth-method>BASIC</auth-method> <realm-name>Grouper Application</realm-name> </login-config> <security-role> <description>Web service</description> </security-role> 4/13/2017 University of Pennsylvania 107

108 Authentication – container authN (continued)
Configure in tomcat-users.xml (servlet container specific) <?xml version='1.0' encoding='utf-8'?> <tomcat-users> <role rolename="grouper_user"/> <user username="mchyzer" password="mchyzer1" roles="grouper_user"/> </tomcat-users> 4/13/2017 University of Pennsylvania 108

109 Authentication – Kerberos
Alternative builtin authentication: Kerberos String authHeader = request.getHeader("Authorization"); //if no header, we cant go to kerberos if (StringUtils.isBlank(authHeader)) { LOG.error("No authorization header in HTTP"); return null; } Matcher matcher = regexPattern.matcher(authHeader); String authHeaderBase64Part = null; if (matcher.matches()) { authHeaderBase64Part =; if (StringUtils.isBlank(authHeaderBase64Part)) { LOG.error("Cant find base64 part in auth header"); return null; } 4/13/2017 University of Pennsylvania 109

110 Authentication – Kerberos (continued)
//unencrypt this byte[] base64Bytes = authHeaderBase64Part.getBytes(); byte[] unencodedBytes = Base64.decodeBase64(base64Bytes); String unencodedString = new String(unencodedBytes); //split based on user/pass String user = GrouperUtil.prefixOrSuffix(unencodedString, ":", true); String pass = GrouperUtil.prefixOrSuffix(unencodedString, ":", false); if (authenticateKerberos(user, pass)) { return user; } 4/13/2017 University of Pennsylvania 110

111 Authentication – Pluggable
Deployers can use their own authentication # to provide custom authentication (instead of the default # httpServletRequest.getUserPrincipal() # for non-Rampart authentication. Class must implement the # interface: # # .WsCustomAuthentication # class must be fully qualified. e.g. # # blank means use default: # .WsGrouperDefaultAuthentication # kerberos: # .WsGrouperKerberosAuthentication = 4/13/2017 University of Pennsylvania 111

112 Authentication – Pluggable
Implement this interface public interface WsCustomAuthentication { /** * retrieve the current username (subjectId) from the request * object. httpServletRequest the logged in username (subjectId) WsInvalidQueryException if there is a problem */ public String retrieveLoggedInSubjectId(HttpServletRequest httpServletRequest) throws WsInvalidQueryException; } 4/13/2017 University of Pennsylvania 112

113 Authentication – Caching
It is a TODO to provide the option for the an HTTP basic authN to cache a hashed version of the authentication header (and translation to subject ID) Hash this so a Base64 version is not held in memory Do not want to load down the authentication service (e.g. kerberos) if the web service is under high load 4/13/2017 University of Pennsylvania 113

114 Authentication – Ws- Security
Contributed by Sanjay Vivek at Newcastle University Axis has a module called Rampart which implements WS –security standards Important if the service requires multiple hops or proxying Useful if not using SSL Only for Soap, not for Rest Useful for authenticating without a user/pass (e.g. Kerberos ticket or x.509) Cannot run rampart along with HTTP basic authn (need multiple webapps) in Axis2 4/13/2017 University of Pennsylvania 114

115 Authentication – Ws- Security (continued)
Set a param in the web.xml <servlet> <servlet-name>AxisServlet</servlet-name> <display-name>Apache-Axis Servlet</display-name> <servlet-class></servlet-class> <load-on-startup>1</load-on-startup> <init-param> <param-name>wssec</param-name> <param-value>true</param-value> </init-param> </servlet> 4/13/2017 University of Pennsylvania 115

116 Authentication – Ws- Security (continued)
Configure axis2 GrouperServiceWssec.aar/services.xml <module ref="rampart" /> <wsp:Policy> <sp:HttpsToken RequireClientCertificate="false"/> </wsp:Policy> <ramp:RampartConfig xmlns:ramp=""> <ramp:passwordCallbackClass></ramp:passwordCallbackClass> </ramp:RampartConfig> 4/13/2017 University of Pennsylvania 116

117 Authentication – Ws- Security (continued)
Default services.xml requires implementation of interface public interface GrouperWssecAuthentication { /** * <pre> * authenticate the user, and find the subject and return. * See GrouperWssecSample for an example * </pre> wsPasswordCallback true if callback type is supported, false if not IOException if there is a problem or if user is * not authenticated correctly */ public boolean authenticate(WSPasswordCallback wsPasswordCallback) throws IOException; } 4/13/2017 University of Pennsylvania 117

118 Authentication – Ws- Security (continued)
Then you also need to configure this on the client There is an example in Includes service.xml And RampartPwHandlerClient callback for credentials 4/13/2017 University of Pennsylvania 118

119 University of Pennsylvania
Authorization Grouper has built in authorization about who is allowed to do what Who can add to which stems Who can view memberships Who can view that groups exist Who can edit group attributes Another layer is which users are allowed to use web services Since Grouper is all about groups, use a group 4/13/2017 University of Pennsylvania 119

120 Authorization (continued)
Setting in config file lists a group that users must be in to use WS (if blank allow all) 4/13/2017 University of Pennsylvania 120

121 Authorization (continued)
If not in group, log it and alert the caller (should make option to suppress this) HTTP/ Internal Server Error X-Grouper-resultCode: EXCEPTION X-Grouper-success: F <?xml version='1.0' encoding='iso '?> , params: null, java.lang.RuntimeException: User is not authorized at edu.internet2… Caused by: edu.internet2… GroupNotFoundException: Cannot find group with name: 'etc:webServiceClientUsers' Note: it is a security best practice not to give too much information to clients, so they don’t know what the next hurdle is to hack the application. However, the administrators running the service must know errors (logs), and if clients can get information which does not help them hack in, but which reduces support time, then it is good. This should able to be suppressed so that deployers can decide. 4/13/2017 University of Pennsylvania 121 121

122 Authorization (continued)
Given that callers must be in group, make it easy to setup This setting in will auto-create groups and auto-populate for ease of startup and testing 4/13/2017 University of Pennsylvania 122 122

123 Authorization (continued)
If there is an auto-created group on startup, log it :28:03,327: [main] WARN GrouperCheckConfig.checkGroup(130) - cannot find group from config: key etc:webServiceClientUsers :28:04,952: [main] WARN GrouperCheckConfig.checkGroup(149) - auto-created key etc:webServiceClientUsers :28:05,015: [main] WARN GrouperCheckConfig.checkGroups(469) - auto-added subject mchyzer to group: etc:webServiceClientUsers 4/13/2017 University of Pennsylvania 123 123

124 Authorization – actAs proxy
Sometimes you need run as a different user than logged in as E.g. to run things as the “system” [root] user Or if there is an app users are using, and you call the web service as that user Or if an admin user needs to proxy another user in the office (maybe someone on vacation) All calls have an optional actAs input <WsRestAddMemberRequest> <actAsSubjectLookup> <subjectId>GrouperSystem</subjectId> </actAsSubjectLookup> </WsRestAddMemberRequest> 4/13/2017 University of Pennsylvania 124 124

125 Authorization – actAs proxy (continued)
Any user can actAs themselves (why? Due to examples  ) GrouperSystem [root] can act as anyone You can configure groups of users who can actAs anyone (in = etc:webServiceActAsGroup Or you can specify groups who can act as users in another group (to clamp down a bit) = orgs:admins123 :::: orgs:users123 In this case users in orgs:admins123 group can only actAs any user in orgs:users123 4/13/2017 University of Pennsylvania 125 125

126 Authorization – actAs proxy (continued)
Important to audit both the logged in user and actAs user 4/13/2017 University of Pennsylvania 126 126

127 Authorization – two factor (TODO)
Factor 1: What you know (password) Factor 2: Where you are (assumes SSL) 4/13/2017 University of Pennsylvania 127 127

4/13/2017 University of Pennsylvania 128

129 University of Pennsylvania
TCP/IP monitor Need to have a way to see HTTP traffic to/from web service Network level proxy works in non-SSL only Axis has a swing tcp monitor Ethereal Axis has log settings to log the traffic from client (not server?) Eclipse has built-in TCP/IP monitor (my choice) Note: this has more uses than just web services See video (make sure you have the Xvid codec): /eclipseTcp.avi 4/13/2017 University of Pennsylvania 129

4/13/2017 University of Pennsylvania 130

131 University of Pennsylvania
Javadoc in CVS Ant script to make javadoc CVSweb ready 4/13/2017 University of Pennsylvania 131 131

132 Javadoc in CVS (continued)
4/13/2017 University of Pennsylvania 132 132

133 Javadoc in CVS (continued)
4/13/2017 University of Pennsylvania 133 133

4/13/2017 University of Pennsylvania 134

135 University of Pennsylvania
Testing There are lots of places to test web services Best place is at the client See movie (make sure you have the Xvid codec): /testing.avi Most bugs found in Grouper web services could have been avoided with more testing Web services are easier to automatically test than web applications 4/13/2017 University of Pennsylvania 135 135

Download ppt "Java Web Service Servers and Clients in Internet2 Grouper"

Similar presentations

Ads by Google