Java Web Service Servers and Clients in Internet2 Grouper February 2009 Chris Hyzer University of Pennsylvania IT Internet2

Purpose of this presentation Show how Internet2 Grouper developed web services Grouper is open source, feel free to borrow or suggest improvements Discuss the successes and areas for improvement Mention planned enhancements 4/13/2017 University of Pennsylvania

University of Pennsylvania Contents Introduction to Internet2 Grouper Introduction to web services Architecture of REST/SOAP web service SOAP web services with Axis2 (servers and clients) REST web services with xstream (servers and clients) Client Documenting web services Bonus material Security Axis serving Rest Testing 4/13/2017 University of Pennsylvania

Introduction to Internet2 Grouper 4/13/2017 University of Pennsylvania

University of Pennsylvania Internet2 Grouper Open source group management Internet2 has been working on group management for 8 years Generally used in educational institutions, but could be anywhere Funded by Internet2 4/13/2017 University of Pennsylvania

Why central group management with Grouper? Instead of apps managing own groups Reuse group lists Central place to see which groups a person is in Central auditing of group and membership actions Central management of authorization Security: Who can view/edit groups and memberships Opt-in/Opt-out Delegate authority Automatic or manual membership management Composite groups for group match: and / or / minus Groups of groups 4/13/2017 University of Pennsylvania

University of Pennsylvania Grouper architecture Note: the one-way arrow doesn’t mean the traffic originates from one side or the other. E.g. no ldap reads originate from the ldap server. 4/13/2017 University of Pennsylvania

Introduction to WEB SERVICES 4/13/2017 University of Pennsylvania

University of Pennsylvania User web request Person using browser makes a request to a server Person (user) views the results in browser, and types and or clicks to continue 4/13/2017 University of Pennsylvania

University of Pennsylvania Web service request Program makes a request to a web application Program parses the output Note, it doesn’t have to be a server which makes a web service request, it could be any program, e.g. ajax. 4/13/2017 University of Pennsylvania

Overlap of web request and web service? Ajax for example Can be kicked off by a user click Can update the screen similar to a web application However, Ajax is making the request and parsing the response, it is a web service If it doesn’t parse the output, and just puts the resultant HTML into the browser DOM, then not a web service Web service screen scraping a web application A program can “screen scrape” a web application Beware of changes in the HTML! This is not a web service Is a browser an application making requests, are all user requests web services? 4/13/2017 University of Pennsylvania

University of Pennsylvania Why web services http(s) is a well understood protocol by programming languages and programmers Ports 80/443 might already be available in firewall rules Http is text based (easy to debug) Http is not programming language specific, so the server technology can be different than the client (e.g. ajax) Webpages are either XML, XHTML, or XML-like (e.g. HTML) Most programming languages have XML libraries Note: web services do not have to be XML, though generally the are Development and production environments might be similar (or same) to existing web applications Penn generally communicates between systems with WS 4/13/2017 University of Pennsylvania

University of Pennsylvania SOAP web services Simple Object Access Protocol Specifies how web service messages are exchanged W3C standard Must use XML and XML schema for data Messages have XML envelopes, headers, body, exception handling Web Service Description Language (WSDL) describes the SOAP messages in a programmatic way (XML) Many features (security, error handling, caching, resource discovery, etc) Many programming languages generate SOAP Not considered light-weight 4/13/2017 University of Pennsylvania

University of Pennsylvania Example SOAP request <?xml version='1.0' encoding='UTF-8'?> <soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"> <soapenv:Body> <ns1:addMemberLite xmlns:ns1="http://soap.ws.grouper.middleware.internet2.edu/xsd"> <ns1:groupName>aStem:aGroup</ns1:groupName> <ns1:subjectIdentifier>mchyzer</ns1:subjectIdentifier> </ns1:addMemberLite> </soapenv:Body> </soapenv:Envelope> 4/13/2017 University of Pennsylvania

University of Pennsylvania REST web services Representational State Transfer Two definitions: (strict or RESTful): Protocol that specifies how HTTP (perhaps) and XML are used for web services (non-strict): Any web service that does not have the overhead of SOAP. Aka Remote Procedure Call (RPC) 4/13/2017 University of Pennsylvania

University of Pennsylvania RESTful web services The web services are organized like static web resources URL’s represent resources, not operations HTTP methods indicate the operations. Generally: GET, POST (update), PUT (insert), DELETE. Can use more, or custom Messages can be HTML so that systems or browsers can consume them 4/13/2017 University of Pennsylvania

Example REST web service PUT /grouperWs/servicesRest/xhtml/v1_4_000/groups/aStem%3AaGroup/members/mchyzer This means add this member to the group Note, there can be body here, though in this case it isn’t needed 4/13/2017 University of Pennsylvania

HTTP / XML / RPC / Hybrid web services Each service makes its own standards URL can be: Resource: http://localhost/grouperWs/group Operation: http://localhost/grouperWs/addGroup Generally just use POST or GET as the HTTP method The XML document sent can be: Complete object representation: <group><name>myGroup</name><desc>MyGroup</desc></group> Operational: <groupChangeName><groupId>123</groupId> <newName>myGroup2</newName></groupChangeName> 4/13/2017 University of Pennsylvania

University of Pennsylvania GROUPER WEB SERVICES 4/13/2017 University of Pennsylvania

University of Pennsylvania Grouper web services Generally web services are programmed to host a service Grouper is software, so its WS are programmed so institutions can download and host services E.g. Grouper is app server and database server agnostic Requirements Dozen operations SOAP and REST (as close to RESTful as possible) SOAP and REST should deploy in one webapp Simple operations (Lite), and batched operations Pluggable authentication Documented well Versioned (generally Grouper has bi-annual releases) 4/13/2017 University of Pennsylvania

Grouper web service operations findGroups findStems stemSave stemDelete memberChangeSubject getGrouperPrivileges assignGrouperPrivileges addMember deleteMember getMembers hasMember getGroups groupSave groupDelete 4/13/2017 University of Pennsylvania

University of Pennsylvania Lite vs batched One batched operation has less overhead than many smaller operations (performance test for yourself to validate) Benchmarks Add group (inserts), requires many queries 100 batches of 1 add groups take 19.8 seconds 10 batches of 10 add groups take 12.4 seconds 5 batches of 20 add groups take 12.0 seconds Has member, lightweight, readonly 100 batches of 1 hasMember checks take 8.3 seconds 10 batches of 10 checks take 1.9 seconds 5 batches of 20 checks take 1.6 seconds Benchmarks notes Completely run on developer PC, local mysql E.g. WsSampleHasMemberRest100 4/13/2017 University of Pennsylvania

University of Pennsylvania Object model Assume web service data are simple POJOs (Plain Old Java Objects) Use only: Beans Arrays (of beans or simple types) Simple types: String, Integer Note: Don’t use Collections, enums, dates, timestamps, booleans 4/13/2017 University of Pennsylvania

Object model (continued) What makes up the data of a group? Here is a simple group 4/13/2017 University of Pennsylvania

Object model (continued) What makes up the data of a group? Here is a more complex group 4/13/2017 University of Pennsylvania

Object model (continued) 4/13/2017 University of Pennsylvania

Object model (continued) Request object model 4/13/2017 University of Pennsylvania

Object model (continued) Response object model 4/13/2017 University of Pennsylvania

Object model (continued) Metadata object model Response metadata is one per response Result metadata is one per Lite response, or one per each line item in batch Success: T|F Result code: many enums Codes also in HTTP headers 4/13/2017 University of Pennsylvania

Object model (continued) Operations should be idempotent if possible If they are sent twice, generally it is ok Delete member mchyzer from group etc:sysAdminGroup Idempotent Delete the first member of group etc:sysAdminGroup NOT idempotent 4/13/2017 University of Pennsylvania 30 30

SOAP web services with Axis2 4/13/2017 University of Pennsylvania

University of Pennsylvania Axis architecture Axis can do many things, but how Grouper uses Axis is: Servlet to accept web requests and call Grouper’s business logic Generates WSDL from business logic Generates sample client library from WSDL 4/13/2017 University of Pennsylvania

Business logic for Axis Create a class (GrouperService) Contains only instance methods of business logic Each method takes all fields of the input bean, and returns the output bean Each bean is only simple pojo (uses Javabean properties) Note the Lite methods only take scalars are inputs 4/13/2017 University of Pennsylvania

Business logic for Axis (continued) 4/13/2017 University of Pennsylvania

Business logic for Axis (continued) GrouperService isn’t great for Javadoc since enums are strings Delegate to GrouperServiceLogic Decode booleans, dates, enums, etc 4/13/2017 University of Pennsylvania

University of Pennsylvania Axis generate WSDL Generate WSDL from POJOs and GrouperService class <target name="java2wsdl" description="convert the java to a wsdl"> <touch file="${generated.client.project.dir}/GrouperService.wsdl" /> <delete file="${generated.client.project.dir}/GrouperService.wsdl" /> <java classname="org.apache.ws.java2wsdl.Java2WSDL" fork="true"> <classpath refid="ws.class.path" /> <arg value="-o" /><arg value="${generated.client.project.dir}" /> <arg value="-of" /><arg value="GrouperService.wsdl" /> <arg value="-cn" /> <arg value="edu.internet2.middleware.grouper.ws.soap.GrouperService" /> <arg value="-stn" /> <arg value="http://soap.ws.grouper.middleware.internet2.edu/xsd" /> <arg value="-l" /> <arg value="http://localhost:8090/grouper-ws/services/GrouperService" /> </target> 4/13/2017 University of Pennsylvania

Axis generate WSDL (continued) Generate WSDL from POJOs and GrouperService class C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws>ant java2wsdl Buildfile: build.xml java2wsdl: [delete] Deleting: C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws-java -generated-client\GrouperService.wsdl BUILD SUCCESSFUL Total time: 9 seconds C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws> 4/13/2017 University of Pennsylvania

Axis generate WSDL (continued) Result 4/13/2017 University of Pennsylvania

Axis generate WSDL (continued) Result – 2000 lines of SOAP definition This might be idealistic, but Grouper is trying not to touch this WSDL file, and only use the one generated by Axis. Note this is in CVS so we can diff revisions, and it is versioned (e.g. get the WSDL from a particular Grouper branch or version). Features in WSDL which are not in Axis generated WSDL are not being used, e.g. enums. 4/13/2017 University of Pennsylvania

University of Pennsylvania Axis generate client Ant script to generate SOAP client from WSDL (any WSDL) <target name="wsdl2java" description="convert the wsdl to a java client"> <delete><fileset dir="${generated.client.project.dir}"> <include name=“…” /></fileset></delete> <java classname="org.apache.axis2.wsdl.WSDL2Java" fork="true"> <classpath refid="ws.class.path" /> <arg value="-uri" /> <arg file="${generated.client.project.dir}/GrouperService.wsdl" /> <arg value="-t" /><arg value="-u" /><arg value="-p" /> <arg value="edu.internet2.middleware.grouper.webservicesClient" /> <arg value="-o" /><arg value="${generated.client.project.dir}" /> </java> </target> Note: we used to have one large class (with inner classes), but now we do separate classes so we don’t have a class several megs large), this is the –u option 4/13/2017 University of Pennsylvania

Axis generate client (continued) Run ant script to generate client C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws>ant wsdl2java Buildfile: build.xml wsdl2java: [java] Retrieving document at 'C:\dev_inst\eclipse\workspace\grouper_v1_4\g rouper-ws-java-generated-client\GrouperService.wsdl'. BUILD SUCCESSFUL Total time: 9 seconds C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws> 4/13/2017 University of Pennsylvania

Axis generate client (continued) Result: 100 classes, ~5megs of source 4/13/2017 University of Pennsylvania

University of Pennsylvania Axis archive Axis needs an AAR file of logic, create via ant to WEB-INF/services/GrouperService.aar <target name="generate-aar" depends="compile"> <property name="webservice.folder" value="${basedir}/webservices" /> <delete dir="${webservice.folder}/classes" /> <copy toDir="${webservice.folder}/classes" failonerror="false"> <fileset dir="${build.dir.grouper-ws}"> <include name="edu/internet2/middleware/grouper/ws/**/*.class" /> </fileset></copy> <jar destfile="${basedir}/webapp/WEB-INF/services/GrouperService.aar"> <fileset excludes="edu/internet2/middleware/grouper/ws/**/*Test.class" dir="${webservice.folder}/classes" /> <fileset dir="webservices/GrouperService.aar" /> </jar> </target> 4/13/2017 University of Pennsylvania

University of Pennsylvania Axis configuration Configure the web.xml for Axis <servlet> <servlet-name>AxisServlet</servlet-name> <display-name>Apache-Axis Servlet</display-name> <servlet-class> edu.internet2.middleware.grouper.ws.GrouperServiceAxisServlet </servlet-class> <load-on-startup>1</load-on-startup> <!-- hint that this is the wssec servlet --> <!-- init-param> <param-name>wssec</param-name> <param-value>true</param-value> </init-param --> </servlet> <servlet-mapping> <servlet-name>AxisServlet</servlet-name> <url-pattern>/services/*</url-pattern> </servlet-mapping> 4/13/2017 University of Pennsylvania

Axis configuration (continued) Boilerplate WEB-INF/conf/axis2.xml WEB-INF/modules/*.mar WEB-INF/modules/modules.list WEB-INF/services/*.aar (including GrouperService.aar) WEB-INF/service/services.list WEB-INF/lib/ (50 axis jars) 4/13/2017 University of Pennsylvania

University of Pennsylvania Axis Example See video (make sure you have the Xvid codec): https://wiki.internet2.edu/confluence/download/attachments/3014660/soapExample.avi 4/13/2017 University of Pennsylvania

REST web services with xstream 4/13/2017 University of Pennsylvania

University of Pennsylvania Xstream Easy to use Java object to XML processor In this example I alias the class names so they arent so long public class XstreamPocGroup { public XstreamPocGroup(String theName, XstreamPocMember[] theMembers) { this.name = theName; this.members = theMembers; } private String name; private XstreamPocMember[] members; public String getName() { ... 4/13/2017 University of Pennsylvania

University of Pennsylvania Xstream (continued) This is the child bean public class XstreamPocMember { public XstreamPocMember(String theName, String theDescription) { this.name = theName; this.description = theDescription; } private String name; private String description; 4/13/2017 University of Pennsylvania

University of Pennsylvania Xstream (continued) public static void main(String[] args) { XstreamPocGroup group = new XstreamPocGroup("myGroup", new XstreamPocMember[]{ new XstreamPocMember("John", "John Smith - Employee"), new XstreamPocMember("Mary", "Mary Johnson - Student")}); XStream xStream = new XStream(new XppDriver()); xStream.alias("XstreamPocGroup", XstreamPocGroup.class); xStream.alias("XstreamPocMember", XstreamPocMember.class); StringWriter stringWriter = new StringWriter(); xStream.marshal(group, new CompactWriter(stringWriter)); String xml = stringWriter.toString(); System.out.println(GrouperUtil.indent(xml, true)); group = (XstreamPocGroup)xStream.fromXML(xml); System.out.println(group.getName() + ", number of members:" + group.getMembers().length); } 4/13/2017 University of Pennsylvania

University of Pennsylvania Xstream (continued) <XstreamPocGroup> <name>myGroup</name> <members> <XstreamPocMember> <name>John</name> <description>John Smith - Employee</description> </XstreamPocMember> <name>Mary</name> <description>Mary Johnson - Student</description> </members> </XstreamPocGroup> myGroup, number of members: 2 4/13/2017 University of Pennsylvania

University of Pennsylvania Xstream JSON You can control things with annotations @XStreamOmitField private Group group = null; You can convert Java object to JSON XStream xStream = new XStream(new JettisonMappedXmlDriver()); {"XstreamPocGroup": {"name":"myGroup","members": {"XstreamPocMember":[ {"name":"John","description":"John Smith - Employee"}, {"name":"Mary","description":"Mary Johnson - Student"} ]}} } myGroup, number of members: 2 4/13/2017 University of Pennsylvania

University of Pennsylvania XHTML output Grouper had a requirement to have XHTML output Rest people like XHTML, it can turn a web service into a browsable application I have some reservations about that (what are the clients? Browsers, or screen scraping applications? Can you change the output?) I rolled my own bean->XHTML converter Based on XmlStreamWriter for output Based on JDOM for input 4/13/2017 University of Pennsylvania

XHTML output (continued) WsXhtmlOutputConverter wsXhtmlOutputConverter = new WsXhtmlOutputConverter(true, null); StringWriter stringWriter = new StringWriter(); wsXhtmlOutputConverter.writeBean(group, stringWriter); String xhtml = stringWriter.toString(); System.out.println(GrouperUtil.indent(xhtml, true)); WsXhtmlInputConverter wsXhtmlInputConverter = new WsXhtmlInputConverter(); wsXhtmlInputConverter.addAlias("XstreamPocGroup", XstreamPocGroup.class); wsXhtmlInputConverter.addAlias("XstreamPocMember", XstreamPocMember.class); group = (XstreamPocGroup)wsXhtmlInputConverter.parseXhtmlString(xhtml); System.out.println(group.getName() + ", number of members: " + group.getMembers().length); 4/13/2017 University of Pennsylvania

XHTML output (continued) <?xml version='1.0' encoding='iso-8859-1'?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title>XstreamPocGroup</title> </head> <body> <div title="XstreamPocGroup"> <ul class="members"> <li title="XstreamPocMember"> <p class="name">John</p> <p class="description">John Smith - Employee</p> </li> 4/13/2017 University of Pennsylvania

XHTML output (continued) <li title="XstreamPocMember"> <p class="name">Mary</p> <p class="description">Mary Johnson - Student</p> </li> </ul> <p class="name">myGroup</p> </div> </body> </html> myGroup, number of members: 2 Note: Im not sure why this would be useful… 4/13/2017 University of Pennsylvania

University of Pennsylvania HTTP input Similar to Axis input, REST (Lite) should accept HTTP params (URL params or in HTTP body) PUT /grouperWs/servicesRest/v1_4_000/groups /aStem%3AaGroup/members/10021368 HTTP/1.1 Connection: close Authorization: Basic xxxxxxxxxxxxxxxxx== User-Agent: Jakarta Commons-HttpClient/3.1 Host: localhost:8092 Content-Length: 72 wsLiteObjectType=WsRestAddMemberLiteRequest &actAsSubjectId=GrouperSystem 4/13/2017 University of Pennsylvania

Indenting of XML, XHTML, JSON Couldn’t find a 3rd party indenter, rolled my own System.out.println(GrouperUtil.indent(xhtml, true)); System.out.println(GrouperUtil.indent(json, true)); System.out.println(GrouperUtil.indent(xml, true)); This is valuable when showing examples 4/13/2017 University of Pennsylvania

Make the operations more Restful URLs need to represent hierarchical resources: /servicesRest/v1_4_000/groups/aStem%3AaGroup/members Servlet, client version, top level “folder”, name of item, inner “folder” The client version is in there since that is the first part of versioning GET that URL would retrieve all the members of that group POST would update the list (send all new members) PUT would create a new list DELETE would delete all the members …/groups/aStem%3AaGroup/members/mchyzer Drilling down further GET could see if mchyzer is a member PUT would add mchyzer to group etc 4/13/2017 University of Pennsylvania

University of Pennsylvania Problems with Rest Some things are programmatic, not resource based (e.g. a fixBadMemberships diagnostic tool) The input params could be complicated, e.g. Get the membership list of a group Get only immediate members Act as a different user (proxy) Get the student list, not the faculty list There might be many different ways to ID a resource E.g. name or UUID Composite ID’s of resources are not convenient /groups/aStem:aGroup/members/sourceId/someSource/subjectId/1234 4/13/2017 University of Pennsylvania

Problems with Rest (continued) The only HTTP methods which take a body are POST and PUT How can you delete something with XML constraints (actAs proxy, certain list type, include details on result) if a DELETE method does not take an XML body??? Rest is supposed to “model the web” Which web? Static? When is the last time you deleted a static resource with an HTTP DELETE method? The web is more and more dynamic, so Rest/HTTP/XML/hybrid might be more similar for developers 4/13/2017 University of Pennsylvania

University of Pennsylvania Grouper Rest URLs are resource based HTTP methods are honored (GET/POST/PUT/DELETE) If there is a body (POST/PUT), then the object type sent will trump the HTTP method (can DELETE with metadata) HTTP status codes are sent Though they really shouldn’t be read, they don’t mean much, 404 could be a success HTTP headers on the response boolean to determine if the operation is a success or failure Enum based text status code which is specific to the operation 4/13/2017 University of Pennsylvania

Grouper Rest (continued) The same operations are exposed as SOAP The XML document can specify the data, or the URL/HTTP method Could add member like this: PUT /grouperWs/servicesRest/xhtml/v1_4_000 /groups/aStem%3AaGroup/members/10021368 HTTP/1.1 Or you could add a member with a POST and a body of the right object type: POST /grouperWs/servicesRest/v1_4_001/groups /aStem%3AaGroup/members HTTP/1.1 <WsRestAddMemberLiteRequest><subjectId>10021368</subjectId> <actAsSubjectId>GrouperSystem</actAsSubjectId> </WsRestAddMemberLiteRequest> 4/13/2017 University of Pennsylvania 63 63

Grouper Rest Architecture 4/13/2017 University of Pennsylvania

Grouper Rest (continued) Client can use same beans as server (not true with Axis) Enums for content types Use Jakarta HTTP client for communication (Axis uses this too) Show movie of Rest client (make sure you have the Xvid codec): https://wiki.internet2.edu/confluence/download/attachments/3014660/restDemo.avi 4/13/2017 University of Pennsylvania

WHICH WEB SERVICE ARCHITECTURE SHOULD I USE? 4/13/2017 University of Pennsylvania 66

University of Pennsylvania When SOAP More complex operations (describe in WSDL) Clients can handle SOAP Client code generation WS-* security (e.g. kerberos ticket authentication) 4/13/2017 University of Pennsylvania 67

University of Pennsylvania When Restful Simple operations Non batched Simple resources Nice to not have composite identifiers Operations with little or no metadata (e.g. actAs) Clients are known to handle Rest HTTP methods Perhaps not for Ajax (limitation might be HTTP response code and HTTP methods) 4/13/2017 University of Pennsylvania 68

When Rest / HTTP / XML hybrid / POX When supporting “Rest” and SOAP Since SOAP is not resource based Disparate clients Need to send a body of metadata with a GET or DELETE 4/13/2017 University of Pennsylvania 69

University of Pennsylvania Client 4/13/2017 University of Pennsylvania

University of Pennsylvania GrouperClient Architecture 4/13/2017 University of Pennsylvania

University of Pennsylvania GrouperClient Its nice to give a packaged client with web services Writing HTTP/XML does not make a quick start Anyone can use it command line Can use as Java library Can make custom XML samples Debugging tools More samples Rest only (since easiest to version, most lightweight… doesn’t have 50 Axis jars!) Since command line, its one jar Can be used along side other jars 4/13/2017 University of Pennsylvania

GrouperClient (continued) Very simple POJOs Refactor 3rd party libs Xstream HttpClient Commons-logging Jexl: expression language morphString So they don’t conflict Want only one jar 4/13/2017 University of Pennsylvania

GrouperClient (continued) Example: C:\gc>java -jar grouperClient.jar --operation=getGroupsWs --subjectIds=10021368 SubjectIndex 0: success: T: code: SUCCESS: subject: 10021368: groupIndex: 0: aStem:aGroup SubjectIndex 0: success: T: code: SUCCESS: subject: 10021368: groupIndex: 1: etc:webServiceClientUsers SubjectIndex 0: success: T: code: SUCCESS: subject: 10021368: groupIndex: 2: etc:sysadmingroup Show movie (make sure you have the Xvid codec): https://wiki.internet2.edu/confluence/download/attachments/3014660/grouperClient.avi 4/13/2017 University of Pennsylvania

GrouperClient (continued) Java API No longer need to deal with httpClient or authentication Error handling built in public static void main(String[] args) { WsGetGroupsResults wsGetGroupsResults = new GcGetGroups() .addSubjectLookup(new WsSubjectLookup("10021368", null, null)).execute(); WsGetGroupsResult wsGroupsResult = wsGetGroupsResults.getResults()[0]; for (WsGroup wsGroup : wsGroupsResult.getWsGroups()) { System.out.println(wsGroup.getName()); } } aStem:aGroup etc:webServiceClientUsers etc:sysadmingroup 4/13/2017 University of Pennsylvania

GrouperClient – external encrypted passwords Set encrypt key in grouper.client.properties encrypt.key = fnh453hfbdw Encrypt the password: C:\gc>java -jar grouperClient.jar --operation=encryptPassword Type the string to encrypt (note: pasting might echo it back): Encrypted password: mpAdW53ekchSGAX3vq1UiQ== C:\gc> Put this in a file, refer to file in grouper.client.properties grouperClient.webService.password = c:/gc/ws.pass (more) sanitized config files for email or source control Perhaps auditing requirement forbidding clear text passwords 4/13/2017 University of Pennsylvania

GrouperClient – JEXL output templates If using command line utility in prod, will screenscrape STDOUT C:\gc>java -jar grouperClient.jar --operation=getGroupsWs --subjectIds=10021368 SubjectIndex 0: success: T: code: SUCCESS: subject: 10021368: groupIndex: 0: aStem:aGroup To give flexibility, and ability to change defaults, template --subjectIds=10021368 -–outputTemplate ="${groupIndex+1} groupName: ${wsGroup.name}$newline$" 1 groupName: aStem:aGroup 2 groupName: etc:webServiceClientUsers 3 groupName: etc:sysadmingroup 4/13/2017 University of Pennsylvania

Grouper client traffic capture To help troubleshoot or create samples grouperClient.logging.webService.documentDir = c:/gc/xmls grouperClient.logging.webService.indent = true Organize files so they are easy to archive or delete 4/13/2017 University of Pennsylvania

Grouper client traffic capture (continued) 4/13/2017 University of Pennsylvania

DOCUMENTING WEB SERVICES: AUTOMATIC SAMPLES 4/13/2017 University of Pennsylvania

University of Pennsylvania Automatic samples Samples are tedious to maintain Too many combinations: Rest and Soap Lite and Batched Within Rest: XML, XHTML, JSON, HTTP params 12 operations Program automatically generates 150+ samples 4/13/2017 University of Pennsylvania

Automatic samples (continued) The sample generator starts up a Java TCP listener/proxy, tunnels traffic through that, captures the request and response to the server, and creates documents 4/13/2017 University of Pennsylvania

University of Pennsylvania Automatic samples See movie (make sure you have the Xvid codec): https://wiki.internet2.edu/confluence/download/attachments/3014660/sampleCapture.avi 4/13/2017 University of Pennsylvania

University of Pennsylvania References and links http://middleware.internet2.edu/dir/groups/grouper/ https://wiki.internet2.edu/confluence/display/GrouperWG/v1.4.0+Grouper+Web+Services cvs -d:pserver:anoncvs@anoncvs.internet2.edu:/home/cvs/i2mi login cvs -d:pserver:anoncvs@anoncvs.internet2.edu:/home/cvs/i2mi export -r GROUPER_1_4_BRANCH grouper-ws http://www.w3schools.com/soap/default.asp http://en.wikipedia.org/wiki/Representational_State_Transfer 4/13/2017 University of Pennsylvania 84

University of Pennsylvania Questions? 4/13/2017 University of Pennsylvania

University of Pennsylvania BONUS MATERIAL: MISC 4/13/2017 University of Pennsylvania 86

Xstream words of warning The default JSON outputted is awful An array of size 1 is the same as a field A string with an number value is the same as a number If you use this, find settings so that it makes sense Xstream by default works by fields, not Javabean properties This has surfaced from mismatches between Xstream and Axis (which goes by Javabean properties) There is a setting to enable this, it is on my TODO list You can override some settings in Xstream to ignore extraneous XML fields (for backwards compatibility) Fields are lower case, Classes are uppercase (as aliased), and a user reported difficulty with Xpath 4/13/2017 University of Pennsylvania 87 87

Grouper Rest (continued) GrouperRestServlet <servlet> <servlet-name>RestServlet</servlet-name> <display-name>WS REST Servlet</display-name> <servlet-class> edu.internet2.middleware.grouper.ws.rest.GrouperRestServlet </servlet-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <url-pattern>/servicesRest/*</url-pattern> </servlet-mapping> 4/13/2017 University of Pennsylvania 88 88

JEXL output templates (continued) Easy to setup, get jakarta commons jexl.jar I have a utility method to substitute, something like this: JexlContext jc = JexlHelper.createContext(); for (String key: variableMap.keySet()) { jc.getVars().put(key, variableMap.get(key)); } Pattern pattern = Pattern.compile("\\$\\{(.*?)\\}"); Matcher matcher = pattern.matcher(stringToParse); StringBuilder result = new StringBuilder(); while(matcher.find()) { result.append(stringToParse.substring(index,matcher.start())); String script = matcher.group(1); Expression e = ExpressionFactory.createExpression(script); Object o = e.evaluate(jc); result.append(o); index = matcher.end(); } 4/13/2017 University of Pennsylvania 89 89

BONUS MATERIAL: GROUPER OBJECT MODEL 4/13/2017 University of Pennsylvania 90

Grouper data model (simplified) Oversimplified data model Stems are the folders, which are hierarchical. Non-top-level stems refer to their parent stem A group is in a stem. It has many attributes (e.g. name), which are fields. Composite groups link up 3 groups and a type (union, intersection, minus). Memberships are a combination of group, member, and field (field is the list type) Members are an internal representation of externally managed people (or groups). 4/13/2017 University of Pennsylvania 91 91

Object model (continued) How to represent “Group” in a web service pojo? Where do we need “Group” in web services? Input examples: Add member to a group Save a group Delete a group See if a member is in a group Find a group Output examples: List groups for member Find group Save a group? Delete a group? 4/13/2017 University of Pennsylvania 92

Object model (continued) Break “Group” into two cases Lookup Object representation Lookup (allow multiple ways) Lookup by UUID Lookup by name There are several lookups in Grouper WS: Stem lookup Subject lookup Group lookup How to handle object representation? 4/13/2017 University of Pennsylvania 93

Object model (continued) Each request has its own object One for Lite request Note: each Lite request has only scalars One for Batch request Each response has its own object One for Lite response One for Batch response One for each line item of a batch response 4/13/2017 University of Pennsylvania 94 94

Object model (continued) Option 1: minimal, make another request for more info For list groups for member, send back the group UUID and name If more info needed make another request for more info Option 2: send it all Just send everything about a group on all requests Option 3: pick and choose When an operation returns groups, tell the server what to return Option 4: two levels to decide Generally send back basic data If includeGroupDetail in a request, send it all Note: for saveGroup, same object goes back and forth, including an optional groupLookup on request. We chose Option 4 to minimize the number of requests if more info is needed. If there is any operation done on an object, return that object. Option 3 is not very restful. 4/13/2017 University of Pennsylvania 95 95

Lite vs batched (continued) Original vision was Lite operations would require only scalars Do not need XML in request Only use HTTP params Response would still have a body Originally we used Axis REST It didn’t satisfy our requirements, so we moved to custom REST 4/13/2017 University of Pennsylvania 96

BONUS MATERIAL: AXIS SERVING REST 4/13/2017 University of Pennsylvania 97

University of Pennsylvania Axis serving Rest Axis can serve “Rest” services (loose-Rest) Basically it is the same service without the SOAP envelope Can also pass params as HTTP params Set this in axis2.xml: <parameter name="disableREST" locked="true">false</parameter> Set this option in the client: options.setProperty( Constants.Configuration.ENABLE_REST, Constants.VALUE_TRUE); Show movie (make sure you have the Xvid codec): https://wiki.internet2.edu/confluence/download/attachments/3014660/axisRestExample.avi 4/13/2017 University of Pennsylvania 98 98

Axis serving Rest (continued) Not clear how to customize the XML, and it is not great looking XML (not what you would create by hand) Not real Rest, since the HTTP method is still GET or POST, not PUT or DELETE Still resembles RPC calls, not resource centric calls There is a bad bug with Axis SOAP and REST where if you skip params it marshals them in the wrong order. Hopefully this will be fixed soon. https://issues.apache.org/jira/browse/AXIS2-3364 Grouper abandoned Axis Rest in favor of custom Rest 4/13/2017 University of Pennsylvania 99 99

BONUS MATERIAL: WEB SERVICE SECURITY 4/13/2017 University of Pennsylvania 100

University of Pennsylvania Authentication Easiest way to go is HTTP basic authentication Assumes using SSL (since only Base64 encoded) Send this with Commons Http client httpClient.getParams().setAuthenticationPreemptive(true); Credentials defaultcreds = new UsernamePasswordCredentials(RestClientSettings.USER, RestClientSettings.PASS); //e.g. localhost and 8093 httpClient.getState() .setCredentials(new AuthScope(RestClientSettings.HOST, RestClientSettings.PORT), defaultcreds); 4/13/2017 University of Pennsylvania 101

Authentication (continued) Results in this HTTP PUT /grouperWs/servicesRest/v1_4_000/groups/aStem%3AaGroup/members/10021368 HTTP/1.1 Connection: close Authorization: Basic SDF423SFD423xxxx== User-Agent: Jakarta Commons-HttpClient/3.1 Host: localhost:8092 Content-Length: 72 4/13/2017 University of Pennsylvania 102

Authentication (continued) Handle this on the server Solution should work with Axis or Rest Make a servlet filter for Soap and Rest: <filter><filter-name>Grouper service filter</filter-name> <filter-class>edu.internet2.middleware.grouper.ws.GrouperServiceJ2ee</filter-class></filter> <filter-mapping> <filter-name>Grouper service filter</filter-name> <url-pattern>/services/*</url-pattern> </filter-mapping> <url-pattern>/servicesRest/*</url-pattern> 4/13/2017 University of Pennsylvania 103

Authentication (continued) Keep threadlocals of request and response threadLocalRequest.set((HttpServletRequest) request); threadLocalResponse.set((HttpServletResponse) response); threadLocalRequestStartMillis.set(System.currentTimeMillis()); try { filterChain.doFilter(request, response); } finally { threadLocalRequest.remove(); threadLocalResponse.remove(); threadLocalRequestStartMillis.remove(); } 4/13/2017 University of Pennsylvania 104

Authentication (continued) Utility method, called from business logic, to authenticate String authenticationClassName = GrouperWsConfig.getPropertyString( GrouperWsConfig.WS_SECURITY_NON_RAMPART_AUTHENTICATION_CLASS, WsGrouperDefaultAuthentication.class.getName()); Class<? extends WsCustomAuthentication> theClass = GrouperUtil.forName(authenticationClassName); WsCustomAuthentication wsAuthentication = GrouperUtil.newInstance(theClass); userIdLoggedIn = wsAuthentication .retrieveLoggedInSubjectId(retrieveHttpServletRequest()); // cant be blank! if (StringUtils.isBlank(userIdLoggedIn)) { throw new WsInvalidQueryException("No user is logged in"); } 4/13/2017 University of Pennsylvania 105

Authentication – container authN Two built in authentication methods, first is container auth public String retrieveLoggedInSubjectId(HttpServletRequest httpServletRequest) throws RuntimeException { // use this to be the user connected, or the user act-as String userIdLoggedIn = GrouperServiceJ2ee.retrieveUserPrincipalNameFromRequest(); return userIdLoggedIn; } 4/13/2017 University of Pennsylvania 106

Authentication – container authN Configure in web.xml <security-constraint><web-resource-collection> <web-resource-name>Web services</web-resource-name> <url-pattern>/servicesRest/*</url-pattern> </web-resource-collection><auth-constraint> <role-name>grouper_user</role-name> </auth-constraint></security-constraint> <login-config> <auth-method>BASIC</auth-method> <realm-name>Grouper Application</realm-name> </login-config> <security-role> <description>Web service</description> </security-role> 4/13/2017 University of Pennsylvania 107

Authentication – container authN (continued) Configure in tomcat-users.xml (servlet container specific) <?xml version='1.0' encoding='utf-8'?> <tomcat-users> <role rolename="grouper_user"/> <user username="mchyzer" password="mchyzer1" roles="grouper_user"/> </tomcat-users> 4/13/2017 University of Pennsylvania 108

Authentication – Kerberos Alternative builtin authentication: Kerberos String authHeader = request.getHeader("Authorization"); //if no header, we cant go to kerberos if (StringUtils.isBlank(authHeader)) { LOG.error("No authorization header in HTTP"); return null; } Matcher matcher = regexPattern.matcher(authHeader); String authHeaderBase64Part = null; if (matcher.matches()) { authHeaderBase64Part = matcher.group(1); if (StringUtils.isBlank(authHeaderBase64Part)) { LOG.error("Cant find base64 part in auth header"); return null; } 4/13/2017 University of Pennsylvania 109

Authentication – Kerberos (continued) //unencrypt this byte[] base64Bytes = authHeaderBase64Part.getBytes(); byte[] unencodedBytes = Base64.decodeBase64(base64Bytes); String unencodedString = new String(unencodedBytes); //split based on user/pass String user = GrouperUtil.prefixOrSuffix(unencodedString, ":", true); String pass = GrouperUtil.prefixOrSuffix(unencodedString, ":", false); if (authenticateKerberos(user, pass)) { return user; } 4/13/2017 University of Pennsylvania 110

Authentication – Pluggable Deployers can use their own authentication # to provide custom authentication (instead of the default # httpServletRequest.getUserPrincipal() # for non-Rampart authentication. Class must implement the # interface: # edu.internet2.middleware.grouper.ws.security # .WsCustomAuthentication # class must be fully qualified. e.g. # edu.school.whatever.MyAuthenticator # blank means use default: # .WsGrouperDefaultAuthentication # kerberos: edu.internet2.middleware.grouper.ws.security # .WsGrouperKerberosAuthentication ws.security.non-rampart.authentication.class = 4/13/2017 University of Pennsylvania 111

Authentication – Pluggable Implement this interface public interface WsCustomAuthentication { /** * retrieve the current username (subjectId) from the request * object. * @param httpServletRequest * @return the logged in username (subjectId) * @throws WsInvalidQueryException if there is a problem */ public String retrieveLoggedInSubjectId(HttpServletRequest httpServletRequest) throws WsInvalidQueryException; } 4/13/2017 University of Pennsylvania 112

Authentication – Caching It is a TODO to provide the option for the an HTTP basic authN to cache a hashed version of the authentication header (and translation to subject ID) Hash this so a Base64 version is not held in memory Do not want to load down the authentication service (e.g. kerberos) if the web service is under high load 4/13/2017 University of Pennsylvania 113

Authentication – Ws- Security Contributed by Sanjay Vivek at Newcastle University Axis has a module called Rampart which implements WS –security standards Important if the service requires multiple hops or proxying Useful if not using SSL Only for Soap, not for Rest Useful for authenticating without a user/pass (e.g. Kerberos ticket or x.509) Cannot run rampart along with HTTP basic authn (need multiple webapps) in Axis2 4/13/2017 University of Pennsylvania 114

Authentication – Ws- Security (continued) Set a param in the web.xml <servlet> <servlet-name>AxisServlet</servlet-name> <display-name>Apache-Axis Servlet</display-name> <servlet-class>edu.internet2.middleware.grouper.ws.GrouperServiceAxisServlet</servlet-class> <load-on-startup>1</load-on-startup> <init-param> <param-name>wssec</param-name> <param-value>true</param-value> </init-param> </servlet> 4/13/2017 University of Pennsylvania 115

Authentication – Ws- Security (continued) Configure axis2 GrouperServiceWssec.aar/services.xml … <module ref="rampart" /> <wsp:Policy> <sp:HttpsToken RequireClientCertificate="false"/> </wsp:Policy> <ramp:RampartConfig xmlns:ramp="http://ws.apache.org/rampart/policy"> <ramp:passwordCallbackClass>edu.internet2.middleware.grouper.ws.security.RampartHandlerServer</ramp:passwordCallbackClass> </ramp:RampartConfig> 4/13/2017 University of Pennsylvania 116

Authentication – Ws- Security (continued) Default services.xml requires implementation of interface public interface GrouperWssecAuthentication { /** * <pre> * authenticate the user, and find the subject and return. * See GrouperWssecSample for an example * </pre> * @param wsPasswordCallback * @return true if callback type is supported, false if not * @throws IOException if there is a problem or if user is * not authenticated correctly */ public boolean authenticate(WSPasswordCallback wsPasswordCallback) throws IOException; } 4/13/2017 University of Pennsylvania 117

Authentication – Ws- Security (continued) Then you also need to configure this on the client There is an example in RampartSampleGetGroupsLite.java Includes service.xml And RampartPwHandlerClient callback for credentials 4/13/2017 University of Pennsylvania 118

University of Pennsylvania Authorization Grouper has built in authorization about who is allowed to do what Who can add to which stems Who can view memberships Who can view that groups exist Who can edit group attributes Another layer is which users are allowed to use web services Since Grouper is all about groups, use a group 4/13/2017 University of Pennsylvania 119

Authorization (continued) Setting in config file lists a group that users must be in to use WS (if blank allow all) 4/13/2017 University of Pennsylvania 120

Authorization (continued) If not in group, log it and alert the caller (should make option to suppress this) HTTP/1.1 500 Internal Server Error X-Grouper-resultCode: EXCEPTION X-Grouper-success: F <?xml version='1.0' encoding='iso-8859-1'?> , params: null, java.lang.RuntimeException: User is not authorized at edu.internet2… Caused by: edu.internet2… GroupNotFoundException: Cannot find group with name: 'etc:webServiceClientUsers' Note: it is a security best practice not to give too much information to clients, so they don’t know what the next hurdle is to hack the application. However, the administrators running the service must know errors (logs), and if clients can get information which does not help them hack in, but which reduces support time, then it is good. This should able to be suppressed so that deployers can decide. 4/13/2017 University of Pennsylvania 121 121

Authorization (continued) Given that callers must be in group, make it easy to setup This setting in grouper.properties will auto-create groups and auto-populate for ease of startup and testing 4/13/2017 University of Pennsylvania 122 122

Authorization (continued) If there is an auto-created group on startup, log it 2009-02-14 21:28:03,327: [main] WARN GrouperCheckConfig.checkGroup(130) - cannot find group from config: grouper.properties key configuration.autocreate.group.name.1: etc:webServiceClientUsers 2009-02-14 21:28:04,952: [main] WARN GrouperCheckConfig.checkGroup(149) - auto-created grouper.properties key configuration.autocreate.group.name.1: etc:webServiceClientUsers 2009-02-14 21:28:05,015: [main] WARN GrouperCheckConfig.checkGroups(469) - auto-added subject mchyzer to group: etc:webServiceClientUsers 4/13/2017 University of Pennsylvania 123 123

Authorization – actAs proxy Sometimes you need run as a different user than logged in as E.g. to run things as the “system” [root] user Or if there is an app users are using, and you call the web service as that user Or if an admin user needs to proxy another user in the office (maybe someone on vacation) All calls have an optional actAs input <WsRestAddMemberRequest> <actAsSubjectLookup> <subjectId>GrouperSystem</subjectId> </actAsSubjectLookup> </WsRestAddMemberRequest> 4/13/2017 University of Pennsylvania 124 124

Authorization – actAs proxy (continued) Any user can actAs themselves (why? Due to examples  ) GrouperSystem [root] can act as anyone You can configure groups of users who can actAs anyone (in grouper-ws.properties) ws.act.as.group = etc:webServiceActAsGroup Or you can specify groups who can act as users in another group (to clamp down a bit) ws.act.as.group = orgs:admins123 :::: orgs:users123 In this case users in orgs:admins123 group can only actAs any user in orgs:users123 4/13/2017 University of Pennsylvania 125 125

Authorization – actAs proxy (continued) Important to audit both the logged in user and actAs user 4/13/2017 University of Pennsylvania 126 126

Authorization – two factor (TODO) Factor 1: What you know (password) Factor 2: Where you are (assumes SSL) 4/13/2017 University of Pennsylvania 127 127

BONUS MATERIAL: DEVELOPMENT ENVIRONMENTS 4/13/2017 University of Pennsylvania 128

University of Pennsylvania TCP/IP monitor Need to have a way to see HTTP traffic to/from web service Network level proxy works in non-SSL only Axis has a swing tcp monitor Ethereal Axis has log settings to log the traffic from client (not server?) Eclipse has built-in TCP/IP monitor (my choice) Note: this has more uses than just web services See video (make sure you have the Xvid codec): https://wiki.internet2.edu/confluence/download/attachments/3014660/eclipseTcp.avi 4/13/2017 University of Pennsylvania 129

BONUS MATERIAL: JAVADOC IN CVS 4/13/2017 University of Pennsylvania 130

University of Pennsylvania Javadoc in CVS Ant script to make javadoc CVSweb ready 4/13/2017 University of Pennsylvania 131 131

Javadoc in CVS (continued) 4/13/2017 University of Pennsylvania 132 132

Javadoc in CVS (continued) 4/13/2017 University of Pennsylvania 133 133

BONUS MATERIALS: WEB SERVICE TESTING 4/13/2017 University of Pennsylvania 134

University of Pennsylvania Testing There are lots of places to test web services Best place is at the client See movie (make sure you have the Xvid codec): https://wiki.internet2.edu/confluence/download/attachments/3014660/testing.avi Most bugs found in Grouper web services could have been avoided with more testing Web services are easier to automatically test than web applications 4/13/2017 University of Pennsylvania 135 135