Presentation is loading. Please wait.

Presentation is loading. Please wait.

Grouper Web Services Java Web Service Servers and Clients in Internet2 Grouper February 2009 Chris Hyzer University of Pennsylvania IT Internet2.

Similar presentations


Presentation on theme: "Grouper Web Services Java Web Service Servers and Clients in Internet2 Grouper February 2009 Chris Hyzer University of Pennsylvania IT Internet2."— Presentation transcript:

1 Grouper Web Services Java Web Service Servers and Clients in Internet2 Grouper February 2009 Chris Hyzer University of Pennsylvania IT Internet2

2 Grouper Web Services  Show how Internet2 Grouper developed web services  Grouper is open source, feel free to borrow or suggest improvements  Discuss the successes and areas for improvement  Mention planned enhancements Purpose of this presentation 4/29/2015 University of Pennsylvania2

3 Grouper Web Services  Introduction to Internet2 Grouper  Introduction to web services  Architecture of REST/SOAP web service  SOAP web services with Axis2 (servers and clients)  REST web services with xstream (servers and clients)  Client  Documenting web services  Bonus material – Security – Axis serving Rest – Testing Contents 4/29/2015 University of Pennsylvania3

4 Grouper Web Services INTRODUCTION TO INTERNET2 GROUPER 4/29/2015 University of Pennsylvania4

5 Grouper Web Services  Open source group management  Internet2 has been working on group management for 8 years  Generally used in educational institutions, but could be anywhere  Funded by Internet2 Internet2 Grouper 4/29/2015 University of Pennsylvania5

6 Grouper Web Services  Instead of apps managing own groups  Reuse group lists  Central place to see which groups a person is in  Central auditing of group and membership actions  Central management of authorization  Security: – Who can view/edit groups and memberships – Opt-in/Opt-out – Delegate authority  Automatic or manual membership management  Composite groups for group match: and / or / minus  Groups of groups Why central group management with Grouper? 4/29/2015 University of Pennsylvania6

7 Grouper Web Services Grouper architecture 4/29/2015 University of Pennsylvania7

8 Grouper Web Services INTRODUCTION TO WEB SERVICES 4/29/2015 University of Pennsylvania8

9 Grouper Web Services  Person using browser makes a request to a server  Person (user) views the results in browser, and types and or clicks to continue User web request 4/29/2015 University of Pennsylvania9

10 Grouper Web Services  Program makes a request to a web application  Program parses the output Web service request 4/29/2015 University of Pennsylvania10

11 Grouper Web Services  Ajax for example – Can be kicked off by a user click – Can update the screen similar to a web application – However, Ajax is making the request and parsing the response, it is a web service – If it doesn’t parse the output, and just puts the resultant HTML into the browser DOM, then not a web service  Web service screen scraping a web application – A program can “screen scrape” a web application – Beware of changes in the HTML! – This is not a web service  Is a browser an application making requests, are all user requests web services? Overlap of web request and web service? 4/29/2015 University of Pennsylvania11

12 Grouper Web Services  http(s) is a well understood protocol by programming languages and programmers  Ports 80/443 might already be available in firewall rules  Http is text based (easy to debug)  Http is not programming language specific, so the server technology can be different than the client (e.g. ajax)  Webpages are either XML, XHTML, or XML-like (e.g. HTML) – Most programming languages have XML libraries – Note: web services do not have to be XML, though generally the are  Development and production environments might be similar (or same) to existing web applications  Penn generally communicates between systems with WS Why web services 4/29/2015 University of Pennsylvania12

13 Grouper Web Services  Simple Object Access Protocol  Specifies how web service messages are exchanged  W3C standard  Must use XML and XML schema for data  Messages have XML envelopes, headers, body, exception handling  Web Service Description Language (WSDL) describes the SOAP messages in a programmatic way (XML)  Many features (security, error handling, caching, resource discovery, etc)  Many programming languages generate SOAP  Not considered light-weight SOAP web services 4/29/2015 University of Pennsylvania13

14 Grouper Web Services aStem:aGroup mchyzer Example SOAP request 4/29/2015 University of Pennsylvania14

15 Grouper Web Services  Representational State Transfer  Two definitions: 1.(strict or RESTful): Protocol that specifies how HTTP (perhaps) and XML are used for web services 2.(non-strict): Any web service that does not have the overhead of SOAP. Aka Remote Procedure Call (RPC) REST web services 4/29/2015 University of Pennsylvania15

16 Grouper Web Services  The web services are organized like static web resources  URL’s represent resources, not operations  HTTP methods indicate the operations. Generally: GET, POST (update), PUT (insert), DELETE. Can use more, or custom  Messages can be HTML so that systems or browsers can consume them RESTful web services 4/29/2015 University of Pennsylvania16

17 Grouper Web Services  PUT /grouperWs/servicesRest/xhtml/v1_4_000/groups/aStem%3A aGroup/members/mchyzer  This means add this member to the group Example REST web service 4/29/2015 University of Pennsylvania17

18 Grouper Web Services  Each service makes its own standards  URL can be: – Resource: – Operation:  Generally just use POST or GET as the HTTP method  The XML document sent can be: – Complete object representation: myGroup MyGroup – Operational: 123 myGroup2 HTTP / XML / RPC / Hybrid web services 4/29/2015 University of Pennsylvania18

19 Grouper Web Services GROUPER WEB SERVICES 4/29/2015 University of Pennsylvania19

20 Grouper Web Services  Generally web services are programmed to host a service  Grouper is software, so its WS are programmed so institutions can download and host services  E.g. Grouper is app server and database server agnostic  Requirements – Dozen operations – SOAP and REST (as close to RESTful as possible) – SOAP and REST should deploy in one webapp – Simple operations (Lite), and batched operations – Pluggable authentication – Documented well – Versioned (generally Grouper has bi-annual releases) Grouper web services 4/29/2015 University of Pennsylvania20

21 Grouper Web Services  addMember  deleteMember  getMembers  hasMember  getGroups  groupSave  groupDelete Grouper web service operations 4/29/2015 University of Pennsylvania21  findGroups  findStems  stemSave  stemDelete  memberChangeSubject  getGrouperPrivileges  assignGrouperPrivileges

22 Grouper Web Services  One batched operation has less overhead than many smaller operations (performance test for yourself to validate)  Benchmarks – Add group (inserts), requires many queries 100 batches of 1 add groups take 19.8 seconds 10 batches of 10 add groups take 12.4 seconds 5 batches of 20 add groups take 12.0 seconds – Has member, lightweight, readonly 100 batches of 1 hasMember checks take 8.3 seconds 10 batches of 10 checks take 1.9 seconds 5 batches of 20 checks take 1.6 seconds  Benchmarks notes – Completely run on developer PC, local mysql – E.g. WsSampleHasMemberRest100 Lite vs batched 4/29/2015 University of Pennsylvania22

23 Grouper Web Services  Assume web service data are simple POJOs (Plain Old Java Objects)  Use only: – Beans – Arrays (of beans or simple types) – Simple types: String, Integer  Note: – Don’t use Collections, enums, dates, timestamps, booleans Object model 4/29/2015 University of Pennsylvania23

24 Grouper Web Services  What makes up the data of a group?  Here is a simple group Object model (continued) 4/29/2015 University of Pennsylvania24

25 Grouper Web Services  What makes up the data of a group?  Here is a more complex group Object model (continued) 4/29/2015 University of Pennsylvania25

26 Grouper Web Services Object model (continued) 4/29/2015 University of Pennsylvania26

27 Grouper Web Services  Request object model Object model (continued) 4/29/2015 University of Pennsylvania27

28 Grouper Web Services  Response object model Object model (continued) 4/29/2015 University of Pennsylvania28

29 Grouper Web Services  Metadata object model  Response metadata is one per response  Result metadata is one per Lite response, or one per each line item in batch  Success: T|F  Result code: many enums  Codes also in HTTP headers Object model (continued) 4/29/2015 University of Pennsylvania29

30 Grouper Web Services  Operations should be idempotent if possible – If they are sent twice, generally it is ok  Delete member mchyzer from group etc:sysAdminGroup – Idempotent  Delete the first member of group etc:sysAdminGroup – NOT idempotent Object model (continued) 4/29/2015 University of Pennsylvania30

31 Grouper Web Services SOAP WEB SERVICES WITH AXIS2 4/29/2015 University of Pennsylvania31

32 Grouper Web Services Axis architecture 4/29/2015 University of Pennsylvania32

33 Grouper Web Services  Create a class (GrouperService)  Contains only instance methods of business logic  Each method takes all fields of the input bean, and returns the output bean  Each bean is only simple pojo (uses Javabean properties)  Note the Lite methods only take scalars are inputs Business logic for Axis 4/29/2015 University of Pennsylvania33

34 Grouper Web Services Business logic for Axis (continued) 4/29/2015 University of Pennsylvania34

35 Grouper Web Services  GrouperService isn’t great for Javadoc since enums are strings  Delegate to GrouperServiceLogic  Decode booleans, dates, enums, etc Business logic for Axis (continued) 4/29/2015 University of Pennsylvania35

36 Grouper Web Services  Generate WSDL from POJOs and GrouperService class Axis generate WSDL 4/29/2015 University of Pennsylvania36

37 Grouper Web Services  Generate WSDL from POJOs and GrouperService class C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws>ant java2wsdl Buildfile: build.xml java2wsdl: [delete] Deleting: C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws-java -generated-client\GrouperService.wsdl BUILD SUCCESSFUL Total time: 9 seconds C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws> Axis generate WSDL (continued) 4/29/2015 University of Pennsylvania37

38 Grouper Web Services  Result Axis generate WSDL (continued) 4/29/2015 University of Pennsylvania38

39 Grouper Web Services  Result – 2000 lines of SOAP definition Axis generate WSDL (continued) 4/29/2015 University of Pennsylvania39

40 Grouper Web Services  Ant script to generate SOAP client from WSDL (any WSDL) Axis generate client 4/29/2015 University of Pennsylvania40

41 Grouper Web Services  Run ant script to generate client C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws>ant wsdl2java Buildfile: build.xml wsdl2java: [java] Retrieving document at 'C:\dev_inst\eclipse\workspace\grouper_v1_4\g rouper-ws-java-generated-client\GrouperService.wsdl'. BUILD SUCCESSFUL Total time: 9 seconds C:\dev_inst\eclipse\workspace\grouper_v1_4\grouper-ws> Axis generate client (continued) 4/29/2015 University of Pennsylvania41

42 Grouper Web Services  Result: 100 classes, ~5megs of source Axis generate client (continued) 4/29/2015 University of Pennsylvania42

43 Grouper Web Services  Axis needs an AAR file of logic, create via ant to WEB- INF/services/GrouperService.aar Axis archive 4/29/2015 University of Pennsylvania43

44 Grouper Web Services  Configure the web.xml for Axis AxisServlet Apache-Axis Servlet edu.internet2.middleware.grouper.ws.GrouperServiceAxisServlet 1 wssec true AxisServlet /services/* Axis configuration 4/29/2015 University of Pennsylvania44

45 Grouper Web Services  Boilerplate WEB-INF/conf/axis2.xml  WEB-INF/modules/*.mar  WEB-INF/modules/modules.list  WEB-INF/services/*.aar (including GrouperService.aar)  WEB-INF/service/services.list  WEB-INF/lib/ (50 axis jars) Axis configuration (continued) 4/29/2015 University of Pennsylvania45

46 Grouper Web Services  See video (make sure you have the Xvid codec):Xvid codec  https://wiki.internet2.edu/confluence/download/attachments/ /soapExample.avi https://wiki.internet2.edu/confluence/download/attachments/ /soapExample.avi Axis Example 4/29/2015 University of Pennsylvania46

47 Grouper Web Services REST WEB SERVICES WITH XSTREAM 4/29/2015 University of Pennsylvania47

48 Grouper Web Services  Easy to use Java object to XML processor  In this example I alias the class names so they arent so long public class XstreamPocGroup { public XstreamPocGroup(String theName, XstreamPocMember[] theMembers) { this.name = theName; this.members = theMembers; } private String name; private XstreamPocMember[] members; public String getName() {... Xstream 4/29/2015 University of Pennsylvania48

49 Grouper Web Services  This is the child bean public class XstreamPocMember { public XstreamPocMember(String theName, String theDescription) { this.name = theName; this.description = theDescription; } private String name; private String description; } Xstream (continued) 4/29/2015 University of Pennsylvania49

50 Grouper Web Services public static void main(String[] args) { XstreamPocGroup group = new XstreamPocGroup("myGroup", new XstreamPocMember[]{ new XstreamPocMember("John", "John Smith - Employee"), new XstreamPocMember("Mary", "Mary Johnson - Student")}); XStream xStream = new XStream(new XppDriver()); xStream.alias("XstreamPocGroup", XstreamPocGroup.class); xStream.alias("XstreamPocMember", XstreamPocMember.class); StringWriter stringWriter = new StringWriter(); xStream.marshal(group, new CompactWriter(stringWriter)); String xml = stringWriter.toString(); System.out.println(GrouperUtil.indent(xml, true)); group = (XstreamPocGroup)xStream.fromXML(xml); System.out.println(group.getName() + ", number of members:" + group.getMembers().length); } Xstream (continued) 4/29/2015 University of Pennsylvania50

51 Grouper Web Services myGroup John John Smith - Employee Mary Mary Johnson - Student myGroup, number of members: 2 Xstream (continued) 4/29/2015 University of Pennsylvania51

52 Grouper Web Services  You can control things with private Group group = null;  You can convert Java object to JSON XStream xStream = new XStream(new JettisonMappedXmlDriver()); {"XstreamPocGroup": {"name":"myGroup","members": {"XstreamPocMember":[ {"name":"John","description":"John Smith - Employee"}, {"name":"Mary","description":"Mary Johnson - Student"} ]}} } myGroup, number of members: 2 Xstream JSON 4/29/2015 University of Pennsylvania52

53 Grouper Web Services  Grouper had a requirement to have XHTML output  Rest people like XHTML, it can turn a web service into a browsable application – I have some reservations about that (what are the clients? Browsers, or screen scraping applications? Can you change the output?)  I rolled my own bean->XHTML converter  Based on XmlStreamWriter for output  Based on JDOM for input XHTML output 4/29/2015 University of Pennsylvania53

54 Grouper Web Services WsXhtmlOutputConverter wsXhtmlOutputConverter = new WsXhtmlOutputConverter(true, null); StringWriter stringWriter = new StringWriter(); wsXhtmlOutputConverter.writeBean(group, stringWriter); String xhtml = stringWriter.toString(); System.out.println(GrouperUtil.indent(xhtml, true)); WsXhtmlInputConverter wsXhtmlInputConverter = new WsXhtmlInputConverter(); wsXhtmlInputConverter.addAlias("XstreamPocGroup", XstreamPocGroup.class); wsXhtmlInputConverter.addAlias("XstreamPocMember", XstreamPocMember.class); group = (XstreamPocGroup)wsXhtmlInputConverter.parseXhtmlString(xhtml); System.out.println(group.getName() + ", number of members: " + group.getMembers().length); XHTML output (continued) 4/29/2015 University of Pennsylvania54

55 Grouper Web Services XstreamPocGroup John John Smith - Employee XHTML output (continued) 4/29/2015 University of Pennsylvania55

56 Grouper Web Services Mary Mary Johnson - Student myGroup myGroup, number of members: 2  Note: Im not sure why this would be useful… XHTML output (continued) 4/29/2015 University of Pennsylvania56

57 Grouper Web Services  Similar to Axis input, REST (Lite) should accept HTTP params (URL params or in HTTP body) PUT /grouperWs/servicesRest/v1_4_000/groups /aStem%3AaGroup/members/ HTTP/1.1 Connection: close Authorization: Basic xxxxxxxxxxxxxxxxx== User-Agent: Jakarta Commons-HttpClient/3.1 Host: localhost:8092 Content-Length: 72 wsLiteObjectType=WsRestAddMemberLiteRequest &actAsSubjectId=GrouperSystem HTTP input 4/29/2015 University of Pennsylvania57

58 Grouper Web Services  Couldn’t find a 3 rd party indenter, rolled my own System.out.println(GrouperUtil.indent(xhtml, true)); System.out.println(GrouperUtil.indent(json, true)); System.out.println(GrouperUtil.indent(xml, true));  This is valuable when showing examples Indenting of XML, XHTML, JSON 4/29/2015 University of Pennsylvania58

59 Grouper Web Services  URLs need to represent hierarchical resources:  /servicesRest/v1_4_000/groups/aStem%3AaGroup/members – Servlet, client version, top level “folder”, name of item, inner “folder” – The client version is in there since that is the first part of versioning – GET that URL would retrieve all the members of that group – POST would update the list (send all new members) – PUT would create a new list – DELETE would delete all the members  …/groups/aStem%3AaGroup/members/mchyzer – Drilling down further – GET could see if mchyzer is a member – PUT would add mchyzer to group – etc Make the operations more Restful 4/29/2015 University of Pennsylvania59

60 Grouper Web Services  Some things are programmatic, not resource based (e.g. a fixBadMemberships diagnostic tool)  The input params could be complicated, e.g. – Get the membership list of a group – Get only immediate members – Act as a different user (proxy) – Get the student list, not the faculty list  There might be many different ways to ID a resource – E.g. name or UUID  Composite ID’s of resources are not convenient – /groups/aStem:aGroup/members/sourceId/someSource/subjectId/1234 Problems with Rest 4/29/2015 University of Pennsylvania60

61 Grouper Web Services  The only HTTP methods which take a body are POST and PUT – How can you delete something with XML constraints (actAs proxy, certain list type, include details on result) if a DELETE method does not take an XML body???  Rest is supposed to “model the web” – Which web? Static? When is the last time you deleted a static resource with an HTTP DELETE method? – The web is more and more dynamic, so Rest/HTTP/XML/hybrid might be more similar for developers Problems with Rest (continued) 4/29/2015 University of Pennsylvania61

62 Grouper Web Services  URLs are resource based  HTTP methods are honored (GET/POST/PUT/DELETE)  If there is a body (POST/PUT), then the object type sent will trump the HTTP method (can DELETE with metadata)  HTTP status codes are sent – Though they really shouldn’t be read, they don’t mean much, 404 could be a success  HTTP headers on the response – boolean to determine if the operation is a success or failure – Enum based text status code which is specific to the operation Grouper Rest 4/29/2015 University of Pennsylvania62

63 Grouper Web Services  The same operations are exposed as SOAP  The XML document can specify the data, or the URL/HTTP method  Could add member like this: PUT /grouperWs/servicesRest/xhtml/v1_4_000 /groups/aStem%3AaGroup/members/ HTTP/1.1  Or you could add a member with a POST and a body of the right object type: POST /grouperWs/servicesRest/v1_4_001/groups /aStem%3AaGroup/members HTTP/ GrouperSystem Grouper Rest (continued) 4/29/2015 University of Pennsylvania63

64 Grouper Web Services Grouper Rest Architecture 4/29/2015 University of Pennsylvania64

65 Grouper Web Services  Client can use same beans as server (not true with Axis)  Enums for content types  Use Jakarta HTTP client for communication (Axis uses this too)  Show movie of Rest client (make sure you have the Xvid codec): https://wiki.internet2.edu/confluence/download/attachments/ /restDemo.aviXvid codec https://wiki.internet2.edu/confluence/download/attachments/ /restDemo.avi Grouper Rest (continued) 4/29/2015 University of Pennsylvania65

66 Grouper Web Services WHICH WEB SERVICE ARCHITECTURE SHOULD I USE? 4/29/2015 University of Pennsylvania66

67 Grouper Web Services  More complex operations (describe in WSDL)  Clients can handle SOAP  Client code generation  WS-* security (e.g. kerberos ticket authentication) When SOAP 4/29/2015 University of Pennsylvania67

68 Grouper Web Services  Simple operations – Non batched  Simple resources – Nice to not have composite identifiers  Operations with little or no metadata (e.g. actAs)  Clients are known to handle Rest HTTP methods  Perhaps not for Ajax (limitation might be HTTP response code and HTTP methods) When Restful 4/29/2015 University of Pennsylvania68

69 Grouper Web Services  When supporting “Rest” and SOAP – Since SOAP is not resource based  Disparate clients  Need to send a body of metadata with a GET or DELETE When Rest / HTTP / XML hybrid / POX 4/29/2015 University of Pennsylvania69

70 Grouper Web Services CLIENT 4/29/2015 University of Pennsylvania70

71 Grouper Web Services  Architecture GrouperClient 4/29/2015 University of Pennsylvania71

72 Grouper Web Services  Its nice to give a packaged client with web services  Writing HTTP/XML does not make a quick start  Anyone can use it command line  Can use as Java library  Can make custom XML samples  Debugging tools  More samples  Rest only (since easiest to version, most lightweight… doesn’t have 50 Axis jars!)  Since command line, its one jar  Can be used along side other jars GrouperClient 4/29/2015 University of Pennsylvania72

73 Grouper Web Services  Very simple POJOs  Refactor 3 rd party libs – Xstream – HttpClient – Commons-logging – Jexl: expression language – morphString  So they don’t conflict  Want only one jar GrouperClient (continued) 4/29/2015 University of Pennsylvania73

74 Grouper Web Services  Example: C:\gc>java -jar grouperClient.jar --operation=getGroupsWs --subjectIds= SubjectIndex 0: success: T: code: SUCCESS: subject: : groupIndex: 0: aStem:aGroup SubjectIndex 0: success: T: code: SUCCESS: subject: : groupIndex: 1: etc:webServiceClientUsers SubjectIndex 0: success: T: code: SUCCESS: subject: : groupIndex: 2: etc:sysadmingroup Show movie (make sure you have the Xvid codec):Xvid codec https://wiki.internet2.edu/confluence/download/attachments/ /grouperClient.avi GrouperClient (continued) 4/29/2015 University of Pennsylvania74

75 Grouper Web Services  Java API  No longer need to deal with httpClient or authentication  Error handling built in public static void main(String[] args) { WsGetGroupsResults wsGetGroupsResults = new GcGetGroups().addSubjectLookup(new WsSubjectLookup(" ", null, null)).execute(); WsGetGroupsResult wsGroupsResult = wsGetGroupsResults.getResults()[0]; for (WsGroup wsGroup : wsGroupsResult.getWsGroups()) { System.out.println(wsGroup.getName()); } } aStem:aGroup etc:webServiceClientUsers etc:sysadmingroup GrouperClient (continued) 4/29/2015 University of Pennsylvania75

76 Grouper Web Services  Set encrypt key in grouper.client.properties encrypt.key = fnh453hfbdw  Encrypt the password: C:\gc>java -jar grouperClient.jar --operation=encryptPassword Type the string to encrypt (note: pasting might echo it back): Encrypted password: mpAdW53ekchSGAX3vq1UiQ== C:\gc>  Put this in a file, refer to file in grouper.client.properties grouperClient.webService.password = c:/gc/ws.pass  (more) sanitized config files for or source control  Perhaps auditing requirement forbidding clear text passwords GrouperClient – external encrypted passwords 4/29/2015 University of Pennsylvania76

77 Grouper Web Services  If using command line utility in prod, will screenscrape STDOUT C:\gc>java -jar grouperClient.jar --operation=getGroupsWs --subjectIds= SubjectIndex 0: success: T: code: SUCCESS: subject: : groupIndex: 0: aStem:aGroup  To give flexibility, and ability to change defaults, template C:\gc>java -jar grouperClient.jar --operation=getGroupsWs --subjectIds= –outputTemplate ="${groupIndex+1} groupName: ${wsGroup.name}$newline$" 1 groupName: aStem:aGroup 2 groupName: etc:webServiceClientUsers 3 groupName: etc:sysadmingroup GrouperClient – JEXL output templates 4/29/2015 University of Pennsylvania77

78 Grouper Web Services  To help troubleshoot or create samples grouperClient.logging.webService.documentDir = c:/gc/xmls grouperClient.logging.webService.indent = true  Organize files so they are easy to archive or delete Grouper client traffic capture 4/29/2015 University of Pennsylvania78

79 Grouper Web Services Grouper client traffic capture (continued) 4/29/2015 University of Pennsylvania79

80 Grouper Web Services DOCUMENTING WEB SERVICES: AUTOMATIC SAMPLES 4/29/2015 University of Pennsylvania80

81 Grouper Web Services  Samples are tedious to maintain  Too many combinations: – Rest and Soap – Lite and Batched – Within Rest: XML, XHTML, JSON, HTTP params – 12 operations  Program automatically generates 150+ samples Automatic samples 4/29/2015 University of Pennsylvania81

82 Grouper Web Services Automatic samples (continued) 4/29/2015 University of Pennsylvania82

83 Grouper Web Services  See movie (make sure you have the Xvid codec):Xvid codec  https://wiki.internet2.edu/confluence/download/attachments/ /sampleCapture.avi https://wiki.internet2.edu/confluence/download/attachments/ /sampleCapture.avi Automatic samples 4/29/2015 University of Pennsylvania83

84 Grouper Web Services   https://wiki.internet2.edu/confluence/display/GrouperWG/v Grouper+Web+Services https://wiki.internet2.edu/confluence/display/GrouperWG/v Grouper+Web+Services cvs login cvs export -r GROUPER_1_4_BRANCH grouper-ws   References and links 4/29/2015 University of Pennsylvania84

85 Grouper Web Services Questions? 4/29/2015 University of Pennsylvania85

86 Grouper Web Services BONUS MATERIAL: MISC 4/29/2015 University of Pennsylvania86

87 Grouper Web Services  The default JSON outputted is awful – An array of size 1 is the same as a field – A string with an number value is the same as a number – If you use this, find settings so that it makes sense  Xstream by default works by fields, not Javabean properties – This has surfaced from mismatches between Xstream and Axis (which goes by Javabean properties) – There is a setting to enable this, it is on my TODO list  You can override some settings in Xstream to ignore extraneous XML fields (for backwards compatibility)  Fields are lower case, Classes are uppercase (as aliased), and a user reported difficulty with Xpath Xstream words of warning 4/29/2015 University of Pennsylvania87

88 Grouper Web Services  GrouperRestServlet RestServlet WS REST Servlet edu.internet2.middleware.grouper.ws.rest.GrouperRestServlet 1 RestServlet /servicesRest/* Grouper Rest (continued) 4/29/2015 University of Pennsylvania88

89 Grouper Web Services  Easy to setup, get jakarta commons jexl.jar  I have a utility method to substitute, something like this: JexlContext jc = JexlHelper.createContext(); for (String key: variableMap.keySet()) { jc.getVars().put(key, variableMap.get(key)); } Pattern pattern = Pattern.compile("\\$\\{(.*?)\\}"); Matcher matcher = pattern.matcher(stringToParse); StringBuilder result = new StringBuilder(); while(matcher.find()) { result.append(stringToParse.substring(index,matcher.start())); String script = matcher.group(1); Expression e = ExpressionFactory.createExpression(script); Object o = e.evaluate(jc); result.append(o); index = matcher.end(); } JEXL output templates (continued) 4/29/2015 University of Pennsylvania89

90 Grouper Web Services BONUS MATERIAL: GROUPER OBJECT MODEL 4/29/2015 University of Pennsylvania90

91 Grouper Web Services Grouper data model (simplified) 4/29/2015 University of Pennsylvania91

92 Grouper Web Services  How to represent “Group” in a web service pojo?  Where do we need “Group” in web services?  Input examples: – Add member to a group – Save a group – Delete a group – See if a member is in a group – Find a group  Output examples: – List groups for member – Find group – Save a group? – Delete a group? Object model (continued) 4/29/2015 University of Pennsylvania92

93 Grouper Web Services  Break “Group” into two cases – Lookup – Object representation  Lookup (allow multiple ways) – Lookup by UUID – Lookup by name – There are several lookups in Grouper WS: Stem lookup Subject lookup Group lookup  How to handle object representation? Object model (continued) 4/29/2015 University of Pennsylvania93

94 Grouper Web Services  Each request has its own object – One for Lite request Note: each Lite request has only scalars – One for Batch request  Each response has its own object – One for Lite response – One for Batch response – One for each line item of a batch response Object model (continued) 4/29/2015 University of Pennsylvania94

95 Grouper Web Services  Option 1: minimal, make another request for more info – For list groups for member, send back the group UUID and name – If more info needed make another request for more info  Option 2: send it all – Just send everything about a group on all requests  Option 3: pick and choose – When an operation returns groups, tell the server what to return  Option 4: two levels to decide – Generally send back basic data – If includeGroupDetail in a request, send it all  Note: for saveGroup, same object goes back and forth, including an optional groupLookup on request. Object model (continued) 4/29/2015 University of Pennsylvania95

96 Grouper Web Services  Original vision was Lite operations would require only scalars  Do not need XML in request  Only use HTTP params  Response would still have a body  Originally we used Axis REST  It didn’t satisfy our requirements, so we moved to custom REST Lite vs batched (continued) 4/29/2015 University of Pennsylvania96

97 Grouper Web Services BONUS MATERIAL: AXIS SERVING REST 4/29/2015 University of Pennsylvania97

98 Grouper Web Services  Axis can serve “Rest” services (loose-Rest)  Basically it is the same service without the SOAP envelope  Can also pass params as HTTP params  Set this in axis2.xml: false  Set this option in the client: options.setProperty( Constants.Configuration.ENABLE_REST, Constants.VALUE_TRUE);  Show movie (make sure you have the Xvid codec): https://wiki.internet2.edu/confluence/download/attachments/ /axisRestExample.aviXvid codec https://wiki.internet2.edu/confluence/download/attachments/ /axisRestExample.avi Axis serving Rest 4/29/2015 University of Pennsylvania98

99 Grouper Web Services  Not clear how to customize the XML, and it is not great looking XML (not what you would create by hand)  Not real Rest, since the HTTP method is still GET or POST, not PUT or DELETE  Still resembles RPC calls, not resource centric calls  There is a bad bug with Axis SOAP and REST where if you skip params it marshals them in the wrong order. Hopefully this will be fixed soon. https://issues.apache.org/jira/browse/AXIS https://issues.apache.org/jira/browse/AXIS  Grouper abandoned Axis Rest in favor of custom Rest Axis serving Rest (continued) 4/29/2015 University of Pennsylvania99

100 Grouper Web Services BONUS MATERIAL: WEB SERVICE SECURITY 4/29/2015 University of Pennsylvania100

101 Grouper Web Services  Easiest way to go is HTTP basic authentication  Assumes using SSL (since only Base64 encoded)  Send this with Commons Http client httpClient.getParams().setAuthenticationPreemptive(true); Credentials defaultcreds = new UsernamePasswordCredentials(RestClientSettings.USER, RestClientSettings.PASS); //e.g. localhost and 8093 httpClient.getState().setCredentials(new AuthScope(RestClientSettings.HOST, RestClientSettings.PORT), defaultcreds); Authentication 4/29/2015 University of Pennsylvania101

102 Grouper Web Services  Results in this HTTP PUT /grouperWs/servicesRest/v1_4_000/groups/aStem %3AaGroup/members/ HTTP/1.1 Connection: close Authorization: Basic SDF423SFD423xxxx== User-Agent: Jakarta Commons-HttpClient/3.1 Host: localhost:8092 Content-Length: 72 Authentication (continued) 4/29/2015 University of Pennsylvania102

103 Grouper Web Services  Handle this on the server  Solution should work with Axis or Rest  Make a servlet filter for Soap and Rest: Grouper service filter edu.internet2.middleware.grouper.ws.GrouperServiceJ2ee Grouper service filter /services/* Grouper service filter /servicesRest/* Authentication (continued) 4/29/2015 University of Pennsylvania103

104 Grouper Web Services  Keep threadlocals of request and response threadLocalRequest.set((HttpServletRequest) request); threadLocalResponse.set((HttpServletResponse) response); threadLocalRequestStartMillis.set(System.currentTimeMillis()); try { filterChain.doFilter(request, response); } finally { threadLocalRequest.remove(); threadLocalResponse.remove(); threadLocalRequestStartMillis.remove(); } Authentication (continued) 4/29/2015 University of Pennsylvania104

105 Grouper Web Services  Utility method, called from business logic, to authenticate String authenticationClassName = GrouperWsConfig.getPropertyString( GrouperWsConfig.WS_SECURITY_NON_RAMPART_AUTHENTICATION_CLASS, WsGrouperDefaultAuthentication.class.getName()); Class theClass = GrouperUtil.forName(authenticationClassName); WsCustomAuthentication wsAuthentication = GrouperUtil.newInstance(theClass); userIdLoggedIn = wsAuthentication.retrieveLoggedInSubjectId(retrieveHttpServletRequest()); // cant be blank! if (StringUtils.isBlank(userIdLoggedIn)) { throw new WsInvalidQueryException("No user is logged in"); } Authentication (continued) 4/29/2015 University of Pennsylvania105

106 Grouper Web Services  Two built in authentication methods, first is container auth public String retrieveLoggedInSubjectId(HttpServletRequest httpServletRequest) throws RuntimeException { // use this to be the user connected, or the user act-as String userIdLoggedIn = GrouperServiceJ2ee.retrieveUserPrincipalNameFromRequest(); return userIdLoggedIn; } Authentication – container authN 4/29/2015 University of Pennsylvania106

107 Grouper Web Services  Configure in web.xml Web services /servicesRest/* grouper_user BASIC Grouper Application Web service grouper_user Authentication – container authN 4/29/2015 University of Pennsylvania107

108 Grouper Web Services  Configure in tomcat-users.xml (servlet container specific) Authentication – container authN (continued) 4/29/2015 University of Pennsylvania108

109 Grouper Web Services  Alternative builtin authentication: Kerberos String authHeader = request.getHeader("Authorization"); //if no header, we cant go to kerberos if (StringUtils.isBlank(authHeader)) { LOG.error("No authorization header in HTTP"); return null; } Matcher matcher = regexPattern.matcher(authHeader); String authHeaderBase64Part = null; if (matcher.matches()) { authHeaderBase64Part = matcher.group(1); } if (StringUtils.isBlank(authHeaderBase64Part)) { LOG.error("Cant find base64 part in auth header"); return null; } Authentication – Kerberos 4/29/2015 University of Pennsylvania109

110 Grouper Web Services //unencrypt this byte[] base64Bytes = authHeaderBase64Part.getBytes(); byte[] unencodedBytes = Base64.decodeBase64(base64Bytes); String unencodedString = new String(unencodedBytes); //split based on user/pass String user = GrouperUtil.prefixOrSuffix(unencodedString, ":", true); String pass = GrouperUtil.prefixOrSuffix(unencodedString, ":", false); if (authenticateKerberos(user, pass)) { return user; } Authentication – Kerberos (continued) 4/29/2015 University of Pennsylvania110

111 Grouper Web Services  Deployers can use their own authentication # to provide custom authentication (instead of the default # httpServletRequest.getUserPrincipal() # for non-Rampart authentication. Class must implement the # interface: # edu.internet2.middleware.grouper.ws.security #.WsCustomAuthentication # class must be fully qualified. e.g. # edu.school.whatever.MyAuthenticator # blank means use default: # edu.internet2.middleware.grouper.ws.security #.WsGrouperDefaultAuthentication # kerberos: edu.internet2.middleware.grouper.ws.security #.WsGrouperKerberosAuthentication ws.security.non-rampart.authentication.class = Authentication – Pluggable 4/29/2015 University of Pennsylvania111

112 Grouper Web Services  Implement this interface public interface WsCustomAuthentication { /** * retrieve the current username (subjectId) from the request * object. httpServletRequest the logged in username (subjectId) WsInvalidQueryException if there is a problem */ public String retrieveLoggedInSubjectId(HttpServletRequest httpServletRequest) throws WsInvalidQueryException; } Authentication – Pluggable 4/29/2015 University of Pennsylvania112

113 Grouper Web Services  It is a TODO to provide the option for the an HTTP basic authN to cache a hashed version of the authentication header (and translation to subject ID)  Hash this so a Base64 version is not held in memory  Do not want to load down the authentication service (e.g. kerberos) if the web service is under high load Authentication – Caching 4/29/2015 University of Pennsylvania113

114 Grouper Web Services  Contributed by Sanjay Vivek at Newcastle University  Axis has a module called Rampart which implements WS – security standards  Important if the service requires multiple hops or proxying  Useful if not using SSL  Only for Soap, not for Rest  Useful for authenticating without a user/pass (e.g. Kerberos ticket or x.509)  Cannot run rampart along with HTTP basic authn (need multiple webapps) in Axis2 Authentication – Ws- Security 4/29/2015 University of Pennsylvania114

115 Grouper Web Services  Set a param in the web.xml AxisServlet Apache-Axis Servlet edu.internet2.middleware.grouper.ws.GrouperServiceAxisS ervlet 1 wssec true Authentication – Ws- Security (continued) 4/29/2015 University of Pennsylvania115

116 Grouper Web Services  Configure axis2 GrouperServiceWssec.aar/services.xml … … edu.internet2.middleware.grouper. ws.security.RampartHandlerServer … Authentication – Ws- Security (continued) 4/29/2015 University of Pennsylvania116

117 Grouper Web Services  Default services.xml requires implementation of interface public interface GrouperWssecAuthentication { /** * * authenticate the user, and find the subject and return. * See GrouperWssecSample for an example * wsPasswordCallback true if callback type is supported, false if not IOException if there is a problem or if user is * not authenticated correctly */ public boolean authenticate(WSPasswordCallback wsPasswordCallback) throws IOException; } Authentication – Ws- Security (continued) 4/29/2015 University of Pennsylvania117

118 Grouper Web Services  Then you also need to configure this on the client  There is an example in RampartSampleGetGroupsLite.java  Includes service.xml  And RampartPwHandlerClient callback for credentials Authentication – Ws- Security (continued) 4/29/2015 University of Pennsylvania118

119 Grouper Web Services  Grouper has built in authorization about who is allowed to do what – Who can add to which stems – Who can view memberships – Who can view that groups exist – Who can edit group attributes  Another layer is which users are allowed to use web services  Since Grouper is all about groups, use a group Authorization 4/29/2015 University of Pennsylvania119

120 Grouper Web Services  Setting in config file lists a group that users must be in to use WS (if blank allow all) Authorization (continued) 4/29/2015 University of Pennsylvania120

121 Grouper Web Services  If not in group, log it and alert the caller (should make option to suppress this) HTTP/ Internal Server Error X-Grouper-resultCode: EXCEPTION X-Grouper-success: F, params: null, java.lang.RuntimeException: User is not authorized at edu.internet2… Caused by: edu.internet2… GroupNotFoundException: Cannot find group with name: 'etc:webServiceClientUsers' Authorization (continued) 4/29/2015 University of Pennsylvania121

122 Grouper Web Services  Given that callers must be in group, make it easy to setup  This setting in grouper.properties will auto-create groups and auto-populate for ease of startup and testing Authorization (continued) 4/29/2015 University of Pennsylvania122

123 Grouper Web Services  If there is an auto-created group on startup, log it :28:03,327: [main] WARN GrouperCheckConfig.checkGroup(130) - cannot find group from config: grouper.properties key configuration.autocreate.group.name.1: etc:webServiceClientUsers :28:04,952: [main] WARN GrouperCheckConfig.checkGroup(149) - auto-created grouper.properties key configuration.autocreate.group.name.1: etc:webServiceClientUsers :28:05,015: [main] WARN GrouperCheckConfig.checkGroups(469) - auto-added subject mchyzer to group: etc:webServiceClientUsers Authorization (continued) 4/29/2015 University of Pennsylvania123

124 Grouper Web Services  Sometimes you need run as a different user than logged in as  E.g. to run things as the “system” [root] user  Or if there is an app users are using, and you call the web service as that user  Or if an admin user needs to proxy another user in the office (maybe someone on vacation) All calls have an optional actAs input GrouperSystem Authorization – actAs proxy 4/29/2015 University of Pennsylvania124

125 Grouper Web Services  Any user can actAs themselves (why? Due to examples )  GrouperSystem [root] can act as anyone  You can configure groups of users who can actAs anyone (in grouper-ws.properties) ws.act.as.group = etc:webServiceActAsGroup  Or you can specify groups who can act as users in another group (to clamp down a bit) ws.act.as.group = orgs:admins123 :::: orgs:users123  In this case users in orgs:admins123 group can only actAs any user in orgs:users123 Authorization – actAs proxy (continued) 4/29/2015 University of Pennsylvania125

126 Grouper Web Services  Important to audit both the logged in user and actAs user Authorization – actAs proxy (continued) 4/29/2015 University of Pennsylvania126

127 Grouper Web Services  Factor 1: What you know (password)  Factor 2: Where you are (assumes SSL) Authorization – two factor (TODO) 4/29/2015 University of Pennsylvania127

128 Grouper Web Services BONUS MATERIAL: DEVELOPMENT ENVIRONMENTS 4/29/2015 University of Pennsylvania128

129 Grouper Web Services  Need to have a way to see HTTP traffic to/from web service  Network level proxy works in non-SSL only  Axis has a swing tcp monitorswing tcp monitor  Ethereal  Axis has log settings to log the traffic from client (not server?)  Eclipse has built-in TCP/IP monitor (my choice)  Note: this has more uses than just web services  See video (make sure you have the Xvid codec): https://wiki.internet2.edu/confluence/download/attachments/ /eclipseTcp.aviXvid codec https://wiki.internet2.edu/confluence/download/attachments/ /eclipseTcp.avi TCP/IP monitor 4/29/2015 University of Pennsylvania129

130 Grouper Web Services BONUS MATERIAL: JAVADOC IN CVS 4/29/2015 University of Pennsylvania130

131 Grouper Web Services  Ant script to make javadoc CVSweb ready Javadoc in CVS 4/29/2015 University of Pennsylvania131

132 Grouper Web Services Javadoc in CVS (continued) 4/29/2015 University of Pennsylvania132

133 Grouper Web Services Javadoc in CVS (continued) 4/29/2015 University of Pennsylvania133

134 Grouper Web Services BONUS MATERIALS: WEB SERVICE TESTING 4/29/2015 University of Pennsylvania134

135 Grouper Web Services  There are lots of places to test web services  Best place is at the client  See movie (make sure you have the Xvid codec):Xvid codec  https://wiki.internet2.edu/confluence/download/attachments/ /testing.avi https://wiki.internet2.edu/confluence/download/attachments/ /testing.avi  Most bugs found in Grouper web services could have been avoided with more testing  Web services are easier to automatically test than web applications Testing 4/29/2015 University of Pennsylvania135


Download ppt "Grouper Web Services Java Web Service Servers and Clients in Internet2 Grouper February 2009 Chris Hyzer University of Pennsylvania IT Internet2."

Similar presentations


Ads by Google