1. File: 05_RETS_Software_Development.PPT 1 RETS Software Development  RETS the standard  RETS transactions – Identify issues, concerns and recommendations  Sample session workflow

2. File: 05_RETS_Software_Development.PPT 2 RETS Software Development RETS Software Development Topic Objectives  Improve understanding of the RETS architecture  Build familiarity with the RETS document  Enable more productive RETS software development

3. File: 05_RETS_Software_Development.PPT 3 RETS Software Development RETS Principles - 1  Stateless communications protocol  Carry state using a cookie: – Basically a token that points to a persistent store on a server somewhere – RETS defines an optional cookie for this purpose  Recommendations: – Return all cookies received – Client applications using browsers will need to accept cookies

4. File: 05_RETS_Software_Development.PPT 4 RETS Software Development RETS Principles - 2  Uses Http-based transport  Uses DMQL - an open query language  Supports XML and site-specific output data formats  Supports site-specific metadata to describe the local MLS system  Recommendation: – Effort to support metadata produces interoperability benefits

5. File: 05_RETS_Software_Development.PPT 5 RETS Software Development RETS Document  Organization follows a typical real estate industry workflow: – Initiate single session – Multiple data requests – Terminate session  References standards produced by other standards organizations RETS document can be challenging to read

6. File: 05_RETS_Software_Development.PPT 6 RETS Software Development Simple Collaboration Diagram

7. File: 05_RETS_Software_Development.PPT 7 RETS Software Development Login Transaction Making the Initial Connection  Authentication – Response with authorized resources  Types: – Basic – Digest using MD5 or SHA1 – SSL  Recommendations: – Implement Digest using MD5 – SHA1 if you can

8. File: 05_RETS_Software_Development.PPT 8 RETS Software Development Login Transaction Authentication  Most common authentication type is digest using MD5 – Somewhat tricky to implement algorithm – Easiest to borrow the implementation from one of the open source projects  When in doubt: – Ask the RETS-DEV list – Ask your software vendor

9. File: 05_RETS_Software_Development.PPT 9 RETS Software Development Login Transaction Activity Diagram < >

10. File: 05_RETS_Software_Development.PPT 10 RETS Software Development Login Transaction Digest Authentication  Message Digest algorithms are one-way functions  Easy to perform the calculation  Difficult to reverse the calculation  Both sides calculate the same message digest  Transmit only the digest  Compare your calculated digest with the received digest

11. File: 05_RETS_Software_Development.PPT 11 RETS Software Development Login Transaction UserAgent  Application name accessing the MLS server – All client requests MUST include this field – This is a standard HTTP header field as defined in RFC 2616  MLS server should validate application name presented against list of allowable applications maintained on a table  Application name and userid is a useful key to a table that tracks MLS server usage

12. File: 05_RETS_Software_Development.PPT 12 RETS Software Development Login Transaction Gotchas  Using operating system libraries for MD5 can cause authentication failure  Proxies and tunnels can be a problem  Re-authenticate is part of the RFC but few vendors enforce it  Case may matter

13. File: 05_RETS_Software_Development.PPT 13 RETS Software Development Sample Login Challenge Response Header 1 HTTP/1.1 401 Unauthorized xxx Date: Mon, 02 Aug 2004 05:23:54 GMT Server: WebLogic XMLX Module 8.1 SP1 Fri Jun 20 23:06:40 PDT 2003 271009 with WWW-Authenticate: Digest realm="users@ftc2.com", nonce="313035", opaque="6e"

14. File: 05_RETS_Software_Development.PPT 14 RETS Software Development Sample Login Challenge Response Header 1 cont. Content-Length: 0 Content-Type: text/html RETS-Version: RETS/1.0 Cache-Control: private

15. File: 05_RETS_Software_Development.PPT 15 RETS Software Development Sample Login Nonce nonce= "313035393731353435373630342058dd631265e3 360724c45d15f23aec7a"

16. File: 05_RETS_Software_Development.PPT 16 RETS Software Development Sample Login Opaque opaque="6e6f742075736564"

17. File: 05_RETS_Software_Development.PPT 17 RETS Software Development Sample Login Challenge Response Header 2 HTTP/1.1 200 OK Date: Mon, 02 Aug 2004 05:31:03 GMT Server: WebLogic XMLX Module 8.1 SP1 Fri Jun 20 23:06:40 PDT 2003 271009 with Content-Type: text/plain

18. File: 05_RETS_Software_Development.PPT 18 RETS Software Development Sample Login Challenge Response Header 2 cont. Set-Cookie: RETS-Session-ID=20000439936 RETS-Version: RETS/1.0 Transfer-Encoding: Chunked Cache-Control: private 0230

19. File: 05_RETS_Software_Development.PPT 19 RETS Software Development Sample Login Challenge Response Body 2 MemberName=Mark Crawford User=632344, NULL, NULL, 632344 Broker=National,1 MetadataVersion=1.2.0 MinMetadataVersion=1.1.7 OfficeList=National;1 TimeoutSeconds=900

20. File: 05_RETS_Software_Development.PPT 20 RETS Software Development Sample Login Challenge Response Body 2 cont. Search=http://rets.ftc2.com:6103/search GetObject=http://rets.ftc2.com:6103/get object Login=http://rets.ftc2.com:6103/login GetMetadata=http://rets.ftc2.com:6103/getmetadata Logout=http://rets.ftc2.com:6103/logout 0000

21. File: 05_RETS_Software_Development.PPT 21 RETS Software Development Login Transaction Capability URL List  Response from the Login transaction includes a list of known resources on the server – Each URL points to a RETS transaction type  Capability URL list may be different based on the user and their authorization level  Required that the server return a list containing at least Search  The URL references are not permanent locations – the standard allows the location to change

22. File: 05_RETS_Software_Development.PPT 22 RETS Software Development Action-URL and Get Transaction  Included in the Capability URL List is an Action-URL  Points to a human readable message – Most useful if changed regularly; perhaps daily  Immediately following successful login, may need to do a Get on the Action-URL – Get Transaction is a file retrieval from the server

23. File: 05_RETS_Software_Development.PPT 23 RETS Software Development Metadata  Description of the structure of the data on a specific server  Unique to each site – There will be common elements  Most powerful feature of the RETS specification Perhaps the most challenging part of the RETS specification

24. File: 05_RETS_Software_Development.PPT 24 RETS Software Development GetMetadata Transaction Overview  Transaction to retrieve system metadata  Can select a specific level, the level and descendants or a specific metadata ID  Formats supported are COMPACT and Standard-XML  Metadata id value of 0 – Request is for all types at the same level – breadth  Metadata id value of * – Request is for all types at the same level and all child types - depth

25. File: 05_RETS_Software_Development.PPT 25 RETS Software Development Metadata Format ID  Hierarchy of types of metadata – See Figure 11.1 on page 11-2 of the RETS specification  When referencing hierarchy as ID – Parent: Child is the form – Example: Type: METADATA-UPDATE_TYPE ID: Property: RES: Add

26. File: 05_RETS_Software_Development.PPT 26 RETS Software Development Metadata Format Versioning - 1  Hierarchy of metadata is versioned  Changes to lower level of the hierarchy propagate upward – Example: Changed METADATA-TABLE attribute will propagate up through CLASS, RESOURCE and SYSTEM

27. File: 05_RETS_Software_Development.PPT 27 RETS Software Development Metadata Format Versioning - 2  Caching of metadata is permitted  Use the login information to determine metadata changes  Recommendation: – If possible, get all the metadata on a change instead of attempting to get a specific type

28. File: 05_RETS_Software_Development.PPT 28 RETS Software Development Metadata Format Version 1.5 changes  Added the Foreign Keys metadata – Provides relationships between the offered resources of the form parent:child  Clarified a number of issues with Metadata Format  Recommendation: – Use 1.5 as a reference even when implementing 1.0.1 compliant software

29. File: 05_RETS_Software_Development.PPT 29 RETS Software Development Application Responses to Metadata Changes  Ignore metadata – Change breaks the application, intervention required to fix  Static metadata – Change is invisible, applications keep working, intervention to required view change  Dynamic metadata – Change is visible to applications without intervention

30. File: 05_RETS_Software_Development.PPT 30 RETS Software Development Search Transaction Overview  Most important RETS transaction  Can use either http GET method or http POST method – Example – GET method request http://rets.server.com:6103/search?Class=RES  Issue: GET method is limited in the character length of the request  Recommendation: POST method

31. File: 05_RETS_Software_Development.PPT 31 RETS Software Development Search Transaction Client Diagram

32. File: 05_RETS_Software_Development.PPT 32 RETS Software Development Search Transaction Server Diagram

33. File: 05_RETS_Software_Development.PPT 33 RETS Software Development Sample Search Search Request Header HTTP/1.1 200 OK Date: Fri, 31 Jul 2004 23:22:43 GMT Server: WebLogic XMLX Module 8.1 SP1 Fri Jun 20 23:06:40 PDT 2003 271009 with Content-Type: text/xml

34. File: 05_RETS_Software_Development.PPT 34 RETS Software Development Sample Search Search Request Header cont. Set-Cookie: RETS-Session-ID=20000439934 Set-Cookie: RETS-Request-ID="42" RETS-Version: RETS/1.0 Transfer-Encoding: Chunked Cache-Control: private

35. File: 05_RETS_Software_Development.PPT 35 RETS Software Development Sample Search Search Response Body 0fe8

36. File: 05_RETS_Software_Development.PPT 36 RETS Software Development Sample Search Search Response Body cont. 1 Shows Well RETS Software Development

37. File: 05_RETS_Software_Development.PPT 37 RETS Software Development Search Transaction SearchType Required  SearchType: The resource id to search on – Example – SearchType=Property

38. File: 05_RETS_Software_Development.PPT 38 RETS Software Development Search Transaction Class Required  The class of data within the resource to search against – Example – Class=RES  Issues: – Some of the data may be in system specific Class names rather than the Class name that represents an XML Standard Name

39. File: 05_RETS_Software_Development.PPT 39 RETS Software Development Search Transaction QueryType Required  QueryType: The query language – Only two types: DMQL and DMQL2 – Example – QueryType=DMQL2  Issues: – Mainly stylistic difference, DMQL2 supports quoted text – Not suitable for highly complex searches – Not possible to perform cross Class searches Example – NOT Residential and Lot Land

40. File: 05_RETS_Software_Development.PPT 40 RETS Software Development Search Transaction Query Required  Query: – Key – value pairs – Uses field name based on metadata or on standard names – Example Query=(ListPrice=200000-300000),(Status=|A)  Issues: – Lookup type queries can execute for a long time

41. File: 05_RETS_Software_Development.PPT 41 RETS Software Development Search Transaction Optional Arguments  Optional argument are just that  A vendor MAY choose to implement none, one or more  Software is RETS compliant without any of the optional arguments implemented  Issue: – Mismatch between client & server feature sets  Recommendation: – Do NOT assume, verify the feature set

42. File: 05_RETS_Software_Development.PPT 42 RETS Software Development Search Transaction Count  Default is no count  Behavior outside of given three values not defined

43. File: 05_RETS_Software_Development.PPT 43 RETS Software Development Search Transaction Format  Return records in XML or column header/row values format – XML is more verbose, but contains structure information – Some issues with site specific XML – clarification in RETS needed  Default is Standard-XML  Issue: – Site specific XML is not defined in the current standard

44. File: 05_RETS_Software_Development.PPT 44 RETS Software Development Search Transaction Format Sidebar: XML - 1  XML is a meta-language that defines a markup language  RETS is an example of an XML application  An application consists of a DTD and documents that obey the rules embodied in the DTD  The XML DTD is a document that describes the structure and elements of an XML document

45. File: 05_RETS_Software_Development.PPT 45 RETS Software Development Search Transaction Format Sidebar: XML - 2  The XML document takes data and applies the DTD  RETS will eventually move from a DTD based vocabulary to a Schema-based vocabulary

46. File: 05_RETS_Software_Development.PPT 46 RETS Software Development Search Transaction Format Sidebar: Compact Formats  Provides a delimited column description  Provides delimited data in the same position as the column description  Field delimiter is chosen by the server  Gotcha: – Delimiter tag is a two character representation of the ASCII character – Delimiter value may collide with data values in earlier implementations

47. File: 05_RETS_Software_Development.PPT 47 RETS Software Development Search Transaction Limit  Limits the number of records returned: – The NONE well known name requests a suspension of limit – Behavior is implementation dependent – Requesting a Limit should make the query faster – May still search for all records but only transmit records to a Limit

48. File: 05_RETS_Software_Development.PPT 48 RETS Software Development Search Transaction Offset  Return records starting from the specified point in the result set: – One method to handle very large data pulls – Given that the underlying transport mechanism is not guaranteed, very large results can fail with transport errors – Given that the data volumes can be very large, out of memory errors can occur

49. File: 05_RETS_Software_Development.PPT 49 RETS Software Development Search Transaction StandardNames  Indicates that the query uses the standard names or the system names: – Servers are not required to implement standard names to meet RETS compliance criteria – Standard Names are intended to be common across all RETS systems Refining the list of Standard Names is an ongoing process

50. File: 05_RETS_Software_Development.PPT 50 RETS Software Development Search Transaction Restricted Indicator  Certain fields may have restricted visibility: – Client may request fields that are restricted be replaced by a specific character string – Example - compensation amount field – Client may request that this amount be masked by ### – Server default is to return a null value

51. File: 05_RETS_Software_Development.PPT 51 RETS Software Development Search Transaction Select  Provides the ability to create a subset of the fields accessible to the client  Default is to return all accessible fields  Gotcha: Select field names are not specified as either the system names or the standard names – Ambiguity in RETS standard – Working assumption is that the server will understand both

52. File: 05_RETS_Software_Development.PPT 52 RETS Software Development Search Transaction DMQL  Defines the search criteria for the search result set  Based on a BNF grammar to provide a neutral representation  RETS Server parses the DMQL and creates a server specific query  Trade off between domain specific query forms and more complex query grammars like SQL-99  Issues around the lexical parsing of the grammar

53. File: 05_RETS_Software_Development.PPT 53 RETS Software Development Update Transaction Overview  Allows addition of records to the RETS system  Supports Auto-population of fields  References only by System Name, not Standard Name  Validation based on rules described in the metadata  Single record per update – no bulk update

54. File: 05_RETS_Software_Development.PPT 54 RETS Software Development Update Transaction Validation  Basic validation on data types  Additional validation for single lookup, multi-lookup and range  Additional validation for Boolean expression  Additional validation using external validation  Certain validation steps may be server only

55. File: 05_RETS_Software_Development.PPT 55 RETS Software Development Update Transaction Error Block  Validation tests that fail result in an error block that provides some information to assist in correction  One error block per failed validation test – Means a single field may trigger multiple errors

56. File: 05_RETS_Software_Development.PPT 56 RETS Software Development Update Transaction Gotchas and Recommendations  Gotcha: – RETS 1.7d1 adds a third state to the validate-flag “2” meaning “validate the record but do not save it” – Be aware of this coming specification change  Recommendation: – Client-side validation is fast fail – use it

57. File: 05_RETS_Software_Development.PPT 57 RETS Software Development GetObject Transaction Overview  Provides a way for users to get objects defined in the metadata  Main use is to return pictures although there are other media types  Returns the object in the appropriate mime type format - this may be a binary format  Recommendation: – Ordering of the object ID changed in 1.5D2 – Implement using the most recent RETS version

58. File: 05_RETS_Software_Development.PPT 58 RETS Software Development ChangePassword Transaction Overview  Allows a user to change their password  Some ambiguity in the 1.5 and earlier standard  Cryptographically weak  Recommendations: – Be very careful when implementing – Use the 1.7 standard addition as described in RCP 041

59. File: 05_RETS_Software_Development.PPT 59 RETS Software Development Logout Transaction Overview  Courtesy transaction  Communicates that a user has finished the session  A mechanism for the server to: – dispose of any local resources – communicate session and billing information  Recommendation: – Clients should log out – Servers should be aware that clients may never log out

60. File: 05_RETS_Software_Development.PPT 60 RETS Software Development Extensions to the RETS Standard  RETS allows vendors and sites to extend the standard to meet the specific needs of a site  Additional Capability List URLs have no suggested form  Additional parameters to an existing transaction and additional metadata have should have prefix ‘X’ – Example: Add sort direction to Search: XSortDir=Asc  Almost everything else is covered by metadata

61. File: 05_RETS_Software_Development.PPT 61 RETS Software Development Role of Parser Generators  Used to produce parsers for: – DMQL – Validation Expressions  DMQL Java code will be used by RETS server to generate SQL from DMQL  Validation expression Java code will be used by RETS server to execute the validation expression

62. File: 05_RETS_Software_Development.PPT 62 RETS Software Development Benefits for using a Parser Generator  Improves productivity for software developers – Not easy to write a good parser – Relatively easy to write the grammar input file for the parser generator  Avoids having to develop a parser – Difficult, demanding, time-consuming  Makes use of available parser generators – Examples: javacc, ANTLR, lexx  Produces more robust Java source code

63. File: 05_RETS_Software_Development.PPT 63 RETS Software Development Generating Java Source Code from a BNF Grammar File BNF Grammar File (produced by developer) Javacc Parser Generator Java Source code BNF grammar for DMQL - specified in RETS specification - see example on next two slides Grammar file syntax - specified in parser generator specification - see example on next two slides

64. File: 05_RETS_Software_Development.PPT 64 RETS Software Development BNF Grammar File Fragment DMQL Javacc Input File - 1 PARSER_BEGIN(DMQL) // The DMQL Javacc file which is used to generate the the primary class for parsing DMQL queries public class DMQL { } PARSER_END(DMQL) // Parse the DMQL statement. void Input() : { } // Match the the DMQL Query at the highest level. void Query() : { }

65. File: 05_RETS_Software_Development.PPT 65 RETS Software Development BNF Grammar File Fragment DMQL Javacc Input File - 2 { Search() ( | )}... BNF and Javacode Productions and additional Java code for processing tokens for parsing DMQL based on the RETS DMQL Grammar... // Terminals (Tokens) TOKEN : /* The DMQL Tokens (See RETS Specification) */ { |. | }

66. File: 05_RETS_Software_Development.PPT 66 RETS Software Development Application of Parser Output Java code to process DMQL SQL Input rules for update (produced by MLS) Java Validation Expression Code to evaluate Rules Validation Outcome Input DMQL (produced by client software) Parser Generator Output

67. File: 05_RETS_Software_Development.PPT 67 RETS Software Development Resources for Parser Generators  ANTLR, ANother Tool for Language Recognition – http://www.antlr.org  Java Compiler Compiler tm – JavaCC tm – https://javacc.dev.java.net  Lexers and Parser Generators – http://catalog.compilertools.net/lexparse.html

68. File: 05_RETS_Software_Development.PPT 68 RETS Software Development Topic Summary  RETS provides a rich set of features for implementing a service or client  Careful design will result in a successful project  Work with the vendor or customer to ensure that the correct features are built  Read the RETS document and watch for forward and reverse references  Use RETS 1.5 as a supplement to RETS 1.0.1 – it includes many clarifications

69. File: 05_RETS_Software_Development.PPT 69 RETS Software Development Software Quality

70. File: 05_RETS_Software_Development.PPT 70 RETS Software Development Top 10 signs Technology Has Taken Over Your Life 10.You need to fill out a form that must be typewritten, but you can't because there isn't one typewriter in your house - only computers with laser printers. 9.You would rather get more dots per inch than miles per gallon. 8.You rotate your screen savers more frequently than your automobile tires. 7.You think of the gadgets in your office as "friends," but you forget to send your father a birthday card. 6.You know Bill Gates' e-mail address, but you have to look up your own social security number. 5.On vacation, you are reading a computer manual and turning the pages faster than everyone else who is reading John Grisham novels. 4.The thought that a CD could refer to finance or music never enters your mind. 3.Your letterhead lists a phone number, a fax number, e-mail addresses for two online services, and your web site address but no street address. 2.You have never sat through an entire movie without having at least one device on your body beep or buzz. 1.You sign Christmas cards by putting :-) next to your signature.

71. File: 05_RETS_Software_Development.PPT 71 RETS Software Development Typical Session Workflow Login Search Get Object Logout Update Metadata ? Get Metadata Get Objects ? Data about available server data Data about Offices, Agents, Listings Pictures, Video Documents