1. DEV-9: Using the ProDataSet™ in OpenEdge® 10
John Sadd
Engineering Fellow and OpenEdge Evangelist

2. Agenda Where we came from ProDataSet Overview
Using a ProDataSet as a data cache
Batching data retrieval with ProDataSets
Extending ProDataSet definitions
Mapping ProDataSet data to XML
Conclusions
DEV-9: Using the ProDataSet in OpenEdge 10

3. Temp-Table Parameters
Define a temp-table
Based on a database table or not
Populate the temp-table on the server
One row or many
Pass the temp-table as a parameter
One parameter for each table
Modify the temp-table on the client and pass it back
These are basically the techniques available to you in V9 to pass data between client and AppServer.
DEV-9: Using the ProDataSet in OpenEdge 10

4. Temp-Table Limitations
Separate table and parameter for each two-dimensional data definition
Parent-child and other relationships need multiple parameters
No standard mechanism to record changes
DEV-9: Using the ProDataSet in OpenEdge 10

5. The direction
Define a single core OpenEdge object to represent complex data relationships
One handle, one parameter
Associate validation and other business logic
Built-in events
Map directly to data objects on other platforms
These are the basic goals of the ProDataSet
DEV-9: Using the ProDataSet in OpenEdge 10

6. ProDataSet to ADO.NET DataSet
ProDataSet is an OpenEdge in-memory data store
ProDataSet maps directly to an ADO.NET DataSet, allowing you to:
Design a .NET™ interface in terms of .NET DataSets
Pass data back and forth to OpenEdge AppServer™
ProDataSet also maps to Java™ SDO in OpenEdge 10.1A
One basic goal is to provide a direct mapping between Progress and the .NET DataSet for developers requiring access to their Progress applications from .NET.
DEV-9: Using the ProDataSet in OpenEdge 10

7. Agenda Where we came from ProDataSet Overview
Using a ProDataSet as a data cache
Batching data retrieval with ProDataSets
Extending ProDataSet definitions
Mapping ProDataSet data to XML
Conclusions
DEV-9: Using the ProDataSet in OpenEdge 10

8. A ProDataSet is made of Temp-Tables
Define temp-tables in the usual way
DEFINE TEMP-TABLE ttOrder
/* fields from Order db table */
FIELD OrderTotal AS DECIMAL
FIELD CustName LIKE Customer.Name
FIELD RepName LIKE SalesRep.RepName.
DEFINE TEMP-TABLE ttOrderLine…
DEFINE TEMP-TABLE ttItem…
The ProDataSet is composed of already familiar Progress objects, fundamentally the temp-table. So you define your temp-tables first. We will encourage you to define your temp-tables to represent the right internal view of your data, even when it is different from the physical database storage. You can define the mapping between the two and adjust that as you change data sources or improve your database design.
DEV-9: Using the ProDataSet in OpenEdge 10

9. Define the DataSet
Define the DataSet in terms of the temp-tables it combines
DEFINE DATASET dsOrder FOR ttOrder, ttOline, ttItem
DATA-RELATION OrderLine FOR ttOrder, ttOline
RELATION-FIELDS (OrderNum, OrderNum)
DATA-RELATION LineItem FOR ttOline, ttItem
RELATION-FIELDS (ItemNum, ItemNum) REPOSITION.
Then you define the DataSet in terms of those tables. The DataSet can be valuable even for data requiring only one table, because of the event support and other extended features of the DataSet as compared to a temp-table.
DEV-9: Using the ProDataSet in OpenEdge 10

10. Data-Relations
The Data-Relation defines the relationship between parent-child tables
The Relation-Fields define fields in parent and child to use for selection
…DATA-RELATION OrderLine FOR ttOrder, ttOline
RELATION-FIELDS (OrderNum, OrderNum)…
Relation-Fields are used to determine how to populate the DataSet by default
They also tell OpenEdge how to navigate the DataSet by default
When the business object requires multiple levels of data definition, DATA-RELATIONs define those relationships. As with every aspect of the DataSet, this provides powerful default behavior that you can extend or override where the relationship is more complex than a simple field mapping between tables.
DEV-9: Using the ProDataSet in OpenEdge 10

11. Data-Sources tell OpenEdge where the data comes from
The Data-Source is a separate object from the ProDataSet
Define a Data-Source for each temp-table buffer in the DataSet
Each Data-Source can define database buffers or a query or both
Data-Sources are separate objects so that they can be replaced, and so that the db definitions are not really part of the DataSet. The DataSet must be independent of any single data-source, both for flexibility and so that it can be passed as a parameter without db-specific definitions going along with it.
DEV-9: Using the ProDataSet in OpenEdge 10

12. Populating a ProDataSet
Use the FILL method to populate the DataSet from the Data-Sources
FILL on the DataSet fills every table starting with top-level buffers
FILL on a buffer fills that table and its descendents
FILL-MODE
NO-FILL – skip this table
EMPTY – empty the table first
APPEND – add more records to the table
MERGE – add recs & eliminate dups
REPLACE – replace existing recs
You can FILL the entire DataSet or a subset of it. You can disable Data-relations to further control this. You can also specify via FILL-MODE whether and how to fill each individual temp-table.
DEV-9: Using the ProDataSet in OpenEdge 10

13. Populating with Data-Relations
…DATA-RELATION OrderLine FOR ttOrder, ttOline
RELATION-FIELDS (OrderNum, OrderNum)…
When a DataSet is populated, OpenEdge retrieves children of the current parent
ProDataSet
Order
Data-Source
OrderTT
6 1 01/05/ /19/ /10/93
Application
Database
When you are populating a DataSet (using the FILL method), Progress automatically interleaves reads of parent and child data when the Data-Relations describe the relationship. Again, there are events to let you intercept this to read your own set of records, filter data, etc. All the children for all the parents in the top-level query are retrieved. You can disable relations or fill at different levels to avoid filling all the data at once if that is too expensiv.
Data-Relation
OrderLineTT
Order
Data-Source
DEV-9: Using the ProDataSet in OpenEdge 10

14. Event Callback Procedures
You can define event procedures for:
Before and After FILL of the ProDataSet
Before and After FILL of any temp-table buffer
Before and After FILL of any record
Use ProDataSet events to prepare queries, attach Data-Sources, etc.
Use Buffer events to manipulate the table
Use Record events to populate calculated fields
You can use these events when there is no Data-Source at all
Callback procedures let you define code for many events on the DataSet.
You could connect to a db in a BEFORE-FILL event on the DataSet.
You could ATTACH a data source in the BEFORE-FILL event on a buffer and DETACH it in the AFTER-FILL event.
You could filter records or calculate extra fields in the BEFORE/AFTER-RECORD-FILL event.
DEV-9: Using the ProDataSet in OpenEdge 10

15. Dynamic ProDataSets Every element of a ProDataSet can be dynamic
CREATE DATASET
ADD-BUFFER()
ADD-RELATION()
A Data-Source can also be dynamic:
CREATE DATA-SOURCE
And a DataSet can use dynamic temp-tables
CREATE TEMP-TABLE
Attributes and methods manipulate the DataSet through its handle
There is a complete set of dynamic statements and methods for DataSets as for other objects such as queries and temp-tables. Attributes let you access the DataSet through its handle, and most operations on the DataSet (such as FILL) are implemented as methods on the handle.
DEV-9: Using the ProDataSet in OpenEdge 10

16. Updating Through the ProDataSet
ProDataSet tables can have a companion before-image temp-table to record updates (add/change/delete)
Used to pass changes to the server to be made in the Data-Source
Maps to .NET and Java support for update versions
FCS of 10.0A has support for keeping track of updates to the DataSet after it has been filled.
DEV-9: Using the ProDataSet in OpenEdge 10

17. Before-Image Table for a DataSet
Original values for every modified record
Initial values for every added record
Original values for every deleted record
ProDataSet
OrderTT
6 1 01/05/ /19/ /10/93
This done using “shadow” temp-tables for each updated table that are created and maintained automatically and passed from client to server as part of the DataSet.
Data-Relation
OrderLineTT
A M D
Before-Table
DEV-9: Using the ProDataSet in OpenEdge 10

18. Saving Changes To The Database
SAVE-ROW-CHANGES method for saving a row to the database
Uses the before-table for optimistic locking
Allows full control in the event of a conflict
ERROR, ERROR-STRING, REJECTED, DATA-SOURCE-MODIFIED attributes
DEV-9: Using the ProDataSet in OpenEdge 10

19. ProDataSet demo… See demo notes for a more detailed description.
Most basic example of 4GL client using a dataset to read multi-table data and display.
Same example but now a GUI version where data is displayed in fill-in and browse objects. Also, logic is applied to calculate the order total and to manipulate the item data.
DEV-9: Using the ProDataSet in OpenEdge 10

20. ProDataSet Overview Summary
Multi-level data object built from temp-tables and their relations
Separate Data-Source to define the physical data
Automated FILL and navigation support
Update tracking within the ProDataSet
Save support for writing updates back to the Data-Source
DEV-9: Using the ProDataSet in OpenEdge 10

21. DEV-9: Using the ProDataSet in OpenEdge 10

22. Agenda Where we came from ProDataSet Overview
Using a ProDataSet as a data cache
Batching data retrieval with ProDataSets
Extending ProDataSet definitions
Mapping ProDataSet data to XML
Conclusions
DEV-9: Using the ProDataSet in OpenEdge 10

23. Passing a ProDataSet as a Parameter
You can pass a DataSet parameter as:
DATASET – static reference like TABLE
DATASET-HANDLE – dynamic reference to the DataSet and its definition, like TABLE-HANDLE HANDLE – passes a simple DataSet handle to a local procedure
Note!
By default the DATASET and DATASET-HANDLE forms copy the ProDataSet contents
DATASET parameter is similar to temp-table parameter set, NOTE the However! Bullet. This allows Progress to do the most efficient thing by default both for local and distributed calls, unless specifically told to copy the DataSet. This is different from temp-table parameters, which copy the temp-table when passed as TABLE or TABLE-HANDLE.
DEV-9: Using the ProDataSet in OpenEdge 10

24. Passing a ProDataSet BY-REFERENCE: Local to a single session
RUN OrderDset.p ON hSession
(INPUT iOrderNum,
OUTPUT DATASET dsOrder
BY-REFERENCE).
ProDataSet
OrderTT
6 1 01/05/ /19/ /10/93
So when run locally, a procedure references the already existing DataSet passed in as a parameter.
Parameter mode is effectively INPUT-OUTPUT.
Always uses the caller’s instance
/* OrderDset.p -- Test proc for Order Dataset */
{dsOrderTT.i}
{dsOrder.i}
DEFINE INPUT PARAMETER iOrderNum AS INTEGER.
DEFINE OUTPUT PARAMETER DATASET FOR dsOrder.
DEV-9: Using the ProDataSet in OpenEdge 10

25. Remote DataSet Parameter BY-REFERENCE
AppServer
RUN OrderDset.p ON hSession(INPUT iOrderNum, OUTPUT DATASET dsOrder
BY-REFERENCE).
/* OrderDset.p */
DEFINE OUTPUT PARAMETER
DATASET FOR dsOrder.
ProDataSet
OrderTT
6 1 01/05/ /19/ /10/93
ProDataSet
OrderTT
6 1 01/05/ /19/ /10/93
The same procedure call done remotely copies the definitions of all tables and relations, along with all the data, from server to client or vice-versa.
Client
DEV-9: Using the ProDataSet in OpenEdge 10

26. Illustration of BY-REFERENCE call
Proc1.p
Proc2.p
holds data
unused
DEFINE DATASET dsRef…
DEFINE DATASET dsRef…
dsRef:FILL().
PROCEDURE fetchData: DEFINE OUTPUT PARAMETER FOR DATASET dsRef.
dsRef:FILL().
END PROCEDURE.
RUN fetchData IN hProc2 (OUTPUT DATASET dsRef BY-REFERENCE).
Always the caller’s instance used
Called procedure instantiates a ProDataSet that is never used
References are redirected for the duration of that procedure call only
DEV-9: Using the ProDataSet in OpenEdge 10

27. New options for parameters in 10.1A: Using ProDataSet for data caching
Proc1.p
Proc2.p
data cache
Define DATASET dsRef REFERENCE-ONLY…
unused
Define DATASET dsRef…
PROCEDURE bindData: DEFINE OUTPUT PARAMETER FOR DATASET dsRef BIND. END PROCEDURE.
RUN bindData IN hProc2 (OUTPUT DATASET dsRef BIND)
RUN fetchData IN hProc2. DISPLAY <dsRef fields>.
PROCEDURE fetchData:
dsRef:FILL().
ProDataSet – or temp-table – on one side of the call is REFERENCE-ONLY
This tells OpenEdge not to instantiate it
It is like an uninitialized handle until bound
This new option can be used with BY-REFERENCE to eliminate the unused instance
Use the BIND option on both sides of a single procedure call
This binds references in the REFERENCE-ONLY side to the other definition for their lifetime
No need to pass the DataSet again!
ProDataSet – or temp-table – on one side of the call is defined REFERENCE-ONLY
Use the BIND option on both sides of a single procedure call
No need to pass the ProDataSet again!
DEV-9: Using the ProDataSet in OpenEdge 10

28. Summary of BY-REFERENCE and BIND
Use BY-REFERENCE to use the caller’s ProDataSet instance in the called procedure
Use REFERENCE-ONLY for the unused instance
Use BIND to identify called or caller procedure’s instance to use
BIND is useful for data caching and separation of data management from user interface
DEV-9: Using the ProDataSet in OpenEdge 10

29. Agenda Where we came from ProDataSet Overview
Using a ProDataSet as a data cache
Batching data retrieval with ProDataSets
Extending ProDataSet definitions
Mapping ProDataSet data to XML
Conclusions
DEV-9: Using the ProDataSet in OpenEdge 10

30. Batching in a distributed stateless environment
Client
AppServer
Database
dsCustomer
BATCH-SIZE
QUERY CustBrowse
APPEND
BATCH-SIZE attribute for a buffer
Maximum number of rows to FILL
LAST-BATCH attribute signals end of data
True when end of database query reached
NEXT-ROWID for repositioning a query
Set by OpenEdge at the end of a batch
ROWID of the next row in the database table
RESTART-ROWID
Set by the application when requesting the next batch
dsCustomer
LAST-BATCH
PROCEDURE offEnd:
OFF-END event
NEXT-ROWID
PROCEDURE fetchCustomers:
RUN fetchWhere ON hAppServer
(OUTPUT DATASET dsCustomer)
Procedure fetchWhere:
DATASET dsCustomer:FILL()
RESTART-ROWID
APPEND)
DEV-9: Using the ProDataSet in OpenEdge 10

31. ProDataSet events to support data batching
The goal is to make data retrieval with batching transparent
OFF-END event for a ProDataSet query
Occurs when a query goes off the end
Occurs before Browse:OFF-END
Occurs before Query:QUERY-OFF-END
FIND-FAILED for a FIND on a buffer
Use to respond to individual missing rows
DEV-9: Using the ProDataSet in OpenEdge 10

32. Sample code for batching
Set up an event handler for OFF-END
QUERY CustBrowse:SET-CALLBACK(“OFF-END”, “OffEnd”).
Define the event handler
PROCEDURE OffEnd: DEFINE INPUT PARAMETER DATASET FOR dsCustomer. IF NOT BUFFER eCustomer:LAST-BATCH THEN DO:
RUN fetchCustomers RETURN NO-APPLY. /* Cancel OFF-END */ END.
END PROCEDURE.
DEV-9: Using the ProDataSet in OpenEdge 10

33. Requesting a batch from the server
Make the request
RUN fetchWhere ON hServer (INPUT “Customer”, INPUT iBatchSize, INPUT-OUTPUT rRestartRowid, INPUT cWhereString, OUTPUT DATASET dsCustomer APPEND).
Handle the request for data on the server
hCustBuffer:BATCH-SIZE = iBatchSize. hDataSource:RESTART-ROWID(1) = rRestartRowid. hCustQuery:QUERY-PREPARE(“FOR EACH Customer WHERE “ + cWhereString).
DATASET dsCustomer:FILL().
IF hCustBuffer:LAST-BATCH = NO THEN rRestartRowid = hDataSource:NEXT-ROWID(1).
DEV-9: Using the ProDataSet in OpenEdge 10

34. Data batching demo… See demo notes for a more detailed description.
Most basic example of 4GL client using a dataset to read multi-table data and display.
Same example but now a GUI version where data is displayed in fill-in and browse objects. Also, logic is applied to calculate the order total and to manipulate the item data.
DEV-9: Using the ProDataSet in OpenEdge 10

35. DEV-9: Using the ProDataSet in OpenEdge 10

36. Agenda Where we came from ProDataSet Overview
Using a ProDataSet as a data cache
Batching data retrieval with ProDataSets
Extending ProDataSet definitions
Mapping ProDataSet data to XML
Conclusions
DEV-9: Using the ProDataSet in OpenEdge 10

37. Arguments to the methods can be expressions
Using new ABL syntax in OpenEdge 10.1A to mix dynamic and static references
In OpenEdge 10.0:
hDataSet:GET-BUFFER-HANDLE(“ttOrder”)
hDataSet:GET-BUFFER-HANDLE(“ttOrder”):BUFFER-FIELD(“OrderNum”):BUFFER-VALUE
hOrderBuf:BUFFER-FIELD(“OrderNum”):BUFFER-VALUE
Arguments to the methods can be expressions
1 TO NUM-BUFFERS or NUM-FIELDS
pcBufferName or pcFieldName
Supports highly dynamic access to buffers, temp-tables, and ProDataSets
DEV-9: Using the ProDataSet in OpenEdge 10

38. When to use a combination of dynamic and static
Sometimes you know the names of buffers within a ProDataSet or fields in a buffer
But you may not know the exact ProDataSet or buffer definition
The same logic may be used for multiple ProDataSets that share a buffer
Or for multiple buffers that share some field
hDataSet:GET-BUFFER-HANDLE(“ttOrder”)
DEV-9: Using the ProDataSet in OpenEdge 10

39. Providing known arguments to dynamic references in OpenEdge 10.1A
Instead of all this syntax:
hDataSet:GET-BUFFER-HANDLE(“ttOrder”):
BUFFER-FIELD(“OrderNum”):BUFFER-VALUE
In 10.1A you can code just this:
hDataSet::ttOrder::OrderNum
Dynamic reference
Literal value
Etcetera…
Double colon!
DEV-9: Using the ProDataSet in OpenEdge 10

40. Syntax variants hDataSet::ttOrder hDataSet::ttOrder::OrderNum
Yields the buffer handle for the temp-table ttOrder
hDataSet::ttOrder::OrderNum
Yields the field value for the field OrderNum in ttOrder
Note the second ‘::’ because ttOrder yields a handle
hBuffer::OrderNum
Yields the field value for the field OrderNum
Note that you get the BUFFER-FIELD value, not the field handle
DEV-9: Using the ProDataSet in OpenEdge 10

41. Why do you need literal references from handles?
PROCEDURE eItemAfterRowFill:
DEFINE INPUT PARAMETER DATASET-HANDLE phDataSet.
DEFINE VARIABLE cItemTypes AS CHAR INIT “BASEBALL,CROQUET,FISHING,FOOTBALL,GOLF,SKI,SWIM”.
DO iType = 1 TO NUM-ENTRIES(cItemTypes): cType = ENTRY(iType, cItemTypes). IF INDEX(phDataSet::eItem::ItemName, cType) NE 0 THEN phDataSet::eItem::ItemName = REPLACE(phDataSet::eItem::ItemName, cType, cType).
END.
END PROCEDURE.
DEV-9: Using the ProDataSet in OpenEdge 10

42. Summary of new double-colon syntax
Lets you code static references from a handle
Use for business logic against variable ProDataSet and temp-table definitions
Use with extended definitions
DEV-9: Using the ProDataSet in OpenEdge 10

43. Agenda Where we came from ProDataSet Overview
Using a ProDataSet as a data cache
Batching data retrieval with ProDataSets
Extending ProDataSet definitions
Mapping ProDataSet data to XML
Conclusions
DEV-9: Using the ProDataSet in OpenEdge 10

44. Why should you care about data as XML?
XML represents data in character format
Hold data independent of a database
Context or other data outside the DB
XML is a standard data interchange format
Web services and other integration
XML for the Integration Layer of the Reference Architecture
DEV-9: Using the ProDataSet in OpenEdge 10

45. Converting between XML and ProDataSets
New ProDataSet (and temp-table) methods in OpenEdge 10.1A:
READ-XML – takes XML source and loads it into a ProDataSet
It can create the ProDataSet definition after a CREATE DATASET statement
READ-XMLSCHEMA – creates or verifies XML Schema information
WRITE-XML – creates XML stream from data
WRITE-XMLSCHEMA – creates XML Schema information based on the data definition
DEV-9: Using the ProDataSet in OpenEdge 10

46. WRITE-XML example writeOrder.p
DEFINE DATASET dsOrder FOR eOrder, eOrderLine DATA-RELATION OrderLine FOR eOrder, eOrderLine RELATION-FIELDS (OrderNum, OrderNum).
hOrderBuf = DATASET dsOrder::eOrder.
hOlineBuf = DATASET dsOrder::eOrderLine.
hOrderBuf:ATTACH-DATA-SOURCE(DATA-SOURCE srcOrder:HANDLE, “Customer.Name,CustName”).
hOlineBuf:ATTACH-DATA-SOURCE(DATA-SOURCE srcOline:HANDLE,””).
QUERY OrderQuery:QUERY-PREPARE(“FOR EACH Order WHERE…).
DATASET dsOrder:FILL().
DATASET dsOrder:WRITE-XML (“FILE”,”dsOrder.xml”, TRUE).
DEV-9: Using the ProDataSet in OpenEdge 10

47. The resulting XML document
DEV-9: Using the ProDataSet in OpenEdge 10

48. Summary of ProDataSet-to-XML support
Auto-convert ProDataSet or temp-table data to XML
Optionally generate schema for it
Auto-convert XML data into a temp-table or ProDataSet
Optionally generate the definition from a dynamically CREATEd object
DEV-9: Using the ProDataSet in OpenEdge 10

49. OpenEdge 10.1A User Endorsement
“The absolutely incredible power of XML to and from temp-tables and ProDataSets has increased my productivity at least one order of magnitude and has made the application incredibly more flexible .”
Jeffrey Kodman , Manuvis
DEV-9: Using the ProDataSet in OpenEdge 10

50. Agenda Where we came from ProDataSet Overview
Using a ProDataSet as a data cache
Batching data retrieval with ProDataSets
Extending ProDataSet definitions
Mapping ProDataSet data to XML
Conclusions
DEV-9: Using the ProDataSet in OpenEdge 10

51. Summary – what did you just learn about?
How to build ProDataSets to express complex data, relations, and data sources
How this maps to .NET and other platforms
How to hold and manage updates using ProDataSets
How to extend your logic with built-in events
How to share ProDataSet instances
How to handle data batching
How to mix dynamic and static references
How to use XML for integration
DEV-9: Using the ProDataSet in OpenEdge 10

52. Supporting materials available on PSDN
DEV-9: Using the ProDataSet in OpenEdge 10

53. In Conclusion…
The ProDataSet is a key part of OpenEdge applications starting in OpenEdge 10
It is fundamental to our focus on helping you organize complex data definitions and business logic in the ABL
…In conformance with the OpenEdge Reference Architecture
The ProDataSet is a standards-conformant object that integrates with .NET, Sonic and Web services as part of a global Service-Oriented Architecture
DEV-9: Using the ProDataSet in OpenEdge 10

54. Additional Resources Expert Series book: Using ProDataSets
Now available through amazon.com!
PSDN white paper on Using Advanced ProDataSet Language Features
Part of a series on implementing the OERA
PSDN “patterns” papers on topics I touched on today
Two-day instructor-led or self-study course: Using ProDataSets
OpenEdge online documentation
**The whitepapers are currently in .pdf format only – they are not in the DynaWeb HTML doc set (yet).
DEV-9: Using the ProDataSet in OpenEdge 10

55. Questions?
DEV-9: Using the ProDataSet in OpenEdge 10

56. Thank you for your time
DEV-9: Using the ProDataSet in OpenEdge 10

57. DEV-9: Using the ProDataSet in OpenEdge 10