1. Chapter 17: Application Recovery
17.2 Stateless Applications Based on Queues
17.3 Stateful Applications Based on Queues
17.4 Workflows Based on Queues
17.5 General Client-Server Applications
17.6 Lessons Learned
“Basic research is what I‘m doing when I don‘t know what I‘m doing.”
(Wernher von Braun)
“Research is the act of going up alleys to see if they are blind.” (Plutarch)
4/8/2019
Transactional Information Systems

2. From ACID To Recovery Guarantees
Problem:
Client that does not receive a returncode from transactional server
cannot easily find out the transaction outcome and
may be tempted to re-initiate the (non-idempotent) transaction,
thus producing unacceptable effects.
Approach:
In addition to atomicity, the transactional server needs to
guarantee the exactly-once execution of the transaction,
where execution includes the server‘s reply message.
 (almost) perfect failure masking
4/8/2019
Transactional Information Systems

3. Stateless Applications Based on Queues
(running on client, or app server or data server):
user sends input message
app program sends request message to data server
data server executes transaction and sends reply message to app
app program sends output message to user
there are no conversations with the user within a transaction,
and subsequent transactions are independent
Solution Queued Transactions:
message recovery by queue manager
with persistent, recoverable message queues
exactly-once execution by enclosing
message dequeue and enqueue into transaction
4/8/2019
Transactional Information Systems

4. Illustration of 2-Tier Queued Transaction
User
Application
Process
(Client)
Database
Server
...
input
output
enqueue
request
dequeue
reply
server transaction
4/8/2019
Transactional Information Systems

5. Illustration of 3-Tier Queued Transaction
User
Client
Database
Server
...
input
output
enqueue
request
dequeue
reply
distributed server transaction
Application
4/8/2019
Transactional Information Systems

6. Correctness of Queued Transaction Protocol
Theorem 17.1:
With the queued transaction protocol for stateless applications,
the following guarantees hold:
1. Once the user-input transaction is committed, a request is
executed by the server exactly once.
2. Once the user-input transaction is committed, the user output
is delivered at least once.
3. If user output is testable, the user output is delivered exactly once,
provided the user-input transaction has been committed.
Inherent (small window of) uncertainty:
(last) user input may get lost
(last) user output may be sent more than once
 can be eliminated with testable output (using special hardware)
4/8/2019
Transactional Information Systems

7. Client During Normal Operation
user-input processing by client:
begin transaction;
enqueue (request);
commit transaction;
user-output processing by client:
wait until reply queue is not empty;
dequeue (reply);
while user has not acknowledged the reply
or sent the next request do
present reply to user;
end /*while*/;
4/8/2019
Transactional Information Systems

8. Server During Normal Operation
request-reply processing by data server:
begin transaction;
dequeue (request);
perform data operations and generate reply;
enqueue (reply);
commit transaction;
4/8/2019
Transactional Information Systems

9. Client and Server Restart
Client restart:
check reply queue;
if not empty then
process reply like during normal operation;
end /*if*/;
Server restart:
check request queue;
if not empty then
initiate processing of requests
like during normal operation
end /*if*/;
4/8/2019
Transactional Information Systems

10. Chapter 17: Application Recovery
17.2 Stateless Applications Based on Queues
17.3 Stateful Applications Based on Queues
17.4 Workflows Based on Queues
17.5 General Client-Server Applications
17.6 Lessons Learned
4/8/2019
Transactional Information Systems

11. Pseudo-Conversational Transactions for Stateful Applications
Queue-based message recovery for entire conversations
Conversational “logical unit of work”
broken down into chain of stateless transactions
with (small) application state maintained in the queue
(analogously to Cookies, but more general and much more reliable)
Dequeue of reply and enqueue of next request
combined into one transaction for exactly-once execution guarantee
good for apps such as travel reservation, electronic shopping, etc.
4/8/2019
Transactional Information Systems

12. Illustration of Pseudo-Conversational Transactions
User
Application
Process
(Client)
Database
Server
...
4/8/2019
Transactional Information Systems

13. Correctness of Pseudo-Conversational Transaction Protocol
Theorem 17.2:
With the queue-based message recovery for conversational
multi-step transaction chains, the following guarantees hold:
Once the initial user-input transaction that starts the entire
conversation is committed, the entire transaction chain is
executed by the server exactly once.
Once the initial user-input transaction is committed, each
user-output message throughout the conversation is
delivered at least once.
If user output is testable, each user-output message is
delivered exactly once,
provided the initial user-input transaction has been committed.
4/8/2019
Transactional Information Systems

14. Chapter 17: Application Recovery
17.2 Stateless Applications Based on Queues
17.3 Stateful Applications Based on Queues
17.4 Workflows Based on Queues
17.5 General Client-Server Applications
17.6 Lessons Learned
4/8/2019
Transactional Information Systems

15. Queue-based Message Recovery for Exactly-Once Workflow Execution
At end of activity execute transaction that combines:
writing the activity‘s modifications of workflow state and context
to persistent store
writing the state modifications that result from the firing of
outgoing transitions to persistent store
writing the context modifications that result from the actions of
firing transitions to persistent store
notifying the follow-up activities by enqueueing messages
Newly invoked activity executes transaction that combines:
dequeueing of notification message
writing the workflow state and context to persistent store
4/8/2019
Transactional Information Systems

16. Use of Queued Transactions in Travel Planning Workflow
Select
Conference
Check
Flight
Hotel
Cost Go No
/ Budget:=1000;
Trials:=1;
/ Cost =
ConfFee +
TravelCost
[Cost  Budget]
[Cost > Budget & Trials < 3] / Trials++
[Cost > Budget
& Trials  3]
[!ConfFound]
[ConfFound]
/ Cost:=0
Tutorials
Compute
Fee
Áirfare
CheckTravelCost
CheckConfFee
4/8/2019
Transactional Information Systems

17. Chapter 17: Application Recovery
17.2 Stateless Applications Based on Queues
17.3 Stateful Applications Based on Queues
17.4 Workflows Based on Queues
17.5 General Client-Server Applications
17.6 Lessons Learned
4/8/2019
Transactional Information Systems

18. Problem Scenario with C/S Application Failures
User
Application
Process
(Client)
Database
Server
...
input
output
request
reply
2nd App ?
replay
crash
during normal operation
during client restart
4/8/2019
Transactional Information Systems

19. General Considerations for Client-Server Stateful Application Recovery
Message logging for message recovery and
deterministic program replay
(of piecewise deterministic program)
Installation points for process recovery and
reduced program replay
Server processes concurrent threads on behalf of many clients
Server “commits its state” upon sending a reply to a client
Forced logging should be minimized
Server should be able to perform independent recovery
4/8/2019
Transactional Information Systems

20. Server Reply Logging Method
Client and server each
maintain a message lookup table (MLT) and
write message log entries to a stable log
Client performs lazy, non-forced, logging,
and periodically creates intallation point,
and force-logs user-input messages
Server forces its log buffer before sending a reply message
Server recovery rebuilds message lookup table
and replays incomplete requests to produce reply
may need logging of read/write interleaving among threads
Client recovery rebuilds MLT,
reloads app from last installation point and
replays application, intercepting message events and
obtaining the contents of messages from local MLT or the server
Client sends stability notifications to facilitate server log truncation
4/8/2019
Transactional Information Systems

21. Data Structures for Server Reply Logging
client
server
stable log file
message
lookup
table
MSN
Type
... 10
request 20 30
reply 40 70 80 15
input 45
output
installation
point
lazy logging
force log upon reply 65
4/8/2019
Transactional Information Systems

22. Replaying Incomplete Requests with Server Reply Logging
client
server
MSN
Type
... 10
request 20 30
reply 15
input
... R(x)W(x)R(x)R(y)W(y)R(y) ...
4/8/2019
Transactional Information Systems

23. Log Truncation with Server Reply Logging
client c
server
...
MSN
Type 15
input 20
request 40
reply 45
output 70
request 70 +
stability notification
RedoMSN for client c 80
server log
other clients
client log
client message lookup table
4/8/2019
Transactional Information Systems

24. Chapter 17: Application Recovery
17.2 Stateless Applications Based on Queues
17.3 Stateful Applications Based on Queues
17.4 Workflows Based on Queues
17.5 General Client-Server Applications
17.6 Lessons Learned
4/8/2019
Transactional Information Systems

25. Transactional Information Systems
Lessons Learned
Application recovery needs to reconcile
data, process, and message recovery
Queue-based message recovery sufficient
for stateless and small-state, pseudo-conversational apps
(including workflows but disregarding the actual workflow activities)
Rich-state applications require more general approach to
message logging and process installation points
efficiently solved for client-server applications
hot research topic for advanced multi-tier e-services
(e.g., electronic auctions)
4/8/2019
Transactional Information Systems