1. Scalable network services - Java non-blocking IO - Reactor pattern

2. Classic ServerSocket Loop
class Server implements Runnable {
public void run() {
try {
ServerSocket ss = new ServerSocket(PORT);
while (!Thread.interrupted())
new Thread(new Handler(ss.accept())).start();
// or, single-threaded, or a thread pool
} catch (IOException ex) { /* ... */ } }
static class Handler implements Runnable {
final Socket socket;
Handler(Socket s) { socket = s; }
byte[] input = new byte[MAX_INPUT];
socket.getInputStream().read(input);
byte[] output = process(input);
socket.getOutputStream().write(output);
private byte[] process(byte[] cmd) { /* ... */ }
Note: most exception handling avoided in code examples
Classic ServerSocket Loop

3. Graceful degradation under increasing load (more clients)
Scalability Goals
Graceful degradation under increasing load (more clients)
Continuous improvement with increasing resources (CPU, memory, disk, bandwidth)
Also, to meet availability and performance goals
Response time, Peak demand

4. java.nio
Server Side

5. java.nio Support Channels Buffers Selectors SelectionKeys
Connections to files, sockets etc that support non-blocking operations (read, write)
Buffers
Array-like objects that can be directly read or written by Channels
Selectors
Tell which of a set of Channels have IO events
SelectionKeys
Maintain IO event status and bindings

6. Event-driven Designs Usually more efficient than alternatives
Fewer resources
Don't usually need a thread per client
Less overhead
Less context switching, often less locking
But dispatching can be slower
Must manually bind actions to events
Usually harder to program
Must break up into simple non-blocking actions
Similar to GUI event-driven actions
Cannot eliminate all blocking: GC, page faults, etc
Must keep track of logical state of service

7. Program structure with Java non-blocking IO
Create Selector and one or more Channels
Register Channels to a Selector
Mark the events of interest to detect in each channel.
Loop:
Call to the select operation of Selector.
Returns a number of the keys whose ready set has changed.
(each key corresponds to a channel)
Using the selected keys:
Check ready operations for each key.
Remove the key from the selected keys set.
That clears the ready operations set of the key.
Do appropriate processing.

8. Program structure with Java non-blocking IO
// Create the server socket channel
ServerSocketChannel server = ServerSocketChannel.open();
// nonblocking I/O
server.configureBlocking(false);
// host-port 8000
server.socket().bind(new java.net.InetSocketAddress(host,8000));
// Create the selector
Selector selector = Selector.open();
// Recording server to selector (type OP_ACCEPT)
server.register(selector,SelectionKey.OP_ACCEPT);

9. Program structure with Java non-blocking IO
for(;;) {
// Waiting for events
selector.select();
// Get keys
Set keys = selector.selectedKeys();
Iterator i = keys.iterator();
// For each keys...
while(i.hasNext()) {
SelectionKey key = (SelectionKey) i.next();
// Remove the current key
i.remove();

10. Program structure with Java non-blocking IO
// if isAccetable = true - then a client required a connection
if (key.isAcceptable()) {
// get client socket channel
SocketChannel client = server.accept();
// Non Blocking I/O
client.configureBlocking(false);
// recording to the selector (reading)
client.register(selector, SelectionKey.OP_READ);
continue; }
// if isReadable = true - then the server is ready to read
if (key.isReadable()) {
SocketChannel client = (SocketChannel) key.channel();
try {
client.read(buffer);
} catch (Exception e) {… }

11. Buffers Buffer  a container for a fixed amount of data.
Similar to a byte [] array
But encapsulated such that internal storage can be a block of system memory.
Direct memory mapping
Zero-copy read/receive  high performance communication

12. Creating buffers Factory methods ByteBuffer allocate(int capacity)
creates a ByteBuffer with an ordinary Java backing array of size capacity.
ByteBuffer allocateDirect(int capacity)
Does its best to provide zero-copy IO
ByteBuffer wrap(byte [] array)
wrap() methods create ByteBuffer’s backed by all or part of an array allocated by the user
ByteBuffer wrap(byte [] array, int offset, length)
all static methods of the ByteBuffer class.

13. Channels
high-level version of file-descriptors from POSIX of operating systems.
a handle for performing I/O and control operations on an open file/socket.
java.nio associates a channel with any RandomAccessFile, FileInputStream, FileOutputStream, Socket, ServerSocket or DatagramSocket object.
the channel just provides extra NIO-specific functionality.
NIO buffer objects can written to or read from channels directly.

14. Opening Channels
Socket channel classes have static factory methods called open(), e.g.:
SocketChannel sc = SocketChannel.open() ;
Sc.connect(new InetSocketAddress(hostname, portnumber)) ;

15. Using Channels
All channels that implement the ByteChannel interface provide a read() and a write() instance method:
int read(ByteBuffer dst)
int write(ByteBuffer src)
The Java read() attempts to read from the channel as many bytes as there are remaining to be written in the dst buffer. Returns number of bytes actually read, or -1 if end-of-stream. Also updates dst buffer position.
Similarly write() attempts to write to the channel as many bytes as there are remaining in the src buffer. Returns number of bytes actually read, and updates src buffer position.

16. Using Channels – Example
Assume a source channel src and a destination channel dest:
ByteBuffer buffer = ByteBuffer.allocateDirect(BUF_SIZE) ;
while(src.read(buffer) != -1) {
buffer.flip() ; // Prepare read buffer for “draining”
while(buffer.hasRemaining())
dest.write(buffer) ;
buffer.clear() ; // Empty buffer, ready to read next chunk. }
Notes:
a write() call (or a read() call) may or may not succeed in transferring whole buffer in a single call. Hence need for inner while loop.
Example introduces two new methods on Buffer: hasRemaining() returns true if position < limit; clear() sets position to 0 and limit to buffer’s capacity.

17. Basic Socket Channel Operations
Typical use of a server socket channel:
ServerSocketChannel ssc = ServerSocketChannel.open() ;
ssc.socket().bind( new InetSocketAddress(port) ) ;
while(true) {
SocketChannel sc = ssc.accept() ;
… process a transaction with client through sc … }
The client:
SocketChannel sc = SocketChannel.open() ;
sc.connect( new InetSocketAddr(serverName, port) ) ;
… initiate a transaction with server through sc …
Then typically use read() and write() calls on the SocketChannel
 four operations: accept(), connect(), write(), read()

18. Nonblocking Operations
socket.configureBlocking(false) ;
A read() operation transfers data that is immediately available. If no data immediately available, returns 0.
If data cannot be immediately written, a write() will immediately return 0.
For a server socket, if no client is currently trying to connect, the accept() method immediately returns null.
connect() is more complicated
generally connections would always block for some interval waiting for the server to respond.
In non-blocking mode connect() generally returns false. But the negotiation with the server is nevertheless started. The finishConnect() method on the same socket should be called later. It also returns immediately. Repeat until it return true.

19. Setting Up Selectors To create: To add channel:
open() factory method.
To add channel:
SelectionKey register(Selector sel, int ops)
ops  bit-set representing the interest set for this channel: composed by oring together one or more of:
SelectionKey.OP_READ
SelectionKey.OP_WRITE
SelectionKey.OP_CONNECT
SelectionKey.OP_ACCEPT
A channel added to a selector must be in nonblocking mode!
The register() method returns the SelectionKey created
This automatically gets stored in the Selector, so in most cases you probably don’t need to save the result yourself.

20. Example Create a selector and register three pre-existing channels:
Selector selector = Selector.open() ;
channel1.register (selector, SelectionKey.OP_READ) ;
channel2.register (selector, SelectionKey.OP_WRITE) ;
channel3.register (selector, SelectionKey.OP_READ |
SelectionKey.OP_WRITE) ;
Notes
channel1, channel2, channel3 must all be in non-blocking mode at this time, and must remain in that mode as long as they are registered in any selector.
You remove a channel from a selector by calling the cancel() method of the associated SelectionKey.

21. select() and the Selected Key Set
call select()
To inspect the set of registered channels
Return value: zero if no status changes occurred.
Side effect: set the selected keys embedded in the selector.
A selector maintains a Set object representing this selected keys set.
Each key is associated with a channel,  set of selected channels.
The set of selected keys is different from (presumably a subset of) the registered key set.
Each time the select() method is called it may add new keys to the selected key set, as operations become ready to proceed.
The programmer is responsible for explicitly removing keys from the selected key set belonging to the selector

22. Ready Sets
Each key in the registered key set has an associated interest set
subset of the 4 possible operations on channels.
Similarly each key in the selected key set has an associated ready set,
a subset of the interest set—representing the actual operations that have been found ready to proceed.
You can extract the ready set from a SelectionKey as a bit-set, by using the method readyOps(). Or you can use the convenience methods:
isReadable()
isWriteable()
isConnectable()
isAcceptable()

23. A Pattern for Using select()
// … register some channels with selector …
while(true) {
selector.select() ;
Iterator it = selector.selectedKeys().iterator() ;
while( it.hasNext() ) {
SelectionKey key = it.next() ;
if( key.isReadable() )
… perform read() operation on key.channel() …
if( key.isWriteable() )
… perform write() operation on key.channel() …
if( key.isConnectable() )
… perform connect() operation on key.channel() …
if( key.isAcceptable() )
… perform accept() operation on key.channel() …
it.remove() ; }

24. Remarks
This general pattern will probably serve for most uses of select():
Perform select() and extract the new selected key set
For each selected key, handle the actions in its ready set
Remove the processed key from the selected key set
Note the remove() operation on an Iterator removes the current item from the underlying container.
More generally, the code that handles a ready operation may also alter the set of channels registered with the selector
e.g after doing an accept() you may want to register the returned SocketChannel with the selector, to wait for read() or write() operations.
In many cases only a subset of the possible operations read, write, accept, connect are ever in interest sets of keys registered with the selector, so you won’t need all 4 tests.

25. Key Attachments
Problem: when it.next() returns a key, there is no convenient way of getting information about the context in which the associated channel was registered with the selector.
Solution: specify an arbitrary object as an attachment to the channel register it;
Later when you get the key from the selected set, you can extract the attachment, and use its content in to decide what to do.

26. Simplistic Use of Key Attachments
channel1.register (selector, SelectionKey.OP_READ,
new Integer(1) ) ; // attachment …
channel3.register (selector, SelectionKey.OP_READ |
SelectionKey.OP_WRITE,
new Integer(3) ) ; // attachment
while(true) {
Iterator it = selector.selectedKeys().iterator() ;
SelectionKey key = it.next() ;
if( key.isReadable() )
switch( ((Integer) key.channel().attachment() ).value() ) {
case 1 :
… action appropriate to channel1 …
case 3 :
… action appropriate to channel3 … }

27. Reactor pattern

28. Classic Network Service Design
Each handler started in its own thread
Problems?

29. Divide processing into small tasks Execute each task when it is ready
Divide and Conquer
Divide processing into small tasks
Each task performs an action without blocking
Execute each task when it is ready
Here, an IO event usually serves as trigger

30. Reactor Pattern Purpose Avantages:
Receive incoming messages from multiple concurrent clients.
Process these messages using event handlers one at a time.
Useful in e.g. servers, graphics systems, component frameworks. (example: Java AWT)
Avantages:
Serialisation of events  simpler concurrency in application.
Modularity/portability improved.

31. Reactor Pattern – Structure

32. Reactor Pattern Structure (2)
Handle
Receives events; E.g. a network connection, timer, user interface device
Synchronous Event Demultiplexer
select() waits until an event is received on a Handle and returns the event.
Often implemented as part of an operating system.
Initiation Dispatcher
Uses the Synchronous Event Demultiplexer to wait for events.
Dispatches events to the Event Handlers.
Event Handler
Application-specific event processing code.

33. Reactor Pattern – Dynamics
Setup
Create Initiation Dispatcher.
Register Event Handlers with Initiation Dispatcher.
Main loop
Call handleEvents in Initiation Dispatcher repeatedly.
Initiation Dispatcher calls select in Synchronous Event Demultiplexer, blocking until an event is received.
The Initiation Dispatcher calls handleEvent in the corresponding Event Handler, passing it the event.
End
Unregister Event Handlers from Initiation Dispatcher.

34. Summary Scalable network services Reactor pattern Java non-blocking IO
_______________________________________Typical program structure
Create Selector and one or more Channels
Register Channels to a Selector
Mark the events of interest to detect in each channel.
Loop:
Call to the select operation of Selector.
Returns a number of the keys whose ready set has changed.
(each key corresponds to a channel)
Using the selected keys:
Check ready operations for each key.
Remove the key from the selected keys set.
Do appropriate processing.

35. Example Code Introducing Nonblocking Sockets

36. Implementing Reactor 2: Dispatch loop
// class Reactor continued
public void run() { // normally in a new Thread
try {
while (!Thread.interrupted()) {
selector.select();
Set selected = selector.selectedKeys();
Iterator it = selected.iterator();
while (it.hasNext())
dispatch((SelectionKey)(it.next());
selected.clear(); }
} catch (IOException ex) { /* ... */ }
void dispatch(SelectionKey k) {
Runnable r = (Runnable)(k.attachment());
if (r != null) r.run();

37. Implementing Reactor 3: Acceptor
// class Reactor continued
class Acceptor implements Runnable { // inner
public void run() {
try {
SocketChannel c = serverSocket.accept();
if (c != null)
new Handler(selector, c); }
catch(IOException ex) { /* ... */ }
} // end class Acceptor
} // end class reactor

38. Implementing Reactor 4: Handler setup
final class Handler implements Runnable {
final SocketChannel socket;
final SelectionKey sk;
ByteBuffer input = ByteBuffer.allocate(MAXIN);
ByteBuffer output = ByteBuffer.allocate(MAXOUT);
static final int READING = 0, SENDING = 1;
int state = READING;
Handler(Selector sel, SocketChannel c) throws IOException {
socket = c; c.configureBlocking(false);
// Optionally try first read now
sk = socket.register(sel, 0);
sk.attach(this);
sk.interestOps(SelectionKey.OP_READ);
sel.wakeup(); }
boolean inputIsComplete() { /* ... */ }
boolean outputIsComplete() { /* ... */ }
void process() { /* ... */ }

39. Reactor 5: Request handling
// class Handler continued
public void run() {
try {
if (state == READING) read();
else if (state == SENDING) send();
} catch (IOException ex) { /* ... */ } }
void read() throws IOException {
socket.read(input);
if (inputIsComplete()) {
process();
state = SENDING;
// Normally also do first write now
sk.interestOps(SelectionKey.OP_WRITE);
void send() throws IOException {
socket.write(output);
if (outputIsComplete()) sk.cancel();

40. Simple NIO Client Example
SocketChannel client = SocketChannel.open(); // Create client SocketChannel
client.configureBlocking(false); // nonblocking I/O
client.connect(new java.net.InetSocketAddress(host,8000));
Selector selector = Selector.open(); // Create selector
// register to selector (OP_CONNECT type)
SelectionKey clientKey = client.register(selector, SelectionKey.OP_CONNECT);
while (selector.select(500)> 0) { // Waiting for the connection
Set keys = selector.selectedKeys(); // Get keys
Iterator i = keys.iterator();
while (i.hasNext()) { // process keys
SelectionKey key = (SelectionKey)i.next();
i.remove(); // Remove the current key
SocketChannel channel = (SocketChannel)key.channel();
if (key.isConnectable()) { // Attempt a connection
if (channel.finishConnect()) { // attempt to finalize connect
// Write continuously on the buffer
buffer = ByteBuffer.wrap(new String(" Client " + id + " ").getBytes());
channel.write(buffer); buffer.clear(); }
Simple NIO Client Example