Scalable network services - Java non-blocking IO - Reactor pattern

Classic ServerSocket Loop class Server implements Runnable { public void run() { try { ServerSocket ss = new ServerSocket(PORT); while (!Thread.interrupted()) new Thread(new Handler(ss.accept())).start(); // or, single-threaded, or a thread pool } catch (IOException ex) { /* ... */ } } static class Handler implements Runnable { final Socket socket; Handler(Socket s) { socket = s; } byte[] input = new byte[MAX_INPUT]; socket.getInputStream().read(input); byte[] output = process(input); socket.getOutputStream().write(output); private byte[] process(byte[] cmd) { /* ... */ } Note: most exception handling avoided in code examples Classic ServerSocket Loop

Graceful degradation under increasing load (more clients) Scalability Goals Graceful degradation under increasing load (more clients) Continuous improvement with increasing resources (CPU, memory, disk, bandwidth) Also, to meet availability and performance goals Response time, Peak demand

java.nio Server Side

java.nio Support Channels Buffers Selectors SelectionKeys Connections to files, sockets etc that support non-blocking operations (read, write) Buffers Array-like objects that can be directly read or written by Channels Selectors Tell which of a set of Channels have IO events SelectionKeys Maintain IO event status and bindings

Event-driven Designs Usually more efficient than alternatives Fewer resources Don't usually need a thread per client Less overhead Less context switching, often less locking But dispatching can be slower Must manually bind actions to events Usually harder to program Must break up into simple non-blocking actions Similar to GUI event-driven actions Cannot eliminate all blocking: GC, page faults, etc Must keep track of logical state of service

Program structure with Java non-blocking IO Create Selector and one or more Channels Register Channels to a Selector Mark the events of interest to detect in each channel. Loop: Call to the select operation of Selector. Returns a number of the keys whose ready set has changed. (each key corresponds to a channel) Using the selected keys: Check ready operations for each key. Remove the key from the selected keys set. That clears the ready operations set of the key. Do appropriate processing.

Program structure with Java non-blocking IO // Create the server socket channel ServerSocketChannel server = ServerSocketChannel.open(); // nonblocking I/O server.configureBlocking(false); // host-port 8000 server.socket().bind(new java.net.InetSocketAddress(host,8000)); // Create the selector Selector selector = Selector.open(); // Recording server to selector (type OP_ACCEPT) server.register(selector,SelectionKey.OP_ACCEPT);

Program structure with Java non-blocking IO for(;;) { // Waiting for events selector.select(); // Get keys Set keys = selector.selectedKeys(); Iterator i = keys.iterator(); // For each keys... while(i.hasNext()) { SelectionKey key = (SelectionKey) i.next(); // Remove the current key i.remove();

Program structure with Java non-blocking IO // if isAccetable = true - then a client required a connection if (key.isAcceptable()) { // get client socket channel SocketChannel client = server.accept(); // Non Blocking I/O client.configureBlocking(false); // recording to the selector (reading) client.register(selector, SelectionKey.OP_READ); continue; } // if isReadable = true - then the server is ready to read if (key.isReadable()) { SocketChannel client = (SocketChannel) key.channel(); try { client.read(buffer); } catch (Exception e) {… }

Buffers Buffer  a container for a fixed amount of data. Similar to a byte [] array But encapsulated such that internal storage can be a block of system memory. Direct memory mapping Zero-copy read/receive  high performance communication

Creating buffers Factory methods ByteBuffer allocate(int capacity) creates a ByteBuffer with an ordinary Java backing array of size capacity. ByteBuffer allocateDirect(int capacity) Does its best to provide zero-copy IO ByteBuffer wrap(byte [] array) wrap() methods create ByteBuffer’s backed by all or part of an array allocated by the user ByteBuffer wrap(byte [] array, int offset, length) all static methods of the ByteBuffer class.

Channels high-level version of file-descriptors from POSIX of operating systems. a handle for performing I/O and control operations on an open file/socket. java.nio associates a channel with any RandomAccessFile, FileInputStream, FileOutputStream, Socket, ServerSocket or DatagramSocket object. the channel just provides extra NIO-specific functionality. NIO buffer objects can written to or read from channels directly.

Opening Channels Socket channel classes have static factory methods called open(), e.g.: SocketChannel sc = SocketChannel.open() ; Sc.connect(new InetSocketAddress(hostname, portnumber)) ;

Using Channels All channels that implement the ByteChannel interface provide a read() and a write() instance method: int read(ByteBuffer dst) int write(ByteBuffer src) The Java read() attempts to read from the channel as many bytes as there are remaining to be written in the dst buffer. Returns number of bytes actually read, or -1 if end-of-stream. Also updates dst buffer position. Similarly write() attempts to write to the channel as many bytes as there are remaining in the src buffer. Returns number of bytes actually read, and updates src buffer position.

Using Channels – Example Assume a source channel src and a destination channel dest: ByteBuffer buffer = ByteBuffer.allocateDirect(BUF_SIZE) ; while(src.read(buffer) != -1) { buffer.flip() ; // Prepare read buffer for “draining” while(buffer.hasRemaining()) dest.write(buffer) ; buffer.clear() ; // Empty buffer, ready to read next chunk. } Notes: a write() call (or a read() call) may or may not succeed in transferring whole buffer in a single call. Hence need for inner while loop. Example introduces two new methods on Buffer: hasRemaining() returns true if position < limit; clear() sets position to 0 and limit to buffer’s capacity.

Basic Socket Channel Operations Typical use of a server socket channel: ServerSocketChannel ssc = ServerSocketChannel.open() ; ssc.socket().bind( new InetSocketAddress(port) ) ; while(true) { SocketChannel sc = ssc.accept() ; … process a transaction with client through sc … } The client: SocketChannel sc = SocketChannel.open() ; sc.connect( new InetSocketAddr(serverName, port) ) ; … initiate a transaction with server through sc … Then typically use read() and write() calls on the SocketChannel  four operations: accept(), connect(), write(), read()

Nonblocking Operations socket.configureBlocking(false) ; A read() operation transfers data that is immediately available. If no data immediately available, returns 0. If data cannot be immediately written, a write() will immediately return 0. For a server socket, if no client is currently trying to connect, the accept() method immediately returns null. connect() is more complicated generally connections would always block for some interval waiting for the server to respond. In non-blocking mode connect() generally returns false. But the negotiation with the server is nevertheless started. The finishConnect() method on the same socket should be called later. It also returns immediately. Repeat until it return true.

Setting Up Selectors To create: To add channel: open() factory method. To add channel: SelectionKey register(Selector sel, int ops) ops  bit-set representing the interest set for this channel: composed by oring together one or more of: SelectionKey.OP_READ SelectionKey.OP_WRITE SelectionKey.OP_CONNECT SelectionKey.OP_ACCEPT A channel added to a selector must be in nonblocking mode! The register() method returns the SelectionKey created This automatically gets stored in the Selector, so in most cases you probably don’t need to save the result yourself.

Example Create a selector and register three pre-existing channels: Selector selector = Selector.open() ; channel1.register (selector, SelectionKey.OP_READ) ; channel2.register (selector, SelectionKey.OP_WRITE) ; channel3.register (selector, SelectionKey.OP_READ | SelectionKey.OP_WRITE) ; Notes channel1, channel2, channel3 must all be in non-blocking mode at this time, and must remain in that mode as long as they are registered in any selector. You remove a channel from a selector by calling the cancel() method of the associated SelectionKey.

select() and the Selected Key Set call select() To inspect the set of registered channels Return value: zero if no status changes occurred. Side effect: set the selected keys embedded in the selector. A selector maintains a Set object representing this selected keys set. Each key is associated with a channel,  set of selected channels. The set of selected keys is different from (presumably a subset of) the registered key set. Each time the select() method is called it may add new keys to the selected key set, as operations become ready to proceed. The programmer is responsible for explicitly removing keys from the selected key set belonging to the selector

Ready Sets Each key in the registered key set has an associated interest set subset of the 4 possible operations on channels. Similarly each key in the selected key set has an associated ready set, a subset of the interest set—representing the actual operations that have been found ready to proceed. You can extract the ready set from a SelectionKey as a bit-set, by using the method readyOps(). Or you can use the convenience methods: isReadable() isWriteable() isConnectable() isAcceptable()

A Pattern for Using select() // … register some channels with selector … while(true) { selector.select() ; Iterator it = selector.selectedKeys().iterator() ; while( it.hasNext() ) { SelectionKey key = it.next() ; if( key.isReadable() ) … perform read() operation on key.channel() … if( key.isWriteable() ) … perform write() operation on key.channel() … if( key.isConnectable() ) … perform connect() operation on key.channel() … if( key.isAcceptable() ) … perform accept() operation on key.channel() … it.remove() ; }

Remarks This general pattern will probably serve for most uses of select(): Perform select() and extract the new selected key set For each selected key, handle the actions in its ready set Remove the processed key from the selected key set Note the remove() operation on an Iterator removes the current item from the underlying container. More generally, the code that handles a ready operation may also alter the set of channels registered with the selector e.g after doing an accept() you may want to register the returned SocketChannel with the selector, to wait for read() or write() operations. In many cases only a subset of the possible operations read, write, accept, connect are ever in interest sets of keys registered with the selector, so you won’t need all 4 tests.

Key Attachments Problem: when it.next() returns a key, there is no convenient way of getting information about the context in which the associated channel was registered with the selector. Solution: specify an arbitrary object as an attachment to the channel register it; Later when you get the key from the selected set, you can extract the attachment, and use its content in to decide what to do.

Simplistic Use of Key Attachments channel1.register (selector, SelectionKey.OP_READ, new Integer(1) ) ; // attachment … channel3.register (selector, SelectionKey.OP_READ | SelectionKey.OP_WRITE, new Integer(3) ) ; // attachment while(true) { Iterator it = selector.selectedKeys().iterator() ; SelectionKey key = it.next() ; if( key.isReadable() ) switch( ((Integer) key.channel().attachment() ).value() ) { case 1 : … action appropriate to channel1 … case 3 : … action appropriate to channel3 … }

Reactor pattern

Classic Network Service Design Each handler started in its own thread Problems?

Divide processing into small tasks Execute each task when it is ready Divide and Conquer Divide processing into small tasks Each task performs an action without blocking Execute each task when it is ready Here, an IO event usually serves as trigger

Reactor Pattern Purpose Avantages: Receive incoming messages from multiple concurrent clients. Process these messages using event handlers one at a time. Useful in e.g. servers, graphics systems, component frameworks. (example: Java AWT) Avantages: Serialisation of events  simpler concurrency in application. Modularity/portability improved.

Reactor Pattern – Structure

Reactor Pattern Structure (2) Handle Receives events; E.g. a network connection, timer, user interface device Synchronous Event Demultiplexer select() waits until an event is received on a Handle and returns the event. Often implemented as part of an operating system. Initiation Dispatcher Uses the Synchronous Event Demultiplexer to wait for events. Dispatches events to the Event Handlers. Event Handler Application-specific event processing code.

Reactor Pattern – Dynamics Setup Create Initiation Dispatcher. Register Event Handlers with Initiation Dispatcher. Main loop Call handleEvents in Initiation Dispatcher repeatedly. Initiation Dispatcher calls select in Synchronous Event Demultiplexer, blocking until an event is received. The Initiation Dispatcher calls handleEvent in the corresponding Event Handler, passing it the event. End Unregister Event Handlers from Initiation Dispatcher.

Summary Scalable network services Reactor pattern Java non-blocking IO _______________________________________Typical program structure Create Selector and one or more Channels Register Channels to a Selector Mark the events of interest to detect in each channel. Loop: Call to the select operation of Selector. Returns a number of the keys whose ready set has changed. (each key corresponds to a channel) Using the selected keys: Check ready operations for each key. Remove the key from the selected keys set. Do appropriate processing.

Example Code Introducing Nonblocking Sockets http://www.onjava.com/pub/a/onjava/2002/09/04/nio.html?page=1

Implementing Reactor 2: Dispatch loop // class Reactor continued public void run() { // normally in a new Thread try { while (!Thread.interrupted()) { selector.select(); Set selected = selector.selectedKeys(); Iterator it = selected.iterator(); while (it.hasNext()) dispatch((SelectionKey)(it.next()); selected.clear(); } } catch (IOException ex) { /* ... */ } void dispatch(SelectionKey k) { Runnable r = (Runnable)(k.attachment()); if (r != null) r.run();

Implementing Reactor 3: Acceptor // class Reactor continued class Acceptor implements Runnable { // inner public void run() { try { SocketChannel c = serverSocket.accept(); if (c != null) new Handler(selector, c); } catch(IOException ex) { /* ... */ } } // end class Acceptor } // end class reactor

Implementing Reactor 4: Handler setup final class Handler implements Runnable { final SocketChannel socket; final SelectionKey sk; ByteBuffer input = ByteBuffer.allocate(MAXIN); ByteBuffer output = ByteBuffer.allocate(MAXOUT); static final int READING = 0, SENDING = 1; int state = READING; Handler(Selector sel, SocketChannel c) throws IOException { socket = c; c.configureBlocking(false); // Optionally try first read now sk = socket.register(sel, 0); sk.attach(this); sk.interestOps(SelectionKey.OP_READ); sel.wakeup(); } boolean inputIsComplete() { /* ... */ } boolean outputIsComplete() { /* ... */ } void process() { /* ... */ }

Reactor 5: Request handling // class Handler continued public void run() { try { if (state == READING) read(); else if (state == SENDING) send(); } catch (IOException ex) { /* ... */ } } void read() throws IOException { socket.read(input); if (inputIsComplete()) { process(); state = SENDING; // Normally also do first write now sk.interestOps(SelectionKey.OP_WRITE); void send() throws IOException { socket.write(output); if (outputIsComplete()) sk.cancel();

Simple NIO Client Example SocketChannel client = SocketChannel.open(); // Create client SocketChannel client.configureBlocking(false); // nonblocking I/O client.connect(new java.net.InetSocketAddress(host,8000)); Selector selector = Selector.open(); // Create selector // register to selector (OP_CONNECT type) SelectionKey clientKey = client.register(selector, SelectionKey.OP_CONNECT); while (selector.select(500)> 0) { // Waiting for the connection Set keys = selector.selectedKeys(); // Get keys Iterator i = keys.iterator(); while (i.hasNext()) { // process keys SelectionKey key = (SelectionKey)i.next(); i.remove(); // Remove the current key SocketChannel channel = (SocketChannel)key.channel(); if (key.isConnectable()) { // Attempt a connection if (channel.finishConnect()) { // attempt to finalize connect // Write continuously on the buffer buffer = ByteBuffer.wrap(new String(" Client " + id + " ").getBytes()); channel.write(buffer); buffer.clear(); } Simple NIO Client Example