1. Chapter 17: Distributed Systems CSS503 Systems Programming
Prof. Munehiro Fukuda
Computing & Software Systems University of Washington Bothell
CSS503
Chapter 17: Distributed Systems

2. Chapter 17: Distributed Systems
Network Structure
Local-Area Network (LAN)
Covering a campus or a site
Ethernet, Token Ling, etc.
Speed  10 – 100Mbps
Fast and cheap broadcast
Nodes:
Desktops and laptops
Peripherals
A few servers
Wide-Area Network (WAN)
Links geographically separated sites
P2P long-haul line connection
Speed  – 45Mbps
Broadcast emulated with multiple messages
Nodes:
Routers
CSS503
Chapter 17: Distributed Systems 1

3. Network Structure Cont’d
computing nodes
router
switch
Layer 2: Mac Address
router
router
router
Layer 3: IP Address
servers
servers
Layer 3: IP Address
switch
Layer 2: Mac Address
router
Supercomputer
Layer 3: IP Address
computing nodes
CSS503
Chapter 17: Distributed Systems

4. Chapter 17: Distributed Systems
OSI 7 Layers
Site A
Site B
Peer-to-peer interface
Application protocol 7
Application
Application
rsh, ftp, Telnet
Service interface
Presentation protocol
Dealing with heterogeneity
(Java object serialization)
and cryptography (SSL) 6
Presentation
Presentation
Session protocol 5
Session
Session
Dialog control
(rarely supported)
Transport protocol 4
Transport
Transport
UDP, TCP
Network protocol 3
Network
Network IP
Data link protocol
IEEE802.2
connection or connectionless 2
Data link
Data link
Ethernet
Physical protocol 1
Physical
Physical
Network
CSS503
Chapter 17: Distributed Systems 3 3

5. Data Link Layer Ethernet
CSMA/CD
Carrier sense multiple access with collision detection
Listening to the shared medium
Transmitting a data packet
Detecting collision on the medium
Deferring and retransmitting a packet in 2k–time base collision window
Frame format
MAC (Media Access Control) Addresses
unique, 48-bit unicast address assigned to each adapter
example: 8:0:e4:b1:2
broadcast: all 1s, multicast first bit is 1
Bandwidth: 10Mbps, 100Mbps, 1Gbps
1 listen
3. detect Ⅹ
2 transmit
bytes
46 ~ 1500 8 6 6 2 4 12
Preamble
Dest
MAC Addr
Src
MAC addr
Type
for layer 3
IP: 0x0800
Body
CRC
Next
frame
Inter-frame gap
Min: 64bytes ~ Max: 1518bytes
CSS503
Chapter 17: Distributed Systems

6. Network Layer Internet
Global addressing
IP address
Best-effort delivery (unreliable service)
Connectionless datagram-based
packets are lost, delivered out of order, delayed for a long time, and even duplicated.
Datagram format
TTL: time to live (#hops)
Protocol: TCP, UDP
Checksum
SourceAddr: source IP address
DestinationAddr: destination IP address
frame type
Ethernet
preamble
dest addr
src addr
0x0800
CRC V
ersion
HLen
TOS
Length
Ident
Flags
Offset
TTL
Protocol
Checksum
SourceAddr
DestinationAddr
Options (variable)
Pad
(variable) 4 8 16 19 31
Data
CSS503
Chapter 17: Distributed Systems

7. Internet as a Focal Point…
FTP
HTTP NV
TFTP
TCP
UDP IP
NET 1 2 n
Connection-oriented stream
data delivery
Connection-less datagram
delivery
Focal point for the architecture
Internet Protocol
Ethernet, FDDI, etc.
CSS503
Chapter 17: Distributed Systems 6 6

8. Internet as a Focal Point Cont’dH7 H8 H1 H2 H3
Network 4 (Ethernet)
Network 1 (Ethernet) R1 H4 H5 R3
Network 3
Network 2 (FDDI) H1
(point-to-point) H8 R2
FTP
FTP H6
TCP
TCP R1 R2 R3 IP IP IP IP IP
ETH
ETH
FDDI
FDDI
PPP
PPP
ETH
ETH
Network 3
Network 1 (Ethernet)
Network 2 (FDDI)
(point-to-point)
Network 4 (Ethernet)
CSS503
Chapter 17: Distributed Systems

9. Chapter 16: Distributed Systems
Transport Layer TCP (Connection-Oriented Stream) vs UDP (Connectionless Datagram)
Protocol
OSI layer
User/Kernel mode
IP address
IP port
Message length
Message delivery
Unix socket type
TCP
Layer 4
User
IP address
IP port
Any
FIFO order
SOCK_STREAM
UDP
65507 bytes
Unreliable
SOCK_DATAGRAM IP
Layer 3
Kernel
No IP port
65535 bytes
(practically 1500B)
SOCK_RAW . .
Seq 2
IP port
data
IP addr
TCP message
Seq 1
IP port
data
IP addr
TCP message
port 1 … 65535
UDP packet
IP port
data
IP addr
IP packet
data
IP addr
data
IP addr
IP address:
IP address:
Network address:
CSS503
Chapter 16: Distributed Systems 8 8

10. Chapter 16: Distributed Systems
Sockets and Ports
message
agreed port
any port
socket
Internet address =
Internet address =
other ports
client
server
CSS503
Chapter 16: Distributed Systems

11. UDP/TCP Socket Initialization
Client Socket:
Server Socket:
// Client assumes argv[1] and argv[2] as the server IP name
// and port.
char *server = argv[1];
int port = atoi( argv[2] );
// retrieve the server information from DNS.
struct hostent* host = gethostbyname( server );
if ( host == NULL ) {
fprintf( stderr, "cannot find %s.\n", server );
exit( -1 ); }
// allocate a socket address data structure.
struct sockaddr_in sendSockAddr;
bzero( ( char* )&sendSockAddr, sizeof( sendSockAddr ) );
// initialize the address with Internet(AF_INET), the server
// address(host), and the server port(port).
sendSockAddr.sin_family = AF_INET;
sendSockAddr.sin_addr.s_addr =
inet_addr( inet_ntoa( *(struct in_addr*)*host->h_addr_list ) );
sendSockAddr.sin_port = htons( port );
// allocate a UDP(SOCK_DGRAM) or TCP(SOCK_STREAM) socket.
int clientSd = socket( AF_INET, SOCK_DGRAM, 0 );
SOCK_STREAM,
// Server assumes argv[1] as its local port.
int port = atoi( argv[1] );
// allocate a socket address data structure.
struct sockaddr_in acceptSockAddr;
bzero( ( char* )&acceptSockAddr, sizeof( acceptSockAddr ) );
// initialize the address with Internet(AF_INET), the client
// address(INADDR_ANY), and the server port(port).
acceptSockAddr.sin_family = AF_INET;
acceptSockAddr.sin_addr.s_addr = htonl( INADDR_ANY );
acceptSockAddr.sin_port = htons( port );
// allocate a UDP(SOCK_DGRAM) or TCP(SOCK_STREAM) socket.
int serverSd = socket( AF_INET, SOCK_DGRAM, 0 );
SOCK_STREAM,
// bind this socket to a given port.
if( bind( serverSd, (struct sockaddr*)&acceptSockAddr,
sizeof( acceptSockAddr ) ) < 0 ) {
perror( "Cannot bind the local address to the server socket." );
exit( -1 ); }
CSS503
Chapter 16: Distributed Systems

12. UDP: User Datagram Protocol
client
server
Connectionless
May be lost
No FIFO order
Multicast feature
Unix datagram
Example: TFTP, rwho
socket()
Create a sock descriptor
socket()
(if a client needs to receive
a response)
bind()
Bind it to an IP address
bind()
recvfrom()
sendto()
Blocks until data received
recvfrom()
sendto()
CSS503
Chapter 16: Distributed Systems 11 11

13. Chapter 17: Distributed Systems
UDP Example Code
socket()
socket()
socket()
Port:12345
Port: 13579
Port:12345
bind()
bind()
struct sockaddr_in srcAddr;
socklen_t addrlen = sizeof( srcAddr );
bzero( ( char *)&srcAddr, sizeof( srcAddr ) );
char buf[256];
int nRead = recvfrom( serverSd, buf, 256, 0, (struct sockaddr*)&srcAddr,
&addrlen );
char buf[256];
int buf_len = sizeof( buf );
sendto( clientSd, buf, length, 0, (struct sockaddr *)sendSockAddr,
sizeof( struct sockaddr ) );
recvfrom()
recvfrom()
sendto()
Packets
demultiplexed
UDP
M5, M4, M3, M2, M1
CSS503
Chapter 17: Distributed Systems 12 12

14. TCP: Transport Control Protocol
client
server
Connection-oriented
Reliable
FIFO order
No Multicast feature
Unix stream socket
Example: ftp, http, rsh all major applications
socket()
Create a sock descriptor
socket()
Bind it to an IP address
bind()
Prepare a queue to pool requests
liseten()
Wait for a connection
accept()
Connection established
connect()
Blocks until connection established
write()
read()
read()
wrte()
CSS503
Chapter 17: Distributed Systems 13 13

15. Passing a TCP Connection to a Child
socket()
socket()
socket()
bind()
listen( serverSd, 5 );
connect( clientSd, ( struct sockaddr* )sendSockAddr,
sizeof( struct sockaddr ) );
listen()
connect()
connect()
int newSd = 0;
struct sockaddr_in newSockAddr;
socklen_t newSockAddrSize = sizeof( newSockAddr );
while( true ) {
newSd = accept( serverSd, (struct sockaddr*)&newSockAddr,
&newSockAddrSize ) );
char buf1[256], buf2[1024];
write( clientSd, buf1, sizeof( buf ) );
write( clientSd, buf2, sizeof( buf ) );
accept()
if ( fork( ) == 0 ) {
close( serverSd );
char buf1[256], buf2[1024];
read( newSd, buf1, sizeof( buf1 ) );
read( newSd, buf2, sizeof( buf2 ) );
close( newSd );
exit( 0 ); }
write()
write()
buf2, buf1
buf2, buf1
read()
read()
close( newsd); }
CSS503
Chapter 17: Distributed Systems 14

16. Socket Implementation
Creating a socket: int socket(int domain, int type, int protocol)
domain = PF_INET, PF_UNIX
type = SOCK_STREAM, SOCK_DGRAM
Active Open (Client)
connect( int sd, struct sockaddr *name, socklen_t namelen )
Passive Open (Server)
bind( int sd, struct sockaddr *name, socklen_t namelen );
listen( sd, 5 );
accept( sd, struct sockaddr *addr, socket_t *addrlen );
file
Client process
inode
Server process
int fd = open( “fileA”, O_RDONLY, 0 );
type
protocol
so_pcb
so_rcv
so_snd
Socket data structure
type
protocol
so_pcb
so_rcv
so_snd
Socket data structure
File structure table 1 2 3 4
User file descriptor table 1 2 3 4
User file descriptor table
File structure table
packet
copied
copied
packet
packet
packet
packet
type
protocol
so_pcb
so_rcv
so_snd
Socket data structure
packet
l_addr
l_port
f_addr
f_port
socket
inpcb: Internet protocol control block
l_addr
l_port
f_addr
inpcb: Internet protocol control block
f_port
socket
CSS503
Chapter 17: Distributed Systems

17. Chapter 16: Distributed Systems
Socket Structure
Process
Disk
int fd = open(“fileA”, flags);
read(fd, …);
int sd = socket( AF_INET, SOCK_STREAM, 0 );
connect( sd, &sockaddr, sizof( sockaddr ) );
struct inode {
length
count: 1
direct[12]
indirect[3] }
struct file {
count: 1
f_ops: file pointer
f_data
f_type: DTYPE_INODE }
blocks
Inode Table
File System
Process Control Block
IP Implementation
Internet Protocol
Control Block 1 2 3 4
stdin
struct file {
count: 1
f_ops: file pointer
f_data
f_type: DTYPE_SOCKET }
stdout
struct inpcb {
inp_faddr
inp_fport
inp_laddr
inp_lport
inp_socket }
stderr
struct socket {
so_type: SOCK_STREAM
count: 1
so_pcb
so_snd
so_rcv }
mbuf { data }
User File
Descriptor Table
Socket Structure Table
File Structure Table
CSS503
Chapter 16: Distributed Systems 16 16

18. Sharing a Socket among Processes
stdin
stdout
stderr 1 2 3 4
User File
Descriptor Table
Process Control Block
int sd = socket( AF_INET, SOCK_STREAM, 0 );
bind( sd, … );
listen( sd, 5 );
int new_sd = accept( sd, &sockaddr, sizeof( sockaddr ) );
if ( fork( ) == 0 ) {
close( sd );
execl( “program2”, … ); }
close( new_sd );
Internet Protocol
Control Block
Socket Structure Table
struct inpcb {
inp_faddr
inp_fport
inp_laddr
inp_lport
inp_socket }
struct file {
count: 1
f_ops: file pointer
f_data
f_type: DTYPE_SOCKET }
struct socket {
so_type: SOCK_STREAM
count: 1
so_pcb
so_snd
so_rcv } X
struct file {
count: 1
f_ops: file pointer
f_data
f_type: DTYPE_SOCKET }
fork and execl X
struct socket {
so_type: SOCK_STREAM
count: 1
so_pcb
so_snd
so_rcv }
struct inpcb {
inp_faddr
inp_fport
inp_laddr
inp_lport
inp_socket }
copied
File Structure Table
Process 2
stdin
stdout
stderr 1 2 3 4
User File
Descriptor Table
Process Control Block
int main(int argc, char** argv) {
int fd = 4;
read( fd, … ); }
mbuf { data }
CSS503
Chapter 16: Distributed Systems 17 17

19. Socket Sharing Example: RSH
Client
Server
shell
inetd
TCP connection
request
Command
rsh ls- l
rshd
shell
TCP connection
Inherited all the way
To a child
Command
ls -l
CSS503
Chapter 17: Distributed Systems 18

20. User Processes and Messaging
The source process completes a message transfer by itself.
The destination process is blocked until the current process delivers a message to it.
Destination Process
Source Process
context switch
user message
Application Layer
user message
Current Process
tcp
header
user data 1
tcp
header
user data 2
Transport Layer
tcp
header
user data 1
tcp
header
user data 2 ip
header
tcp
header
user data 1 ip
header
tcp
header
user data 2
Network Layer ip
header
tcp
header
user data 1 ip
header
tcp
header
user data 2
eth
frame ip
header
tcp
header
user data 1
eth
frame ip
header
tcp
header
user data 2
Data Link Layer
eth
frame ip
header
tcp
header
user data 1
eth
frame ip
header
tcp
header
user data 2
CSS503
Chapter 17: Distributed Systems

21. Programming Assignment 4 Inter-Segment UDP Broadcast
CSS503
Chapter 17: Distributed Systems

22. Programming Assignment 4 Inter-Segment UDP Broadcast
message
UDP
message
UDP
message
UDP
message
BroadcastServer.java
BroadcastServer.java
-32
-31
-30 2
BroadcastClient.java
BroadcastServer.java
-32
-31
-30 1
A’s IP
-32
-31
-30 2
-32
-31
-30 2
A’s IP
B’s IP
A’s IP
A’s IP
UDP
message
UDP
message
-32
-31
-30
-32
-31
-30
B’s IP
B’s IP
UDP
message
UDP
message
UDP
message
UDP
message
UdpRelay.cpp (A)
UdpRelay.cpp (B)
BroadcastSocket.java
BroadcastSocket.java
BroadcastSocket.java
BroadcastSocket.java
TCP Link
packet not looped
CSS503
Chapter 17: Distributed Systems

23. Programming Assignment 4 Inter-Segment UDP Broadcast
CSS503
Chapter 17: Distributed Systems

24. Programming Assignment 4 Inter-Segment UDP Broadcast
UdpRelay.java
commandThread
relayOutThread
User
remote
segment 1
read
relayOutThread
read
remote
segment 2
multicast
write
local segment
multicast
write
relayInThread
acceptThread
recv
message
remote
segment 3
new connection
CSS503
Chapter 17: Distributed Systems

25. Programming Assignment 4 Inter-Segment UDP Broadcast
TCP Link
TCP Link
UdpRelay
UdpRelay
UdpRelay
BroadcastServer
BroadcastServer
BroadcastClient
BroadcastServer
Node 2
Node 1
Node 6
Node 5
Node 7
Node 3
Node 4
CSS503
Chapter 17: Distributed Systems

26. Chapter 17: Distributed Systems
Discussion
Under what circumstances is a token-passing network more effective than an Ethernet network?
Compare multi-process versus multi-threaded servers in terms of their pros and cons.
Solve textbook Ex 17.18: why does HTTP use TCP/IP? Why not UDP or any other performance improvement?
CSS503
Chapter 17: Distributed Systems