1. Computer Networks An Open Source Approach
Chapter 6: Internet Services
Chapter 6: Application Layer

2. Chapter 6: Application Layer
Content
6.1 Issues
6.2 Domain Name System
6.3
6.4 World Wide Web
6.5 File Transfer Protocol
6.6 Simple Network Management Protocol
6.7 Voice over IP (VoIP)
6.8 Streaming
6.9 Peer-to-Peer Applications (P2P)
6.10 Summary
Chapter 6: Application Layer

3. Chapter 6: Application Layer
6.1 Issues
How ports work?
How servers start?
Classification of servers
Characteristics of application layer protocols
Chapter 6: Application Layer

4. Chapter 6: Application Layer
How Ports Work?
Well known ports: 0~1023
Registered ports: 1024~49151
Dynamic and/or private ports: 49152~65535
Chapter 6: Application Layer

5. Chapter 6: Application Layer
How Servers Start?
Is a server better to run as standalone or via (x)inetd?
Standalone:
requires explicit restarting for configuration changes to take effect
crashed standalone server will probably stay unnoticed and the service will become unavailable
better performance
(x)inetd:
reads configuration files each time a client attempts to access their information
fresh copies will be spawned on demand
Chapter 6: Application Layer

6. Classification of Servers
Concurrent connectionless server
Concurrent connection-oriented server
Iterative connectionless server
Iterative connection-oriented server
Chapter 6: Application Layer

7. Underlying Transport Protocols
Application
Application layer protocol
Underlying transport protocol
electronic mail
SMTP, POP3, IMAP
TCP
remote terminal access
Telnet
file transfer
FTP
web
HTTP
web caching
ICP
typically UDP
name resolution
DNS
network file system
NFS
network management
SNMP
routing protocol
RIP
Internet telephony
proprietary (e.g. Vocaltec)
streaming multimedia
proprietary (e.g. RealNetworks)
P2P
proprietary (e.g. BitTorrent, eDonkey)
UDP for queries and TCP for data transfer
Chapter 6: Application Layer

8. Iterative Connectionless Server
Chapter 6: Application Layer

9. Concurrent Connection-Oriented Server
Chapter 6: Application Layer

10. Characteristics of Application Layer Protocols
Variable message formats and length
Message formats and length of application layer protocols vary with different applications and requirements
Variable data types
Messages can be transmitted in textual or non-textual formats.
E.g., a Web server replies textual Web pages and binary images
Statefulness
The server retains information of the session with the client
E.g., an FTP server remembers the client’s current working directory
Chapter 6: Application Layer

11. Chapter 6: Application Layer
6.2 Domain Name System
Domain name space and domain delegation
Zones and name servers
Zone data files and resource records
Resolvers and Name resolution
Open source: BIND
Chapter 6: Application Layer

12. Top Level Domains Domain Description com
Commercial organizations, such as Intel (intel.com).
org
Non-profit organizations, such as WWW consortium (w3.org).
gov
Government organizations, reserved for U.S government such as National Science Foundation (nsf.gov).
edu
Educational organizations, such as UCLA (ucla.edu).
net
Networking organizations, such as Internet Assigned Numbers Authority which maintains the DNS root servers (gtld-servers.net) .
int
Organizations established by international treaties between governments. For example, International Telecommunication Union (itu.int).
Mil
Reserved exclusively for the United States Military. For example, Network Information Center, Department of Defense (nic.mil).
Two-letter country code
The two-letter country code top level domains (ccTLDs) are based on the ISO two-letter country codes. Examples are tw (Taiwan), uk (United Kingdom).
arpa
Mostly unused now, except for the in-addr.arpa domain, which is used to maintain a database for reverse DNS queries.
Others
Such as .biz (business), .name (for individuals), .info (similar with .com).
Chapter 6: Application Layer

13. Chapter 6: Application Layer
Domain Delegation
Easier management
Load distribution
root domain tw
edu
com uk
org
com
edu co
nthu
nctu
nsysu
csie
cis ee
cis.nctu.edu.tw
Chapter 6: Application Layer

14. Chapter 6: Application Layer
Zones and Name Servers
Differences between a zone and a domain
A zone contains some part of the domain name space
A domain is a subtree in the domain name space
A name server may be authoritative for multiple zones
Zone data files in a name server
Contains Resource Records (RR) describing all the hosts within that zone
Six major types of RR: “SOA”, “NS”, “A”, “PTR”, “CNAME” and “MX”
Master and slave name servers
Availability and load balance
Data duplication by zone transfer
Request from slave
Notify by master
Chapter 6: Application Layer

15. Master/Slave Name Servers
1. Redundancy for heavy load
2. Availability achieved
slave
master (A) | slave (B)
master
A large domain name space
zone transfer
df of A and B
df of A
zone transfer
df of B
clients
clients
zone A
zone B
df: zone data file
Chapter 6: Application Layer

16. Chapter 6: Application Layer
Resource Records
SOA (Start Of Authority)
Specify a best name server to be authoritative (master) for a zone
Ex: cis.nctu.edu.tw IN SOA cisserv.cis.nctu.edu.tw.
Options for refresh, retry, expire ..etc are supported
NS (Name Server)
Specify name servers for a domain
Ex: cis.nctu.edu.tw IN NS cisserv.cis.nctu.edu.tw.
A (Address)
Map names to addresses,
Example (multi-homed):
linux.cis.nctu.edu.tw IN A
linux.cis.nctu.edu.tw IN A
Chapter 6: Application Layer

17. Resource Records (cont)
CNAME (Canonical Name)
Create name-to-name alias
Ex: IN CNAME cache.cis.nctu.edu.tw.
PTR (Pointer)
Point addresses to names (canonical name only)
Ex: in-addr.arpa IN PTR laser0.cis.nctu.edu.tw.
MX (Mail Exchanger)
Provide mail-routing information
The one of best (lowest) preference value is chosen
cis.nctu.edu.tw IN MX 0 mail.cis.nctu.edu.tw.
cis.nctu.edu.tw IN MX 10 mail1.cis.nctu.edu.tw.
Chapter 6: Application Layer

18. Chapter 6: Application Layer
Name Resolution
Query type
Forward query – name-to-address
Reverse query – address-to-name
Resolution method
Iterative (by referral, commonly adopted)
Recursive (site-by-site recursion)
Caching for speeding up the lookup process
Expire according to TTL
Tradeoff between performance and consistency
Chapter 6: Application Layer

19. Name Resolution (Iterative)1
root
name server
query for “ 2
client (resolver)
local name server 3
“uk”
name server 4 5
5M IN A
5M IN A
”gov.uk”
name server 6 7 8
“dti.gov.uk”
name server
other candidate name servers
Chapter 6: Application Layer

20. Chapter 6: Application Layer
Message Format
bit ID QR
Opcode AA TC RD RA
Reserved
Rcode
QDCOUNT
ANCOUNT
NSCOUNT
ARCOUNT
Question
Answer
Authority
Additional
Header
Chapter 6: Application Layer

21. Execution Flowchart of Named
Install a handler for signals
Cache initialization
Command line parsing
chroot and
daemonize
1. Task manager
2. Timer manager
3. Socket manager
Create managers
Initialize server configurations and data structures
For possible restart or termination signals
Wait for signals
Chapter 6: Application Layer

22. Chapter 6: Application Layer

23. Basic Data Structure in Named
viewlist …
view1
view2
viewN
NULL
zone_table
zone_table
zone_table …
zone1
zone3
Red/Black Tree for zones
zone3
zone2
zone4
zone8
zoneN
RR1
Red/Black Tree of RRs for zone4
RR2 …
RRN
Chapter 6: Application Layer

24. Domain Information Gopher (DIG)
; <<>> DiG <<>>
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26027
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3
;; QUESTION SECTION:
; IN A
;; ANSWER SECTION:
IN A
;; AUTHORITY SECTION:
nctu.edu.tw IN NS ns.nctu.edu.tw.
nctu.edu.tw IN NS ns2.nctu.edu.tw.
nctu.edu.tw IN NS ns3.nctu.edu.tw.
;; ADDITIONAL SECTION:
ns.nctu.edu.tw IN A
ns2.nctu.edu.tw IN A
ns3.nctu.edu.tw IN A
Chapter 6: Application Layer

25. Chapter 6: Application Layer
6.3
Introduction
Message Formats
Internet Mail Protocols
Open Source: Qmail
Chapter 6: Application Layer

26. Chapter 6: Application Layer
Introduction
What is ?
A method of sending messages from one user to another via computer networks
Internet mail addressing:
format:
e.g.
Chapter 6: Application Layer

27. Chapter 6: Application Layer
Introduction (cont.)
Components of Internet mail system:
Mail User Agent (MUA)
Mail Transfer Agent (MTA)
Mail Delivery Agent (MDA)
Mail Retrieval Agent (MRA)
Chapter 6: Application Layer

28. Chapter 6: Application Layer
Message Formats
Internet Message Format (RFC 822)
Multipurpose Internet Mail Extensions (RFC )
Chapter 6: Application Layer

29. RFC 822 – Internet Message Format
A message consists of:
An envelope which contains information needed to accomplish transport and delivery
Contents which compose the object to be delivered to the recipient
Chapter 6: Application Layer

30. RFC 822 – Internet Message Format (cont.)
Common message header fields:
Chapter 6: Application Layer

31. RFC 822 – Internet Message Format (cont.)
An example of message header:
From: "Dr. Ying-Dar Lin"
To:
Date: Mon, 30 Jul :42: (CST)
Subject: paper
Return-Path:
Received: (from by mail.cis.nctu.edu.tw (8.11.4/8.11.4) id f6TIgct33454 for Mon, 30 Jul :42: (CST) (envelope-from
Received: from cissun53.cis.nctu.edu.tw [ ]) by mail.cis.nctu.edu.tw (8.11.4/8.11.4av) with ESMTP id f6TIgaI33445 for Mon, 30 Jul :42: (CST) (envelope-from
Received: (from by cissun53.cis.nctu.edu.tw (8.9.3/8.9.3) id CAA16191 for juvenia; Mon, 30 Jul :42: (CST)
Message-Id:
X-Virus-Scanned: by AMaViS perl
X-UIDL: F5%#!88!"!>RG"!l!G!!
Chapter 6: Application Layer

32. Chapter 6: Application Layer
MIME
What is MIME?
MIME is a specification for enhancing the capabilities of conventional Internet message format .
MIME enables messages to have:
textual header information and message bodies in character sets other than 7-bit ASCII
multiple objects within a single message
binary or application-specific files
multimedia files, such as images, audio, and video files
Chapter 6: Application Layer

33. Chapter 6: Application Layer
MIME (cont.)
MIME message header:
Chapter 6: Application Layer

34. Chapter 6: Application Layer
MIME (cont.)
The MIME content type set:
Chapter 6: Application Layer

35. Chapter 6: Application Layer
MIME (cont.)
Possible values for the Content-Transfer-Encoding: field:
Quoted-Printable
Base64
7bit
8bit
Binary
X-Encoding
Chapter 6: Application Layer

36. Chapter 6: Application Layer
MIME (cont.)
An example of MIME message:
From: 'Ching-Ming Tien'
To:
Subject: Cover
MIME-Version: 1.0
Content-Type: image/jpg;
name=cover.jpg'
Content-Transfer-Encoding: base64
Content-Description: The front cover of the book
<.....base64 encoded jpg image of cover...>
Chapter 6: Application Layer

37. Internet Mail Protocols
Simple Mail Transfer Protocol (RFC 2821)
Post Office Protocol (RFC 1939)
Internet Message Access Protocol (RFC 2060)
Chapter 6: Application Layer

38. Simple Mail Transfer Protocol
What is SMTP?
A standard host-to-host mail transport protocol and traditionally operates over TCP on port 25
Chapter 6: Application Layer

39. Simple Mail Transfer Protocol (cont.)
Important SMTP commands:
SMTP replies:
Response
Description
2xx
Command accepted and processed.
3xx
General flow control.
4xx
Critical system or transfer failure.
5xx
Errors with the SMTP command.
Chapter 6: Application Layer

40. Simple Mail Transfer Protocol (cont.)
An SMTP transaction scenario:
Chapter 6: Application Layer

41. Chapter 6: Application Layer
Post Office Protocol
What is POP?
A protocol used to retrieve from a mail server.
POP3 Session States:
Authorization:
Must log in with password before entering transaction state.
Transaction:
Client can request actions of server, get mail for example.
Update:
Updates mail box to reflect actions taken in transaction state.
Chapter 6: Application Layer

42. Post Office Protocol (cont.)
Minimal POP3 commands:
Chapter 6: Application Layer

43. Post Office Protocol (cont.)
POP3 replies:
+OK, -ERR
Example POP session:
Chapter 6: Application Layer

44. Internet Message Access Protocol
What is IMAP?
A replacement for the POP3 protocol
Differences between IMAP4 and POP3:
IMAP4 allows messages being stored and manipulated on the mail system
POP3 only allows users to download their messages and store and manipulate messages on the client’s machines
Chapter 6: Application Layer

45. Internet Message Access Protocol (cont.)
Four states in the IMAP4 server:
Non-authenticated state
When a connection is established between the server and client
Authenticated state
When a pre-authenticated connection starts,
When acceptable authentication credentials have been provided
After an error in selecting a mailbox
Selected state
When a mailbox has been successfully selected
Logout state
When the client asks to exit the server
Chapter 6: Application Layer

46. Internet Message Access Protocol (cont.)
IMAP4 command summary:
Chapter 6: Application Layer

47. Internet Message Access Protocol (cont.)
An IMAP4 transaction scenario:
Chapter 6: Application Layer

48. Chapter 6: Application Layer

49. Chapter 6: Application Layer
Open Source: qmail
Introduction to qmail:
qmail is a secure, reliable, efficient, simple MTA designed for Unix-like operating systems
qmail is a replacement for the sendmail
qmail is the second most common SMTP server
qmail has by far the fastest growth of any SMTP server
Chapter 6: Application Layer

50. qmail System Structure
Core modules of qmail:
Chapter 6: Application Layer

51. Chapter 6: Application Layer
qmail Data Flow
Chapter 6: Application Layer

52. Chapter 6: Application Layer
qmail Control Files
Control files are in /var/qmail/control
Some control files of qmail:
Chapter 6: Application Layer

53. Chapter 6: Application Layer
qmail Queue Structure
The qmail queue directory is in /var/qmail/quque
Subdirectories in the qmail queue and contents :
Chapter 6: Application Layer

54. How Messages Pass Through the qmail Queue
Chapter 6: Application Layer

55. Chapter 6: Application Layer
6.4 World Wide Web
Introduction
Web Naming and Addressing
Web Data Formats
Hypertext Transfer Protocol
Open Source: Apache
Chapter 6: Application Layer

56. Chapter 6: Application Layer
Introduction
What is WWW?
A system of Internet servers that support specially formatted documents.
How does it work?
Chapter 6: Application Layer

57. Web Naming and Addressing
Uniform Resource Identifier (RFC 2396)
Uniform Resource Locator (RFC 1738)
Uniform Resource Name (RFC 2141)
Chapter 6: Application Layer

58. Uniform Resource Identifier
What is URI?
A compact string of characters for identifying an abstract or physical resource.
URI syntax:
Absolute URI: <scheme>:<scheme-specific-part>
Generic URI: <scheme>://<authority><path>?<query>
URI examples:
ftp://ftp.cis.nctu.edu.tw/Documents/IETF/rfc2300~2399/rfc2396.txt
mailto:
news: comp.os.linux
telnet://bbs.cis.nctu.edu.tw/
../icons/logo.gif
Chapter 6: Application Layer

59. Uniform Resource Locator
What is URL?
A compact string representation of the location for a resource that is available via the Internet
URL syntax:
Chapter 6: Application Layer

60. Uniform Resource Locator (cont.)
Some URL examples:
nntp://news.cis.nctu.edu.tw/cis.course.computer-networks/5238
telnet://mail.cis.nctu.edu.tw:110/
Chapter 6: Application Layer

61. Chapter 6: Application Layer
Uniform Resource Name
What is URN?
A name that identifies a resource of unit of information independent of its location
URN syntax:
<URN> ::= "urn:" <NID> ":" <NSS>
NID: Namespace Identifier
NSS: Namespace Specific String
URN examples:
urn:path:/A/B/C/doc.html
urn:ans:cis.nctu.edu.tw/ydlin/Resource
urn:isbn:
URN resolutioin:
Chapter 6: Application Layer

62. Chapter 6: Application Layer
Web Data Formats
Evolution of Web Data Formats
SGML
HTML
XML
XHTML
Chapter 6: Application Layer

63. Standard Generalized Markup Language
What is SGML?
A system for organizing and tagging elements of a document.
Characteristics of SGML
Descriptive Markup
Types of Document
Data Independence
Chapter 6: Application Layer

64. HyperText Makeup Language
What is HTML?
The authoring language used to create documents on the World Wide Web.
A brief history of HTML
HTML 2.0, 3.0, 3.2, 4.0
HTML document structure
HTML version information
Header
Body
Chapter 6: Application Layer

65. Extensible Markup Language
What is XML?
A pared-down version of SGML, designed especially for Web documents.
Why XML?
How to use XML?
Traditional data processing
Document-driven programming (DDP)
Archiving
Binding
Chapter 6: Application Layer

66. Extensible HyperText Markup Language
What is XHTML?
A hybrid between HTML and XML specifically designed for Net device displays.
Why XHTML?
Using XHTML with other W3C tag sets:
XHTML for structural markup of documents
SMIL for multimedia
MathML for mathematics
SVG for scalable vector graphics
XForms for smart web forms
Chapter 6: Application Layer

67. Hypertext Transfer Protocol
What is HTTP?
HTTP Conversation
Client Request
Server Response
HTTP 1.1
Chapter 6: Application Layer

68. Chapter 6: Application Layer
What is HTTP?
A comprehensive addressing scheme
Client-Server Architecture
The HTTP protocol is connectionless and stateless
An extensible and open representation for data types
Chapter 6: Application Layer

69. Chapter 6: Application Layer
HTTP Conversation
Client request
Request headers
Request body
Server status
Response headers
Requested data
Disconnected
Chapter 6: Application Layer

70. Chapter 6: Application Layer
Client Request
HTTP request methods:
Chapter 6: Application Layer

71. Chapter 6: Application Layer
Client Request
Request example:
Chapter 6: Application Layer

72. Chapter 6: Application Layer
Server Response
Response example:
Server status codes:
Chapter 6: Application Layer

73. Chapter 6: Application Layer

74. Chapter 6: Application Layer
HTTP 1.1
HTTP 1.1 features:
Persistent connections
Chunked encoding
byte ranges
Caching
Non-IP virtual hosting
Chapter 6: Application Layer

75. Chapter 6: Application Layer
Web Caching
Web caching is a mechanism to expedite document downloading in WWW.
To achieve maximum satisfaction from Web caching, some aspects need to be considered.
Candidates to be cached
Content replacement
Cache coherence
Copyright Reserved 2010
Chapter 6: Application Layer 75

76. Chapter 6: Application Layer
Transparent Proxy
A cache server can also act as a proxy server
With port redirection, transparent proxy does not require manual configuration
Two types of transparent proxy
integrated within a gateway
standalone server box
Copyright Reserved 2010
Chapter 6: Application Layer 76

77. Chapter 6: Application Layer
Open Source: Apache
Introduction to Apache:
Open-Source Web server originally based on NCSA server
Available on over 160 varieties of Unix -- and Windows NT
Over 58% of Internet Web servers run Apache or an Apache derivative
Chapter 6: Application Layer

78. Chapter 6: Application Layer

79. Chapter 6: Application Layer

80. Chapter 6: Application Layer

81. Chapter 6: Application Layer

82. Chapter 6: Application Layer

83. Chapter 6: Application Layer

84. Ration of sendfile ( ) to ap_invoke_handler ( )
File size
1 kb
4 kb
16 kb
64 kb
256 kb
1024 kb
# of calling sendfile ( ) 1 2 7 15
Total execution time (μs) of sendfile ( ) 37 42 78
215
527
Time ratio of sendfile ( ) to ap_invoke_handler ( )
35%
38%
40%
53%
77%
87%
Chapter 6: Application Layer

85. Apache Server Life Cycle
On Unix systems, Apache creates multiple processes to handle requests.
The Windows and OS/2 ports are multithreaded..
Chapter 6: Application Layer

86. The Request Processing Cycle
Chapter 6: Application Layer

87. 6.5 File Transfer Protocol
File transfer service
Operation model
Inside the connections
Open source: wu-ftpd
Chapter 6: Application Layer

88. Chapter 6: Application Layer
File Transfer Service
Goal
File sharing
Data replication for backup
Efficiency and reliability during transmission
Use client-server model based on TCP/IP
Authenticated and anonymous accesses
Relationship between FTP and Telnet
Access an ftp server with browser
ftp://ftp.cis.nctu.edu.tw (anony)
(authen)
Chapter 6: Application Layer

89. Some Application FTP Commands
Description
OPEN
Connect to a remote host
CAT
View a file in a remote host
GET
Retrieve files in a remote host
RENAME
Change the name of a file in a remote host RM
Delete a file in a remote host
QUIT
Terminate an FTP session
Chapter 6: Application Layer

90. Chapter 6: Application Layer
Operation Model
PORT: Send the IP and port of the client to which the data is retrieved
Client as both control host and receiver
Chapter 6: Application Layer

91. Inside the Connections
Establishing control/data connections
Active Mode
Control connection initiated by client
Data connection initiated by server
Passive Mode
When client is behind a firewall
Both control/data connections are initiated by client
FTP Reply
Example FTP session
Error Recovery
Chapter 6: Application Layer

92. Active/Passive Mode Active mode client server Control firewall Data
Request
(Connect to me at port P of host H)
Initiate a data connection
client
server
Control
firewall
Data
Passive mode
Request
(Ask server to listen)
Reply
(Listening on port P of host H)
Initiate a data connection
client
server
firewall
Chapter 6: Application Layer

93. Some Proper FTP Commands
Description
Type
USER
Send the user name
Access Control
PASS
Send the password
PORT
Send the IP and port of the client to which the data is retrieved
Transfer Parameter
PASV
Tell the server to listen on a data port rather than initiate a data connection
RETR
Ask server to transfer a copy of the requested file to the client
File service
STOR
Cause the server to accept and receive the data and store it as a file
RNFR
Specify the path of a source file to rename from
RNTO
Specify the path of a destination file to rename to
ABOR
Tell the server to abort the previous command and the corresponding data transfer
Chapter 6: Application Layer

94. FTP Reply Reply Description Type 1yz
The requested action is being initiated; expect another reply before proceeding with a new command.
Positive Preliminary reply
2yz
The requested action has been successfully completed.
Positive Complete reply
3yz
The command has been accepted, but the requested action is being held, waiting for further information from another command.
Positive Intermediate reply
4yz
The command is not accepted the and the requested action did not take place. The action can be requested again.
Transient Negative Completion reply
5yz
Similar with 4yz, except that the error condition is permanent so that the action cannot be requested again.
Permanent Negative Completion reply
Chapter 6: Application Layer

95. Chapter 6: Application Layer
Error Recovery
The restart mechanism
Sender inserts ‘marker’ (used to identify the checkpoint) in the data stream
Receiver marks the position of the marker and reply the latest marker position of both sender and receiver to user
When error, user issues ‘restart’ with the position of the marker to the sender
* User (control host) and receiver may/may not exist in the same machine
Chapter 6: Application Layer

96. Chapter 6: Application Layer
Example FTP Session
STATUS:> Connecting to (ip = )
STATUS:> Socket connected. Waiting for welcome message...
220 FTP server (Version wu-2.6.0(1) Mon Feb 28 10:30:36 EST 2000) ready.
COMMAND:> USER www
331 Password required for www.
COMMAND:> PASS ********
230 User www logged in.
COMMAND:> TYPE I
200 Type set to I.
COMMAND:> REST 100
350 Restarting at 100. Send STORE or RETRIEVE to initiate transfer.
COMMAND:> REST 0
350 Restarting at 0. Send STORE or RETRIEVE to initiate transfer.
COMMAND:> pwd
257 "/home/www" is current directory.
COMMAND:> TYPE A
200 Type set to A.
COMMAND:> PORT 140,113,189,29,10,27  tell the server where to connect to
200 PORT command successful.
COMMAND:> LIST  retrieve directory listing
150 Opening ASCII mode data connection for /bin/ls.  File status okay; about to open data connection
…….list of files….
COMMAND:> PORT 140,113,189,29,10,31
COMMAND:> RETR test  retrieve the file “test”
150 Opening BINARY mode data connection for test (5112 bytes).
Chapter 6: Application Layer

97. Chapter 6: Application Layer
Open Source: wu-ftpd
Introduction to wu-ftpd
Originally developed at Washington University
Most popular ftp daemon
Maintained by WU-FTPD Development Group.
Features and configurations
Virtual ftp servers
On-the-fly compression
Important configuration files
Inside wu-ftpd
Chapter 6: Application Layer

98. Inside wu-ftpd start ftp server with some options read ACL files
no (under (x)inetd)
stand-alone?
yes
service initialization
fork off
(parent exits)
reverse DNS check
listen to
requests no
connection
accepted?
parse & execute commands
yes
fork a handler
loop until termination signal is received
Chapter 6: Application Layer

99. Chapter 6: Application Layer
Virtual FTP Servers
1. Manage two or more FTP servers in one machine
2. Support guest groups for different virtual FTP servers
clients
FTP server
ftp.site1.com.tw
Lookup the ftpaccess file
Configuration file of each virtual server
ftp.site2.com.tw …
ftp.site3.com.tw
# Virtual Server setup for ftp.site1.com.tw virtual ftp.site1.com.tw root /var/ftp/virtual/site1 virtual ftp.site1.com.tw banner /var/ftp/virtual/site1/banner.msg virtual ftp.site1.com.tw logfile /var/log/ftp/virtual/site1/xferlog
# Virtual Server setup for ftp.site2.com.tw …
A rule segment in ftpaccess
Chapter 6: Application Layer

100. On-the-Fly File Compressions
Server compress a file (or directory) right before retrieved by users
Example
User ynlin logged in.
Logged in to wwwpc.cis.nctu.edu.tw.
ncftp /home/ynlin > ls
1.tar.gz Desktop/ ucd-snmp-4.2.1/
ncftp /home/ynlin > get ucd-snmp tar.gz
ucd-snmp tar.gz: bytes kB/s
ncftp /home/ynlin >lls -l
drwxr-xr-x 24 gis88559 gis Oct 8 12:18 .
drwxr-xr-x 88 root gis Sep 10 17:48 ..
-rw-r gis88559 gis Oct 8 12:18 ucd-snmp tar.gz
Chapter 6: Application Layer

101. Important Configuration Files
File name
Description
ftpaccess
Used to configure the operations of the ftp daemon.
ftpconversions
Specify the postfix of a retrieved file and its corresponding operations.
ftphosts
Used to deny/allow some hosts to login as certain accounts.
ftpservers
List the virtual servers and the corresponding directories containing their own configuration files.
Chapter 6: Application Layer

102. 6.6 Simple Network Management Protocol
Background
Architectural framework
MIB
SMI
SNMP, SNMPv2, SNMPv3
Open source: NET-SNMP
Chapter 6: Application Layer

103. Chapter 6: Application Layer
Background
Control over network systems are demanded
Small tools: ping, traceroute, netstate..etc (base on ICMP)
SNMP:
Remote control without being physically attached to managed entities
Exchange management information between network devices
Short history
SNMPv1 (1989)– Management framework
SNMPv2 (1993)– Functionality enhancement
SNMPv3 (1998)– Security add-on
Chapter 6: Application Layer

104. Architectural Framework
Management station
Agent
Managed device
Managed object (specified in MIB)
Management protocols
management station
trap or response
SNMP
poll or request
agent & managed device in one machine
master agent
MIB
Protocols simpler than SNMP
subagent (managed device )
MIB
MIB
MIB
Chapter 6: Application Layer

105. Management Information Base (MIB)
Tree-like virtual information store
Identified by object identifiers
Extensible in “experimental” and “private” branches
Register at IANA (
MIB-II: for network management of TCP/IP-based internets (RFC 1213)
Chapter 6: Application Layer

106. Chapter 6: Application Layer
MIB-II
iso (1) ….
org (3) ….
dod (6) …. ….
internet (1)
snmpv2 (6)
security (5)
private (4)
experimental (3) ….
mgmt (2)
directory (1)
mib-2 (1)
snmp (11)
transmission (10)
egp (8)
udp (7)
tcp (6)
icmp (5)
ip (4)
at (3)
interface (2)
system (1)
OBJECT IDENTIFIER:
Chapter 6: Application Layer

107. Chapter 6: Application Layer
Object Groups in Mib-2
Group
Description
system
General information about the managed system
interface
Configuration information and statistics of each physical interface at
Address translation between network address and physical address ip
Information of implementation and operation of IP in a local system. For example, routing table, default TTL.
icmp
Information about the implementation and operation of ICMP. For example, number of messages ICMP sent and received.
tcp
Information about the implementation and operation of TCP. For example, the number of maximum and active connections in the system.
udp
Information about the implementation and operation of UDP. For example, the number of datagrams sent.
egp
Information about the implementation and operation of EGP.
tranmission
Related information and statistics of different transmission schemes.
snmp
Information about the accesses and errors of SNMP operations.
Chapter 6: Application Layer

108. Structure of Management Information (SMI)
Define the structure of a particular MIB
Place restrictions on the types of the objects allowed in the MIB
Abstract Syntax Notation one (ASN.1)
Exchange of information between application components in different systems
Syntax – integer, octet string, object identifier
Encode the object values (by Basic Encoding Rule)
Define the way data is represented during transmission
Three categories of data types
Simple: Primitive ASN.1 data types
Application-wide: Special data types used in particular Applications
Simply constructed: table, row
Chapter 6: Application Layer

109. TCP Connection Table (in MIB-II)
-- the TCP Connection table
-- The TCP connection table contains information about this
-- entity's existing TCP connections.
tcpConnTable OBJECT-TYPE
SYNTAX SEQUENCE OF TcpConnEntry
ACCESS not-accessible
STATUS mandatory
DESCRIPTION
"A table containing TCP connection-specific information."
::= { tcp 13 }
tcpConnEntry OBJECT-TYPE
SYNTAX TcpConnEntry
"Information about a particular current TCP connection. An object of this type is transient, in that it ceases to exist when (or soon after)
the connection makes the transition to the CLOSED state."
INDEX { tcpConnLocalAddress,
tcpConnLocalPort,
tcpConnRemAddress,
tcpConnRemPort }
::= { tcpConnTable 1 }
TcpConnEntry ::=
SEQUENCE {
tcpConnState INTEGER,
tcpConnLocalAddress IpAddress,
tcpConnLocalPort INTEGER ( ),
tcpConnRemAddress IpAddress,
tcpConnRemPort INTEGER ( ) }
tcpConnState OBJECT-TYPE
SYNTAX INTEGER {
closed(1), listen(2), synSent(3), synReceived(4)
established(5), finWait1(6), finWait2(7), closeWait(8),
lastAck(9), closing(10), timeWait(11), deleteTCB(12) }
ACCESS read-write
STATUS mandatory
DESCRIPTION
"The state of this TCP connection.."
::= { tcpConnEntry 1 }
tcpConnLocalAddress OBJECT-TYPE
SYNTAX IpAddress
ACCESS read-only
"The local IP address for this TCP connection. In the case of a connection in the listen state which is willing to accept connections for any IP interface associated with the node, the value is used."
::= { tcpConnEntry 2 }
tcpConnLocalPort OBJECT-TYPE
SYNTAX INTEGER ( )
"The local port number for this TCP connection."
::= { tcpConnEntry 3 }
tcpConnRemAddress OBJECT-TYPE
"The remote IP address for this TCP connection."
::= { tcpConnEntry 4 }
tcpConnRemPort OBJECT-TYPE
"The remote port number for this TCP connection."
::= { tcpConnEntry 5 }
Chapter 6: Application Layer

110. TCP Connection Table (tabular view)
tcpConnTable ( )
tcpConnEntry = (x.1)
tcpConnState
(x.1.1)
tcpConnLocalAddress
(x.1.2)
tcpConnLocalPort
(x.1.3)
tcpConnRemoteAddress
(x.1.4)
tcpConnRemotePort
(x.1.5)
Listen 23
161
closeWait
1029
established
3082
x.1
x.1
x.1
x.1
INDEX
Chapter 6: Application Layer

111. Basic Operations in SNMP
PDU
Descriptions
Version
GetRequest
Retrieve the value of a leaf object V1
GetNextRequest
Get the object lexicographically next to the one specified
SetRequest
Set (update) a leaf object with a value
GetResponse
Response for GetRequest (value) or SetRequest (ACK)
Trap
Issued by agent to notify the management station of
some significant event asynchronously
GetBulkRequest
Retrieve large blocks of data, such as multiple rows in a
table. V2
InformRequest
Allows one MS to send trap information to another MS
and receive a response
PDU: Basic data unit in SNMP operations
MS : Management Station
Variable-binding list: A list of variables and corresponding values in a PDU
Chapter 6: Application Layer

112. TCP Connection Table (lexicographical view)
tcpConnTable ( =x)
Traverse the tree using Depth First Search
tcpConnEntry = (x.1) …
tcpConnState
(x.1.1)
tcpConnLocalAddress
(x.1.2)
tcpConnLocalPort
(x.1.3) (x
.0.0.0) 23 (x
.0.0.0)
Listen (x
.0.0.0)
Listen (x
.0.0.0) (x
.0.0.0)
161 (x
.0.0.0)
closeWait (x ) (x )
161 (x )
established (x ) (x ) 23 (x )
Chapter 6: Application Layer

113. User-based Security Model (USM, RFC2574)
Four major threats
Modification of Information (between agent and MS)
Masquerade (pretend to be an authorized user)
Disclosure (eavesdropping)
Message Stream Modification (reorder, delay, replay)
Solutions
Timeliness module (for 1st, 3rd and 4th threats)
Authentication protocol—use MD5 (for the 2nd threat)
Privacy protocol—Use DES (for 3rd)
Chapter 6: Application Layer

114. View-based Access Control Model (VACM RFC 2575)
MIB view -- a collection of MIB objects
Elements in the model
Group– Categorization of managers
securityLevel– Help distinguish the access rights of a group
Context– A collection of managed object accessible by an snmp entity.
MIB view– A subset of a particular context
Access policy– Decide the final access rights of a group to a context
Chapter 6: Application Layer

115. An SNMP Entity and its Component (RFC 2571)
SNMP engine
Message Processing Subsystem
Access Control Subsystem
Security Subsystem
Dispatcher
Application(s)
Command Generator
Notification Receiver
Proxy Forwarder
Command Responder
Notification Originator
Other
Chapter 6: Application Layer

116. Comparisons Between Three SNMP Versions
Description and improvement
SNMPv1
1. Define the SMI (RFC1155)
2. A more concise MIB definition (RFC 1212)
3. SNMP framework and its related operations (RFC 1157)
4. Concept of security (authentication) is proposed
SNMPv2
1. Improved SMI (support 64-bit counter, and other types of address than IP)
2. Inclusion of ‘GetBulkRequest’ to improve the efficiency for retrieving large
blocks of data
3. ‘InformRequest’ for communication between management stations
SNMPv3
1. Security and administration add-on’s (RFC2571)
2. Multi-version SNMP message processing and dispatch capability (RFC 2572)
3. The five type of applications within an SNMP engine (RFC 2573)
4. User-based security model (RFC 2574)
5. View-based access control (RFC 2575)
Chapter 6: Application Layer

117. Chapter 6: Application Layer
Open Source: Net-SNMP
Introduction to Net-SNMP
Some commands for query, set and trap
Extensible architecture
Ways of extending the MIB
Subagent protocol– AgentX (RFC2741)
Build and include a private MIB
Inside Net-SNMP
Snmpd and snmptrapd
Chapter 6: Application Layer

118. Chapter 6: Application Layer
Introduction
Started at CMU, moved to UCDavis (1995) and is now based at SourceForge (2000~)
What it provides:
An extensible agent
SNMP library for further development
Tools to get or set information from SNMP agents
Tools to generate and handle SNMP traps
Support SNMP V1, V2 and V3
Runs on many Unix-like systems and Windows
Chapter 6: Application Layer

119. Some Commands for Query, Set and Trap
Name
Description and example
PDU used
SNMPGET
Retrieve the value of a leaf object using get
GetRequest
SNMPSET
Set (update) a leaf object with a value
SetRequest
SNMPBULKGET
Get multiple objects at a time. Possibly
under different subtrees
GetBulkRequest
SNMPWALK
Explore all the objects under a subtree
of the MIB
GetNextRequest
SNMPTRAP
Uses the TRAP Request to send information
to a network manager. More than one object
identifiers can be applied as arguments
Trap
SNMPSTATUS
Used to retrieve several important statistics
from a network entity. Errors will also be
reported, if any
SNMPNETSTAT
Displays the values of various network-
related information retrieved from a remote
system using the SNMP protocol
Chapter 6: Application Layer

120. Chapter 6: Application Layer
Command Line Examples
$ snmpget -v 3 -u ynlin -l authNoPriv -a MD5 -A ynlinsnmp localhost system.sysContact.0
system.sysContact.0 =
$ snmpset -v 3 -u ynlin -l authNoPriv -a MD5 -A ynlinsnmp localhost system.sysContact.0 s gis88559
system.sysContact.0 = gis88559
$ /usr/local/bin/snmpbulkwalk -v 3 -u ynlin -l authNoPriv –a MD5 -A ynlinpasswd localhost system
system.sysDescr.0 = Linux ynlin2.cis.nctu.edu.tw #5 SMP 週四 11月 22 23:6
system.sysObjectID.0 = OID: enterprises.ucdavis.ucdSnmpAgent.linux
system.sysUpTime.0 = Timeticks: ( ) 3 days, 12:28:34.50
system.sysContact.0 = gis88559
system.sysName.0 = ynlin2.cis.nctu.edu.tw
system.sysLocation.0 = ynlin2
system.sysORLastChange.0 = Timeticks: (0) 0:00:00.00
system.sysORTable.sysOREntry.sysORID.1 = OID: ifMIB
system.sysORTable.sysOREntry.sysORID.2 = OID: .iso.org.dod.internet.snmpV2.snmpB
system.sysORTable.sysOREntry.sysORID.3 = OID: tcpMIB
system.sysORTable.sysOREntry.sysORID.4 = OID: ip
system.sysORTable.sysOREntry.sysORID.5 = OID: udpMIB
Chapter 6: Application Layer

121. Chapter 6: Application Layer
Snmpd and Snmptrapd
Snmpd
Process snmp packets and respond with the required information
Can be either master agent or subagent
Basic access control with a port/addr pair
Load dynamical MIB module at start
Snmptrapd
Receive and log trap messages
Configured to launch an external program when receiving a trap matching a particular object identifier
Chapter 6: Application Layer

122. Processing flow inside Net-SNMP
Chapter 6: Application Layer

123. Data Structure of a Session
Chapter 6: Application Layer

124. Chapter 6: Application Layer
6.7 VoIP
Introduction
Elements in a VoIP environment
Underlying protocols
H.323
SIP
Challenges for VoIP technology
Open source: openphone
Chapter 6: Application Layer

125. Chapter 6: Application Layer
Introduction
Three ways of transferring voice
Circuit switched
Voice over frame relay
Voice over IP
Pros and cons of VoIP compared with PSTN
Inexpensive cost
Simplicity
Less bandwidth consumption
Extensibility
Chapter 6: Application Layer

126. VoIP Environment (H.323 adopted)
Gateway
Router
MCU
H.323 gatekeeper
Telephone
H.323 terminal
Packet switched network
Circuit switched network
Chapter 6: Application Layer

127. Chapter 6: Application Layer
H.323 Protocol Stack
Control
Data
Audio
Video
A / V
control
Control
(by GK)
G.711
G.723
G.729
H.263
RAS
(H.225.0)
Q.931
H.245
T.120
RTCP
RTP
TCP
UDP IP
Recommendation
Description
RAS(H.225.0)
Stands for Registration, Admission and Status.
H.245
Control protocol for capability negotiation and channel setup.
Q.931
Used for call setup and teardown between two terminals.
T.120
Data protocols for multimedia conferencing. (application sharing, whiteboarding)
RTP/RTCP
Used for real time traffic synchronization and transportation.
Chapter 6: Application Layer

128. Setup Procedure of an H.323 Call
Registration and admission
RAS
Call setup
Q.931
Terminal capability negotiation ,channel setup and master-slave detection
H.245
Stable call established and proceeds
RTP/RTCP
Close channel
H.245
Call teardown
Q.931
Disengagement
RAS
Chapter 6: Application Layer

129. Session Initialization Protocol (SIP)
Targeted to replace the H.323
Simplicity
Compatibility
Characterize a multimedia session
With supplementary SDP/SAP protocols
HTTP-like characteristics
Text-based protocol
Message syntax and header fields identical to HTTP/1.1
Client-server scheme
Chapter 6: Application Layer

130. Elements in an SIP Environment
Redirect server
Location server
Internet
User Agent Client (UAC)
Local
proxy server
Remote proxy server
User Agent Server (UAS)
Chapter 6: Application Layer

131. Chapter 6: Application Layer
Protocol Stacks of SIP
Control plane
Data plane
SIP
Multimedia Traffic
SAP/SDP
RTP
RTCP
TCP
UDP IP
Chapter 6: Application Layer

132. Operators and Reply Codes in SIP
Description
INVITE
Invite a user to a call
ACK
Confirmation for the final response
BYE
Terminate a call between endpoints
CANCEL
Terminate the search for a user or request for a call
OPTIONS
Features supported for a call
REGISTER
Register current location of the client with location server
INFO
Use for mid-session signaling
Reply Code
Description
1xx (Informational)
Trying, ringing and queued
2xx (Successful)
The request was successful
3xx (Redirection)
Give information about the receiver’s new location
4xx (Request Failures)
Failure responses from a particular server
5xx (Server Failures)
Failure responses given when a server itself has erred
6xx (Global Failures)
Busy, decline, requests not acceptable
Chapter 6: Application Layer

133. Example Operation in SIP
UAC INVITE UAS with the SIP URL
Destination IP address
known => Send the request directly to the destination
unknown => Redirected to the local proxy server (with location server).
3. In redirection mode, send back the callee’s new location
4. In proxy mode, forward the request to the destination.
The UAS answers 200 (OK), then the UAC ACKs
Session established
Chapter 6: Application Layer

134. Chapter 6: Application Layer
H.323 V.S. SIP
Simplicity in operation and implementation
Message encoding
Capability exchange
Supported data types
Capability of handling a conference
Chapter 6: Application Layer

135. Chapter 6: Application Layer
Open Source: Asterisk
An integrated PBX (Private Branch eXchange) system
Chapter 6: Application Layer

136. Chapter 6: Application Layer
Block Diagram
Asterisk provides a framework to build a customized VoIP system.
Flexible to add/remove modules to establish the basic transport service
Chapter 6: Application Layer
136
136

137. Chapter 6: Application Layer
Data Structures
Two concepts, context and extension, are introduced
Chapter 6: Application Layer
137
137

138. Chapter 6: Application Layer
Call flow
Chapter 6: Application Layer
138
138

139. Chapter 6: Application Layer

140. Chapter 6: Application Layer
6.8 Streaming
Introduction to streaming
Availability
Short fetching time
Avoid storage overhead
Live broadcasting
Architecture and components for streaming
Compression scheme
Protocol stack for streaming
QoS and synchronization mechanism
Open source: Darwin streaming server
Chapter 6: Application Layer

141. Architecture and Components for Streaming
Internet
Server
Client
Transport Protocols
Transport Protocols
QoS Control
Video/Audio raw data
Video Files
QoS Control
synchronization
Audio Files
Video Decoder
Audio Decoder
Chapter 6: Application Layer

142. Chapter 6: Application Layer
Compression Schemes
Temporal or spatial
Temporal – describe changes between frames
Spatial – describe similar patterns in a frame
Lossless or lossy
Recoverability when decompressed
GIF (lossless) and JPG (lossy)
Symmetrical or Asymmetrical
The time for compression and decompression
According to the load of the server
Chapter 6: Application Layer

143. Protocol Stacks of Streaming
RTSP (Real Time Streaming Protocol)
VCR-Style remote control functionalities
Establish and control streams
Description file before a session
Methods in a session (SETUP, PLAY, TEARDOWN)
Reuse of HTTP concepts (Ex: ASCII messages)
HTTP
Mostly by buffering
Low performance
Control plane
Data plane
HTTP
RTSP
RTCP
RTP
TCP
UDP IP
Chapter 6: Application Layer

144. QoS and Synchronization Mechanism
QoS mechanism
Source-based rate control
Receiver-based rate control
Synchronization mechanism
Intra-stream synchronization
Continuity of audio/video data frames inside a stream
Inter-stream synchronization
Consistency between cooperative streams
Inter-object synchronization
Consistency between streams and other objects like text and still images
Chapter 6: Application Layer

145. Open Source: Darwin Streaming Server
Open source version of Apple's QuickTime Streaming Server (QTSS)
Deliver streaming media with RTP and RTSP
Provide both live and on-demand programs
Supports many formats: H.264/MPEG-4 AVC, MPEG-4 Part 2, 3GP and MP3
Provide Web-based administration, authentication, relay support, and integrated broadcaster administration, etc.
Chapter 6: Application Layer

146. Chapter 6: Application Layer
Block Diagram
Two parts: core server and modules
Core server provides task scheduling
Module provides specific functions
Chapter 6: Application Layer
146
146

147. Chapter 6: Application Layer
Data Structures
Chapter 6: Application Layer
147
147

148. Call Flow: Task Handling
Chapter 6: Application Layer
148
148

149. RTSP Handling State Transition Diagram
Chapter 6: Application Layer
149
149

150. 6.9 Peer-to-Peer Applications (P2P)
Introduction
P2P Architectures
Performance Issues of P2P Applications
Case Study: BitTorrent
Open source: Vuze (BitTorrent Client)
Chapter 6: Application Layer
150
150

151. Chapter 6: Application Layer
Introduction to P2P
A distributed network architecture in which participants act as both a client and a server
Participants construct a virtual overlay network at the application layer on top of the underlying IP network
Node: peers
Edge: TCP/UDP connection
Chapter 6: Application Layer
151
151

152. Chapter 6: Application Layer
Introduction to P2P
Operations in P2P systems
Joining the P2P overlay network,
Resource discovery
The most challenging problem for P2P applications
Resource retrieval
Chapter 6: Application Layer

153. Chapter 6: Application Layer
P2P Architectures
Way of forming a P2P overlay network could be classified into three categories
Centralized
Decentralized and unstructured
Decentralized but structured
Chapter 6: Application Layer

154. Chapter 6: Application Layer
Centralized P2P
Operation overview
Chapter 6: Application Layer

155. Chapter 6: Application Layer
Centralized P2P
Utilizes a central directory server for locating objects in the P2P network
Peers join by registering to the directory server
Peers inform the directory server of objects to be shared
A peer sends query to the directory server to search an object
The peer receives reply from the directory server
The peer selects one or more peer in the reply to download the object directly from
Example: Napster
Chapter 6: Application Layer

156. Chapter 6: Application Layer
Centralized P2P
Advantages
Simple
Easy to implement
Support various kinds of search such as keyword, full-text, and metadata search
Disadvantages
Not a true P2P system as it relies on a central directory server which becomes performance bottleneck, single point of failure, not scalable and vulnerable to DoS attacks
Chapter 6: Application Layer

157. Decentralized and Unstructured P2P
To get rid of server, this approach floods query messages to peers to search for shared objects
Limited-scope flooding is adopted to reduce flooding messages
A query hit message is returned along the reverse path back to the inquirer
Example: Gnutella
Chapter 6: Application Layer

158. Decentralized and Unstructured P2P
Join procedure
A peer needs to know at least one of the peers already on the overlay network.
The peer sends a join message to a peer already on the overlay.
The existing peer then replies its identity as well as a list of its neighbors. It may also forward the join message to its neighbors.
Upon receiving join reply messages, the newcomer knows more peers on the overlay.
Chapter 6: Application Layer

159. Decentralized and Unstructured P2P
Hierarchical overlay with super peers
Flooding is apparently not scalable
FastTrack adopts a hierarchical overlay
A super peer acts as a local directory database which stores the indexes of objects shared by ordinary peers
Two-level hierarchical overlay
The lower level adopts the central
server approach
The upper level (super peers) adopts the
decentralized and unstructured approach.
Chapter 6: Application Layer

160. Decentralized and Unstructured P2P
Advantages
Fully distributed
Reliable, fault-tolerant
No single point of failure
Disadvantages
Excessive query traffic make it not scalable
May fail to find content that is actually in the system
Super peer may become overloaded or been attacked
Chapter 6: Application Layer

161. Decentralized but Structured
Combine the distributed directory service with an efficient query routing scheme
Key ideas
For distributed directory service, a hash function maps peers and objects into the same address space so that objects can be deterministically assigned to peers in a distributive manner.
For efficient query routing, peers are organized into a structured overlay based on their positions in the address space.
Chapter 6: Application Layer

162. Decentralized but Structured
Operations overview
Each peer generates its own node ID by a predefined hash function.
For each object being held and to be shared, the peer generates the object ID by the same or another hash function.
For each object, the peer sendd a register message to the node that has the node ID same as the object’s ID.
If a peer wants to query an object, it uses the hash function to generate the object ID and sends the query message to the node that hosts the object’s ID.
Chapter 6: Application Layer

163. Decentralized but Structured
Message routing (use Chord as an example)
Key idea: have each peer maintain a specially designed routing table such that every peer could forward the arriving message to a neighboring peer with node ID that is further closer to the destination.
Consider a 10-node Chord overlay in a 6-bit address space
Chord views its address space as a one-dimensional circular space such that peers in the space form a ring overlay.
Chapter 6: Application Layer

164. Message Routing in Chord
The routing table in Chord is called a finger table.
For an m-bit address space, the finger table of a node with ID=x consists of at most m entries and the i-th entry points to the first node with ID following the ID of x+2i-1 modulo 2m, for 1≤i≤m.
Chapter 6: Application Layer

165. Chapter 6: Application Layer
Finger Table of Chord
Finger table of node N8, where m =6.
Chapter 6: Application Layer

166. Routing a Query Message
Routing a query message for object 54 from N8
Chapter 6: Application Layer

167. Performance Issues of P2P Applications
Free Riding
Flash Crowd
Topology Awareness
NAT Traversal
Churn
Security
Copyright Infringement
Chapter 6: Application Layer
Copyright Reserved 2010
167
167

168. Chapter 6: Application Layer
Free Riding
Scalability of P2P systems relies on the contribution from peers
If a peer only consumes but contributes little or no resources, it becomes a free rider of the system.
85% of peers share no files in Gnutella in 2005
A common solution is to implement some incentive mechanisms.
For example, tit-for-tat in BitTorrent.
Other solutions: reward-based and credit-based mechanisms
Chapter 6: Application Layer

169. Chapter 6: Application Layer
Flash Crowd
A sudden, unanticipated growth in the demand of a particular object
e.g., a new release of a DVD video or mp3 file
Issues
how to deal with a sudden large amount of query messages
how long to find and download the object within a short time period
Solutions
Cache, duplicating popular objects
Chapter 6: Application Layer

170. Chapter 6: Application Layer
Topology Awareness
A virtual link in a P2P overlay could be a long end-to-end connection across continents or a short one within a local area network
How to avoid serious topology mismatch
Solutions
Many route-proximity and neighbor-proximity enhancements for P2P overlay systems have been proposed based on RTT measurement, preference of routing domain or ISP, or geographical information.
Chapter 6: Application Layer

171. Chapter 6: Application Layer
NAT Traversal
Basic requirement for a P2P system is to provide peers with NAT traversal mechanisms
If both peers are behind NAT devices, they cannot connect to each other without help from other peers or STUN servers
Solutions
In most cases, NAT traversal is solved by relay peers or super peers that have public IP addresses
Chapter 6: Application Layer

172. Chapter 6: Application Layer
Churn
Churn refers to the phenomenon that peers dynamically join and leave the system at will.
high churn rate seriously affects the stability and scalability of a P2P system.
e.g., a high churn rate may cause a tremendous overlay maintenance overhead and dramatic routing performance degradation in DHT-based system
Solutions
Avoid a rigid structure or relation among peers
Peers maintain a list of potential neighbors for quick and dynamic neighbor replacement
Chapter 6: Application Layer

173. Chapter 6: Application Layer
Security
Security issues
P2P programs with back hole (Trojan Horse), spurious content, leaking of files not to be shared.
Solutions to content pollution
protect the content with message digest such as MD5
In BitTorrent, the MD5 digest of each piece of a shared file is stored in the metadata file
peer reputation system
object reputation system
Chapter 6: Application Layer

174. Copyright Infringement
Sharing copyrighted objects through P2P systems is a serious problem which hinders the promotion and usage of P2P systems.
Not only P2P users are responsible for copyright infringement, so are the companies that host P2P applications
Especially in the case where P2P systems will not be able to exist without their serves, e.g., Napster.
Chapter 6: Application Layer

175. Case Study: BitTorrent
BitTorrent (BT) is originally designed by Bram Cohen in 2001
Well thought-out protocol with several unique features
use tit-for-tat as an incentive mechanism to cope with free riders
use out-of-band search to avoid copyright infringement issue
use pull-based swarming for load balancing
use hash check to prevent propagation of spurious pieces
after a peer has successfully downloaded a file, it becomes a seeder to distribute the file
Chapter 6: Application Layer
Copyright Reserved 2010
175
175

176. Chapter 6: Application Layer
BT Terminologies
A file is cut into pieces of a fixed size.
A piece is further divided into chunks, the basic data unit for a peer to request for content.
The integrity of a piece is protected by a SHA-1
A peer becomes a seeder if it has successfully downloaded the file.
A tracker for each file or group of files to be shared. The tracker tracks the downloading peers and seeds, and coordinates the file distribution among peers.
Chapter 6: Application Layer

177. Chapter 6: Application Layer
BT Operation Overview
Chapter 6: Application Layer

178. Chapter 6: Application Layer
Piece Selection
Random first piece selection
For the first few pieces, the client just randomly selects a piece to download.
Rarest first policy
Selects the most scarce piece to download first
End-game mode
To speed up the completion of a file download at the end, a peer with only a few pieces missing will send requests for all missing pieces to all the peers
Chapter 6: Application Layer

179. Chapter 6: Application Layer
Peer Selection
Choking/unchoking
Choking refers to a temporal refusal to upload to a peer.
At the beginning, all peers are chocked
Tit-for-tat algorithm selects a fixed number of peers from which the peer downloaded most to unchoke
Optimistic unchoking
new peer needs to move its first step when initially joined the system
select one peer at random
Anti-snubbing
If a peer is choked by all of its peers (snubbed), it is better to run optimistic unchoking more often to explore more peers that are willing to cooperate.
Chapter 6: Application Layer

180. Open source: Vuze (BitTorrent Client)
Many BT client programs are open-source
Some popular client programs
Vuze
uTorrent
BitComet
Chapter 6: Application Layer
Copyright Reserved 2010
180
180

181. Chapter 6: Application Layer
Core Packages
Most of Vuze’s core packages are located under the .\com\aelitis\azureus\core directory
Chapter 6: Application Layer

182. Chapter 6: Application Layer
Data Structure
Classes for the peer and piece objects
Chapter 6: Application Layer

183. Algorithm Implementation
Main program for controlling the piece and peer selection is the PEPeerControlImpl class
Chapter 6: Application Layer

184. Implementation of Peer and Piece Selection
Unchoking algorithms are implemented in DownloadingUnchocker.java and SeedingUnchocker.java
tit-for-tat is implemented in calculateUnchokes()
Optimistic unchoking is implemented in UnchokerUtil.getNextOptimisticPeer()
getRequestCandidate() defined in PiecePickerImpl.java is the core method for deciding which block to download
Chapter 6: Application Layer

185. Chapter 6: Application Layer
6.10 Summary
General issues
well-known ports, concurrent connection-oriented servers and iterative connectionless servers, statefulness/statelessness
Major application protocols
DNS, SMTP, POP3, IMAP4, HTTP, FTP, SNMP
Real time, multimedia application protocols
SIP, RTP, RTCP, RTSP
P2P protocols
Napster, Gnutella, DHT, Chord, BT
Chapter 6: Application Layer