1. Horizontal Scaling and Geographic Distribution of OnSIP
John Riordan, CTO Junction Networks
August 9, 2011

2. Introduction to Junction Networks
Mike Oeth, CEO
Junction Networks was founded in 2004 on the premise that the future of business communications would rely heavily on IP Technologies.
Junction Networks is privately owned, with no outside investors.
John Riordan is CTO and lead of the Engineering Team at Junction Networks. 
This slide is for Mike to expand a bit on the JN background, vision, and John.
John first got involved with computers in the late 70′s. He studied computer science and cognitive science at MIT in the late 80′s. Since time, he’s been an Internet entrepreneur and engineer, building an Internet Service Provider in New York City, Interport Communications, before it was purchased by RCN in the late 1990's and then co-founding Junction Networks in 2004.
Horizontal Scaling and Geographic Distribution of OnSIP  Page 2

3. Abstract
OnSIP is a hosted SIP and XMMP service targeted at small and medium sized businesses. The hosted SIP platform presents itself as a single outbound proxy to end users who utilize standards based user agents to register one or more contacts for an AOR which they select. Horizontal scalability across an undefined number of geographically disparate data centers was one of the goals in the development of the current platform. Many open source projects such as OpenSIPS, FreeSWITCH and Asterisk are used in the implementation of the current platform. The "core" of the current SIP production implementation consists of a distributed set of dialog stateful SIP proxies based on OpenSIPS. State information in this implementation is maintained and propagated between proxies via integration with Cassandra, MySQL, and multicast event notifications. User agents are affiliated via Geo-DNS to a home proxy with which they register (Path headers). User agents can transition from one home proxy to another with no interruption of service. NAT issues are handled server side via message rewriting with the support of RTP proxies and keep-alives. To support sequential and future out of dialog requests, contacts which are not globally routable are mapped in initial requests and responses to GRUU's. The architecture and implementation of this distributed proxy core will be reviewed.
Horizontal Scaling and Geographic Distribution of OnSIP  Page 3

4. Introduction to OnSIP
Hosted SIP and XMPP service targeted at small and medium sized businesses.
OnSIP Platform
Marketed as Hosted PBX service
VM, ACD, AA, MOH, E911, Conferencing
PSTN IN/OUT
Integrated XMPP based IM and Presence
Web based user interfaces
XML API
XMPP API
Horizontal Scaling and Geographic Distribution of OnSIP  Page 4

5. Introduction to OnSIP
Hosted SIP and XMPP service targeted at small and medium sized businesses.
The hosted SIP platform presents itself as a single outbound proxy to end users who utilize user agents to register one or more contacts for an AOR that they select.
End users are assigned globally reachable SIP AORs (e.g.
Any standards compliant user agent may be utilized
Many open source projects such as OpenSIPS, FreeSWITCH and Asterisk are used in the implementation of the current platform.
Horizontal Scaling and Geographic Distribution of OnSIP  Page 5

6. Horizontal Scaling
OnSIP platform is built with horizontal scalability across an undefined number of geographically disparate data centers
Horizontal vs Vertical Scaling
Vertical scaling: buying bigger and bigger hardware to run the same software, thus giving you more resources to use.
Horizontal scaling: buying many of the same type of smaller hardware and distribute load.
General Issues
SIP-specific Issues
Geographic distribution
Out of dialog requests
Maintaining global state
NAT traversal
Horizontal Scaling and Geographic Distribution of OnSIP  Page 6

7. Geographic Distribution
New York
Core
Proxies
App
Application
Servers
Los Angeles
Internet
Miami
London
sip.onsip.com UA
Geographic Distribution
GeoDNS directs UAs to closest service location
UAs for same user can register with different locations
UAs can switch locations at anytime without disruption
Horizontal Scaling and Geographic Distribution of OnSIP  Page 7

8. Core Proxies Internet New York Los Angeles Read Write Read Only
OpenSIPS
Proxies
sip.onsip.com
Cassandra
Nodes
New York
Los Angeles
MySQL
Slaves
PubSubContact
Dialog
Events
Read Write
Read Only
Write Only
Proxy Local
Horizontal Scaling and Geographic Distribution of OnSIP  Page 8

9. Core Proxies Internet Read Write Read Only Read Write Write Only
sip.onsip.com
sip.onsip.com
OpenSIPS
Proxies
OpenSIPS
Proxies
Cassandra
Nodes
Cassandra
Nodes
Read Write
MySQL
Slaves
MySQL
Slaves
Read Only
MySQL
Proxy Local
MySQL
Proxy Local
Read Write
PubSubContact
Dialog
Events
PubSubContact
Dialog
Events
Write Only
New York
Los Angeles
Horizontal Scaling and Geographic Distribution of OnSIP  Page 9

10. OpenSIPS
sip.onsip.com
Outbound proxy for user agents
GeoDNS
Home proxy for inbound requests
NAT detection and handling
GRUU mapping of contacts
Dialog stateful
Forking
The "core" of the current SIP production implementation consists of a distributed set of dialog stateful SIP proxies based on OpenSIPS.
Horizontal Scaling and Geographic Distribution of OnSIP  Page 10

11. GeoDNS A record for sip.onsip.com
ISC Bind customization
Response varies based on source IP of request
IP2Location database
Based on kernel.org paper*
“GeoDNS is the idea of taking an incoming DNS request, doing the geographic look-up at the request time, and returning different results based on the incoming IP ad- dress.” – Hawley*
Hawley, John. GeoDNS—Geographically-aware, protocol-agnostic load balancing at the DNS level.
Linux Foundation:
Horizontal Scaling and Geographic Distribution of OnSIP  Page 11

12. NAT Handling Contact and SDP rewriting REGISTER
NAT info stored in location database
Path information is stored on registration
INVITE and other requests
NAT info carried in Route header
UAC behind a NAT
UAS behind a NAT
UAC and UAS behind same NAT
When a proxy receives an INVITE request for an AOR, it will do a lookup through the user location service provided by the registrar.
Horizontal Scaling and Geographic Distribution of OnSIP  Page 12

13. GRUU Mapping Out of dialog Request to Contact URI
Globally Routable UA URU (GRUU)
Out of dialog Request to Contact URI
i.e. REFER & INVITE w/Replaces
On Ingress of Request/Response
Contact URI is translated to a GRUU
GRUU hostport references proxy hostport
On Egress of Request
RURI GRUU translated Contact URI
Horizontal Scaling and Geographic Distribution of OnSIP  Page 13

14. Core Proxies Internet Read Write Read Only Read Write Write Only
sip.onsip.com
sip.onsip.com
OpenSIPS
Proxies
OpenSIPS
Proxies
Cassandra
Nodes
Cassandra
Nodes
Read Write
MySQL
Slaves
MySQL
Slaves
Read Only
MySQL
Proxy Local
MySQL
Proxy Local
Read Write
PubSubContact
Dialog
Events
PubSubContact
Dialog
Events
Write Only
New York
Los Angeles
Horizontal Scaling and Geographic Distribution of OnSIP  Page 14

15. Cassandra Open source column oriented DBMS Distributed key-value store
A highly scalable, second-generation distributed database
Open source column oriented DBMS
initially developed by Facebook
Distributed key-value store
Eventual Consistency
Google BigTable data model
Amazon Dynamo like infrastructure
Expiring Columns (TTL)
User location data
custom OpenSIPS db module
Path header
Registration structure
Cassandra
Nodes
Horizontal Scaling and Geographic Distribution of OnSIP  Page 15

16. CAP theorem Consistency Availability Partition tolerance
(can only have 2 out of 3 of the following)
Consistency
all nodes see the same data at the same time
Availability
a guarantee that every request receives a response about whether it was successful or failed
Partition tolerance
the system continues to operate despite arbitrary message loss
Horizontal Scaling and Geographic Distribution of OnSIP  Page 16

17. Core Proxies Internet Read Write Read Only Read Write Write Only
sip.onsip.com
sip.onsip.com
OpenSIPS
Proxies
OpenSIPS
Proxies
Cassandra
Nodes
Cassandra
Nodes
Read Write
MySQL
Slaves
MySQL
Slaves
Read Only
MySQL
Proxy Local
MySQL
Proxy Local
Read Write
PubSubContact
Dialog
Events
PubSubContact
Dialog
Events
Write Only
New York
Los Angeles
Horizontal Scaling and Geographic Distribution of OnSIP  Page 17

18. Slaved to master over Internet Addresses
MySQL Slaves
Slaved to master over Internet
ssh reverse tunneling
Addresses
CPL
Aliases
PBX Extensions
E.164 Numbers
Domains
MySQL
Slaves
Horizontal Scaling and Geographic Distribution of OnSIP  Page 18

19. Core Proxies Internet Read Write Read Only Read Write Write Only
sip.onsip.com
sip.onsip.com
OpenSIPS
Proxies
OpenSIPS
Proxies
Cassandra
Nodes
Cassandra
Nodes
Read Write
MySQL
Slaves
MySQL
Slaves
Read Only
MySQL
Proxy Local
MySQL
Proxy Local
Read Write
PubSubContact
Dialog
Events
PubSubContact
Dialog
Events
Write Only
New York
Los Angeles
Horizontal Scaling and Geographic Distribution of OnSIP  Page 19

20. MySQL Proxy Local Regular OpenSIPS database tables Dialog state
System local instance
Dialog state
dialog module
Location state
custom OpenSIPS db module
usrloc module
in support of NAT keep-alive
MySQL Proxy
Local
Horizontal Scaling and Geographic Distribution of OnSIP  Page 20

21. Core Proxies Internet Read Write Read Only Read Write Write Only
sip.onsip.com
sip.onsip.com
OpenSIPS
Proxies
OpenSIPS
Proxies
Cassandra
Nodes
Cassandra
Nodes
Read Write
MySQL
Slaves
MySQL
Slaves
Read Only
MySQL
Proxy Local
MySQL
Proxy Local
Read Write
PubSubContact
Dialog
Events
PubSubContact
Dialog
Events
Write Only
New York
Los Angeles
Horizontal Scaling and Geographic Distribution of OnSIP  Page 21

22. PubSub XMPP Publish-Subscribe (XEP-0060) User Agents Active Calls
Contact dialog events
XMPP Publish-Subscribe (XEP-0060)
User Agents
usrloc module events (customized module)
Active Calls
dialog/tm module events (customized module)
Application level events (voic )
Recent call history
PubSub
Horizontal Scaling and Geographic Distribution of OnSIP  Page 22

23. Thank You Contact developer@junctionnetworks.com OnSIP API
Visit onsip.com/api
Horizontal Scaling and Geographic Distribution of OnSIP  Page 23