one 2 one ERLANG USER CONFERENCE 3rd October 2000

Use of Erlang for IN Service Creation Sean Hinde IN Services Manager One 2 One

A Bit about Intelligent Networks IN provides a high level abstraction of the functionality of a Telephony Network by defining a set of building blocks which make up a call, and allowing them to be put together in new and interesting ways. It should not be possible to break the network by doing something wrong in an IN service! What is it?

A Bit About Intelligent Networks Architecture and Abbreviations SCP (Service Control Point) MSC/SSF (Mobile Switching Centre/Service Switching Function) C7 INAP Signalling SDP (Service Data Point)

A Bit about Intelligent Networks Put a new box in the network (SCP) and interface this into the existing switching network using a standard (ish) protocol. Provide a GUI based Service Creation Environment for the SCP. The SCP contains the real time data base and logic program which controls the SSF. Optional Service Data Point (SDP) - external database for holding large amounts of customer data How does it work?

Properties of IN and GSM systems Real Time Response - a few millisecs Massively High Availability % Enormous complexity - GSM! Great Flexibility - Service differentiation Is it impossible to achieve all these at once??

One 2 One SDP Implementation One 2 One started out with a home grown SDP implementation which existed before the introduction of IN into the network. Based on a commercial relational database/ big multiprocessor UNIX machine. The SCP interface and some service logic is home grown and coded in straight C. Uses a separate c7 to TCP/IP protocol converter IN Service fails safe if SDP times out

Experiences of own SDP One read failure led to others - multithread! Random explosions in the number of threads - file descriptors.. Commercial Failover never worked CPU load 100% - buying more processors didn’t help - spinlock problem. Still slows down - Full time DBAs to keep it running/optimised.

Experiences of own SDP - 2 Now have team of 20 IT contractors writing C code Conclusion from the telecoms designer perspective was that the experiment was not a great success. Hard and expensive to do well. What was the alternative though? Commercial SDPs not flexible enough SCP internal database not suitable/flexible enough Needed a Telecoms tool running on IT platforms with IT flexibility...

ERLANG Browsing Ericsson Web site - stumbled across a new platform - Open Telecom Platform.. It seemed to have many of the properties we were looking for, but what actually was it?? Downloaded and read more and more Some time approved to investigate further

ERLANG - The learning experience Started out as a non programmer (except for Ericsson SMAS IN SCE) and learned by reading the Erlang book part 1 PDF and the source. Three weeks to get multithreaded TCP/IP server running. Real confidence that code would carry on working. Bought commercial licence and got The Book.

ERLANG - Initial Conclusions The conclusion from the initial investigations was that there was minimal work required to turn Erlang/OTP into a high availability SDP with sophisticated logic. SCP TCP/IP Interface and service logic wrapper Alarm Interface - TCP/IP (now SNMP) Stats Interface - Files Provisioning - Same as SCP Event logging via Socket. Web based monitoring Service logic written in very few lines of easily testable and maintainable code

Current Applications Corporate VPN service Short Code dialling Black/White lists Short code presentation number Intranet based provisioning system (uses enhanced esp) 3 Nodes WAP RADIUS Whole customer base - 24 nodes Authentication with non standard logic mnesia table split into 64 fragments across 8 pairs

Soon to be in service Mini Prepayment system Checks usage at start of call and writes new total at the end Intranet Page to view customers with highest usage Uses several ordered set tables to achieve ordered queries 3 Nodes Number Translation Another with whole customer base - 22 nodes All voic calls will absolutely rely on it. mnesia table split into 64 fragments across 8 pairs

RADIUS Management Customer Details

RADIUS Management TCP/IP Client status

RADIUS Management Current Stats

RADIUS Management System Alarms

Operational Experience First 5 months of WAP system - several node failures, no missed calls. The day after sending the paper for publication - disaster. Major LAN problems affected Solaris/OTP so badly that heart or h/w watchdog kicked in on several nodes. Erlang went into cyclic restart on several nodes due to corrupt SNMP ets disk file preventing clean startup. LAN up and down - so many partitioned network entries we lost count. Failed to recover database consistency despite repeated attempts to set_master_nodes, mnesia deadlock Disk Failure on one machine meant we couldn’t load fallback until this was also fixed.

Lessons from Outage Plan to put a check to stop cyclic restarts. Need to apply an SNMP fix which was already available - and any others which may affect startup. More thought needed about how to fix partitioned networks. Proved that restore from fallback works faultlessly :-) 40 mins for 370M backup file.

Having Said That... Very few problems have been found with the whole system over two years of use. Wishes: Multipath distribution mechanism or something similar. Better memory handling - Occasional excessive memory usage with potentially serious consequences. Mnesia online table transform

Future Plans Plan to continue the use of Erlang/OTP in the Telecoms domain in One 2 One: Mass SMS Sending Tool More functionality for existing systems Tuxedo interface.. Possible migration of existing work into 3G architecture Other odd projects!

Conclusions Erlang/OTP has proved itself to be well up to the task of being an IN Service Data Point. The Erlang language has proved to be an ideal and safe environment for writing and maintaining IN service logic. It has also proved very quick and easy to write simple Internet protocol front ends to mnesia with no software faults found to date. Quote from Support Engineer: “We didn’t know much about how to fix it because it never goes wrong”...

one 2 one