1. TCB 9 RK/MSFT Benchmark Results
X-IO rebranded slides as published by RedKnee document TCB_9 x_RK_MSFT_Benchmark_Report_v1.0.
X-IO rebranded slides as published by RedKnee document TCB_9 x_RK_MSFT_Benchmark_Report_v1.0.
X-IO Confidential – do not distribute

2. Executive Summary Achievements Scope Throughputs
Successfully demonstrated the scalability and performance of Redknee TCB 9.1 (Turnkey Converged Billing System), running on Microsoft Windows 2008 R2 Enterprise Edition and SQL Server one billing system to support 250 million subscribers.
Much less HW was used compared to the original estimation*.
Scope
Five subscriber steps (1M/10M/50M/100M/250M, 90/10 prepaid/postpaid split) and two key billing scenarios (Invoicing and Mediation) were tested, linear scalability is achieved and performance/throughput objectives are exceeded.
Throughputs
Finished 26 million subscriber invoices in a 6 hours invoicing window, on average 1,250 invoices were generated per second. This is enough for invoicing for all postpaid subscribers in one bill run. Multiple bill runs can be configured to support higher percentage of postpaid subscribers. For example, with four bill runs it is able to generate 100M invoices.
Processed 4.9 billion CDRs within 12 hours , on average 113,400 CDR were processed per second, this is enough to process CDR’s generated by 250M subscribers in real-time.
Note*: Refer to page 7 for more details.

3. Technologies Highlights
Applications
Most recently released Redknee TCB 9.1, a complete real-time billing solution that manages any payment type, any service, and any technology, empowered with most advanced Database, Hardware and other technologies.
Database
SQL server 2012 (“Denali”) - the upcoming latest and greatest database platform from Microsoft.
SQL compression is enabled to:
Significantly reduce storage space up to 80%.
Significantly reduce physical IO.
70% database license cost reduction* compared with Oracle.
Server
Application Tier is built by entry level severs that can linearly scale with subscriber and traffic growth.
DB Tier is built by a single NEC Express5800, leveraging the latest generation of Intel® Xeon® processor E7 series.
Storage
All database tables and logs are stored on the Hyper ISE arrays from XIO, a true SHD storage system that fuses together SSD and HDD into a single pool of capacity.
Note*: Microsoft SQL Server 2012 (Denali) core based licensing model and Oracle 10gR2 core based licensing model is used in cost calculations.

4. Testing Methodologies (1)
Test Steps
Five subscriber steps (1M/10M/50M/100M/250M, 90/10 prepaid/postpaid split). The first step (1M subs) focused on sanity/functional aspect, whereas the rest focused on performance/scalability.
Test cases
Mediation: Pre-generated CDR files are polled and processed by BSS Mediation module and inserted in to SQL server database. Total throughput (CDR/sec) across all mediation instances are measured as the result.
Invoicing: Postpaid account invoices in xml format are generated with all monthly CDR’s associated with the test subscribers in SQL Server database. Total throughput (invoices/sec) across all invoice server instances are measured as the result.
Major Obstacles and Solutions
During the index building on “calldetail” tables in the 10M run, we experienced much slower than expected performance, with help from Peter on tuning SQL server max degree of parallelism, index building time was significantly reduced.
During the 10M mediation run, a hot spot (page) in SQL server was found and this prevented the database to go faster. With re-design on clustered index and the pre-generated CDR files, we successfully removed the hot spot in SQL server with closer-to-reality CDR file pattern.
From the 10M run result analysis, it was highlighted that total available storage space would not be sufficient for 100M/250M run, consulted with Microsoft and enabled SQL server compression in the 50M run, storage space utilization was reduced by 50%.

5. Testing Methodologies (2)
Major Obstacles and Solutions (Continued…)
During the 50M run, Mediation performance was identified to be the potential bottleneck of the benchmark testing, a few code changes and configuration changes in applications were done and successfully drive the throughput up by three times.
During the 50M run, a bottleneck was found in SQL server/IOPS due to high volume of singleton commits used by mediation, with code change to batched commits, we were able to reduce the IO on transaction logs by 10 times which results in 10 times throughput improvements with the same IOPS.
During the 100M run, a network bandwidth bottleneck was identified, upgrading from 1GbE to 10GbE resolved the issue.
From the 100M run result analysis, CPU bottleneck was predicted for the 250M run, consulted with Microsoft and enabled hyper-threading, a 25-30% performance gain was measured.
During the 100M run, we observed a discrepancy between the total DB size shown in SQL server and the total files size in file system. SQL server does not automatically shrink the files to the size which is seen in the SQL server.. By changing from 30 file groups to one big file group and 7 files, SQL server can use the storage space more efficiently (actual storage space reduced another 60%) without any performance degradation.
During the 250M run, a network bottleneck was observed (again), for some reason, it appeared one 10GbE NIC can not go beyond 300Mbps, by adding a second 10GbE and load-share the network traffic across the two NICs we were able to overcome the bandwidth limitation.
During the 250M run, we observed the load is not evenly distributed across all cores, which had a negative impact on the performance. With help from Peter, we successfully tuned the configurations in NIC and SQL server to distribute load evenly to all cores.

6. Test Cases and Objectives
 SCENARIO
# of Subscribers
10,000,000
50,000,000
100,000,000
250,000,000
% of Postpaid *
10%
# of Postpaid subs *
1,000,000
5,000,000
25,000,000
# of CDR/sub per busy hour
1.5
 MEDIATION
Busy hour to daily CDR ratio (%)
Processed CDR/day
150,000,000
750,000,000
1,500,000,000
3,750,000,000
Processed CDR/hour
15,000,000
75,000,000
375,000,000
Processed CDR/second
4,167
20,833
41,667
104,167
# of CDR records per month
3,000,000,000
15,000,000,000
30,000,000,000
75,000,000,000
 INVOICING
# of subscribers per account 1
# of postpaid account
# of Bill runs per Month *
# of hours per invoice window 6
invoices/bill run
invoices/hour
166,667
833,333
1,666,667
4,166,667
invoices/sec
46.3
231
463
1,157
Note*: Multiple bill runs can be configured to support higher postpaid penetration.

7. Test Results Overview 10M 50M 100M 250M Mediation Invoicing
# of CDR processed per hour
15,480,000
97,200,000
187,200,000
408,247,200
# of CDR processed per second
4,300
27,000
52,000
113,402
% of Objectives
103%
129%
124%
108%
# of App Servers 2 4 7 15
% CPU on SQL Server 8%
28%
30%*
90% *
Invoicing
# of invoices generated per hour
374,400
1,152,000
1,944,000
4,500,000
# of invoices generated per second
104
320
540
1,250
224%
138%
116% 1 5 6 13 5%
15%
7%*
11%*
Note*: Intel Hyper-Threading technology enabled in 100M and 250M test runs.

8. Linear Scalability on Application Tier
Note*: Mediation and Invoicing instances can share the same physical server, where billing runs (invoicing) are scheduled during low traffic hours.

9. Cost Comparison Estimated Cost * Actual Cost * App Tier Server 2.1M
Estimated Cost *
Actual Cost *
App Tier
Server
 2.1M
 450K
DB Tier
 800K
 200K
Storage 2M
700K **
DB License
 3.1M
 142K
TOTAL
 8M
1.5M 
Note:
1. Estimate cost is based on initial HW sizing estimation using traditional HW and Oracle database, actual cost is based on the actual HW and SW. Both estimated cost and actual cost are budgetary.
- Estimated cost: 70 app 30K each; 6 DB 130K each; 1 XP24000, 192 Oracle DB licenses 16.4K each
- Actual Cost: 15 app 30K each; 1 DB 200K; 4 XIO 175K each; 20 SQL server DB licenses (EE) @ 7.1K each
2. Hyper ISE / ISE SAN cost reviewed by X-IO

10. Optimizations and Tunings
Application servers, DB server and Storage are all tuned to the max capacity (>90% utilization).
Application Optimization
Mediation
Code change and configuration change to reduce 3 redundant queries.  (50M run)
Configuration change to disable xdb logging. (50M run)
Code change to enable multiple line commit. (100M run)
Increase number of thread from 40 to 60 (250M run)
Invoicing
Code change to home cache price plan and price plan version. (10M run)
Number of thread increase to 40. (100M run)
Disabled audit journal. (250M run)
Change the data pattern to evenly distribute data across 30 days. (250MM run)
Database Tuning
Tune “max degree of parallelism” to reduce bulk-loading time and enable data compression to save storage space (50M run)
Change clustered index on table “calldetail” to columns (“subscriberid”, “posteddate”) ( 10M run)
Merge daily file groups to one file group. (250M run)
Move tables “invoice”, “subscriberinvoice”, “invoicedelivery” from default file group to file group “FG_calldetail” (250M run)
Increased number of tempdb files to 160. (250M run)
Storage Tuning
Windows Server – disable Windows write-caching for optimal performance of Windows Server with XIO storage (1M run)
Emulex HBA – set max queue depth to maximum (from 32 to 254) on all ports (1M run)
Added one Hyper ISE (250M run).
Other
Enable CPU Hyper-threading (100M run)
Upgrade to 10GbE end to end (100M run)
Disable CPU power saving to achieve the maximum CPU performance (250M run)
NIC Configuration optimizations to evenly load share network traffic across all CPUs (250M run)

11. Test Environment

12. Test Environment Overview
 HW Configuration
SW Configuration
Application Tier
15 x Intel x86 entry level servers
4 x Quad-core Intel Xeon 7300 processor
128GB RAM, 2x 146GB HDD
10GbE NIC
Windows 2008 R2 EE
TCB 9.1 (BSS, Mediation, IS)
Database Tier
1 x Intel x86 high-end server
8 x 10-core Intel Xeon E processor
1TB RAM, 2x 146GB HDD
SQL Server 2012
Storage
3 x XIO Hyper ISE SAN for DB
1 x XIO ISE SAN for SQL logs
XIO firmware version 2.2.0
Network
10GbE End-to-End
N/A

13. Application and DB setup

14. DB Partitioning Calldetail (CDR) tables are partitioned by date
One file group, ten files, each file maps to a LUN on the Hyper-ISE to evenly loadshare IO across all LUNs.

15. Backup Slides

16. Detailed Test Results: 250M Subscribers

17. Invoice Generation Throughput (250M Subs)

18. App Server CPU Utilization During Invoicing

19. SQL Server CPU Utilization During Invoicing

20. SQL Server IO Performance During Invoicing

21. SQL Server Network Throughput During Invoicing

22. Mediation Throughput (250M Subs)

23. App Server CPU Utilization During Mediation

24. SQL Server CPU Utilization During Mediation

25. SQL Server IO Performance During Mediation

26. SQL Server Network Throughput During Mediation

27. Thank you!