0. Benchmarking Results from the Tycho Performance Tests
Matthew Grove DSG Seminar 27rd January 2006

1. Outline
Introduction.
Changes made to the Tycho design from last time (June 2005).
Example Tycho setup.
Tycho benchmark motivations and methodology.
Some performance results.
Summary.

2. Introduction
Tycho is a reference implementation of a combined extensible wide-area messaging framework with a built in distributed registry.
Tycho is deigned to reuse existing infrastructure where ever possible.
Tycho aims to simplify and speed application development by freeing developers from the need to use a combination of middleware to provide service discovery and messaging.

3. What’s new for Tycho
We have changed the name from jGMA to Tycho, this lets us focus on the presenting the strengths of Tycho, removing the dependency to explain our motivation in terms of GMA and the Grid.
The architecture has remained the same but we have completed the initial implementation.

4. Tycho’s Architecture

5. Example Tycho Setup

6. Implementation Changes
There are 24 items in the change log, the API has remained the same, highlights are:
The IRC VR-connector has been completed.
Extended the HTTP handler to provide VR boot strapping using a known web site.
Support for JDBC data-stores (like HSQLDB).
Added support for SQL keywords ‘LIKE’ and ‘LIMIT’.
Added caching for inter-VR responses.
Web wasn’t around until DNS was also done in 1988

7. Tycho Benchmarks We performed two rounds of benchmarking:
Message Passing: These tests measured the performance of inter-client and inter-mediator communications for Tycho and NaradaBrokering.
Virtual Registry: These tests measured the performance of the VR (query routing and processing) in a range of different configurations:
These results were compared to other tests which measured performance of Hawkeye, R-GMA and MDS2 and 4.

8. Message Passing Tests (A)
We measured the latency of communication for LAN and simulated WAN messaging between two clients varying message size (ping-pong tests).
Then we measured the affect on scalability of message latency using many producers publishing to a single consumer and single consumer consuming from multiple producers.
We used Starbug to run the tests.

9. Test A - Configuration

10. Latency vs Message Length
Ping-Pong - Latency
Latency vs Message Length

11. Bandwidth vs Message Length
Ping-Pong - Bandwidth
Bandwidth vs Message Length

12. Scalability Results - Producers
Bandwidth vs Message Length

13. Scalability Results - Consumers
Bandwidth vs Message Length

14. Message Passing Tests - Summary
Tycho achieves comparable or lower latency than NaradaBrokering in all of these tests.
The performance of both systems does not drop off after a producer or consumer has become saturated.
NaradaBrokering needs the JVM heap size to be increased as the number of clients increases (due to internal buffers):
Tycho used the default heap for all of the tests.

15. Virtual Registry Tests (B)
We used 4 different configurations to measure the performance of the Tycho VR.
For each test we varied one or more aspect of the configuration and measured the effect on client query latency:
Number of records in a mediator.
Number of simultaneous client queries.
Numbers of mediators in the VR.
Different VR-interconnects between mediators.
In order to make some comparisons between Tycho and some other systems we attempted to reproduce a test used in a related paper.

16. Configuration

17. Variables
Test Parameter Matrix; the components that are varied in each test are highlighted. S1 queries select a single record, S2 queries select every record.
The comparison tests with the other systems used the test 2 configuration.

18. Response Time vs Number of Records
Number of Records
Response Time vs Number of Records

19. Simultaneous Client Queries
Simultaneous Client Queries
Response Time vs Number of Clients

20. Response Time vs Number of Mediators
Number of Mediators
Response Time vs Number of Mediators

21. Response Time vs Number of Mediators
VR-Interconnects
Response Time vs Number of Mediators

22. Virtual Registry Tests - Summary
The Simple store performs best for small numbers of records, but HSQLDB and MySQL scale better with response size.
The response time increases linearly with number of clients.
For routing queries in the VR, IRC has better performance than HTTP.
As response size increases HTTP performs much better than IRC, this is due to the IRC code having to send multiple messages.

23. Comparison Tests
The tests measure the impact of increasing the number of clients simultaneously querying the registry.
We had to use 5 compute nodes instead of 8 to make the test as close a match as possible.
Each system has different sized responses which we matched by adding padding to Tycho registry information.
The main difference is CPU (2.8Ghz vs 1.1Ghz).

24. Response Time vs Number of Clients (2Kbyte Responses)
Comparison Tests (2K)
Response Time vs Number of Clients (2Kbyte Responses)

25. Response Time vs Number of Clients (10Kbyte Responses)
Comparison Tests (10K)
Response Time vs Number of Clients (10Kbyte Responses)

26. Response Time vs Number of Clients (19Kbyte Responses)
Comparison Tests (19K)
Response Time vs Number of Clients (19Kbyte Responses)

27. VR Comparison Tests – Summary
Tycho shows linear increase in response times regardless of message size and number of clients.
Tycho had lower response times than R-GMA, MDS4 and MDS2 without caching.
MDS2 with caching is basically OpenLDAP - which is mature and performs well.
Hawkeye performs better than Tycho for less than 100 clients, but it does not scale well with larger number of clients.

28. Future Work Some performance improvements:
Caching of local mediator queries to reduce response times.
The Simple store could be improved by making it more intelligent.
We could use a hybrid VR-interconnect to use IRC for query routing and HTTP for transporting large responses.
More functionality can be added to provide services that are more advanced:
HTTPS and SSL Sockets for secure comms.
Support for transactions.
Producers and consumers for computational steering.

29. Links Project Web page: The DSG Web page:
The DSG Web page: