1. Performance Evaluation of JXTA-* Communication Layers Mathieu Jan PARIS Research Group Paris, November 2004

2. 2 Outline Overview of JXTA communications layers Experimental setup and benchmarks used Evaluation of JXTA-J2SE Fast-Ethernet Myrinet Evaluation of JXTA-C Fast-Ethernet Myrinet

3. 3 Motivations: performance of JXTA? Papers from Emir Halepovic Pipe service only No in-depth explanations about the results No precise details about the experimental setup Mainly JXTA 1.x Complex topologies (relays, etc) Not clear what is benchmarked Results from the P3 project =~ XtremWeb over JXTA

4. 4

5. 5

6. 6 Project JuxTest A set of benchmarks for JXTA-J2SE 2.x Focuses of JXTA-J2SE Communications layers Uses JXTA Distributed Framework project (JDF) Co-written with David A. Noblet Undergraduate at the University of New Hampshire Source code soon available! (22/11) http://jdf.jxta.org/ Results already on-lineon-line

7. 7 JXTA-J2SE communications layers Endpoint service Pipe service JXTA Socket TCP, HTTP, etc

8. 8 JXTA-J2SE communication layers JXTA Socket Java Socket interface Data-stream interface Reliability Pipe service Pipe ID Unreliable Endpoint service Peer ID Unreliable

9. 9 Experimental setup Nodes 2.4GHz Intel P4, 1GB of RAM, linux 2.4.22 Networks Fast-Ethernet (MTU of 1500) Myrinet (GM 2.0.11, MTU of 9000) JVMs Sun 1.4.2_01-b06 (-server -Xms256M -Xmx256M) IBM 1.4.1 (-Xms256M -Xmx256M) JXTA-J2SE peers via JDF 1 rdv peer and 2 edge peers Benchmarks used RTT and protocol efficiency

10. Performance Evaluation of JXTA-J2SE on Fast-Ethernet

11. 11 Raw bandwidth results of JXTA Socket Java socket = 11.22 MB/s JXTA 2.3 = 9.72 MB/s JXTA 2.2.1 = 9.48 MB/s

12. 12 Raw latency results of JXTA Socket Java socket = < 0.1 ms JXTA 2.3 =~ 3 ms JXTA 2.2.1 =~ 4 ms

13. 13 Protocol efficiency of JXTA Socket 4 message elements in a JXTA message ACK_NUMBER Message payload + reliability EndpointRouterMsg (XML doc) Used by the endpoint router service 565 bytes! Not even needed! EndpointSourceAddress TCP address EndpointDestinationAddress TCP address + EndpointService (Net group) + Endpoint Router 1 byte = 909 bytes sent on the wires!

14. 14 Tunning the output buffer size of JXTA Socket output buffer size = size of a pipe message Java socket = 11.22 MB/s JXTA 2.2.1 (512KB) = 11.12 MB/s

15. 15 Raw bandwidth results of unicast pipe Java socket = 11.22 MB/s JXTA 2.3 = 9.59 MB/s JXTA 2.2.1 = 10.74 MB/s

16. 16 Raw latency result of unicast pipe Java socket = < 0.1 ms JXTA 2.3 =~ 2 ms JXTA 2.2.1 =~ 35 ms

17. 17 Protocol efficiency of unicast pipe 4 message elements in a JXTA message PAYLOAD EndpointRouterMsg (XML doc) EndpointSourceAddress EndpointDestinationAddress 1 byte = 877 bytes sent on the wires!

18. 18 JXTA Socket vs unicast pipe (bandwidth) JXTA Socket = 9.72 MB/s JXTA unicast pipe = 9.59 MB/s

19. 19 JXTA Socket vs unicast pipe (latency) JXTA Socket =~ 3 ms JXTA unicast pipe =~ 2 ms

20. 20 Removing limit on message size (unicast pipe) Java socket = 11.22 MB/s JXTA 2.2.1 = 11.14 MB/s

21. 21 IBM JVM vs Sun JVM on unicast pipe JXTA 2.3 IBM =~ 1.2 ms JXTA 2.3 SUN =~ 1.8 ms

22. 22 Raw bandwidth results of endpoint service Java socket = 11.22 MB/s JXTA 2.3 = 10.47 MB/s JXTA 2.2.1 = 11.01 MB/s

23. 23 Raw latency results of endpoint service Java socket = < 100 us JXTA 2.3 IBM =~ 480 us JXTA 2.3 SUN =~ 730 us Issue 1228: 2 TCP messages for 1 JXTA message!

24. 24 Protocol efficiency of endpoint service 3 message elements in a JXTA message PAYLOAD EndpointSourceAddress EndpointDestinationAddress 1 byte = only 300 bytes sent on the wires! If we can say only … No EndpointRouterMsg message element Removed 565 bytes! Removed XML parsing

25. 25 Cost of each layer (bandwidth)

26. 26 Cost of each layer (latency)

27. Performance Evaluation of JXTA-J2SE on Myrinet Or how to desperately reach the Gb/s!

28. 28 Experimental setup « Ethernet emulation » feature of GM No OS-bypass mode :-( Same benchmarks A different IP address when configuring JXTA Focus on bandwidth Removed limit on message size

29. 29 JXTA Socket bandwidth results Results are not very good! Java socket = 159.20 MB/s JXTA 2.3 = 14.70 MB/s JXTA 2.2.1 = 12.10 MB/s

30. 30 Tuning the output buffer size of JXTA Socket Above 80 MB/s

31. 31 Unicast pipe bandwidth results Above 1 Gb/s! :-) Java socket = 159.20 MB/s JXTA 2.3 = 106.54 MB/s JXTA 2.2.1 = 136.78 MB/s

32. 32 Endpoint service bandwidth results Java socket = 159.20 MB/s JXTA 2.3 = 112.32 MB/s JXTA 2.2.1 = 144.97 MB/s

33. Performance Evaluation of JXTA-C on Fast-Ethernet

34. 34 Short overview of JXTA-C Needs Apache Portable Runtime (APR) Threads, Network, etc Newly added support for TCP communications PeerCom, Seoul (South Korea) Does not support rdv peers Need to use a J2SE rdv peer No JXTA Socket layer Only pipe service and endpoint service Several bug fixes Mostly EndpointRouterMsg TCP_NODELAY on Several TCP messages

35. 35 Raw bandwidth results of unicast pipe Similar results compared to JXTA-J2SE JXTA-J2SE 2.3 = 11.1 MB/s JXTA-J2SE 2.2.1 = 11.1 MB/s JXTA-C = 11.1 MB/s

36. 36 Raw latency results of unicast pipe JXTA-J2SE 2.3 =~ 1.8 ms JXTA-J2SE 2.2.1 =~ 35 ms JXTA-C =~ 2 ms

37. 37 Protocol efficiency of unicast pipe 4 message elements in a JXTA message PAYLOAD EndpointRouterMsg (XML doc) EndpointSourceAddress EndpointDestinationAddress 1 byte = 834 bytes sent on the wires! JXTA-J2SE = 877 bytes + GatewayForward tag in EndpointRouter Waiting for Bernard Traversat answear - charset tag for each message element

38. 38 Raw bandwidth results of endpoint service JXTA-J2SE 2.3 = 11.15 MB/s JXTA-J2SE 2.2.1 = 11.19 MB/s JXTA-C = 11.16 MB/s

39. 39 Raw latency results of endpoint service JXTA-J2SE 2.3 =~ 0.8 ms JXTA-J2SE 2.2.1 =~ 1 ms JXTA-C =~ 0.8 ms

40. 40 Protocol efficiency of endpoint service 3 message elements in a JXTA message PAYLOAD EndpointSourceAddress EndpointDestinationAddress 1 byte = only 239 bytes sent on the wires! If we can say only … JXTA-J2SE = 300 bytes No EndpointRouterMsg Removing 595 bytes! Removed XML parsing

41. 41 Unicast pipe vs endpoint service

42. Performance Evaluation of JXTA-C on Myrinet Still using « Ethernet emulation » feature of GM

43. 43 Unicast pipe bandwidth results JXTA-J2SE 2.3 = 106 MB/s JXTA-J2SE 2.2.1 = 136 MB/s JXTA-C = 107 MB/s

44. 44 Endpoint service bandwidth results JXTA-J2SE 2.3 = 112.32 MB/s JXTA-J2SE 2.2.1 = 144.97 MB/s JXTA-C = 109 MB/s

45. 45 JXTA-C on top of PadicoTM :-) JXTA-C is working on top of PadicoTM! (17/11) « sed » modifications in apr Using marcel threads instead of pthread JXTA-C configured to use GDBM instead of SDBM Goal: stop using « ethernet emulation » of GM … But JXTA-C is not yet zero-copy :-( Performance evaluation in progress …

46. 46 Conclusion Performance evaluation of the communication layers of JXTA-* (J2SE and C) Fast-Ethernet Myrinet Much better results compared to other related work and explained results Known costs of JXTA JXTA-J2SE nearly saturate a Fast-Ethernet link JXTA-J2SE on Myrinet bandwidth greater than 1Gb/s JXTA-J2SE > JXTA-C on Myrinet JXTA-C is not zero-copy

47. 47 Ongoing work Performance evaluation of JXTA-C on top of PadicoTM JuxMem client in C for DIET Better knowledge of JXTA-C tricks ;-) Write papers GP2PC (1/12) on JXTA raw performance Fast-ethernet ? on JXTA tuned performance for grid computing Myrinet

48. 48 Future work Benchmark secure and propagate pipes Update JuxTest results JXTA 2.3.1 Unidirectional throughput test Plug-in for Ethereal An easier analysis of JXTA protocols