Presentation is loading. Please wait.

Presentation is loading. Please wait.

HandlerSocket plugin for MySQL Jun 29, 2010 DeNA Technology Yoyogi IT Platform Dept., System Management Division DeNA Co.,Ltd. Akira Higuchi.

Similar presentations


Presentation on theme: "HandlerSocket plugin for MySQL Jun 29, 2010 DeNA Technology Yoyogi IT Platform Dept., System Management Division DeNA Co.,Ltd. Akira Higuchi."— Presentation transcript:

1 HandlerSocket plugin for MySQL Jun 29, 2010 DeNA Technology Yoyogi IT Platform Dept., System Management Division DeNA Co.,Ltd. Akira Higuchi Akira Higuchi

2 Who am I? Akira Higuchi, Ph.D. in science Akira Higuchi, Ph.D. in science IT Platform Dept., DeNA Co.,Ltd. IT Platform Dept., DeNA Co.,Ltd. system-wide performance optimization system-wide performance optimization middleware development middleware development The creator of HandlerSocket plugin The creator of HandlerSocket plugin Using GNU/Linux since 1993 Using GNU/Linux since 1993 Fedora: yum install KoboDeluxe Fedora: yum install KoboDeluxe Debian: apt-get install kobodeluxe Debian: apt-get install kobodeluxe

3 About HandlerSocket plugin

4 What is HandlerSocket? Non-SQL interface for MySQL Non-SQL interface for MySQL

5 What HandlerSocket aims Executes simple CRUD operations fast Executes simple CRUD operations fast Omit SQL parsing Omit SQL parsing Combine multiple requests on the server side Combine multiple requests on the server side Allows SQL on the same database Allows SQL on the same database Only simple operations can be faster Only simple operations can be faster Seamless migration from SQL queries Seamless migration from SQL queries

6 HandlerSocket plugin Offers a direct and non-SQL interface to MySQL storage engines Offers a direct and non-SQL interface to MySQL storage engines Own TCP/IP listener Own TCP/IP listener Talks a text protocol Talks a text protocol There is a C++ and a Perl client libraries There is a C++ and a Perl client libraries Only works with Linux Only works with Linux The source code is here: The source code is here: https://github.com/ahiguti/HandlerSocket-Plugin-for- MySQL https://github.com/ahiguti/HandlerSocket-Plugin-for- MySQL https://github.com/ahiguti/HandlerSocket-Plugin-for- MySQL https://github.com/ahiguti/HandlerSocket-Plugin-for- MySQL More infos on the DeNA Tech Blog More infos on the DeNA Tech Blog (in Japanese) (in Japanese)

7 Construction Handler Interface InnodbMyISAMOther storage engines … SQL LayerHandlersocket Plugin Listener for libmysql libmysqllibhsclient Applications mysqld client app

8 Other NoSQL interfaces to MySQL mycached mycached /08/mycached-memcac.html /08/mycached-memcac.html /08/mycached-memcac.html /08/mycached-memcac.html Works with any storage engines Works with any storage engines Talks the memcached protocol Talks the memcached protocol NDB API NDB API Dedicated for the ndbcluster engine Dedicated for the ndbcluster engine

9 Performance

10 Performance Handlersocket executes simple read queries 4x faster than mysqld/libmysql Handlersocket executes simple read queries 4x faster than mysqld/libmysql Very effective when many columns are retrieved Very effective when many columns are retrieved The reason is described later The reason is described later

11 Commands supported by HandlerSocket (for reading data) In pseudo-SQL... In pseudo-SQL... SELECT f1,.., fn FROM db.table WHERE k1,..., km = v1,..., vm WHERE k1,..., km = v1,..., vm ORDER BY index_i LIMIT offset, limit ORDER BY index_i LIMIT offset, limit (k1,..., km) are the key fields (or a prefix) of the index_i (k1,..., km) are the key fields (or a prefix) of the index_i =, >=, >, =, >, <=, and < can be used for a comparator

12 Commands supported by HandlerSocket (for modifying data) UPDATE, DELETE, and INSERT UPDATE, DELETE, and INSERT Does not support transactions Does not support transactions Modifications are recorded to the binary log in the row-based format Modifications are recorded to the binary log in the row-based format Modifications are durable Modifications are durable

13 Command example create table db1.table1 (k int key, v char(20)) create table db1.table1 (k int key, v char(20)) insert into db1.table1 values (234, 'foo'), (678, bar) insert into db1.table1 values (234, 'foo'), (678, bar) $ telnet localhost 9998 Trying Connected to localhost. Escape character is '^]'. P 0 db1 table1 PRIMARY k,v = foo 0 = bar opens the PK find k = 234 find k = 678

14 Why fast? No SQL parsing No SQL parsing low CPU usage Executes multiple requests in bulk Executes multiple requests in bulk low CPU/Disk usage Own client/server protocol Own client/server protocol small network transmission size

15 Eliminating CPU consumption

16 oprofile results – libmysql/mysqld Executes SELECT v from table where k = ? many times Executes SELECT v from table where k = ? many times samples| %| mysqld vmlinux libpthread-2.5.so libc-2.5.so e1000e oprofiled oprofile

17 oprofile results – libmysql/mysqld samples % symbol name MYSQLparse(void*) my_pthread_fastmutex_lock make_join_statistics(JOIN*, TABLE_LIST*, btr_search_guess_on_hash JOIN::optimize() row_search_for_mysql lex_one_token(void*, void*) alloc_root read_view_open_now mysql_select(THD*, Item***, TABLE_LIST*, open_table(THD*, TABLE_LIST*, st_mem_root*, mem_pool_fill_free_list build_template(row_prebuilt_struct*, THD*, get_hash_symbol(char const*, unsigned int, CPU usage inside mysqld CPU usage inside mysqld

18 oprofile results – libmysql/mysqld samples % symbol name schedule tcp_sendmsg tcp_recvmsg tcp_v4_rcv tcp_ack system_call dev_queue_xmit __mod_timer tcp_rcv_established text.task_rq_lock CPU usage inside the Linux kernel CPU usage inside the Linux kernel

19 oprofile results – libmysql/mysqld libmysql/mysqld libmysql/mysqld Much CPU time spent in mysqld Much CPU time spent in mysqld Parsing SQL is slow Parsing SQL is slow schedule() is called frequently schedule() is called frequently

20 oprofile results – HandlerSocket samples| %| vmlinux mysqld libpthread-2.5.so e1000e handlersocket.so libc-2.5.so oprofiled oprofile CPU usage inside MySQL with HandlerSocket CPU usage inside MySQL with HandlerSocket

21 oprofile results – HandlerSocket samples % symbol name btr_search_guess_on_hash row_search_for_mysql mutex_delay my_pthread_fastmutex_lock buf_page_get_known_nowait rec_get_offsets_func build_template(row_prebuilt_struct*, THD*, TABLE*, row_sel_store_mysql_rec btr_cur_search_to_nth_level row_sel_convert_mysql_key_to_innobase cmp_dtuple_rec_with_match ha_innobase::index_read(unsigned char*, dict_index_copy_types mtr_memo_slot_release ha_innobase::init_table_handle_for_HANDLER() CPU consumption in mysqld CPU consumption in mysqld

22 oprofile results – HandlerSocket samples % symbol name tcp_sendmsg tcp_v4_rcv dev_queue_xmit text.skb_release_data __qdisc_run tcp_recvmsg ip_output text.skb_release_head_state tcp_ack __alloc_skb ip_queue_xmit tcp_transmit_skb CPU consumption in the Linux kernel CPU consumption in the Linux kernel

23 oprofile results – HandlerSocket HandlerSocket HandlerSocket Most CPU time is consumed in the kernel Most CPU time is consumed in the kernel schedule() is not called frequently schedule() is not called frequently Inside mysqld, innodb eats most CPU time Inside mysqld, innodb eats most CPU time

24 Executing multiple requests in bulk

25 Threading model mysqld: Thread per connection (MySQL 5) Thread per connection (MySQL 5) Thread pooling (MySQL 6?) Thread pooling (MySQL 6?)

26 Threading model HandlerSocket: Small number of threads Small number of threads Many connections per thread Many connections per thread Uses epoll() Uses epoll() Virtually unlimited number of concurrent connections Virtually unlimited number of concurrent connections Small memory footprint Small memory footprint

27 HandlerSocket reader thread reads requests from many clients locks the DB, gets a read view executes many requests unlocks the DB returns responses to clients locks/unlocks (1/#conns) times per request handlersocket reader thread

28 HandlerSocket writer thread reads requests from many clients locks the DB, begins a transaction executes multiple requests commits, and unlocks the DB returns responses to clients handlersocket writer thread executes multiple ops in a single transaction

29 Write throughput Condition: Condition: Durable write Durable write sync_binlog = 1 sync_binlog = 1 innodb_flush_log_at_trx_commit = 1 innodb_flush_log_at_trx_commit = 1 innodb_support_xa = 1 innodb_support_xa = 1 Write-back cache with BBU, or SSD Write-back cache with BBU, or SSD Throughput: Throughput: MySQL: up to 1000 qps MySQL: up to 1000 qps HandlerSocket: up to qps HandlerSocket: up to qps

30 How HandlerSocket locks tables MyISAM: MyISAM: Shared-exclusive lock Shared-exclusive lock InnoDB: InnoDB: Reader threads dont block Reader threads dont block Only one writer thread can be executed at the same time Only one writer thread can be executed at the same time HandlerSocket requests are deadlock-free HandlerSocket requests are deadlock-free Only simple operations are supported Only simple operations are supported

31 Client/server protocol

32 MySQL C/S protocol write(3, "L\0\0\0\3select column0,column1,column2,column3,column4 from db_1.table_1 where k=15", 80) = 80 read(3, "\1\0\0\1\0056\0\0\2\3def\4db_1\7table_1\7table_1\7column0\7column0\f\r\ 0<\0\0\0\375\200\0\0\0\0006\0\0\3\3def\4db_1\7table_1\7table_1\7column 1\7column1\f\r\0<\0\0\0\375\200\0\0\0\0006\0\0\4\3def\4db_1\7table_1\7t able_1\7column2\7column2\f\r\0<\0\0\0\375\200\0\0\0\0006\0\0\5\3def\4d b_1\7table_1\7table_1\7column3\7column3\f\r\0<\0\0\0\375\200\0\0\0\0006 \0\0\6\3def\4db_1\7table_1\7table_1\7column4\7column4\f\r\0<\0\0\0\375\ 200\0\0\0\0\5\0\0\7\376\0\0\"\0\n\0\0\10\0010\0011\0012\0013\0014\5\ 0\0\t\376\0\0\"\0", 16384) = 327 when the above query is executed... SELECT column0, column1, column2, column3, column4 FROM db_1.table_1 where k = 15

33 HandlerSocket C/S protocol write(3, "1\t=\t1\t15\n", 9) = 9 read(3, "0\t5\t0\t1\t2\t3\t4\n", 8192) = 14 when an equivalent query is executed using handlersocket... libmysqlhandlersocket request 80 bytes 9 bytes response 327 bytes 14 bytes

34 MySQL C/S protocol The strace result shows that MySQL C/S protocol is verbose The strace result shows that MySQL C/S protocol is verbose Result-set metadata Result-set metadata r_Protocol#Field_Packet r_Protocol#Field_Packet Result-set metadata become very large if a result-set has many columns Result-set metadata become very large if a result-set has many columns Neither a HANDLER statement nor a server- side prepared statement does not help to avoid this problem Neither a HANDLER statement nor a server- side prepared statement does not help to avoid this problem

35 Client libraries

36 libhsclient Client library for C++ Client library for C++

37 Net::HandlerSocket Client library for Perl Client library for Perl Invokes libhsclient via XS Invokes libhsclient via XS my $cli = new Net::HandlerSocket( {host => localhost, port => 9999}); $cli->open_index(1, db1, table1, PRIMARY, k,v); my $res = $cli->exec_multi([ [ 1, =, [ 33 ], 1, 0 ], [ 1, =, [ 44 ], 1, 0, U, [ 44, hoge ] ], [ 1, >=, [ 55 ], 10, 20 ], ]);

38 Configuration hints

39 HandlerSocket configuration options handlersocket_threads = 16 handlersocket_threads = 16 Number of reader threads Number of reader threads Recommended value is the number of logical CPU Recommended value is the number of logical CPU handlersocket_thread_wr = 1 handlersocket_thread_wr = 1 Number of writer threads Number of writer threads Recommended value is... 1 Recommended value is... 1 handlersocket_port = 9998 handlersocket_port = 9998 Listening port for reader requests Listening port for reader requests handlersocket_port_wr = 9999 handlersocket_port_wr = 9999 Listening port for writer requests Listening port for writer requests

40 Other configuration options innodb_buffer_pool_size innodb_buffer_pool_size As large as possible As large as possible innodb_log_file_size, innodb_log_files_in_group innodb_log_file_size, innodb_log_files_in_group As large as possible As large as possible innodb_thread_concurrency = 0 innodb_thread_concurrency = 0 open_files_limit = open_files_limit = Number of file descriptors mysqld can open Number of file descriptors mysqld can open HandlerSocket can handle up to concurrent connections HandlerSocket can handle up to concurrent connections

41 Other configuration options innodb_adaptive_hash_index = 1 innodb_adaptive_hash_index = 1 Adaptive has index is fast, but consume memory Adaptive has index is fast, but consume memory

42 Options related to durability sync_binlog = 1 sync_binlog = 1 innodb_flush_log_at_trx_commit = 1 innodb_flush_log_at_trx_commit = 1 innodb_support_xa = 1 innodb_support_xa = 1

43 Benchmark results

44 Benchmark Server: Server: Core2Quad Q6600 Core2Quad Q6600 CentOS 5.4 CentOS 5.4 Single EXPI9301CT(e1000e) Single EXPI9301CT(e1000e) Single Intel X25-E (write-back cache disabled) Single Intel X25-E (write-back cache disabled) Schema: Schema: CREATE TABLE table1 (k varchar(32) KEY, v varchar(32)) engine = INNODB; CREATE TABLE table1 (k varchar(32) KEY, v varchar(32)) engine = INNODB; Read benchmark: Read benchmark: records records SELECT v from table1 where k = ? SELECT v from table1 where k = ? Random access Random access Write benchmark: Write benchmark: records records UPDATE table SET v = ? where k = ? UPDATE table SET v = ? where k = ? Random access Random access Durable write Durable write sync_binlog = 1 sync_binlog = 1 innodb_flush_log_at_trx_commit = 1 innodb_flush_log_at_trx_commit = 1 innodb_support_xa = 1 innodb_support_xa = 1

45 Throughput (reads)

46 Throughput (writes)

47 Maximum response time

48 Average response time

49 Issues and future plans

50 Issues Difficult to build Difficult to build Requires the source of mysql Requires the source of mysql MySQL binary compatibility? MySQL binary compatibility?

51 Future plans where clause where clause Atomic read-modify-write operations Atomic read-modify-write operations SQL support? SQL support? More language bindings More language bindings


Download ppt "HandlerSocket plugin for MySQL Jun 29, 2010 DeNA Technology Yoyogi IT Platform Dept., System Management Division DeNA Co.,Ltd. Akira Higuchi."

Similar presentations


Ads by Google