1. MySQL Cluster Tutorial Andrew Hutchings Andrew Morgan Geert Vanderkelen1

2. MySQL Cluster Tutorial Introduction
Geert Vanderkelen
MySQL Support Engineer 2

3. Virtual Machine Setup Open the DVD
Install VirtualBox: see software/VirtualBox
Copy clustervm/ and config/ folder to your hard drive
VirtualBox: File > Import Appliance
In Wizard, locate file Ubuntu 9.10.ovf
.. wait till done 3

4. Agenda Installation and Configuration Running Nodes & Your First Table
Administer MySQL Cluster 4

5. Agenda
MySQL Cluster Manager
Fault Tolerance
Online Backup 5

6. Agenda
NDB Info
NDB API & Connector for Java
Schema Considerations 6

7. Agenda
Scaling & Performance
Geographical Replication
Security 7

8. Speakers
Andrew Hutchings
Andrew Morgan
Geert Vanderkelen 8

9. MySQL Cluster In Short.. Helps you reach High Availability
using a Shared-Nothing infrastructure

10. MySQL Cluster In Short.. Components are processes which
are called Nodes
and Share-Nothing

11. MySQL Cluster In Short..
Data Nodes
storage of data, index info, etc..

12. = MySQL Cluster In Short.. 2 Data Nodes, means data partitioned in 2
NoOfReplicas=2
Id=3
Id=4
Primary
Primary =
Secondary
Secondary

13. MySQL Cluster In Short.. SQL Nodes, the MySQL Server
using SQL to retrieve and update data from the Data Nodes through TCP/IP

14. MySQL Cluster In Short.. API Nodes applications written using NDB API
(SQL Nodes are an example)

15. MySQL Cluster In Short.. Management Nodes
managing and monitoring MySQL Cluster

16. Virtual Machine Setup Open the DVD
Install VirtualBox: see software/VirtualBox
Copy clustervm/ and config/ folder to your hard drive
VirtualBox: File > Import Appliance
In Wizard, locate file Ubuntu 9.10.ovf
.. wait till done 16

17. MySQL Cluster Tutorial Installation & Configuration
Geert Vanderkelen
MySQL Support Engineer 17

18. MySQL Cluster Versioning
.. full version
6.3.33
mysql ndb
7.0.14
mysql ndb
7.1.2a
mysql ndb-7.1.2beta
mysql-cluster-gpl-7.1.2a-beta-linux-i686-glibc23.tar.gz

19. Download
MySQL Cluster still GPL, get it from dev.mysql.com

20. Installation Just like the regular MySQL Server
shell> cd /usr/local
 shell> tar xzf mysql-cluster-gpl- <version>.tar.gz
 shell> ln -s mysql-cluster-gpl-<version> mysql

21. Tools Regular MySQL tools, and.. ndbd (ndbmtd) Data Node Daemon
ndb_mgmd Management Node Daemon
ndb_mgm MySQL Cluster management tool

22. Tutorial Location Global configuration file /etc/mysql/my.cnf
Cluster configuration:
/opt/mysqlcluster/config.ini
MySQL Servers:
/opt/mysqlcluster/my_A.cnf
/opt/mysqlcluster/my_B.cnf
/opt/mysqlcluster/my_Slave.cnf

23. Tutorial Location Cluster Datadirs: /opt/mysqlcluster/ndb
/opt/mysqlcluster/ndb_Slave
MySQL Servers Datadirs:
/opt/mysqlcluster/mysql_A
/opt/mysqlcluster/mysql_B
/opt/mysqlcluster/mysql_Slave

24. Cluster Configuration File
/opt/mysqlcluster/config.ini
[ndbd default]

NoOfReplicas = 2

DataMemory = 80M

IndexMemory = 10M
DataDir = /opt/mysqlcluster/ndb


 [ndb_mgmd default]

DataDir = /opt/mysqlcluster/ndb



25. Cluster Configuration File (cont.)
/opt/mysqlcluster/config.ini
.. 
[ndb_mgmd]
Id = 1

Name = localhost


[ndbd]

Id = 3
 HostName = localhost


[ndbd]

Id = 4
 HostName = localhost
 ..

26. Your Turn
Check /usr/local/mysql/bin for ndb tools. Use the -- help option to get more information.
In the Virtual Machine, explore /opt/mysqlcluster. Check out the various configuration files and directories.

27. MySQL Cluster Tutorial Starting & Using MySQL Cluster
Andrew Morgan
MySQL Product Manager 27

28. Overview 28

29. Starting Cluster Nodes (processes)
Order :
Management Nodes ndb_mgmd
Data Nodes ndbd or ndbmtd
MySQL Server(s) mysqld 29

30. Starting Management Nodes:
ndb_mgmd --initial -f conf/config.ini --configdir=/home/billy/mysql/7_0_6/conf :
ndb_mgmd --initial -f conf/config.ini –-configdir=/home/billy/mysql/7_0_6/conf 30

31. Starting Management Nodes: status
ndb_mgm> SHOW
[ndbd(NDB)] 2 node(s)
id=1 (not connected, accepting connect from )
id=2 (not connected, accepting connect from )
[ndb_mgmd(MGM)] 2 node(s)
(mysql ndb-7.0.6)
(mysql ndb-7.0.6)
[mysqld(API)] 2 node(s)
id=51 (not connected, accepting connect from )
id=52 (not connected, accepting connect from ) 31

32. Starting Data Nodes 192.168.0.3: ndbd -c localhost:1186 192.168.0.4:32

33. Starting Data Nodes: status
ndb_mgm> SHOW
[ndbd(NDB)] 2 node(s)
(mysql ndb-7.0.6, starting, Nodegroup: 0, Master)
(mysql ndb-7.0.6, starting, Nodegroup: 0)
[ndb_mgmd(MGM)] 2 node(s)
(mysql ndb-7.0.6)
(mysql ndb-7.0.6)
[mysqld(API)] 2 node(s)
id=51 (not connected, accepting connect from )
id=52 (not connected, accepting connect from ) 33

34. Starting MySQL Nodes :
mysqld_safe --defaults-file=conf/my.1.conf & :
mysqld_safe --defaults-file=conf/my.2.conf & 34

35. Starting MySQL Nodes: status
ndb_mgm> SHOW
[ndbd(NDB)] 2 node(s)
(mysql ndb-7.0.6, Nodegroup: 0, Master)
(mysql ndb-7.0.6, Nodegroup: 0)
[ndb_mgmd(MGM)] 2 node(s)
(mysql ndb-7.0.6)
(mysql ndb-7.0.6)
[mysqld(API)] 2 node(s)
(mysql ndb-7.0.6)
(mysql ndb-7.0.6) 35

36. Overview 36

37. Create a MySQL Cluster Table
: mysql -h localhost -P 3306
mysql> USE test
mysql> CREATE TABLE assets (
name VARCHAR(30) NOT NULL,
value INT,
PRIMARY KEY (name)
) ENGINE=NDBCluster; 37

38. Using a MySQL Cluster Table
: mysql -h localhost -P 3306
mysql> USE test
mysql> INSERT INTO assets VALUES ('Car','1900');
mysql> SELECT * FROM assets;
| name | value |
| Car | | 38

39. Access From Other MySQL Server
: mysql -h localhost -P 3306
mysql> USE test
mysql> SELECT * FROM assets;
| name | value |
| Car | | 39

40. Exercise (single host)
$: ndb_mgmd --initial -f /opt/mysqlcluster/config.ini
$: ndb_mgm -e show
$: ndbd -c localhost:1186
$: mysqld_safe --defaults-file=/opt/mysqlcluster/my_A.cnf &
$: mysqld_safe --defaults-file=/opt/mysqlcluster/my_B.cnf &
$: mysql
mysql_A:(none)> use test;
mysql_A:test> create table simples (id int) engine=ndb;
mysql_A:test> insert into simples values (1),(2),(3);
mysql_A:test> quit
$: mysql -h P 3307
3307:(none)> select * from test.simples;

41. MySQL Cluster Tutorial Administering MySQL Cluster
Andrew Hutchings
MySQL Support Engineer 41

42. NDB_MGM
A command line interface to the management node 42

43. NDB_MGM – Useful Commands
ndb_mgm> SHOW
ndb_mgm> X STOP
ndb_mgm> X RESTART
ndb_mgm> X STATUS
ndb_mgm> START BACKUP
ndb_mgm> ALL REPORT MEMORY
Where X is a node ID or the keyword 'ALL' (signifying all data nodes) 43

44. Cluster Log Stored in the DataDir of the management node
Filename ndb_{node_id}_cluster.log
Collects information from other nodes
:58:19 [MgmSrvr] INFO Node 6: Node 32: API mysql ndb-7.0.6
:58:29 [MgmSrvr] INFO Node 3: Communication to Node 34 closed
:58:29 [MgmSrvr] ALERT Node 4: Node 34 Disconnected
:12:16 [MgmSrvr] INFO Node 6: Data usage is 7%( K pages of total )
:12:57 [MgmSrvr] INFO Node 3: Index usage is 4%(3425 8K pages of total 81952) 44

45. Data Node Logs – Stdout log
Stored in the DataDir of the data nodes
Filename ndb_{node_id}_out.log
Logs information about the node's state
:45:05 [ndbd] INFO NDB Cluster -- DB node 3
:45:05 [ndbd] INFO mysql ndb
:45:05 [ndbd] INFO WatchDog timer is set to 6000 ms
:45:05 [ndbd] INFO Ndbd_mem_manager::init(1) min: 16596Mb initial: 16980Mb
:16:29 [ndbd] WARNING -- Ndb kernel thread 2 is stuck in: Job Handling elapsed=400
:16:29 [ndbd] INFO Watchdog: User time: 21 System time: 78 45

46. Data Node Logs – Error log
Stored in the DataDir of the data nodes
Filename ndb_{node_id}_error.log
Logs information about node failures
Holds 25 entries by default, self rotates
Time: Monday 24 August :55:51
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: SimulatedBlock.cpp
Error object: DBTUP (Line: 662) 0x a
Program: /usr/local/mysql//mysql/bin//ndbmtd
Pid: 11037
Trace: /data/mysqlcluster//ndb_3_trace.log.14 /data/mysqlcluster//ndb_3_trace.log.14_t1 /data/mysqlcluster//ndb_3_trace.log.14_t2 46

47. Data Node Logs – Trace files
Stored in the DataDir of the data nodes
Filename ndb_{node_id}_trace.log.{error_id}[_t{thread_id}]
Logs signals leading up to a node failure
Really hard to read! (that is what MySQL Support is for ;)
DBDIH
Signal
r.bn: 252 "QMGR", r.proc: 3, r.sigId: 5559 gsn: 142 "CM_REGREF" prio: 1
s.bn: 252 "QMGR", s.proc: 3, s.sigId: 5558 length: 9 trace: 0 #sec: 0 fragInf: 0
H'00fc0003 H' H' H' H'00029df5 H'00029df5 H'
H' H' 47

48. Data node file systems LCP – Local Check Point
GCP – Global Check Point
Disk data files
Undo log files
Backup files
Best kept separate using:
BackupDataDir
FileSystemPathDD
FileSystemPathDataFiles
FileSystemPathUndoFiles 48

49. Your turn! Use ndb_mgm to try out the SHOW and STOP commands
Use ndb_mgm to try ALL REPORT MEMORY
Use ndb_mgm to try ALL DUMP Look in the cluster log for the result
Run 3 DUMP 7022 in ndb_mgm
What happens?
What does the data node error log say? 49

50. MySQL Cluster Tutorial MySQL Cluster Manager
Andrew Morgan
MySQL Product Manager 50

51. MySQL Cluster Manager v1.0
Cluster-Wide Management
Process Management
On-Line Operations (Upgrades / Reconfiguration)
Disk Persistence
Configuration Consistency
HA Agent Operation
Automated
Management
Status Monitoring & Recovery
Monitoring
HA Operations 51

52. Terms Used by MySQL Cluster Manager
Site
Host
Cluster
Process
agent
Site: the set of physical hosts which are to run Cluster processes to be managed by MySQL Cluster Manager. A site can include 1 or more clusters. 52

53. Terms Used by MySQL Cluster Manager
Site
Host
Host
Host
Host
Cluster
Cluster: represents a MySQL Cluster deployment. A Cluster contains 1 or more processes running on 1 or more hosts
Cluster
Process
Process
Process
Process
Process
Process
Process
Process
agent
agent
Process
agent
Process
agent 53

54. Terms Used by MySQL Cluster Manager
Site
Host
Host
Host
Host
Cluster
Host: Physical machine, running the MySQL Cluster Manager agent
Cluster
Process
Process
Process
Process
Process
Process
Process
Process
agent
agent
Process
agent
Process
agent 54

55. Terms Used by MySQL Cluster Manager
Site
Host
Host
Host
Host
Cluster
Agent: The MySQL Cluster Manager process running on each host
Cluster
Process
Process
Process
Process
Process
Process
Process
Process
agent
agent
Process
agent
Process
agent 55

56. Terms Used by MySQL Cluster Manager
Site
Host
Host
Host
Host
Cluster
Process: an individual MySQL Cluster node; one of: ndb_mgmd, ndbd, ndbmtd, mysqld & ndbapi
Cluster
Process
Process
Process
Process
Process
Process
Process
Process
agent
agent
Process
agent
Process
agent 56

57. Terms Used by MySQL Cluster Manager
Site
Host
Host
Host
Host
Cluster
Package: A copy of a MySQL Cluster installation directory as downloaded from mysql.com, stored on each host
Cluster
Process
Process
Process
Process
Process
Process
Process
Process
agent
agent
Process
agent
Process
agent 57

58. Example Configuration
MySQL Cluster Manager agent runs on each physical host
Each host can run multiple MySQL Cluster nodes (processes)
No central process for Cluster Manager – agents co-operate, each one responsible for its local nodes
Agents are responsible for managing all nodes in the cluster
mysqld & ndb_mgmd are managed directly
ndbd managed via ndb_mgmd
Each node allocated an ID
Management responsibilities
Starting, stopping & restarting nodes
Configuration changes
Upgrades
Host & Node status reporting
Recovering failed nodes
agent
1. ndb_mgmd
7. mysqld
2. ndb_mgmd
8. mysqld
5. ndbd
3. ndbd
6. ndbd
4. ndbd
mysql client
n. ndb_mgmd
n. mysqld
n. ndbd
MySQL Server (ID=n)
Management Node (ID=n)
Data Node (ID=n)
MySQL Cluster Manager agent 58

59. Install, config, run & access
The agent must be installed and run on each host in the Cluster:
Expand the tar-ball into a known directory (/usr/local/mcm)
Copy /usr/local/mcm/etc/mysql-cluster-manager.ini to /home/billy/mcm and edit:
“plugins=manager” should always be used
Launch the agent process:
Access any of the agents from any machine (that has the mysql client installed):
agent
MySQL Cluster Manager agent
[mysql-proxy] plugins=manager manager-port = :1862 log-file = mysql-manager-agent.log log-level = message manager-directory=/home/billy/mcm/manager
mysql client
/usr/local/mcm/bin/mysql-cluster-manager --defaults- file=/home/billy/mcm/mysql-cluster-manager.ini
C:\Users\am233268>mysql -h P u admin –p Enter password: super Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1 Server version: agent-manager MySQL Cluster Manager Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> 59

60. Check Status of MySQL Cluster
Request the status of the Cluster processes
Get the status of the hosts & agents
agent
1. ndb_mgmd
7. mysqld
2. ndb_mgmd
8. mysqld
5. ndbd
3. ndbd
6. ndbd
4. ndbd
mysql client
n. ndb_mgmd
n. mysqld
n. ndbd
MySQL Server (ID=n)
Management Node (ID=n)
Data Node (ID=n)
MySQL Cluster Manager agent
mysql> show status --process mycluster;
| Id | Process | Host | Status | Nodegroup | | 1 | ndb_mgmd | | running | | | 2 | ndb_mgmd | | running | | | 3 | ndbd | | running | | | 4 | ndbd | | running | | | 5 | ndbd | | running | | | 6 | ndbd | | running | | | 7 | mysqld | | running | | | 8 | mysqld | | running | |
mysql> list hosts mysite;
| Host | Status | Version | | | Available | | | | Available | | | | Available | | | | Available | | 60

61. Check Cluster Parameters
Fetch all parameters that apply to all data nodes, including defaults:
Fetch the values of parameters (excluding defaults) for mysqld with ID=7:
Fetch the port parameter to connect to mysqld with ID=7:
agent
1. ndb_mgmd
7. mysqld
2. ndb_mgmd
8. mysqld
5. ndbd
3. ndbd
6. ndbd
4. ndbd
mysql client
n. ndb_mgmd
n. mysqld
n. ndbd
MySQL Server (ID=n)
Management Node (ID=n)
Data Node (ID=n)
MySQL Cluster Manager agent
mysql> get -d :ndbd mycluster;
| Name | Value | Process1 | Id1 | Process2 | Id2 | Level | Comment | | __ndbmt_lqh_threads | NULL | ndbd | 3 | | | Default | | | __ndbmt_lqh_workers | NULL | ndbd | 3 | | | Default | | | Arbitration | NULL | ndbd | 3 | | | Default | | : : : : : : : : | __ndbmt_lqh_threads | NULL | ndbd | 4 | | | Default | | | __ndbmt_lqh_workers | NULL | ndbd | 4 | | | Default | | | Arbitration | NULL | ndbd | 4 | | | Default | | | ArbitrationTimeout | | ndbd | 4 | | | Default | | : : : : : : : : | __ndbmt_lqh_threads | NULL | ndbd | 5 | | | Default | | : : : : : : : : | __ndbmt_lqh_threads | NULL | ndbd | 6 | | | Default | | : : : : : : : :
mysql> get :mysqld:7 mycluster;
| Name | Value | Process1 | Id1 | | datadir | /home/billy/mcm/alpha/manager/clusters/mycluster/7/data | mysqld | 7 | | HostName | ws | mysqld | 7 | | ndb-nodeid | | mysqld | 7 | | ndbcluster | | mysqld | 7 | | NodeId | | mysqld | 7 |
mysql> get -d port:mysqld:7 mycluster;
| Name | Value | Process1 | Id1 | Process2 | Id2 | Level | Comment | | port | | mysqld | 7 | | | Default | | 61

62. Configuration Data Management
MySQL Cluster Manager agents hold master copy of configuration data for all nodes
Administrator can view and edit this data from a mysql client connected to any agent
Configuration information for all nodes is held by all agents
Agent creates local configuration files from this data which the Cluster nodes use (each agent creates just those local config files needed on that host)
agent
1. ndb_mgmd
7. mysqld
2. ndb_mgmd
8. mysqld
5. ndbd
3. ndbd
6. ndbd
4. ndbd
manager
clusters
mycluster 1
cfg 7
config.ini
my.cnf 2 8 3 5 4 6 62

63. Upgrade Cluster Upgrade from MySQL Cluster 6.3.26 to 7.0.7:
Automatically upgrades each node and restarts the process – in the correct order to avoid any loss of service
Without MySQL Cluster Manager, the administrator must stop each process in turn, start the process with the new version and wait for the node to restart before moving onto the next one
agent
1. ndb_mgmd
7. mysqld
2. ndb_mgmd
8. mysqld
5. ndbd
3. ndbd
6. ndbd
4. ndbd
mysql client
n. ndb_mgmd
n. mysqld
n. ndbd
MySQL Server (ID=n)
Management Node (ID=n)
Data Node Node (ID=n)
MySQL Cluster Manager agent
mysql> upgrade cluster --package=7.0 mycluster; 63

64. HA Provided by MySQL Cluster Manager
Service is maintained when any Cluster process fails but the processes need to be restarted as soon as possible afterwards to restore redundancy
If a process dies then it will be detected by the local agent
Without MySQL Cluster Manager, an angel process would recreate failed ndbd processes but not mysqld or ndb_mgmd processes
Cluster continues to function if one or more agents fail or is stopped (e.g. To upgrade it)
To restore management functions, the agent should be restarted
agent
1. ndb_mgmd
7. mysqld
2. ndb_mgmd
8. mysqld
5. ndbd
3. ndbd
6. ndbd
4. ndbd
n. ndb_mgmd
n. mysqld
n. ndbd
MySQL Server (ID=n)
Management Node (ID=n)
Data Node (ID=n)
MySQL Cluster Manager agent 64

65. MySQL Cluster Tutorial Fault Tolerance
Andrew Hutchings
MySQL Support Engineer 65

66. MySQL Server Temporary errors in MySQL happen - especially in Cluster
MySQL Cluster implicitly rollback any transaction in which a temporary error happens
SHOW WARNINGS is your friend
mysql> select * from t1;
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
mysql> show warnings\G
*************************** 1. row ***************************
Level: Error
Code: 1297
Message: Got temporary error 274 'Time-out in NDB, probably caused by deadlock' from NDB
*************************** 2. row ***************************
*************************** 3. row ***************************
Code: 1205
Message: Lock wait timeout exceeded; try restarting transaction
*************************** 4. row ***************************
Code: 1622
Message: Storage engine NDB does not support rollback for this statement. Transaction
rolled back and must be restarted
4 rows in set (0.00 sec) 66

67. Data Node Heartbeats
Data nodes send heartbeats in a ring 67

68. Data Node Heartbeats
Failure occurs! 68

69. Data Node Heartbeats
Automatic failover 69

70. Cluster Arbitration A typical MySQL Cluster configuration 70 Mgmt Node
Data Node (NG1)
Data Node (NG2)
Data Node (NG1)
Data Node (NG2) 70

71. Cluster Arbitration Two network links fail, cluster still up! 71
Mgmt Node
Data Node (NG1)
Data Node (NG2)
Data Node (NG1)
Data Node (NG2) 71

72. Cluster Arbitration
No nodes left in node group 1, cluster cannot continue
Mgmt Node
Data Node (NG1)
Data Node (NG2)
Data Node (NG1)
Data Node (NG2) 72

73. Cluster Arbitration One node from each node group fails, cluster lives
Mgmt Node
Data Node (NG1)
Data Node (NG2)
Data Node (NG1)
Data Node (NG2) 73

74. Checkpointing
Local Checkpoint (LCP) – Writes the entire DataMemory to disk
Global Checkpoint (GCP) – Syncs REDO log to disk
By default up to two seconds of data can be lost if all data nodes fail!
UNDO log used for disk data
Data can also been sent between nodes to synchronise missing data
Common misconception that cluster is in-memory and all data is lost upon power failure 74

75. Your turn Insert some data into your cluster
Kill a data node (using the Linux kill command)
Start that node back up again
In ndb_mgm run 'SHUTDOWN CLUSTER' and kill mysqld
Start it all up again
Is all your data still there? 75

76. MySQL Cluster Tutorial Online Backup
Geert Vanderkelen
MySQL Support Engineer 76

77. Tools For Backuping
mysqldump for backing up the meta data (schema, stored routines, ..) and none Cluster tables
ndb_mgm
launch the online backup for NDBCluster tables
mysql restoring meta data and none Cluster tables
ndb_restore restoring data of Cluster tables
Common misconception that cluster is in-memory and all data is lost upon power failure 77

78. Online Backup ndb_mgm> START BACKUP
Waiting for completed, this may take several minutes
Node 3: Backup 2 started from node 1
Node 3: Backup 2 started from node 1 completed
StartGCP: 284 StopGCP: 287
#Records: 2059 #LogRecords: 0
Data: bytes Log: 0 bytes
Common misconception that cluster is in-memory and all data is lost upon power failure 78

79. Online Backup: Where it goes?
ndb_mgm> START BACKUP
Data Nodes
Common misconception that cluster is in-memory and all data is lost upon power failure 79

80. Online Backup: Files e.g. Backup no. 2: BACKUP/BACKUP-2:
<datadir>/BACKUP on each data node
e.g. Backup no. 2: BACKUP/BACKUP-2:
BACKUP Data
BACKUP Data
BACKUP-2.3.ctl
BACKUP-2.3.log
BACKUP-2.4.ctl
BACKUP-2.4.log
Common misconception that cluster is in-memory and all data is lost upon power failure 80

81. Online Backup: Files BACKUP-2-0.3.Data BACKUP-2-0.4.Data
Files of Node 3 & 4 contain actual data
BACKUP-2.3.ctl
BACKUP-2.4.ctl
Contain meta data
BACKUP-2.3.log
BACKUP-2.4.log
Transaction log
Common misconception that cluster is in-memory and all data is lost upon power failure 81

82. Restoring Using ndb_restore
Restore Node 3: meta and records
shell> cd /opt/mysqlcluster/ndb/BACKUP/
shell> ndb_restore -m -r -b 2 -n 3 BACKUP-2/
-m = restore meta
-r = restore records
-b = backup ID
-n = node ID
<location of backup files>
Common misconception that cluster is in-memory and all data is lost upon power failure 82

83. Restoring Using ndb_restore
Restore Node 4: only records (meta already there)
shell> cd /opt/mysqlcluster/ndb/BACKUP/
shell> ndb_restore -r -b 2 -n 4 BACKUP-2/
-r = restore records
-b = backup ID
-n = node ID
<location of backup files>
Common misconception that cluster is in-memory and all data is lost upon power failure 83

84. ndb_restore: It Does Even More
Get Meta data, without actually restoring:
ndb_restore --print-meta -n 3 -b 2 ./BACKUP-2/
Only restore table db1.t1:
ndb_restore --include-tables=db1.t1
See ndb_restore --help for more!
Common misconception that cluster is in-memory and all data is lost upon power failure 84

85. Your turn Start your cluster empty
Create a new table, and insert some data
Do ndb_mgm> START BACKUP
Check the BACKUP directory and backup files
Start your cluster again empty
Restore meta and records using ndb_restore
Restore meta and/or records again, and see what happens 85

86. MySQL Cluster Tutorial NDBINFO
Andrew Morgan
MySQL Product Manager 86

87. ndbinfo mysql> USE ndbinfo mysql> SHOW TABLES;
| Tables_in_ndbinfo |
| blocks |
| config_params |
| counters |
| logbuffers |
| logspaces |
| memoryusage |
| nodes |
| resources |
| transporters |
New database (ndbinfo) which presents real-time metric data in the form of tables
Exposes new information together with providing a simpler, more consistent way to access existing data
Examples include:
Resource usage (memory, buffers)
Event counters (such as number of READ operations since last restart)
Data node status and connection status 87

88. ndbinfo Example 1: Check memory usage/availability
mysql> select * from memoryusage;
| node_id | DATA_MEMORY | used | max |
| | DATA_MEMORY | 594 | 2560 |
| | DATA_MEMORY | 594 | 2560 |
| | INDEX_MEMORY | 124 | 2336 |
| | INDEX_MEMORY | 124 | 2336 |
Note that there is a DATA_MEMORY and INDEX_MEMORY row for each data node in the cluster
If the Cluster is nearing the configured limit then increase the DataMemory and/or IndexMemory parameters in config.ini and then perform a rolling restart 88

89. ndbinfo
Example 2: Check how many table scans performed on each data node since the last restart
mysql> select node_id as 'data node', val as 'Table Scans' from counters where counter_name='TABLE_SCANS';
| data node | Table Scans | | | | | | |
You might check this if your database performance is lower than anticipated
If this figure is rising faster than you expected then examine your application to understand why there are so many table scans 89

90. ndbinfo
Example 3: Check if approaching the point at which the undo log completely fills up between local checkpoints (which could result in delayed transactions or even a database halt if not addressed):
mysql> select node_id as 'data node', total as 'configured undo log buffer size', used as 'used buffer space' from logbuffers where log_type='DD-UNDO';
| data node | configured undo log buffer size | used buffer space | | | | | | | | |
If log buffer is almost full then increase size of log buffer 90

91. ndbinfo.counters
View created from hidden tables to present counters for the number of events since the last node restart
When using a single threaded data node (ndbd), block _instance=0
If using a multi-threaded data node (ndbmtd) then there will be a DBLQH row for each Local Query Handler thread, for the OPERATIONS counter, identified with different values for block_instance
mysql> select * from ndbinfo.counters;
| node_id | block_name | block_instance | counter_id | counter_name | val | | | DBLQH | | | OPERATIONS | 2069 | | | DBLQH | | | OPERATIONS | 28 | | | DBLQH | | | OPERATIONS | 2066 | | | DBLQH | | | OPERATIONS | 27 | | | DBTC | | | ATTRINFO | 140 | | | DBTC | | | TRANSACTIONS | 19 | | | DBTC | | | COMMITS | 19 | | | DBTC | | | READS | 19 | | | DBTC | | | SIMPLE_READS | 0 | | | DBTC | | | WRITES | 0 | | | DBTC | | | ABORTS | 0 | | | DBTC | | | TABLE_SCANS | 0 | | | DBTC | | | RANGE_SCANS | 0 | | | DBTC | | | ATTRINFO | 2 | | | DBTC | | | TRANSACTIONS | 1 | | | DBTC | | | COMMITS | 1 | | | DBTC | | | READS | 1 | | | DBTC | | | SIMPLE_READS | 0 | | | DBTC | | | WRITES | 0 | | | DBTC | | | ABORTS | 0 | | | DBTC | | | TABLE_SCANS | 1 | | | DBTC | | | RANGE_SCANS | 0 | 91

92. ndbinfo.logbuffers
View created from hidden tables to present log buffer usage
log_type=REDO: ‘total’ represents the amount of memory configured for the redo logs and ‘used’ indicates how much of it is currently being used. If the buffer fills up then will see 1221 “REDO log buffers overloaded” errors – avoid this by increasing the RedoBuffer configuration parameter if ‘used’ approaches ‘total’
log_type=DD_UNDO: ‘total’ represents the amount of memory configured for the undo logs (UNDO_BUFFER_SIZE when creating the log group) and ‘used’ indicates how much of it is currently being used. If buffers fill too quickly then transactions can be delayed while they are flushed to disk; in extreme cases causing database to halt
mysql> select * from ndbinfo.logbuffers;
| node_id | log_type | log_id | log_part | total | used | | | REDO | | | | | | | DD-UNDO | | | | | | | REDO | | | | | | | DD-UNDO | | | | | 92

93. ndbinfo.logspaces
View created from hidden tables to present logspace usage
log_type=REDO: 1 row for each of the 4 file sets for each data node. ‘total’=NoOfFragmentLogFiles*FragmentLogFi leSize and ‘used’ is the amount actually used. If the files fill up before a local checkpoint can complete then error code 410 (Out of log file space temporarily) will be observed. Avoid this by increasing NoOfFragmentLogFiles and/or FragmentLogFileSize if used approaches total.
log_type=DD_UNDO: ‘total’=cumulative size of all of the undo log files assigned to the log group, as added using create/alter logfile group or the InitialLogFileGroup parameter. Add extra undo files if ‘used’ approaches ‘total’ to avoid 1501 errors
mysql> select * from logspaces;
| node_id | log_type | log_id | log_part | total | used | | | REDO | | | | | | | REDO | | | | | | | REDO | | | | | | | REDO | | | | | | | DD-UNDO | | | | | | | REDO | | | | | | | REDO | | | | | | | REDO | | | | | | | REDO | | | | | | | DD-UNDO | | | | | 93

94. ndbinfo.memoryusage
View created from hidden tables to present memory limits and usage
Compare the amount of memory and index used to the amount configured for each data node
DATA_MEMORY: ‘max’ = DataMemory parameter. If ‘used’ approaches ‘max’ then increase DataMemory and/or delete obsolete rows run OPTIMIZE TABLE for the Cluster tables
INDEX_MEMORY: ‘max’ = IndexMemory parameter. If ‘used’ approaches ‘max’ then increase IndexMemory (+ DataMemory if needed) and/or delete obsolete rows run OPTIMIZE TABLE for the Cluster tables
mysql> select * from memoryusage;
| node_id | memory_type | used | max | | | DATA_MEMORY | 632 | 3144 | | | DATA_MEMORY | 632 | 3144 | | | INDEX_MEMORY | 38 | 2336 | | | INDEX_MEMORY | 38 | 2336 | 94

95. ndbinfo.nodes
View created from hidden tables for the status of data nodes.
For each data node that is running in the cluster, a corresponding row in this table provides the
Uptime: time in seconds that this node has been running since it was last started or restarted.
Status: One of NOTHING, CMVMI, STARTING, STARTED, SINGLEUSER, STOPPING_1, STOPPING_2, STOPPING_3, or STOPPING_4.
Start-phase: If Status=STARTING then the start-phase (as seen through ndb_mgm), 0 otherwise.
mysql> select * from nodes;
| node_id | uptime | status | start_phase | | | | STARTED | | | | | STARTED | | 95

96. ndbinfo.transporters
View showing the status of the connection between each data node and each of the other nodes in the cluster (data nodes, management nodes, MySQL Server nodes and any NDB API applications).
mysql> select * from transporters;
| node_id | remote_node_id | status | | | | CONNECTED | | | | DISCONNECTED | | | | CONNECTED | | | | CONNECTED | | | | CONNECTING | | | | CONNECTING | | | | CONNECTED | | | | CONNECTED | | | | DISCONNECTED | | | | CONNECTED | | | | CONNECTING | | | | CONNECTING | 96

97. Exercise Connect to Cluster using mysql client
Open the ndbinfo database: mysql> use ndbinfo;
Check what tables are visible:
mysql> show tables;
Query the memory usage table:
mysql> select * from memoryusage;
Increase DataMemory in config.ini by 10% and restart 1 data node
Check memoryusage again

98. MySQL Cluster Tutorial NDB API & Connector for Java
Andrew Morgan
MySQL Product Manager 98

99. NDB API MySQL Cluster SQL Based Applications JDBC/ODBC MySQL Server
Direct API Applications
Web Services
LDAP
Managemen t Client
NDB API
MGM API
MySQL Cluster
Data Node
Data Node
Managemen t Node
NDB API
Data Node
Data Node 99

100. NDB API
C++ based interface that allows applications to connect directly to the data nodes
Used extremely successfully by many mission-critical, real-time customer applications
Consider using the NDB API if:
C++ application
You need maximum real-time performance (latency & throughput)
Willing to accept added complexity in your application and lock-in to a proprietary interface
API documented at:
See tutorial notes for an example
Alternate ways to more easily exploit many of the performance benefits of the NDB API
back-ndb plugin for OpenLDAP
MySQL Cluster Connector for Java (including OpenJPA)
http plugin for Apache
100

101. MySQL Cluster Connector for Java
Apps
Apps
Apps
Apps
Apps
Rich set of existing MySQL connectors (con) exist but performance worse than NDB API
New Domain Object Model Persistence API (ClusterJ) :
Java API
High performance, low latency
Feature rich
JPA interface built upon this new Java layer:
Java Persistence API compliant
Implemented as an OpenJPA plugin
Uses ClusterJ where possible, reverts to JDBC for some operations
Higher performance than JDBC
More natural for most Java designers
Easier Cluster adoption for web applications
OpenJPA
ClusterJPA
JDBC
ClusterJ
MySQL
JNI
NDB API (C++)
Network
Data Nodes
seminars/display-487.html
101

102. MySQL Cluster Connector for Java
High Performance, Easy to Use
In the style of Hibernate / JPA / JDO
Domain Object Model DataMapper pattern
Data is represented as domain objects
Domain objects are separate from business logic
Domain objects are mapped to database tables
Built on ndbjtie
JNI adapter
integral part of MySQL Cluster
Straight mapping of MySQL Cluster API (a.k.a NDB API) to Java
Does not support relationships
Look at JDO / JPA for these modelling patterns
102

103. Connector for Java: Interfaces
SessionFactory
One per Cluster per JVM
Session
Instance per user; represents a Cluster connection
Transaction
Instance per Session
Query
Multiple instances per Session
Session Factory
Configuration properties
Session & Transaction
Domain Object
Annotated interface
103

104. Connector for Java: Configuration
com.mysql.clusterj.connectstring= :1186
com.mysql.clusterj.database=clusterdb
com.mysql.clusterj.connect.retries=4
com.mysql.clusterj.connect.delay=5
com.mysql.clusterj.connect.verbose=1
com.mysql.clusterj.connect.timeout.before=30
com.mysql.clusterj.connect.timeout.after=20
com.mysql.clusterj.max.transactions=1024
104

105. Connector for Java: Annoted Interface
id INT NOT NULL PRIMARY KEY,
CREATE TABLE employee (
@PersistenceCapable(table="employee")
first VARCHAR(64) DEFAULT NULL,
municipality VARCHAR(64) DEFAULT NULL,
last VARCHAR(64) DEFAULT NULL,
started VARCHAR(64) DEFAULT NULL,
ended VARCHAR(64) DEFAULT NULL,
public interface Employee {
department INT NOT NULL DEFAULT 1,
UNIQUE KEY idx_u_hash (first,last) USING HASH,
@Index(name="idx_uhash")
KEY idx_municipality (municipality)
@PrimaryKey int getId(); void setId(int id);
String getFirst(); void setFirst(String first);
String getLast(); void setLast(String last);
@Column(name="municipality")
String getCity();
void setCity(String city)
String getStarted(); void setStarted(String date);
String getEnded(); void setEnded(String date);
Integer getDepartment(); void setDepartment(
Integer department);}
@Index(name="idx_municipality")
) ENGINE=NDBCLUSTER;
105

106. Connector for Java: Code Example
// Create a session (connection to the database) SessionFactory factory = ClusterJHelper.getSessionFactory(props); Session session = factory.getSession();
// Create and initialise an Employee Employee newEmployee = session.newInstance(Employee.class); newEmployee.setId(988); newEmployee.setFirst("John"); newEmployee.setLast("Jones"); newEmployee.setStarted("1 February 2009"); newEmployee.setDepartment(666);
// Write the Employee to the database session.persist(newEmployee);
// Fetch the Employee from the database Employee theEmployee = session.find(Employee.class, 988);
// Make some changes to the Employee & write back to the database theEmployee.setDepartment(777); theEmployee.setCity("London"); session.updatePersistent(theEmployee);
// Retrieve the set all of Employees in department 777 QueryBuilder builder = session.getQueryBuilder(); QueryDomainType<Employee> domain = builder.createQueryDefinition(Employee.class); domain.where(domain.get("department").equal(domain.param("department"))); Query<Employee> query = session.createQuery(domain); query.setParameter("department",777); List<Employee> results = query.getResultList();
106

107. ClusterJPA Removes ClusterJ limitations: Persistent classes
Relationships
Joins in queries
Lazy loading
Table and index creation from object model
Implemented as an OpenJPA plugin
Uses standard JDBC/MySQL code path for queries
(not simple find)
Better JPA performance for insert, update, delete
107

108. JPA: Java Persistence API
The Java standard for persistence
Works in server containers and outside (not CMP!)
Choice of implementations: OpenJPA, Hibernate, TopLink, or......
Depends on application requirements
Select the implementation via configuration file (persistence.xml)
ClusterJPA optimized for OpenJPA today
Other implementations (Hibernate, TopLink, etc) access MySQL Cluster via JDBC
108

109. ClusterJPA: Example Configuration
<persistence xmlns= xmlns:xsi=" version="1.0">
<persistence-unit name="clusterdb" transaction-type="RESOURCE_LOCAL“> <provider> org.apache.openjpa.persistence.PersistenceProviderImpl </provider> <class>Employee</class> <class>Department</class> <properties> <property name="openjpa.jdbc.SynchronizeMappings" value="buildSchema" /> <property name="openjpa.ConnectionDriverName" value="com.mysql.jdbc.Driver" /> <property name="openjpa.ConnectionURL" value="jdbc:mysql://localhost:3306/clusterdb" /> <property name="openjpa.ConnectionUserName" value="root" /> <property name="openjpa.ConnectionPassword" value="" /> <property name="openjpa.BrokerFactory" value="ndb" /> <property name="openjpa.jdbc.DBDictionary" value="TableType=ndb"/> <property name="openjpa.ndb.connectString" value="localhost:1186" /> <property name="openjpa.ndb.database" value="clusterdb" / </properties> </persistence-unit> </persistence>
109

110. ClusterJPA: Annoted Class Example
@Entity(name = "department")
@Id public int getId() {return Id;} public void setId(int id) {Id=id;}
public String getSite() {return Site;} public void setSite(String site) {Site=site;}
public class Department { private int Id; private String Site;
public Department(){}
@Column(name="location") }
mysql> describe department;
| id | int(11) | NO | PRI | NULL | |
| location | varchar(255) | YES | | NULL | |
| Field | Type | Null | Key | Default | Extra |
110

111. Performance
111

112. MySQL Cluster Tutorial Schema Considerations
Geert Vanderkelen
MySQL Support Engineer
112

113. Considerations MySQL Cluster is good at
Short transactions and many in parallel
Keep access to data simple
Reduce network roundtrips
113

114. Reporting Do reporting outside MySQL Cluster
Replicate to Slave using e.g. InnoDB
114

115. Denormalize Avoid doing excessive, big joins. Go Primary Key.
SELECT s.uid, l.location
 FROM subscribers AS s LEFT JOIN locations AS l WHERE s.uid = 3
SELECT uid, location FROM subscribers WHERE uid = 3
115

116. Partitioning Try to set the partition right.
Following will only hit one data node:
CREATE TABLE attendees (
userid INT,
serviceid INT,
data VARCHAR(200),
PRIMARY KEY(userid, serviceid))
ENGINE=NDBCluster
PARTITION BY KEY(userid)
116

117. Develop For MySQL Cluster
ALTER TABLE attendees ENGINE=NDBCluster
YES, But...
117

118. It's In The Details Columns per table 128 1000 4096 Row size
NDBCluster
InnoDB
MyISAM
Columns per table
128
1000
4096
Row size
8052 bytes
±8kB*
64kB
118

119. Fails. Re-Normalize CREATE TABLE t1 ( c1 INT NOT NULL KEY,
c2 VARCHAR(4000),

c3 VARCHAR(4000)

) ENGINE=NDBCluster
DEFAULT CHARSET=UTF-8
Fails.
119

120. MySQL Cluster Tutorial Scaling and Performance
Andrew Hutchings
MySQL Support Engineer
120

121. Scaling is good! Adding SQL nodes = more QPS
SQL nodes can have more than one cluster connection thread (ndb- cluster-connection-pool)
NDBMTD adds mode query handler threads
Adding more data nodes = faster pkey access
Online add node lets you add nodes without rebuilding cluster
121

122. Cluster Performance Pros and Cons
Direct NDBAPI can be much faster than using SQL
Cluster is great for executing many queries in parallel
There are some great cluster settings to improve performance
Source:
122

123. Cluster Performance Pros and Cons
Blobs perform really badly
Blobs use a hidden table to store the blob data
This means many more accesses and some complicated locking
Joins perform really badly
Joins broken down into many calls to the data nodes
123

124. Cluster Performance Pros and Cons
Johan's talk
MySQL Cluster Performance Tuning Best Practices
Wednesday 2pm
Jonas's talk
MySQL Cluster and Pushdown-joins (In Pursuit of the Holy Grail)
Wednesday 3:05pm
124

125. Your turn
Add two data nodes to your cluster
125

126. MySQL Cluster Tutorial Geographical Replication
Geert Vanderkelen
MySQL Support Engineer
126

127. Classic Replication
MySQ L
BinLog
MySQ L
BinLog
127

128. Why Row-Based? Changes From All!
SQ L
SQ L
SQ L
Data Nodes
API
SQL
BinLog
API

129. Enabling Binary Logging
[mysqld]
ndbcluster
log_bin = master_A_binlog
binlog_format = MIXED
server_id = 1
129

130. Peaking Into Binary Logs
shell> mysqlbinlog -vv master_A_binlog
..
Xz/CSxcBAAAAOwAAAGgEAAAAABAAAAAAAAAABR/gAQAAAA8AAACiAAAAAAAAAAAAAAAAAAAAAA AA
AAA= 
### INSERT INTO mysql.ndb_apply_status 
### SET
 ### @1=1 /* INT meta=0 nullable=0 is_null=0 */
 ### @2= /* LONGINT meta=0 nullable=0 is_null=0 */
### @3='' /* VARSTRING(255) meta=255 nullable=0 is_null=0 */
### @4=0 /* LONGINT meta=0 nullable=0 is_null=0 */
### @5=0 /* LONGINT meta=0 nullable=0 is_null=0 */
Xz/CSxcBAAAAMgAAAJoEAAAQACUAAAAAAAEAAgP8AQAAAA9nZWVy dEBsb2NhbGhvc3Q=
'/*!*/;
 ### INSERT INTO test.attendees
 ### SET 
### @1=1 /* INT meta=0 nullable=0 is_null=0 */ 
### /* VARSTRING(200) meta=200 nullable=1 is_null=0 */
130

131. The Problem..
SQL
BinLog
SQ L
Data Nodes
API

132. .. No Binary Logging going..
SQL
BinLog
SQ L
Data Nodes
API
Where are changes
logged now? ?

133. The LOST_EVENTS incident
SQ L
SQ L
SQ L
Data Nodes
LOST_EVENTS logged
when mysqld starts
API
SQL
BinLog
API

134. The LOST_EVENTS incident
shell> mysqlbinlog master_A_binlog.00001
..
 # :48:47 server id 1 end_log_pos 143
 # Incident: LOST_EVENTS
 RELOAD DATABASE; # Shall generate syntax error

135. The LOST_EVENTS: Stops Replication
mysql_Slave> SHOW SLAVE STATUS\G
 ..
 Last_Errno: 1590
 Last_Error: The incident LOST_EVENTS occured on the master. Message: mysqld startup
..

136. LOST_EVENTS: Solution
SQL
BinLog
SQ L
Data Nodes
API

137. Global Replication
S Q L
BinLo g
Data Nodes
A PI
Channel 1
Channel 2

138. Channel 2 Global Replication S Q L S Q L S Q L S Q L A PI S Q L A PI
Data Nodes
Data Nodes
BinLo g
A PI
S Q L
Channel 2
A PI
BinLo g
S Q L

139. Replicatoin Channel Failover
On Slave, get most recent GCP
mysql>
FROM mysql.ndb_apply_status;

140. Replicatoin Channel Failover
On Master mysql_B, binary log position
mysql> SELECT
@file:=SUBSTRING_INDEX(File, '/', -1),
@pos:=Position
FROM mysql.ndb_binlog_index
WHERE epoch
ORDER BY epoch ASC LIMIT 1;

141. Replicatoin Channel Failover
On Slave, change master to mysql_B
mysql> CHANGE MASTER TO
MASTER_LOG_FILE =
MASTER_LOG_POS
MASTER_HOST=' ',
MASTER_PORT=3307;
mysql> START SLAVE;

142. Exercises
Start Slave MySQL Cluster shell> cd /opt/mysqlcluster shell> ndb_mgmd --initial -f config_Slave.ini --configdir=ndb_Slave 
shell> ndbd -c localhost:1187 shell> mysqld_safe --defaults-file=my_Slave.cnf
Setup Replication, with mysql_A and mysql_B master SQL nodes, and mysql_Slave the slave.
Restart mysql_A, check slave.
Make mysql_Slave read from mysql_B.

143. MySQL Cluster Tutorial Security
Andrew Hutchings
MySQL Support Engineer
143

144. What Security?
Cluster has none!
144

145. What Security? Network encryption bad idea, too much overhead
No authentication, not even on ndb_mgm
MySQL authentication does not propagate to other mysqld nodes
145

146. Recommendations Keep all cluster nodes on a private LAN
Keep MySQL authentication maintained manually (or scripted) between MySQL nodes
146

147. That's all folks! Any questions?
Cluster Mailing list:
Cluster Forum:
Andrew Hutchings:
Andrew Morgan:
Geert Vanderkelen:
147

148. The preceding is intended to outline our general product direction
The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
148

149. 149