Detailed introduction to the principle of mydumper

Mydumper installation: http://www.cnblogs.com/lizhi221/p/7010174.html

mydumper introduction

MySQL’s own mysqldump tool supports single threading Work, export multiple tables one by one, without a parallel machine, which makes it impossible to quickly back up data.

As a practical tool, mydumper can well support multi-threaded work. It can read data from tables and write data to different files simultaneously with multiple threads in parallel. , which makes it faster than traditional mysqldump in terms of processing speed. One of its characteristics is that the list needs to be locked during processing, so if we need to perform backup work during working hours, it will cause DML blocking. But generally, current MySQL has a master and slave, and most backups are performed on the slave, so the lock issue does not need to be considered. In this way, mydumper can better complete the backup task.

##mydumper features

①Multi-threading Backup

② Because it is a multi-threaded logical backup, multiple backup files will be generated after the backup

③ Apply FTWRL( to the MyISAM table during backup FLUSH TABLES WITH READ LOCK), will block DML statements

④Ensure the consistency of backup data

⑤Support file compression

⑥Support exporting binlog

⑦Support multi-thread recovery

⑧Support working in daemon mode, timing Snapshots and continuous binary logs

⑨Support cutting backup files into chunks

Detailed explanation of mydumper parameters

$ mydumper --helpUsage:
  mydumper [OPTION...] multi-threaded MySQL dumping

Help Options:  -?, --help                  Show help optionsApplication Options:  -B, --database              要备份的数据库，不指定则备份所有库
  -T, --tables-list           需要备份的表，名字用逗号隔开
  -o, --outputdir             备份文件输出的目录
  -s, --statement-size        生成的insert语句的字节数，默认1000000
  -r, --rows                  Try to split tables into chunks of this many rows. This option turns off --chunk-filesize
  -F, --chunk-filesize        Split tables into chunks of this output file size. This value is in MB
  -c, --compress              Compress output files压缩输出文件
  -e, --build-empty-files     如果表数据是空，还是产生一个空文件（默认无数据则只有表结构文件）
  -x, --regex                 Regular expression for 'db.table' matching 使用正则表达式匹配'db.table'
  -i, --ignore-engines        Comma delimited list of storage engines to ignore忽略的存储引擎，用逗号分割
  -m, --no-schemas            Do not dump table schemas with the data不备份表结构，只备份数据
  -d, --no-data               Do not dump table data备份表结构，不备份数据
  -G, --triggers              Dump triggers备份触发器
  -E, --events                Dump events
  -R, --routines              Dump stored procedures and functions备份存储过程和函数
  -k, --no-locks              不使用临时共享只读锁，使用这个选项会造成数据不一致
  --less-locking              Minimize locking time on InnoDB tables.减少对InnoDB表的锁施加时间
  -l, --long-query-guard      设定阻塞备份的长查询超时时间，单位是秒，默认是60秒（超时后默认mydumper将会退出）
  -K, --kill-long-queries     Kill long running queries (instead of aborting)杀掉长查询 (不退出)
  -D, --daemon                Enable daemon mode启用守护进程模式，守护进程模式以某个间隔不间断对数据库进行备
  -I, --snapshot-interval     dump快照间隔时间，默认60s，需要在daemon模式下
  -L, --logfile               使用的日志文件名(mydumper所产生的日志), 默认使用标准输出
  --tz-utc                    SET TIME_ZONE='+00:00' at top of dump to allow dumping of TIMESTAMP data when a server has data in different time zones or data is being moved between servers with different time zones, defaults to on use --skip-tz-utc to disable.
  --skip-tz-utc               
  --use-savepoints            使用savepoints来减少采集metadata所造成的锁时间，需要 SUPER 权限
  --success-on-1146           Not increment error count and Warning instead of Critical in case of table doesn't exist
  --lock-all-tables           Use LOCK TABLE for all, instead of FTWRL
  -U, --updated-since         Use Update_time to dump only tables updated in the last U days
  --trx-consistency-only      Transactional consistency only
  -h, --host                  连接的主机名
  -u, --user                  用来备份的用户名
  -p, --password              用户密码
  -P, --port                  连接端口
  -S, --socket                使用socket通信时的socket文件
  -t, --threads               开启的备份线程数，默认是4
  -C, --compress-protocol     压缩与mysql通信的数据
  -V, --version               显示版本号
  -v, --verbose               输出信息模式, 0 = silent, 1 = errors, 2 = warnings, 3 = info, 默认为2

Mydumper main process summary

1. The main thread FLUSH TABLES WITH READ LOCK, applies a global read-only lock to prevent DML statements from being written and ensure data Consistency

2. Read the binary log file name and the location of the log writing at the current point in time and record it in the metadata file for immediate point recovery

3. N (the number of threads can be specified, the default is 4) dump thread START TRANSACTION WITH CONSISTENT SNAPSHOT; Turn on read consistent transactions

4. dump non -InnoDB tables, first export the non-transaction engine tables

5. Main thread UNLOCK TABLES After the non-transaction engine backup is completed, release the global read-only lock

6. Dump InnoDB tables, export InnoDB tables based on transactions

7. Transaction end

mydumper less locking mode:

Mydumper uses --less-locking to reduce the lock waiting time. At this time, the execution mechanism of mydumper is roughly

1. Main thread FLUSH TABLES WITH READ LOCK (global lock)

2. Dump thread START TRANSACTION WITH CONSISTENT SNAPSHOT;

3. LL Dump thread LOCK TABLES non-InnoDB (thread internal lock)

4. Main thread UNLOCK TABLES

5. LL Dump thread dump non-InnoDB tables

## 6. LL DUmp thread UNLOCK non-InnoDB

7 , Dump thread dump InnoDB tables

mydumper backup flow chart

The above is the detailed content of Detailed introduction to the principle of mydumper. For more information, please follow other related articles on the PHP Chinese website!