Home Database Mysql Tutorial 缓解MySQL写入压力和主从延迟的尝试

缓解MySQL写入压力和主从延迟的尝试

Jun 07, 2016 pm 03:21 PM

最近单位需要用MySQL存放大量的日志数据,写入压力很大,并且有很大的主从延迟。

最近单位需要用MySQL存放大量的日志数据,写入压力很大,并且有很大的主从延迟。

具体环境如下
MySQL 5.6.14
服务器(单CPU,6核心,12线程 32G内存)
服务器硬盘(共33T,Raid5)

第一个尝试,分散IO
一般我们使用/dbdata挂载点存放数据文件
/data挂载点存放日志文件(redo log file,binlog,relay log等)
这样的好处是将随机IO和顺序IO分开,不形成争用.

缓解MySQL写入压力和主从延迟的尝试

缺点是/data挂载点的IO使用率一般较低.

缓解MySQL写入压力和主从延迟的尝试

当然,这种情况在一般用途的数据库并无大碍.
但是在Insert密集的数据库,数据文件所在的物理设备使用率会一直保持在100%.而日志文件所在的物理设备使用率一般也就是5%左右,甚至更低.

这时,也许可以考虑将一部分表的数据文件移动到日志文件所在的物理设备,以平衡IO资源使用.

缓解MySQL写入压力和主从延迟的尝试

MySQL的datadir在/dbdata挂载点,
police_im_user_mac数据库通过软链接,实际指向/data的挂载点.
这样,会将一部分的随机IO分散到/data挂载点,减轻/dbdata挂载点的压力,并且减小了CPU的等待

缓解MySQL写入压力和主从延迟的尝试


 第二个尝试,一个库一个表
 因为MySQL 5.6的多线程复制是基于数据库的.
而我们每个表的写入压力都很大,所以修改了结构,每个数据库只有一个表.然后启动多线程复制(实际上就是多个SQL线程),并且设置传输压缩

slave_compressed_protocol=on (Master,Slave都需要配置)
slave_parallel_workers=10

第三个尝试,修改参数
innodb_flush_log_at_trx_commit = 0
 innodb_support_xa=0
 sync_binlog=0
增加innodb_buffer_pool_size
将innodb_max_dirty_pages_pct设置的更大
 将redolog file size设置的更大

经过这些修改,写入压力和主从延迟有了很大的缓解,,但是启动slave_parallel_workers,一旦出现SQL线程异常导致的复制中断,恢复之后,可能会出现1062号错误

这是因为多SQL线程的线程同步机制可能有问题,一旦复制中断,现象上看,等同于Slave异常关机.

当然,处理方法也是一样的

本文永久更新链接地址

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Roblox: Bubble Gum Simulator Infinity - How To Get And Use Royal Keys
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Nordhold: Fusion System, Explained
1 months ago By 尊渡假赌尊渡假赌尊渡假赌
Mandragora: Whispers Of The Witch Tree - How To Unlock The Grappling Hook
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Clair Obscur: Expedition 33 - How To Get Perfect Chroma Catalysts
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Java Tutorial
1677
14
PHP Tutorial
1278
29
C# Tutorial
1257
24
MySQL's Role: Databases in Web Applications MySQL's Role: Databases in Web Applications Apr 17, 2025 am 12:23 AM

The main role of MySQL in web applications is to store and manage data. 1.MySQL efficiently processes user information, product catalogs, transaction records and other data. 2. Through SQL query, developers can extract information from the database to generate dynamic content. 3.MySQL works based on the client-server model to ensure acceptable query speed.

MySQL vs. Other Programming Languages: A Comparison MySQL vs. Other Programming Languages: A Comparison Apr 19, 2025 am 12:22 AM

Compared with other programming languages, MySQL is mainly used to store and manage data, while other languages ​​such as Python, Java, and C are used for logical processing and application development. MySQL is known for its high performance, scalability and cross-platform support, suitable for data management needs, while other languages ​​have advantages in their respective fields such as data analytics, enterprise applications, and system programming.

MySQL for Beginners: Getting Started with Database Management MySQL for Beginners: Getting Started with Database Management Apr 18, 2025 am 12:10 AM

The basic operations of MySQL include creating databases, tables, and using SQL to perform CRUD operations on data. 1. Create a database: CREATEDATABASEmy_first_db; 2. Create a table: CREATETABLEbooks(idINTAUTO_INCREMENTPRIMARYKEY, titleVARCHAR(100)NOTNULL, authorVARCHAR(100)NOTNULL, published_yearINT); 3. Insert data: INSERTINTObooks(title, author, published_year)VA

Explain the InnoDB Buffer Pool and its importance for performance. Explain the InnoDB Buffer Pool and its importance for performance. Apr 19, 2025 am 12:24 AM

InnoDBBufferPool reduces disk I/O by caching data and indexing pages, improving database performance. Its working principle includes: 1. Data reading: Read data from BufferPool; 2. Data writing: After modifying the data, write to BufferPool and refresh it to disk regularly; 3. Cache management: Use the LRU algorithm to manage cache pages; 4. Reading mechanism: Load adjacent data pages in advance. By sizing the BufferPool and using multiple instances, database performance can be optimized.

MySQL: Structured Data and Relational Databases MySQL: Structured Data and Relational Databases Apr 18, 2025 am 12:22 AM

MySQL efficiently manages structured data through table structure and SQL query, and implements inter-table relationships through foreign keys. 1. Define the data format and type when creating a table. 2. Use foreign keys to establish relationships between tables. 3. Improve performance through indexing and query optimization. 4. Regularly backup and monitor databases to ensure data security and performance optimization.

Learning MySQL: A Step-by-Step Guide for New Users Learning MySQL: A Step-by-Step Guide for New Users Apr 19, 2025 am 12:19 AM

MySQL is worth learning because it is a powerful open source database management system suitable for data storage, management and analysis. 1) MySQL is a relational database that uses SQL to operate data and is suitable for structured data management. 2) The SQL language is the key to interacting with MySQL and supports CRUD operations. 3) The working principle of MySQL includes client/server architecture, storage engine and query optimizer. 4) Basic usage includes creating databases and tables, and advanced usage involves joining tables using JOIN. 5) Common errors include syntax errors and permission issues, and debugging skills include checking syntax and using EXPLAIN commands. 6) Performance optimization involves the use of indexes, optimization of SQL statements and regular maintenance of databases.

MySQL: Essential Skills for Beginners to Master MySQL: Essential Skills for Beginners to Master Apr 18, 2025 am 12:24 AM

MySQL is suitable for beginners to learn database skills. 1. Install MySQL server and client tools. 2. Understand basic SQL queries, such as SELECT. 3. Master data operations: create tables, insert, update, and delete data. 4. Learn advanced skills: subquery and window functions. 5. Debugging and optimization: Check syntax, use indexes, avoid SELECT*, and use LIMIT.

Explain the purpose of foreign keys in MySQL. Explain the purpose of foreign keys in MySQL. Apr 25, 2025 am 12:17 AM

In MySQL, the function of foreign keys is to establish the relationship between tables and ensure the consistency and integrity of the data. Foreign keys maintain the effectiveness of data through reference integrity checks and cascading operations. Pay attention to performance optimization and avoid common errors when using them.

See all articles