Slave memory leak and trigger oom-killer_MySQL
Bug Description
We have this problem:
we have set innodb_buffer_pool =80GB on both master and slave , master offer usually workload,
but slave with nothing workload except these slave threads, But with the memory consumption is increasing ,
after a few days , it uses about 120GB of memory(RES) and sometime the machine starts swapping out.
and in the /var/log/message(loged :Out of memory: Kill process 43817 (mysqld) score 847 or sacrifice
childKilled process 43817, UID 500, (mysqld) total-vm:126723628kB, anon-rss:125497616kB, file-rss:3388kB)
We have the exact version of PerconaServer on the master (with production workload) and it doesn't
have this problem
This problem is the same for a few recent versions of Percona server, not just the last one.
we do not user MyISAM tables and just use Innodb Tables;TokuDB is not installed.
Other Information:
this one just OOM last night T-T:
xiean@(none) 11:02> \s--------------mysql Ver 14.14 Distrib 5.6.23-72.1, for Linux (x86_64)
using EditLine wrapper
Connection id: 252482Current database:Current user: xiean@localhostSSL: Not in useCurrent pager:
stdoutUsing outfile: ''Using delimiter: ;Server version: 5.6.23-72.1-log Source distributionProtocol
version: 10Connection: Localhost via UNIX socketServer characterset: utf8Db characterset:
utf8Client characterset: utf8Conn. characterset: utf8UNIX socket: /data/mysql/3307/var/run/mysql.sockUptime:
10 hours 32 min 9 sec
Threads: 6 Questions: 5888709 Slow queries: 312 Opens: 200 Flush tables: 1 Open tables: 193 Queries per
second avg: 155.256--------------
CentOS release 6.5 (Final)Linux xxxxxxx 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09
UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
my.cnf-----------------------------------------------------[mysqld]
skip-external-lockingskip-name-resolvedefault-storage-engine = InnoDBcharacter-set-client-handshake =
FALSEcharacter-set-server = utf8collation-server = utf8_unicode_ciinit_connect='set names utf8'
connect_timeout = 20back_log = 600log_bin_trust_function_creators = 1
binlog_format = rowmax_binlog_size = 512Mbinlog_cache_size = 2Mexpire-logs-days = 7
slave-net-timeout = 20log_slave_updates = 1relay_log_recovery = 1sync_master_info = 1sync_relay_log_info =
1master_info_repository = TABLErelay_log_info_repository = TABLE
slow_query_log = 1long_query_time = 1
max_connections = 2000max_user_connections = 1000max_connect_errors = 100000key_buffer_size =
64Mmax_allowed_packet = 512Mtable_open_cache = 6144table_definition_cache = 4096sort_buffer_size =
512Kread_buffer_size = 512Kread_rnd_buffer_size = 512kjoin_buffer_size = 512Ktmp_table_size =
64Mmax_heap_table_size = 128Mquery_cache_type = 0query_cache_size = 0bulk_insert_buffer_size =
32Mthread_cache_size = 64thread_stack = 256K
innodb_data_file_path = ibdata1:1024M:autoextendinnodb_buffer_pool_size = 80Ginnodb_buffer_pool_instances =
8innodb_additional_mem_pool_size = 16Minnodb_use_sys_malloc = 0innodb_log_file_size =
8196Minnodb_log_buffer_size = 64Minnodb_log_files_in_group = 3innodb_lock_wait_timeout =
30innodb_spin_wait_delay = 96innodb_thread_concurrency = 0innodb_max_dirty_pages_pct =
75innodb_support_xa = 0innodb_flush_log_at_trx_commit = 2innodb_flush_method = O_DIRECTinnodb_read_io_threads =
16innodb_write_io_threads = 16innodb_io_capacity = 2000innodb_flush_neighbors = 0innodb_page_size =
4Kinnodb_log_block_size = 4096innodb_file_per_table = 1innodb_file_format = Barracudainnodb_file_format_max =
Barracudainnodb_strict_mode = 1innodb_purge_threads = 2innodb_purge_batch_size = 32innodb_old_blocks_pct =
37innodb_old_blocks_time = 1000innodb_change_buffering = allinnodb_stats_on_metadata = off
sync_binlog = 0
innodb_buffer_pool_dump_at_shutdown = 1innodb_buffer_pool_load_at_startup = 1
show engine innodb status\G
=====================================2016-03-22 11:08:07 7f8496af9700 INNODB MONITOR OUTPUT====================
=================Per second averages calculated from the last 3 seconds-----------------BACKGROUND THREAD------
-----------srv_master_thread loops: 35735 srv_active, 0 srv_shutdown, 2389 srv_idlesrv_master_thread log flush
and writes: 38124----------SEMAPHORES----------OS WAIT ARRAY INFO: reservation count 27319OS WAIT ARRAY INFO:
signal count 694283Mutex spin waits 13779381, rounds 15836207, OS waits 12843RW-shared spins 3653866, rounds
16984184, OS waits 11004RW-excl spins 562597, rounds 1648229, OS waits 2980Spin rounds per wait: 1.15 mutex,
4.65 RW-shared, 2.93 RW-excl
....................................................
----------------------BUFFER POOL AND MEMORY----------------------
Total memory allocated 107356939316; in additional pool allocated 8441088Total memory allocated by
read views 496Internal hash tables (constant factor + variable factor)
Adaptive hash index 2118609544 (1359985288 + 758624256)
Page hash 42500456 (buffer pool 0 only)
Dictionary cache 340576340 (339997712 + 578628)
File system 867904 (812272 + 55632)
Lock system 849995224 (849992296 + 2928)
Recovery system 0 (0 + 0)Dictionary memory allocated 578628Buffer pool size 20971512Buffer pool size,
bytes 85899313152Free buffers 793287Database pages 19993014Old database pages
7380256Modified db pages 857786Pending reads 0Pending writes: LRU 0, flush list 0,
single page 0Pages made young 1500483, not young 7919383.00 youngs/s, 3.67 non-youngs/sPages read 19777369,
created 365977, written 17055406.33 reads/s, 9.00 creates/s, 255.25 writes/sBuffer pool hit rate 1000 / 1000,
young-making rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s, evicted without access 0.00/s,
Random read ahead 0.00/sLRU len: 19993014, unzip_LRU len: 0I/O sum[36104]:cur[1320], unzip sum[0]:cur[0]
----------------------INDIVIDUAL BUFFER POOL INFO-------------------------
BUFFER POOL 0Buffer pool size 2621439Buffer pool size, bytes 10737414144Free buffers 1024Database
pages 2597173Old database pages 958702Modified db pages 110539Pending reads 0Pending writes: LRU 0,
flush list 0, single page 0Pages made young 497074, not young 3051540.67 youngs/s, 3.67 non-youngs/sPages
read 2603703, created 46719, written 2711371.00 reads/s, 0.33 creates/s, 17.33 writes/sBuffer pool hit
rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s, evicted without
access 0.00/s, Random read ahead 0.00/sLRU len: 2597173, unzip_LRU len: 0I/O sum[4513]:cur[165],
unzip sum[0]:cur[0]---BUFFER POOL 1Buffer pool size 2621439Buffer pool size, bytes 10737414144Free
buffers 1024Database pages 2597316Old database pages 958754Modified db pages 115355Pending reads
0Pending writes: LRU 0, flush list 0, single page 0Pages made young 496041, not young 2381371.00 youngs/s,
0.00 non-youngs/sPages read 2593888, created 50425, written 2643540.33 reads/s, 3.00 creates/s, 29.66
writes/sBuffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s,
evicted without access 0.00/s, Random read ahead 0.00/sLRU len: 2597316, unzip_LRU len: 0I/O
sum[4513]:cur[165], unzip sum[0]:cur[0]---BUFFER POOL 2Buffer pool size 2621439Buffer pool size,
bytes 10737414144Free buffers 1024Database pages 2597259Old database pages 958733Modified db pages
115748Pending reads 0Pending writes: LRU 0, flush list 0, single page 0Pages made young 496770,
not young 2486471.00 youngs/s, 0.00 non-youngs/sPages read 2595651, created 51694,
written 2406831.00 reads/s, 0.00 creates/s, 35.99 writes/sBuffer pool hit rate 1000 / 1000,
young-making rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s, evicted without access 0.00/s,
Random read ahead 0.00/sLRU len: 2597259, unzip_LRU len: 0I/O sum[4513]:cur[165],
unzip sum[0]:cur[0]---BUFFER POOL 3Buffer pool size 2621439Buffer pool size, bytes
10737414144Free buffers 117082Database pages 2481220Old database pages 915938Modified db pages
112864Pending reads 0Pending writes: LRU 0, flush list 0, single page 0Pages made young 2305,
not young 00.00 youngs/s, 0.00 non-youngs/sPages read 2429753, created 51467,
written 2489670.00 reads/s, 0.00 creates/s, 40.99 writes/sBuffer pool hit rate 1000 / 1000,
young-making rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s, evicted without access 0.00/s,
Random read ahead 0.00/sLRU len: 2481220, unzip_LRU len: 0I/O sum[4513]:cur[165],
unzip sum[0]:cur[0]---BUFFER POOL 4Buffer pool size 2621439Buffer pool size,
bytes 10737414144Free buffers 207591Database pages 2390638Old database pages
882501Modified db pages 103991Pending reads 0Pending writes: LRU 0, flush list 0, single page
0Pages made young 2163, not young 00.33 youngs/s, 0.00 non-youngs/sPages read 2345049, created 45589,
written 2144341.00 reads/s, 1.00 creates/s, 32.99 writes/sBuffer pool hit rate 1000 / 1000, young-making
rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/sLRU
len: 2390638, unzip_LRU len: 0I/O sum[4513]:cur[165], unzip sum[0]:cur[0]---BUFFER POOL 5Buffer pool size
2621439Buffer pool size, bytes 10737414144Free buffers 202652Database pages 2395634Old database pages
884332Modified db pages 99353Pending reads 0Pending writes: LRU 0, flush list 0, single page 0Pages made
young 2061, not young 00.00 youngs/s, 0.00 non-youngs/sPages read 2354114, created 41520,
written 1473470.67 reads/s, 3.67 creates/s, 30.99 writes/sBuffer pool hit rate 1000 / 1000, young-making
rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/sLRU
len: 2395634, unzip_LRU len: 0I/O sum[4513]:cur[165], unzip sum[0]:cur[0]---BUFFER POOL 6Buffer pool size
2621439Buffer pool size, bytes 10737414144Free buffers 209285Database pages 2389036Old database pages
881910Modified db pages 96899Pending reads 0Pending writes: LRU 0, flush list 0, single page 0Pages
made young 1913, not young 00.00 youngs/s, 0.00 non-youngs/sPages read 2349930, created 39106, written
1501741.33 reads/s, 0.67 creates/s, 38.65 writes/sBuffer pool hit rate 1000 / 1000, young-making rate 0 /
1000 not 0 / 1000Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/sLRU
len: 2389036, unzip_LRU len: 0I/O sum[4513]:cur[165], unzip sum[0]:cur[0]---BUFFER POOL 7Buffer pool
size 2621439Buffer pool size, bytes 10737414144Free buffers 53605Database pages 2544738Old database pages
939386Modified db pages 103037Pending reads 0Pending writes: LRU 0, flush list 0, single page 0Pages made
young 2156, not young 00.00 youngs/s, 0.00 non-youngs/sPages read 2505281, created 39457, written 1684441.00
reads/s, 0.33 creates/s, 28.66 writes/sBuffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 /
1000Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/sLRU len: 2544738, unzip_LRU
len: 0I/O sum[4513]:cur[165], unzip sum[0]:cur[0]
--------------ROW OPERATIONS--------------
0 queries inside InnoDB,
0 queries in queue1 read views open inside InnoDB0 RW transactions
active inside InnoDB0 RO transactions active inside InnoDB0 out of 1000 descriptors
used---OLDEST VIEW---Normal read viewRead view low limit trx n:o 4402678824Read view
up limit trx id 4402678824Read view low limit trx id 4402678824Read view inpidually stored trx ids:
-----------------Main thread process no. 70096, id 140207583508224,
state: sleepingNumber of rows inserted 1347486, updated 6484099, deleted 505223,
read 4665086631106.63 inserts/s, 1017.66 updates/s, 0.00 deletes/s, 1682192.60 reads/s
----------------------------END OF INNODB MONITOR OUTPUT============================以上就是Slave memory leak and trigger oom-killer_MySQL的內容，更多相關內容請關注PHPcn網絡（www.php.cn）！
