Home Backend Development PHP Tutorial 一个Lararel队列引发的报警

一个Lararel队列引发的报警

Jun 20, 2016 pm 12:35 PM

一台服务器报警了,内存占用过高,奇怪的是集群里其它的服务器都没问题。不过从以往的经验来看:每一个匪夷所思的问题背后,都隐藏着一个啼笑皆非的答案。

首先通过「free -m」确认一下内存情况,发现用掉了 6893M,还剩 976M:

free

然后通过「top」查看一下哪些进程占用内存多,通过「shift + m」按内存排序:

top

虽然通过 free 命令我们能确认系统可用内存不足,但是通过 top 命令我们却没有发现有内存占用大户的进程,那么内存到底都去哪里了呢?

开头我们提到过,集群里只有一台服务器有问题,其它服务器皆正常,于是我们比较了一下问题服务器和正常服务器的进程列表,结果发现问题服务器多了几个进程:

/usr/local/bin/php artisan queue:listen

/usr/local/bin/php artisan queue:work

经过确认,它们是 Laravel 队列,虽然直觉告诉我们问题与其有关联,但是进程本身并没有占用多少内存,在不能立刻确诊原因的情况下,我们用排除法把队列换到另外一台正常的服务器上看看会不会重现问题,过了一会,果然再次出现同样问题。

既然 free,top 之类的命令不能确认内存的去向,那么我们不妨看看「meminfo」:

meminfo

如上图所示,大量内存被 Slab 消耗了,更进一步讲是被 SReclaimable 消耗了,也就是说内存被一些可回收的 Slab 消耗了,更进一步的信息可以通过「slabtop」获取:

slabtop

大量内存被 dentry 消耗了,如果你也跟我一样,搞不清楚它意味着什么,搜索吧,能翻墙用 Google,不能翻墙用 AOL,反正别用百度,我找到如下介绍:

  • Linux服务器Cache占用过多内存导致系统内存不足问题的排查解决

  • Linux服务器Cache占用过多内存导致系统内存不足问题的排查解决(续)

简而言之,内存 dentry 里缓存了最近访问过的文件信息,如果频繁的操作大量文件,那么 dentry 就会不断的增加,于是问题就变为确认 Laravel 队列有没有类似问题。

前面提到过,Laravel 队列有一个 listen 进程,还有一个 work 进程,从名字我们就能判断出来,前者是主进程,后者是子进程,子进程是干活的进程,可是当我直接 strace 跟踪子进程的时候,却提示我子进程不存在,进一步调试发现,原来子进程会不断重启!

既然我们不好直接跟踪子进程,那么不妨从父进程入手跟踪子进程的文件操作:

shell> strace -f -e trace=open,close,stat,unlink -p $(    ps aux | grep "[q]ueue:listen" | awk '{print $2}')
Copy after login

可惜 Laravel 本身号称是巨匠框架,依赖一坨一坨的文件,所以跟踪结果里充斥着大量框架文件本身正常的 open,close,stat 操作,改为只跟踪 unlik 操作试试:

shell> strace -f -e trace=unlink -p $(    ps aux | grep "[q]ueue:listen" | awk '{print $2}')
Copy after login

发现 Laravel 队列频繁的执行删除文件操作,每重启一次子进程就执行一次删除:

unlink(“/tmp/.ZendSem.aXaa3Z”)

unlink(“/tmp/.ZendSem.teQG0Y”)

unlink(“/tmp/.ZendSem.Bn3ien”)

unlink(“/tmp/.ZendSem.V4s8RX”)

unlink(“/tmp/.ZendSem.PnNuTN”)

因为临时文件的名字各不相同,所以消耗了大量的 dentry 缓存。查阅 Laravel 队列的 文档 ,发现 Laravel 队列实际上也提供了不重启的守护进程模式,这样就不会频繁创建大量临时文件,进而也就不会消耗大量的 dentry 缓存,推荐使用。

如果频繁创建大量临时文件的情况无法避免,那么按照 Linux 文档 的描述,我们可以通过设置 drop_caches 为 2 来删除可回收的 slab(包括 dentries 和 inodes),较粗野:

shell> echo 2 > /proc/sys/vm/drop_caches
Copy after login

此外还可以通过设置 vfs_cache_pressure 大于 100 来增加回收概率,较温柔:

shell> echo 1000 > /proc/sys/vm/vfs_cache_pressure
Copy after login

从测试结果看,vfs_cache_pressure 的作用有限,当然也可能是我姿势不对。还有一些资料说 min_free_kbytes 也可以解决此类问题,不过鉴于这个参数有一定危险性,我建议大家别乱改,这里就不多说了,有兴趣的可以自行查阅相关资料。

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

11 Best PHP URL Shortener Scripts (Free and Premium) 11 Best PHP URL Shortener Scripts (Free and Premium) Mar 03, 2025 am 10:49 AM

Long URLs, often cluttered with keywords and tracking parameters, can deter visitors. A URL shortening script offers a solution, creating concise links ideal for social media and other platforms. These scripts are valuable for individual websites a

Introduction to the Instagram API Introduction to the Instagram API Mar 02, 2025 am 09:32 AM

Following its high-profile acquisition by Facebook in 2012, Instagram adopted two sets of APIs for third-party use. These are the Instagram Graph API and the Instagram Basic Display API.As a developer building an app that requires information from a

Working with Flash Session Data in Laravel Working with Flash Session Data in Laravel Mar 12, 2025 pm 05:08 PM

Laravel simplifies handling temporary session data using its intuitive flash methods. This is perfect for displaying brief messages, alerts, or notifications within your application. Data persists only for the subsequent request by default: $request-

Build a React App With a Laravel Back End: Part 2, React Build a React App With a Laravel Back End: Part 2, React Mar 04, 2025 am 09:33 AM

This is the second and final part of the series on building a React application with a Laravel back-end. In the first part of the series, we created a RESTful API using Laravel for a basic product-listing application. In this tutorial, we will be dev

Simplified HTTP Response Mocking in Laravel Tests Simplified HTTP Response Mocking in Laravel Tests Mar 12, 2025 pm 05:09 PM

Laravel provides concise HTTP response simulation syntax, simplifying HTTP interaction testing. This approach significantly reduces code redundancy while making your test simulation more intuitive. The basic implementation provides a variety of response type shortcuts: use Illuminate\Support\Facades\Http; Http::fake([ 'google.com' => 'Hello World', 'github.com' => ['foo' => 'bar'], 'forge.laravel.com' =>

cURL in PHP: How to Use the PHP cURL Extension in REST APIs cURL in PHP: How to Use the PHP cURL Extension in REST APIs Mar 14, 2025 am 11:42 AM

The PHP Client URL (cURL) extension is a powerful tool for developers, enabling seamless interaction with remote servers and REST APIs. By leveraging libcurl, a well-respected multi-protocol file transfer library, PHP cURL facilitates efficient execution of various network protocols, including HTTP, HTTPS, and FTP. This extension offers granular control over HTTP requests, supports multiple concurrent operations, and provides built-in security features.

12 Best PHP Chat Scripts on CodeCanyon 12 Best PHP Chat Scripts on CodeCanyon Mar 13, 2025 pm 12:08 PM

Do you want to provide real-time, instant solutions to your customers' most pressing problems? Live chat lets you have real-time conversations with customers and resolve their problems instantly. It allows you to provide faster service to your custom

Announcement of 2025 PHP Situation Survey Announcement of 2025 PHP Situation Survey Mar 03, 2025 pm 04:20 PM

The 2025 PHP Landscape Survey investigates current PHP development trends. It explores framework usage, deployment methods, and challenges, aiming to provide insights for developers and businesses. The survey anticipates growth in modern PHP versio

See all articles