首页 数据库 mysql教程 Issue record: “No thread for socket” about Memcach

Issue record: “No thread for socket” about Memcach

Jun 07, 2016 pm 04:31 PM
for issue record socket thread

文章系本人原创,转载请保持完整性并注明出自《四火的唠叨》 We own a service supplying not-real-time data to the downstream clients. In order to reduce the average latency and the pressure on database, Memcached was imported as the cluster ce

文章系本人原创,转载请保持完整性并注明出自《四火的唠叨》

Issue record: No thread for socket about MemcachedWe own a service supplying not-real-time data to the downstream clients. In order to reduce the average latency and the pressure on database, Memcached was imported as the cluster center cache, and our service consumed it. I encountered an issue when Memcached was firstly deployed on production environment:

[WARN] (Acceptor ServerSocket[addr=/0.0.0.0,port=0,localport=3961]) amazon.platform.servertoolkit.http.jetty.JettyServerManager: No thread for Socket[addr=/10.0.77.98,port=10545,localport=3961] : [Acceptor ServerSocket[addr=/0.0.0.0,port=0,localport=3961]]org.mortbay.util.ThreadPool.run(ThreadPool.java:350)
登录后复制

The server totally gave no response. So I checked the connection number:

netstat -an | grep ESTABLISHED | wc –l
54
登录后复制

And process number:

ps -ef|wc -l
564
登录后复制

The both seemed normal. One teammate remaindered me I should have a look at the threads of Jetty:

ps uH p 31636 | wc -l
登录后复制

I couldn’t find anything abnormal. I knew I might increase the thread up limit but that would not have any help to resolve the issue, at most it might only delay the issue’s happening.

Actually, this issue was a little more complex than we thought indeed.

The threads were running out from this morning, and client could not connect to our service. I took a snapshot (jstack -F -l 31636) for the stacks in JVM, finding that almost all the threads were hung reading from cache server:

Thread 19795: (state = IN_NATIVE)
 - sun.nio.ch.FileDispatcher.read0(java.io.FileDescriptor, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int) @bci=4, line=21 (Compiled frame)
 - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher, java.lang.Object) @bci=116, line=198 (Compiled frame)
 - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher, java.lang.Object) @bci=31, line=166 (Compiled frame)
 - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=101, line=245 (Compiled frame)
 - com.schooner.MemCached.SockInputStream.readFromChannel() @bci=25 (Interpreted frame)
 - com.schooner.MemCached.SockInputStream.read() @bci=31 (Compiled frame)
 - com.schooner.MemCached.SockInputStream.getLine() @bci=9 (Compiled frame)
 - com.schooner.MemCached.AscIIClient.get(java.lang.String, java.lang.String, java.lang.Integer, boolean) @bci=583 (Compiled frame)
 - com.schooner.MemCached.AscIIClient.get(java.lang.String, java.lang.Integer) @bci=6 (Compiled frame)
 - com.schooner.MemCached.AscIIClient.get(java.lang.String) @bci=3 (Compiled frame)
 - com.danga.MemCached.MemCachedClient.get(java.lang.String) @bci=5 (Compiled frame)
 - amazon.forecast.util.cache.memcached.BSFMemcachedClient.get(java.lang.String, java.lang.String) @bci=14, line=80 (Compiled frame)
 - amazon.forecast.distribution.Forecaster.getForecastDistributions(amazon.forecast.distribution.IForecastDistributionProvider, amazon.forecast.distribution.ForecastRequest)
 ...
登录后复制

On my understanding there should be some error on the connection from the client to cache server, then I checked the cache server but found it’s correctly and stably running, and of course, the network was good. We invoked?Memcached service manually several times thru telnet, it did work normally.?I took several snapshots in series again and got the same result — it seemed all the threads were being hung on the “read0″ method.

However, after restarting the client the issue disappeared. By then nothing could explain why all the threads were being hung reading from cache server. One of my teammate told me if the tcp connection was dropped and the client was trying to read, it would not fail until the retransmission time out. The retransmission would happen in a back-off manner, and it would take long time for “read” to fail. Besides,?It didn’t matter whether or not we were able to connect to the cache server. The exact tcp connection was broken on client, and it didn’t detect and was still using the broken socket to read.

Finally, we were still unclear about the root cause, but found a loop discussing about this issue:?

https://github.com/gwhalin/Memcached-Java-Client/issues/20

A working solution is to set socket timeout to release the connection in time if unforeseen issue happens.

文章系本人原创,转载请保持完整性并注明出自《四火的唠叨》

你可能也喜欢:

本站声明
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系admin@php.cn

热AI工具

Undresser.AI Undress

Undresser.AI Undress

人工智能驱动的应用程序,用于创建逼真的裸体照片

AI Clothes Remover

AI Clothes Remover

用于从照片中去除衣服的在线人工智能工具。

Undress AI Tool

Undress AI Tool

免费脱衣服图片

Clothoff.io

Clothoff.io

AI脱衣机

AI Hentai Generator

AI Hentai Generator

免费生成ai无尽的。

热门文章

R.E.P.O.能量晶体解释及其做什么(黄色晶体)
2 周前 By 尊渡假赌尊渡假赌尊渡假赌
仓库:如何复兴队友
4 周前 By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island冒险:如何获得巨型种子
4 周前 By 尊渡假赌尊渡假赌尊渡假赌

热工具

记事本++7.3.1

记事本++7.3.1

好用且免费的代码编辑器

SublimeText3汉化版

SublimeText3汉化版

中文版,非常好用

禅工作室 13.0.1

禅工作室 13.0.1

功能强大的PHP集成开发环境

Dreamweaver CS6

Dreamweaver CS6

视觉化网页开发工具

SublimeText3 Mac版

SublimeText3 Mac版

神级代码编辑软件(SublimeText3)

解决kernel_security_check_failure蓝屏的17种方法 解决kernel_security_check_failure蓝屏的17种方法 Feb 12, 2024 pm 08:51 PM

Kernelsecuritycheckfailure(内核检查失败)就是一个比较常见的停止代码类型,可蓝屏错误出现不管是什么原因都让很多的有用户们十分的苦恼,下面就让本站来为用户们来仔细的介绍一下17种解决方法吧。kernel_security_check_failure蓝屏的17种解决方法方法1:移除全部外部设备当您使用的任何外部设备与您的Windows版本不兼容时,则可能会发生Kernelsecuritycheckfailure蓝屏错误。为此,您需要在尝试重新启动计算机之前拔下全部外部设备。

Python的socket与socketserver怎么使用 Python的socket与socketserver怎么使用 May 28, 2023 pm 08:10 PM

一、基于TCP协议的socket套接字编程1、套接字工作流程先从服务器端说起。服务器端先初始化Socket,然后与端口绑定(bind),对端口进行监听(listen),调用accept阻塞,等待客户端连接。在这时如果有个客户端初始化一个Socket,然后连接服务器(connect),如果连接成功,这时客户端与服务器端的连接就建立了。客户端发送数据请求,服务器端接收请求并处理请求,然后把回应数据发送给客户端,客户端读取数据,最后关闭连接,一次交互结束,使用以下Python代码实现:importso

PHP+Socket系列之IO多路复用及实现web服务器 PHP+Socket系列之IO多路复用及实现web服务器 Feb 02, 2023 pm 01:43 PM

本篇文章给大家带来了关于php+socket的相关知识,其中主要介绍了IO多路复用,以及php+socket如何实现web服务器?感兴趣的朋友下面一起来看一下,希望对大家有帮助。

怎么使用Spring Boot+Vue实现Socket通知推送 怎么使用Spring Boot+Vue实现Socket通知推送 May 27, 2023 am 08:47 AM

SpringBoot端第一步,引入依赖首先我们需要引入WebSocket所需的依赖,以及处理输出格式的依赖com.alibabafastjson1.2.73org.springframework.bootspring-boot-starter-websocket第二步,创建WebSocket配置类importorg.springframework.context.annotation.Bean;importorg.springframework.context.annotation.Config

Win10如何卸载Skype for Business?电脑上的skype怎么彻底卸载方法 Win10如何卸载Skype for Business?电脑上的skype怎么彻底卸载方法 Feb 13, 2024 pm 12:30 PM

Win10skype可以卸载吗是很多用户们都想知道的一个问题,因为很多的用户们发现自己电脑上的默认程序上有这个应用,担心删除后会影响到系统的运行,下面就让本站来为用户们来仔细的介绍一下Win10如何卸载SkypeforBusiness吧。Win10如何卸载SkypeforBusiness1、在电脑桌面点击Windows图标,再点击设置图标进入。2、点击“应用”。3、在搜索框中输入“Skype”,点击选中找到的结果。4、点击“卸载”。5

php socket无法连接怎么办 php socket无法连接怎么办 Nov 09, 2022 am 10:34 AM

php socket无法连接的解决办法:1、检查php是否开启socket扩展;2、打开php.ini文件,检查“php_sockets.dll”是否被加载;3、取消“php_sockets.dll”的注释状态即可。

PHP+Socket系列之实现客户端与服务端数据传输 PHP+Socket系列之实现客户端与服务端数据传输 Feb 02, 2023 am 11:35 AM

本篇文章给大家带来了关于php+socket的相关知识,其中主要介绍了什么是socket?php+socket如何实现客户端与服务端数据传输?感兴趣的朋友下面一起来看一下,希望对大家有帮助。

Linux中如何让所有用户登录时打印指定提示信息 Linux中如何让所有用户登录时打印指定提示信息 Feb 19, 2024 pm 05:12 PM

在Linux系统中,通过配置文件和脚本,可以实现让所有用户登录时显示指定的提示信息。接下来,我们将介绍几种常用的实现方法。方法一:修改/etc/issue文件打开终端,使用文本编辑器(如vi或nano)以root权限编辑/etc/issue文件。sudovi/etc/issue在文件末尾添加你想要显示的提示信息,例如:WelcometoMyLinuxSystem!Pleasebeawarethatallactivitiesaremonitored.保存并关闭文件。现在,当用户登录时,系统将显示/

See all articles