Issue record: “No thread for socket” about Memcach
文章系本人原创，转载请保持完整性并注明出自《四火的唠叨》 We own a service supplying not-real-time data to the downstream clients. In order to reduce the average latency and the pressure on database, Memcached was imported as the cluster ce
文章系本人原创，转载请保持完整性并注明出自《四火的唠叨》
We own a service supplying not-real-time data to the downstream clients. In order to reduce the average latency and the pressure on database, Memcached was imported as the cluster center cache, and our service consumed it. I encountered an issue when Memcached was firstly deployed on production environment:
[WARN] (Acceptor ServerSocket[addr=/0.0.0.0,port=0,localport=3961]) amazon.platform.servertoolkit.http.jetty.JettyServerManager: No thread for Socket[addr=/10.0.77.98,port=10545,localport=3961] : [Acceptor ServerSocket[addr=/0.0.0.0,port=0,localport=3961]]org.mortbay.util.ThreadPool.run(ThreadPool.java:350)
The server totally gave no response. So I checked the connection number:
netstat -an | grep ESTABLISHED | wc –l 54
And process number:
ps -ef|wc -l 564
The both seemed normal. One teammate remaindered me I should have a look at the threads of Jetty:
ps uH p 31636 | wc -l
I couldn’t find anything abnormal. I knew I might increase the thread up limit but that would not have any help to resolve the issue, at most it might only delay the issue’s happening.
Actually, this issue was a little more complex than we thought indeed.
The threads were running out from this morning, and client could not connect to our service. I took a snapshot (jstack -F -l 31636) for the stacks in JVM, finding that almost all the threads were hung reading from cache server:
Thread 19795: (state = IN_NATIVE) - sun.nio.ch.FileDispatcher.read0(java.io.FileDescriptor, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int) @bci=4, line=21 (Compiled frame) - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher, java.lang.Object) @bci=116, line=198 (Compiled frame) - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher, java.lang.Object) @bci=31, line=166 (Compiled frame) - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=101, line=245 (Compiled frame) - com.schooner.MemCached.SockInputStream.readFromChannel() @bci=25 (Interpreted frame) - com.schooner.MemCached.SockInputStream.read() @bci=31 (Compiled frame) - com.schooner.MemCached.SockInputStream.getLine() @bci=9 (Compiled frame) - com.schooner.MemCached.AscIIClient.get(java.lang.String, java.lang.String, java.lang.Integer, boolean) @bci=583 (Compiled frame) - com.schooner.MemCached.AscIIClient.get(java.lang.String, java.lang.Integer) @bci=6 (Compiled frame) - com.schooner.MemCached.AscIIClient.get(java.lang.String) @bci=3 (Compiled frame) - com.danga.MemCached.MemCachedClient.get(java.lang.String) @bci=5 (Compiled frame) - amazon.forecast.util.cache.memcached.BSFMemcachedClient.get(java.lang.String, java.lang.String) @bci=14, line=80 (Compiled frame) - amazon.forecast.distribution.Forecaster.getForecastDistributions(amazon.forecast.distribution.IForecastDistributionProvider, amazon.forecast.distribution.ForecastRequest) ...
On my understanding there should be some error on the connection from the client to cache server, then I checked the cache server but found it’s correctly and stably running, and of course, the network was good. We invoked?Memcached service manually several times thru telnet, it did work normally.?I took several snapshots in series again and got the same result — it seemed all the threads were being hung on the “read0″ method.
However, after restarting the client the issue disappeared. By then nothing could explain why all the threads were being hung reading from cache server. One of my teammate told me if the tcp connection was dropped and the client was trying to read, it would not fail until the retransmission time out. The retransmission would happen in a back-off manner, and it would take long time for “read” to fail. Besides,?It didn’t matter whether or not we were able to connect to the cache server. The exact tcp connection was broken on client, and it didn’t detect and was still using the broken socket to read.
Finally, we were still unclear about the root cause, but found a loop discussing about this issue:?
https://github.com/gwhalin/Memcached-Java-Client/issues/20
A working solution is to set socket timeout to release the connection in time if unforeseen issue happens.
文章系本人原创，转载请保持完整性并注明出自《四火的唠叨》
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
17 ways to solve the kernel_security_check_failure blue screen
Feb 12, 2024 pm 08:51 PM
Kernelsecuritycheckfailure (kernel check failure) is a relatively common type of stop code. However, no matter what the reason is, the blue screen error causes many users to be very distressed. Let this site carefully introduce 17 types to users. Solution. 17 solutions to kernel_security_check_failure blue screen Method 1: Remove all external devices When any external device you are using is incompatible with your version of Windows, the Kernelsecuritycheckfailure blue screen error may occur. To do this, you need to unplug all external devices before trying to restart your computer.
How to use Python's socket and socketserver
May 28, 2023 pm 08:10 PM
1. Socket programming based on TCP protocol 1. The socket workflow starts with the server side. The server first initializes the Socket, then binds to the port, listens to the port, calls accept to block, and waits for the client to connect. At this time, if a client initializes a Socket and then connects to the server (connect), if the connection is successful, the connection between the client and the server is established. The client sends a data request, the server receives the request and processes the request, then sends the response data to the client, the client reads the data, and finally closes the connection. An interaction ends. Use the following Python code to implement it: importso
IO multiplexing of PHP+Socket series and implementation of web server
Feb 02, 2023 pm 01:43 PM
This article brings you relevant knowledge about php+socket, which mainly introduces IO multiplexing and how php+socket implements web server? Friends who are interested can take a look below. I hope it will be helpful to everyone.
How to use Spring Boot+Vue to implement Socket notification push
May 27, 2023 am 08:47 AM
The first step on the SpringBoot side is to introduce dependencies. First we need to introduce the dependencies required for WebSocket, as well as the dependencies for processing the output format com.alibabafastjson1.2.73org.springframework.bootspring-boot-starter-websocket. The second step is to create the WebSocket configuration class importorg. springframework.context.annotation.Bean;importorg.springframework.context.annotation.Config
How to uninstall Skype for Business on Win10? How to completely uninstall Skype on your computer
Feb 13, 2024 pm 12:30 PM
Can Win10 skype be uninstalled? This is a question that many users want to know, because many users find that this application is included in the default program on their computers, and they are worried that deleting it will affect the operation of the system. Let this website help users Let’s take a closer look at how to uninstall Skype for Business in Win10. How to uninstall Skype for Business in Win10 1. Click the Windows icon on the computer desktop, and then click the settings icon to enter. 2. Click "Apply". 3. Enter "Skype" in the search box and click to select the found result. 4. Click "Uninstall". 5
What to do if php socket cannot connect
Nov 09, 2022 am 10:34 AM
Solution to the problem that the php socket cannot be connected: 1. Check whether the socket extension is enabled in php; 2. Open the php.ini file and check whether "php_sockets.dll" is loaded; 3. Uncomment "php_sockets.dll".
How to let all users print specified prompt information when logging in in Linux
Feb 19, 2024 pm 05:12 PM
In Linux systems, through configuration files and scripts, you can display specified prompt information when all users log in. Next, we will introduce several commonly used implementation methods. Method 1: Modify the /etc/issue file. Open the terminal and use a text editor (such as vi or nano) to edit the /etc/issue file with root permissions. sudovi/etc/issue Add the prompt message you want to display at the end of the file, for example: Welcome to MyLinuxSystem! Please beaware that all activities are monitored. Save and close the file. Now when the user logs in, the system will display /
Research on real-time file transfer technology using PHP and Socket
Jun 28, 2023 am 09:11 AM
With the development of the Internet, file transfer has become an indispensable part of people's daily work and entertainment. However, traditional file transfer methods such as email attachments or file sharing websites have certain limitations and cannot meet the needs of real-time and security. Therefore, using PHP and Socket technology to achieve real-time file transfer has become a new solution. This article will introduce the technical principles, advantages and application scenarios of using PHP and Socket technology to achieve real-time file transfer, and demonstrate the implementation method of this technology through specific cases. technology
