hadoop数据节点通信异常
在前几天，我们的 hadoop 集群很不稳定。经常会有1个数据节点挂掉。使用jps查看，tasktracker和datanode均正常，没有crash掉。查看日志： org.apache. hadoop .ipc.Client: Retrying connect to server : xxxxx/192.168.0.xxxx:9001. Already tried 9 time(s
在前几天，我们的hadoop集群很不稳定。经常会有1个数据节点挂掉。使用jps查看，tasktracker和datanode均正常，没有crash掉。查看日志：
org.apache.hadoop.ipc.Client: Retrying connect to server: xxxxx/192.168.0.xxxx:9001. Already tried 9 time(s).
org.apache.hadoop.ipc.Client: Retrying connect to server: xxxxx/192.168.0.xxxx:9001. Already tried 8 time(s).
org.apache.hadoop.ipc.Client: Retrying connect to server: xxxxx/192.168.0.xxxx:9001. Already tried 7 time(s).
org.apache.hadoop.ipc.Client: Retrying connect to server: xxxxx/192.168.0.xxxx:9001. Already tried 6 time(s).
org.apache.hadoop.ipc.Client: Retrying connect to server: xxxxx/192.168.0.xxxx:9001. Already tried 5 time(s).
就是与namenode无法正常通信。
从集群的角度看，最近没有做过任何修改。
先尝试通过hadoop-daemon.sh stop datanode ,hadoop-daemon.sh stop tasktracker停止数据节点。
然后尝试通过hadoop-daemon.sh start datanode ,hadoop-daemon.sh start tasktracker启动数据节点。
均正常，无错误信息。
但是在运行一段时间，或者跑一两个MR程序后，该数据几点所在服务器负载开始暴增。
然后无法与namenode正常通信。
一个一个排查吧。
节点信息配置，HDFS信息，均无异常。在排查服务器配置的时候，看到了一个很奇怪的东西。
在etc/hosts中被增加了一个配置，如下：
127.0.1.1 xxxxxx
127.0.1.1是debian中的本地回环。这个造成了hadoop解析出现问题。而且此项配置不知道是谁增加上的。
在屏蔽该项后，问题依然存在，只能重启该服务器。重启后一切正常。
由此可看，保持hadoop集群的环境清洁是很重要的。而且这对于我以后针对hadoop集群异常检查增加了不少经验。hadoop集群的配置一般不会有很大的变动，hadoop对服务器环境的依赖较大，从服务器环境是否变化来排查问题是一个不错的解决方式，mark一下。
原文地址：hadoop数据节点通信异常, 感谢原作者分享。
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
Use ddrescue to recover data on Linux
Mar 20, 2024 pm 01:37 PM
DDREASE is a tool for recovering data from file or block devices such as hard drives, SSDs, RAM disks, CDs, DVDs and USB storage devices. It copies data from one block device to another, leaving corrupted data blocks behind and moving only good data blocks. ddreasue is a powerful recovery tool that is fully automated as it does not require any interference during recovery operations. Additionally, thanks to the ddasue map file, it can be stopped and resumed at any time. Other key features of DDREASE are as follows: It does not overwrite recovered data but fills the gaps in case of iterative recovery. However, it can be truncated if the tool is instructed to do so explicitly. Recover data from multiple files or blocks to a single
Open source! Beyond ZoeDepth! DepthFM: Fast and accurate monocular depth estimation!
Apr 03, 2024 pm 12:04 PM
0.What does this article do? We propose DepthFM: a versatile and fast state-of-the-art generative monocular depth estimation model. In addition to traditional depth estimation tasks, DepthFM also demonstrates state-of-the-art capabilities in downstream tasks such as depth inpainting. DepthFM is efficient and can synthesize depth maps within a few inference steps. Let’s read about this work together ~ 1. Paper information title: DepthFM: FastMonocularDepthEstimationwithFlowMatching Author: MingGui, JohannesS.Fischer, UlrichPrestel, PingchuanMa, Dmytr
New generation of optical fiber broadband technology - 50G PON
Apr 20, 2024 pm 09:22 PM
In the previous article (link), Xiao Zaojun introduced the development history of broadband technology from ISDN, xDSL to 10GPON. Today, let’s talk about the upcoming new generation of optical fiber broadband technology-50GPON. █F5G and F5G-A Before introducing 50GPON, let’s talk about F5G and F5G-A. In February 2020, ETSI (European Telecommunications Standards Institute) promoted a fixed communication network technology system based on 10GPON+FTTR, Wi-Fi6, 200G optical transmission/aggregation, OXC and other technologies, and named it F5G. That is, the fifth generation fixed network communication technology (The5thgenerationFixednetworks). F5G is a fixed network
Google is ecstatic: JAX performance surpasses Pytorch and TensorFlow! It may become the fastest choice for GPU inference training
Apr 01, 2024 pm 07:46 PM
The performance of JAX, promoted by Google, has surpassed that of Pytorch and TensorFlow in recent benchmark tests, ranking first in 7 indicators. And the test was not done on the TPU with the best JAX performance. Although among developers, Pytorch is still more popular than Tensorflow. But in the future, perhaps more large models will be trained and run based on the JAX platform. Models Recently, the Keras team benchmarked three backends (TensorFlow, JAX, PyTorch) with the native PyTorch implementation and Keras2 with TensorFlow. First, they select a set of mainstream
Slow Cellular Data Internet Speeds on iPhone: Fixes
May 03, 2024 pm 09:01 PM
Facing lag, slow mobile data connection on iPhone? Typically, the strength of cellular internet on your phone depends on several factors such as region, cellular network type, roaming type, etc. There are some things you can do to get a faster, more reliable cellular Internet connection. Fix 1 – Force Restart iPhone Sometimes, force restarting your device just resets a lot of things, including the cellular connection. Step 1 – Just press the volume up key once and release. Next, press the Volume Down key and release it again. Step 2 – The next part of the process is to hold the button on the right side. Let the iPhone finish restarting. Enable cellular data and check network speed. Check again Fix 2 – Change data mode While 5G offers better network speeds, it works better when the signal is weaker
The vitality of super intelligence awakens! But with the arrival of self-updating AI, mothers no longer have to worry about data bottlenecks
Apr 29, 2024 pm 06:55 PM
I cry to death. The world is madly building big models. The data on the Internet is not enough. It is not enough at all. The training model looks like "The Hunger Games", and AI researchers around the world are worrying about how to feed these data voracious eaters. This problem is particularly prominent in multi-modal tasks. At a time when nothing could be done, a start-up team from the Department of Renmin University of China used its own new model to become the first in China to make "model-generated data feed itself" a reality. Moreover, it is a two-pronged approach on the understanding side and the generation side. Both sides can generate high-quality, multi-modal new data and provide data feedback to the model itself. What is a model? Awaker 1.0, a large multi-modal model that just appeared on the Zhongguancun Forum. Who is the team? Sophon engine. Founded by Gao Yizhao, a doctoral student at Renmin University’s Hillhouse School of Artificial Intelligence.
Tesla robots work in factories, Musk: The degree of freedom of hands will reach 22 this year!
May 06, 2024 pm 04:13 PM
The latest video of Tesla's robot Optimus is released, and it can already work in the factory. At normal speed, it sorts batteries (Tesla's 4680 batteries) like this: The official also released what it looks like at 20x speed - on a small "workstation", picking and picking and picking: This time it is released One of the highlights of the video is that Optimus completes this work in the factory, completely autonomously, without human intervention throughout the process. And from the perspective of Optimus, it can also pick up and place the crooked battery, focusing on automatic error correction: Regarding Optimus's hand, NVIDIA scientist Jim Fan gave a high evaluation: Optimus's hand is the world's five-fingered robot. One of the most dexterous. Its hands are not only tactile
Alibaba 7B multi-modal document understanding large model wins new SOTA
Apr 02, 2024 am 11:31 AM
New SOTA for multimodal document understanding capabilities! Alibaba's mPLUG team released the latest open source work mPLUG-DocOwl1.5, which proposed a series of solutions to address the four major challenges of high-resolution image text recognition, general document structure understanding, instruction following, and introduction of external knowledge. Without further ado, let’s look at the effects first. One-click recognition and conversion of charts with complex structures into Markdown format: Charts of different styles are available: More detailed text recognition and positioning can also be easily handled: Detailed explanations of document understanding can also be given: You know, "Document Understanding" is currently An important scenario for the implementation of large language models. There are many products on the market to assist document reading. Some of them mainly use OCR systems for text recognition and cooperate with LLM for text processing.
