Hadoop运维笔记 之 Snappy创建libhadoop.so导致datanode报错
为了解决上一篇文章中提到的Bug，我们将线上的CDH5升级到了目前最新的CDH5.2.0，但升级之后，有一部分服务器的datanode不能正常启动，报错如下： 2014-11-20 19:54:52,071 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected exception in b
为了解决上一篇文章中提到的Bug，我们将线上的CDH5升级到了目前最新的CDH5.2.0，但升级之后，有一部分服务器的datanode不能正常启动，报错如下：
2014-11-20 19:54:52,071 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected exception in block pool Block pool <registering> (Datanode Uuid unassigned) service to idc1-server1/10.100.1.100:8020 com.google.common.util.concurrent.ExecutionError: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO.link0(Ljava/lang/String;Ljava/lang/String;)V at com.google.common.util.concurrent.Futures.wrapAndThrowExceptionOrError(Futures.java:1126) at com.google.common.util.concurrent.Futures.get(Futures.java:1048) at org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocks(DataStorage.java:870) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.linkAllBlocks (BlockPoolSliceStorage.java:570) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.doUpgrade (BlockPoolSliceStorage.java:379) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.doTransition (BlockPoolSliceStorage.java:313) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead (BlockPoolSliceStorage.java:187) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead (DataStorage.java:309) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1109) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1080) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo (BPOfferService.java:320) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake (BPServiceActor.java:220) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:824) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO.link0 (Ljava/lang/String;Ljava/lang/String;)V at org.apache.hadoop.io.nativeio.NativeIO.link0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO.link(NativeIO.java:838) at org.apache.hadoop.hdfs.server.datanode.DataStorage$2.call(DataStorage.java:862) at org.apache.hadoop.hdfs.server.datanode.DataStorage$2.call(DataStorage.java:855) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... 1 more 2014-11-20 19:54:52,073 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to idc1-server1/10.100.1.100:8020 </registering></registering>
但搜遍了Google也未能找到匹配的信息，唯一沾点边的都是一些在Windows平台上因为缺少lib导致的问题。
而在我们的环境中，只有一部分的服务器有以上问题，对比了所有Hadoop相关的软件包之后都没法发现有什么不同，这给我们分析问题带来了很大的干扰。
最后，我们尝试通过strace来跟踪datanode的进程。
yum install strace
strace -f -F -o /tmp/strace.output.txt /etc/init.d/hadoop-hdfs-datanode start
lsof | grep libhadoop.so
java 18527 hdfs mem REG 253,0 122832 270200 /usr/java/jdk1.7.0_45/jre/lib/amd64/libhadoop.so
发现它读取了一个lib文件：/usr/java/jdk1.7.0_45/jre/lib/amd64/libhadoop.so，而其它正常的服务器的datanode进程则是读取的/usr/lib/hadoop/lib/native/libhadoop.so。
经过验证发现/usr/java/jdk1.7.0_45/jre/lib/amd64/libhadoop.so是在安装Snappy软件包时创建的，在移走了它之后，datanode终于正常启动了。
看来，虽然datanode在启动时指定了 -Djava.library.path=/usr/lib/hadoop/lib/native，但jre中的lib被载入的优先级还是要高一些。
原文地址：Hadoop运维笔记 之 Snappy创建libhadoop.so导致datanode报错, 感谢原作者分享。
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1387
52
How to delete Xiaohongshu notes
Mar 21, 2024 pm 08:12 PM
How to delete Xiaohongshu notes? Notes can be edited in the Xiaohongshu APP. Most users don’t know how to delete Xiaohongshu notes. Next, the editor brings users pictures and texts on how to delete Xiaohongshu notes. Tutorial, interested users come and take a look! Xiaohongshu usage tutorial How to delete Xiaohongshu notes 1. First open the Xiaohongshu APP and enter the main page, select [Me] in the lower right corner to enter the special area; 2. Then in the My area, click on the note page shown in the picture below , select the note you want to delete; 3. Enter the note page, click [three dots] in the upper right corner; 4. Finally, the function bar will expand at the bottom, click [Delete] to complete.
Can deleted notes on Xiaohongshu be recovered?
Oct 31, 2023 pm 05:36 PM
Notes deleted from Xiaohongshu cannot be recovered. As a knowledge sharing and shopping platform, Xiaohongshu provides users with the function of recording notes and collecting useful information. According to Xiaohongshu’s official statement, deleted notes cannot be recovered. The Xiaohongshu platform does not provide a dedicated note recovery function. This means that once a note is deleted in Xiaohongshu, whether it is accidentally deleted or for other reasons, it is generally impossible to retrieve the deleted content from the platform. If you encounter special circumstances, you can try to contact Xiaohongshu’s customer service team to see if they can help solve the problem.
What should I do if the notes I posted on Xiaohongshu are missing? What's the reason why the notes it just sent can't be found?
Mar 21, 2024 pm 09:30 PM
As a Xiaohongshu user, we have all encountered the situation where published notes suddenly disappeared, which is undoubtedly confusing and worrying. In this case, what should we do? This article will focus on the topic of "What to do if the notes published by Xiaohongshu are missing" and give you a detailed answer. 1. What should I do if the notes published by Xiaohongshu are missing? First, don't panic. If you find that your notes are missing, staying calm is key and don't panic. This may be caused by platform system failure or operational errors. Checking release records is easy. Just open the Xiaohongshu App and click "Me" → "Publish" → "All Publications" to view your own publishing records. Here you can easily find previously published notes. 3.Repost. If found
How to connect Apple Notes on iPhone in the latest iOS 17 system
Sep 22, 2023 pm 05:01 PM
Link AppleNotes on iPhone using the Add Link feature. Notes: You can only create links between Apple Notes on iPhone if you have iOS17 installed. Open the Notes app on your iPhone. Now, open the note where you want to add the link. You can also choose to create a new note. Click anywhere on the screen. This will show you a menu. Click the arrow on the right to see the "Add link" option. click it. Now you can type the name of the note or the web page URL. Then, click Done in the upper right corner and the added link will appear in the note. If you want to add a link to a word, just double-click the word to select it, select "Add Link" and press
How to create a folder on Realme Phone?
Mar 23, 2024 pm 02:30 PM
Title: Realme Phone Beginner’s Guide: How to Create Folders on Realme Phone? In today's society, mobile phones have become an indispensable tool in people's lives. As a popular smartphone brand, Realme Phone is loved by users for its simple and practical operating system. In the process of using Realme phones, many people may encounter situations where they need to organize files and applications on their phones, and creating folders is an effective way. This article will introduce how to create folders on Realme phones to help users better manage their phone content. No.
How to add product links in notes in Xiaohongshu Tutorial on adding product links in notes in Xiaohongshu
Mar 12, 2024 am 10:40 AM
How to add product links in notes in Xiaohongshu? In the Xiaohongshu app, users can not only browse various contents but also shop, so there is a lot of content about shopping recommendations and good product sharing in this app. If If you are an expert on this app, you can also share some shopping experiences, find merchants for cooperation, add links in notes, etc. Many people are willing to use this app for shopping, because it is not only convenient, but also has many Experts will make some recommendations. You can browse interesting content and see if there are any clothing products that suit you. Let’s take a look at how to add product links to notes! How to add product links to Xiaohongshu Notes Open the app on the desktop of your mobile phone. Click on the app homepage
How to create pixel art in GIMP
Feb 19, 2024 pm 03:24 PM
This article will interest you if you are interested in using GIMP for pixel art creation on Windows. GIMP is a well-known graphics editing software that is not only free and open source, but also helps users create beautiful images and designs easily. In addition to being suitable for beginners and professional designers alike, GIMP can also be used to create pixel art, a form of digital art that utilizes pixels as the only building blocks for drawing and creating. How to Create Pixel Art in GIMP Here are the main steps to create pixel pictures using GIMP on a Windows PC: Download and install GIMP, then launch the application. Create a new image. Resize width and height. Select the pencil tool. Set the brush type to pixels. set up
How to create a family with Gree+
Mar 01, 2024 pm 12:40 PM
Many friends expressed that they want to know how to create a family in Gree+ software. Here is the operation method for you. Friends who want to know more, come and take a look with me. First, open the Gree+ software on your mobile phone and log in. Then, in the options bar at the bottom of the page, click the "My" option on the far right to enter the personal account page. 2. After coming to my page, there is a "Create Family" option under "Family". After finding it, click on it to enter. 3. Next jump to the page to create a family, enter the family name to be set in the input box according to the prompts, and click the "Save" button in the upper right corner after entering it. 4. Finally, a "save successfully" prompt will pop up at the bottom of the page, indicating that the family has been successfully created.
