Tips for using HDFS file system on CentOS

Guide to Installation, Configuration and Optimization of HDFS File System under CentOS System

This article will guide you on how to install, configure, and optimize Hadoop Distributed File System (HDFS) on CentOS systems.

HDFS installation and configuration

1. Java environment installation:

   First, make sure that the appropriate Java environment is installed. Edit the /etc/profile file, add the following, and replace /usr/lib/java-1.8.0/jdk1.8.0_144 with your actual Java installation path:

   export JAVA_HOME=/usr/lib/java-1.8.0/jdk1.8.0_144
   export PATH=$JAVA_HOME/bin:$PATH
   export CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar
   source /etc/profile

2. Hadoop environment variable configuration:

   Edit /etc/profile file and add Hadoop environment variables. Please replace /opt/hadoop/hadoop-2.8.1 with your Hadoop installation path:

   export HADOOP_HOME=/opt/hadoop/hadoop-2.8.1
   export PATH=$HADOOP_HOME/bin:$PATH
   export CLASSPATH=$HADOOP_HOME/lib/*
   source /etc/profile

3. SSH password-free login configuration:

   To facilitate communication between Hadoop nodes, configuring SSH password-free login is crucial. Execute the following command:

   ssh-keygen -t rsa
   ssh-copy-id localhost

4. NameNode formatting:

   When configuring HDFS for the first time, the NameNode must be formatted:

   hdfs namenode -format

5. HDFS startup:

   Start HDFS service:

   $HADOOP_HOME/sbin/start-dfs.sh

6. HDFS web interface access:

   Access NameNode's web interface through a browser (replace<namenode-ip></namenode-ip> Replace with NameNode's IP address):

   <code>http://<namenode-ip> :50070</namenode-ip></code>

Advanced configuration and optimization

1. HDFS High Availability (HA):

   Configuring high availability requires two NameNodes (one active and one passive) and at least three JournalNodes. Edit the hdfs-site.xml file and add the following configuration (replace with your actual node information):

   <property>
     <name>dfs.nameservices</name>
     <value>mycluster</value>
   </property>
   <property>
     <name>dfs.ha.namenodes.mycluster</name>
     <value>nn1,nn2</value>
   </property>
   <property>
     <name>dfs.namenode.rpc-address.mycluster.nn1</name>
     <value>namenode1:8020</value>
   </property>
   <property>
     <name>dfs.namenode.rpc-address.mycluster.nn2</name>
     <value>namenode2:8020</value>
   </property>
   <property>
     <name>dfs.namenode.http-address.mycluster.nn1</name>
     <value>namenode1:50070</value>
   </property>
   <property>
     <name>dfs.namenode.http-address.mycluster.nn2</name>
     <value>namenode2:50070</value>
   </property>
   <property>
     <name>dfs.namenode.shared.edits.dir</name>
     <value>qjournal://journalnode1:8485;journalnode2:8485;journalnode3:8485/mycluster</value>
   </property>

2. Performance Tuning:

   • NameNode memory optimization: According to the Hadoop version, adjust hadoop-env.sh (Hadoop 2.x) or use the memory automatic allocation feature (Hadoop 3.x) to optimize NameNode memory. Use the jmap -heap command to monitor memory usage.
   • Heartbeat concurrency optimization: Edit hdfs-site.xml and increase the value of dfs.namenode.handler.count to improve concurrency processing capability.
   • Enable HDFS Recycle Bin: Modify fs.trash.interval and fs.trash.checkpoint.interval in core-site.xml to enable the Recycle Bin.
   • Multi-directory configuration: Modify dfs.namenode.name.dir and dfs.datanode.data.dir in hdfs-site.xml to configure multiple directories to improve data reliability and performance.

Through the above steps, you can effectively install, configure and optimize the HDFS file system on your CentOS system. Please remember to adjust the path and IP address according to your actual environment.

The above is the detailed content of Tips for using HDFS file system on CentOS. For more information, please follow other related articles on the PHP Chinese website!