What are the configuration steps for CentOS HDFS

Detailed steps to deploy Hadoop Distributed File System (HDFS) on CentOS:

1. Preparation

1. Install Java: Make sure that the system has the appropriate JDK version installed and configure the JAVA_HOME environment variable.

2. Install Hadoop: Download the corresponding version of Hadoop distribution package and unzip it to the specified directory (for example, /usr/local/hadoop ).

2. Environment configuration

1. Set environment variables: Edit /etc/profile file and add the following environment variables:

 export JAVA_HOME=/path/to/your/jdk
export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_HOME=/path/to/hadoop
export PATH=$HADOOP_HOME/bin:$PATH

Replace /path/to/your/jdk and /path/to/hadoop with actual paths. After saving the file, run source /etc/profile to make the configuration take effect.

2. SSH password-free login: Configure SSH password-free login for all Hadoop nodes. Use ssh-keygen -t rsa to generate the key pair, and then use ssh-copy-id user@nodeX to copy the public key to each node (replace user with username and nodeX with nodename).

3. Network configuration

1. Host Name: Ensure that the host name of each node is correctly configured and accessible through the network.

2. Static IP: Configure a static IP address for each node. Edit the network configuration file (for example /etc/sysconfig/network-scripts/ifcfg-eth0 ), and set the static IP, subnet mask, and gateway.

3. Time synchronization: Use NTP service to synchronize the time of all nodes. Install NTP ( yum install ntp ) and synchronize the time using ntpdate ntp.aliyun.com (or other NTP server).

4. HDFS configuration

1. Core configuration file (core-site.xml): Configure the HDFS default file system. Modify the $HADOOP_HOME/etc/hadoop/core-site.xml file and add the following content:

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://namenode_hostname:9000</value>
  </property>
</configuration>

Replace namenode_hostname with the hostname of the NameNode node.

2. HDFS configuration file (hdfs-site.xml): Configure HDFS data storage path and number of copies, etc. Modify the $HADOOP_HOME/etc/hadoop/hdfs-site.xml file and add the following content:

<configuration>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/path/to/namenode/data</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/path/to/datanode/data</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>
</configuration>

Replace /path/to/namenode/data and /path/to/datanode/data with the data storage directory of NameNode and DataNode.

5. Format NameNode

Run the following command on the NameNode node to format NameNode:

 $HADOOP_HOME/bin/hdfs namenode -format

6. Start HDFS

Run the following command on the NameNode node to start the HDFS cluster:

 $HADOOP_HOME/sbin/start-dfs.sh

VII. Verification and Stop

1. Verification: Use the jps command to check whether the HDFS daemon has been started. Visit http://namenode_hostname:50070 to view the HDFS Web UI.

2. Stop: Execute the following command on the NameNode node to stop the HDFS cluster:

 $HADOOP_HOME/sbin/stop-dfs.sh

Note: The above steps are only basic guides, and the actual configuration may vary depending on Hadoop version and cluster size. Be sure to refer to the official Hadoop documentation for more detailed and accurate information. Please modify the path according to actual conditions.

The above is the detailed content of What are the configuration steps for CentOS HDFS. For more information, please follow other related articles on the PHP Chinese website!