How to build a Hadoop development environment on Debian

This guide details how to build a Hadoop development environment on a Debian system.

1. Install Java Development Kit (JDK)

First, install OpenJDK:

 sudo apt update
sudo apt install openjdk-11-jdk -y

Configure the JAVA_HOME environment variable:

 sudo nano /etc/environment

Add at the end of the file (adjust the path according to the actual JDK version):

 <code>JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64"</code>

Save and exit, and then execute:

 source /etc/environment

Verify installation:

 java -version

2. Install Hadoop

Download Hadoop 3.3.6 (or other version):

 wget https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.3.6/hadoop-3.3.6-src.tar.gz

Verify download integrity:

 wget https://downloads.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6-src.tar.gz.sha512
sha256sum -c hadoop-3.3.6-src.tar.gz.sha512

Create a directory and unzip:

 sudo mkdir /opt/hadoops
sudo tar -xzvf hadoop-3.3.6-src.tar.gz -C /opt/hadoops --strip-components 1

3. Configure Hadoop environment variables

Edit /etc/profile file and add:

 export HADOOP_HOME="/opt/hadoops/hadoop-3.3.6"
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Refresh environment variables:

 source /etc/profile

4. Configure Hadoop core configuration file

Edit core-site.xml :

 sudo nano $HADOOP_HOME/etc/hadoop/core-site.xml

Add to:

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

Edit hdfs-site.xml :

 sudo nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml

Add to:

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/opt/hadoops/hdfs/namenode</value>
  </property>
</configuration>

Edit mapred-site.xml :

 sudo nano $HADOOP_HOME/etc/hadoop/mapred-site.xml

Add to:

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

Edit yarn-site.xml :

 sudo nano $HADOOP_HOME/etc/hadoop/yarn-site.xml

Add to:

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
</configuration>

5. Set SSH without password login

Generate SSH key:

 sudo su - hadoop
ssh-keygen -t rsa -P ""

Copy the public key:

 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Test connection:

 ssh localhost

6. Format NameNode

 hdfs namenode -format

7. Start Hadoop service

 start-dfs.sh
start-yarn.sh

8. Verify Hadoop installation

Check cluster status:

 hdfs dfsadmin -report

Visit NameNode web interface: http://localhost:9870

After completing the above steps, the Hadoop development environment on your Debian system is completed. Please adjust the path and Hadoop version according to the actual situation.

The above is the detailed content of How to build a Hadoop development environment on Debian. For more information, please follow other related articles on the PHP Chinese website!