How to view Debian Hadoop logs

This article introduces how to efficiently view Hadoop logs in Debian system. Hadoop's log aggregation function centrally stores the logs of each node for easy management. The following steps will guide you on how to view these logs:

Step 1: Make sure log aggregation is enabled

First, confirm that log aggregation is enabled for all nodes in the Hadoop cluster. This is usually configured in yarn-site.xml configuration file:

<property>
  <name>yarn.log-aggregation-enable</name>
  <value>true</value>
</property>
<property>
  <name>yarn.log-aggregation.retain-seconds</name>
  <value>172800</value>
</property>
<property>
  <name>yarn.nodemanager.remote-app-log-dir</name>
  <value>/tmp/logs</value>
</property>
<property>
  <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
  <value>logs</value>
</property>
<property>
  <name>yarn.log.server.url</name>
  <value>http://hadoop-node0:19888/jobhistory/logs</value>
</property>

Step 2: Access logs through the Web UI

After log aggregation, you can view the logs through Hadoop's ResourceManager Web UI or the Job History Server Web UI:

• ResourceManager Web UI: The default address is http://<resourcemanager-hostname> :8088/cluster/apps/FINISHED</resourcemanager-hostname> .
• Job History Server Web UI: The default address is http://<jobhistory-server-hostname> :19888/jobhistory</jobhistory-server-hostname> .

Step 3: Use the command line tool to view the log

If you need to read the logs directly from HDFS, you can use the hdfs dfs command:

 hdfs dfs -cat /path/to/logs/*

Step 4: Use a text editor to view the log

You can open and view log files using any text editor (such as nano , vim ):

 sudo nano /path/to/logs/application-<application-id> -container-<container-id> .log</container-id></application-id>

Note that log file paths and URLs may vary depending on Hadoop cluster configuration. Please refer to your Hadoop cluster configuration file for the exact path.

The above is the detailed content of How to view Debian Hadoop logs. For more information, please follow other related articles on the PHP Chinese website!