Hadoop 2.0配置
hadoop
yarn
关于
配置
最近要做一次关于yarn的分享，于是想搭建一个Hadoop环境。Hadoop 2.0较之前的Hadoop 0.1x变化比较大，折腾了好久了，终于把环境搞好了。我搭建了一个两节点的集群，只配置了一些必须的参数，让集群勉强跑起来。 1、core-site.xml configurationpropertynamef
最近要做一次关于yarn的分享，于是想搭建一个Hadoop环境。Hadoop 2.0较之前的Hadoop 0.1x变化比较大，折腾了好久了，终于把环境搞好了。我搭建了一个两节点的集群，只配置了一些必须的参数，让集群勉强跑起来。
1、core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://10.232.42.91:19000/</value> </property> </configuration>
登录后复制
2、mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
登录后复制
3、yarn-site.xml
<configuration> <property> <name>yarn.resourcemanager.address</name> <value>hdfs://10.232.42.91:19001/</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>hdfs://10.232.42.91:19002/</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce.shuffle</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>10.232.42.91:8030</value> </property> </configuration>
登录后复制
把JAVA_HOME、HADOOP_HOME都设置到.bashrc里面去，然后运行sbin/start-all.sh。使用jps可以看到两个节点下运行的进程如下。
[master] jps 31318 ResourceManager 28981 DataNode 11580 JobHistoryServer 28858 NameNode 29155 SecondaryNameNode 31426 NodeManager 11016 Jps [slave] jps 12592 NodeManager 11711 DataNode 17699 Jps
登录后复制
上面这个JobHistoryServer需要单独启动，通过它可以看到每个application的详细日志。启动命令如下。
sbin/mr-jobhistory-daemon.sh start historyserver
登录后复制
打开http://10.232.42.91:8088/cluster/cluster这个地址可以看到cluster的介绍信息。这里再也看不到slot相关的数据了。
万事俱备。放点文本数据到hdfs://10.232.42.91:19000/input这个目录下，运行wordcount看看效果。
$ cd hadoop/share/hadoop/mapreduce $ hadoop jar hadoop-mapreduce-examples-2.0.3-alpha.jar wordcount hdfs://10.232.42.91:19000/input hdfs://10.232.42.91:19000/output 13/03/07 21:08:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 13/03/07 21:08:26 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/03/07 21:08:26 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. 13/03/07 21:08:26 INFO input.FileInputFormat: Total input paths to process : 3 13/03/07 21:08:26 INFO mapreduce.JobSubmitter: number of splits:3 13/03/07 21:08:26 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar 13/03/07 21:08:26 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 13/03/07 21:08:26 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class 13/03/07 21:08:26 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 13/03/07 21:08:26 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name 13/03/07 21:08:26 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class 13/03/07 21:08:26 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 13/03/07 21:08:26 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 13/03/07 21:08:26 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 13/03/07 21:08:26 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 13/03/07 21:08:26 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 13/03/07 21:08:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1362658309553_0019 13/03/07 21:08:26 INFO client.YarnClientImpl: Submitted application application_1362658309553_0019 to ResourceManager at /10.232.42.91:19001 13/03/07 21:08:26 INFO mapreduce.Job: The url to track the job: http://search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0019/ 13/03/07 21:08:26 INFO mapreduce.Job: Running job: job_1362658309553_0019 13/03/07 21:08:33 INFO mapreduce.Job: Job job_1362658309553_0019 running in uber mode : false 13/03/07 21:08:33 INFO mapreduce.Job: map 0% reduce 0% 13/03/07 21:08:39 INFO mapreduce.Job: map 100% reduce 0% 13/03/07 21:08:44 INFO mapreduce.Job: map 100% reduce 100% 13/03/07 21:08:44 INFO mapreduce.Job: Job job_1362658309553_0019 completed successfully 13/03/07 21:08:44 INFO mapreduce.Job: Counters: 43 File System Counters FILE: Number of bytes read=12698 FILE: Number of bytes written=312593 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=16947 HDFS: Number of bytes written=8739 HDFS: Number of read operations=12 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=3 Launched reduce tasks=1 Rack-local map tasks=3 Total time spent by all maps in occupied slots (ms)=10750 Total time spent by all reduces in occupied slots (ms)=4221 Map-Reduce Framework Map input records=317 Map output records=2324 Map output bytes=24586 Map output materialized bytes=12710 Input split bytes=316 Combine input records=2324 Combine output records=885 Reduce input groups=828 Reduce shuffle bytes=12710 Reduce input records=885 Reduce output records=828 Spilled Records=1770 Shuffled Maps =3 Failed Shuffles=0 Merged Map outputs=3 GC time elapsed (ms)=376 CPU time spent (ms)=4480 Physical memory (bytes) snapshot=557428736 Virtual memory (bytes) snapshot=2105122816 Total committed heap usage (bytes)=254607360 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=16631 File Output Format Counters Bytes Written=8739
登录后复制
接下来玩玩yarn吧。Hadoop官方文档那篇WritingYarnApplications太让人蛋碎了，好在我领悟到distributedshell就是使用yarn编写的。要研究yarn的话，直接去Hadoop source里面找相应的代码研究即可。
$ hadoop jar hadoop-yarn-applications-distributedshell-2.0.3-alpha.jar --jar hadoop-yarn-applications-distributedshell-2.0.3-alpha.jar org.apache.hadoop.yarn.applications.distributedshell.Client --shell_command uname --shell_args '-a'
13/03/07 21:42:44 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/03/07 21:42:44 INFO distributedshell.Client: Initializing Client
13/03/07 21:42:44 INFO distributedshell.Client: Running Client
13/03/07 21:42:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/03/07 21:42:44 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
13/03/07 21:42:44 INFO distributedshell.Client: Got Cluster metric info from ASM, numNodeManagers=2
13/03/07 21:42:44 INFO distributedshell.Client: Got Cluster node info from ASM
13/03/07 21:42:44 INFO distributedshell.Client: Got node report from ASM for, nodeId=search042091.sqa.cm4:39557, nodeAddresssearch042091.sqa.cm4:8042, nodeRackName/default-rack, nodeNumContainers0, nodeHealthStatusis_node_healthy: true, health_report: "", last_health_report_time: 1362663711950,
13/03/07 21:42:44 INFO distributedshell.Client: Got node report from ASM for, nodeId=search041134.sqa.cm4:49313, nodeAddresssearch041134.sqa.cm4:8042, nodeRackName/default-rack, nodeNumContainers0, nodeHealthStatusis_node_healthy: true, health_report: "", last_health_report_time: 1362663712038,
13/03/07 21:42:44 INFO distributedshell.Client: Queue info, queueName=default, queueCurrentCapacity=0.0, queueMaxCapacity=1.0, queueApplicationCount=17, queueChildQueueCount=0
13/03/07 21:42:44 INFO distributedshell.Client: User ACL Info for Queue, queueName=root, userAcl=SUBMIT_APPLICATIONS
13/03/07 21:42:44 INFO distributedshell.Client: User ACL Info for Queue, queueName=root, userAcl=ADMINISTER_QUEUE
13/03/07 21:42:44 INFO distributedshell.Client: User ACL Info for Queue, queueName=default, userAcl=SUBMIT_APPLICATIONS
13/03/07 21:42:44 INFO distributedshell.Client: User ACL Info for Queue, queueName=default, userAcl=ADMINISTER_QUEUE
13/03/07 21:42:44 INFO distributedshell.Client: Min mem capabililty of resources in this cluster 1024
13/03/07 21:42:44 INFO distributedshell.Client: Max mem capabililty of resources in this cluster 8192
13/03/07 21:42:44 INFO distributedshell.Client: AM memory specified below min threshold of cluster. Using min value., specified=10, min=1024
13/03/07 21:42:44 INFO distributedshell.Client: Setting up application submission context for ASM
13/03/07 21:42:44 INFO distributedshell.Client: Copy App Master jar from local filesystem and add to local environment
13/03/07 21:42:45 INFO distributedshell.Client: Set the environment for the application master
13/03/07 21:42:45 INFO distributedshell.Client: Setting up app master command
13/03/07 21:42:45 INFO distributedshell.Client: Completed setting up app master command ${JAVA_HOME}/bin/java -Xmx1024m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory 10 --num_containers 1 --priority 0 --shell_command uname --shell_args -a --debug 1><log_dir>/AppMaster.stdout 2><log_dir>/AppMaster.stderr
13/03/07 21:42:45 INFO distributedshell.Client: Submitting application to ASM
13/03/07 21:42:45 INFO client.YarnClientImpl: Submitted application application_1362658309553_0020 to ResourceManager at /10.232.42.91:19001
13/03/07 21:42:46 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=N/A, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao
13/03/07 21:42:47 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=N/A, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao
13/03/07 21:42:48 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=RUNNING, distributedFinalState=UNDEFINED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao
13/03/07 21:42:49 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=RUNNING, distributedFinalState=UNDEFINED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao
13/03/07 21:42:50 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=RUNNING, distributedFinalState=UNDEFINED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao
13/03/07 21:42:51 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=RUNNING, distributedFinalState=UNDEFINED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao
13/03/07 21:42:52 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=FINISHED, distributedFinalState=SUCCEEDED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao
13/03/07 21:42:52 INFO distributedshell.Client: Application has completed successfully. Breaking monitoring loop
13/03/07 21:42:52 INFO distributedshell.Client: Application completed successfully
</log_dir></log_dir>登录后复制
运行完成之后，找不到输出在哪儿，费了好大的劲，终于在hadoop/logs/userlogs下面找到输出了。不知道为何运行了两个container。
$ tree hadoop/logs/userlogs/application_1362658309553_0018
application_1362658309553_0018
|-- container_1362658309553_0018_01_000001
| |-- AppMaster.stderr
| `-- AppMaster.stdout
`-- container_1362658309553_0018_01_000002
|-- stderr
`-- stdout
$ cat hadoop/logs/userlogs/application_1362658309553_0018/container_1362658309553_0018_01_000002/stdout
Linux search042091.sqa.cm4 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
登录后复制
好，开始用yarn调度一个程序。我写了一个脚本，里面启动了服务器。
$ cat ~/start_sp.sh #!/bin/env bash source /home/admin/.bashrc /home/admin/sp/bin/sap_server -c /home/admin/sp/sp_worker/etc/sap_server_app.cfg -l /home/admin/sp/sp_worker/etc/sap_server_log.cfg -k restart
登录后复制
启动起来之后，进程关系图如下。
接着我把脚本直接kill掉，期待yarn给我重启脚本。发现application运行结束了，AppMaster.stderr日志里面有如下内容。
13/03/08 21:40:02 INFO distributedshell.ApplicationMaster: Got response from RM for container ask, completedCnt=1 13/03/08 21:40:02 INFO distributedshell.ApplicationMaster: Got container status for containerID=container_1362747551045_0017_01_000002, state=COMPLETE, exitStatus=137, diagnostics= Killed by external signal 13/03/08 21:40:02 INFO distributedshell.ApplicationMaster: Current application state: loop=464, appDone=true, total=1, requested=1, completed=1, failed=1, currentAllocated=1 13/03/08 21:40:02 INFO distributedshell.ApplicationMaster: Application completed. Signalling finish to RM 13/03/08 21:40:02 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.AMRMClientImpl is stopped. 13/03/08 21:40:02 INFO distributedshell.ApplicationMaster: Application Master failed. exiting
登录后复制
原文地址：Hadoop 2.0配置, 感谢原作者分享。
本站声明
本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容，请联系admin@php.cn
热AI工具
Undresser.AI Undress
人工智能驱动的应用程序，用于创建逼真的裸体照片
AI Clothes Remover
用于从照片中去除衣服的在线人工智能工具。
Undress AI Tool
免费脱衣服图片
Clothoff.io
AI脱衣机
AI Hentai Generator
免费生成ai无尽的。
热门文章
R.E.P.O.能量晶体解释及其做什么（黄色晶体）
3 周前
By 尊渡假赌尊渡假赌尊渡假赌
R.E.P.O.最佳图形设置
3 周前
By 尊渡假赌尊渡假赌尊渡假赌
刺客信条阴影：贝壳谜语解决方案
2 周前
By DDD
R.E.P.O.如果您听不到任何人，如何修复音频
3 周前
By 尊渡假赌尊渡假赌尊渡假赌
WWE 2K25：如何解锁Myrise中的所有内容
3 周前
By 尊渡假赌尊渡假赌尊渡假赌
热工具
记事本++7.3.1
好用且免费的代码编辑器
SublimeText3汉化版
中文版，非常好用
禅工作室 13.0.1
功能强大的PHP集成开发环境
Dreamweaver CS6
视觉化网页开发工具
SublimeText3 Mac版
神级代码编辑软件(SublimeText3)
热门话题
PyCharm与PyTorch完美结合：安装配置步骤详解
Feb 21, 2024 pm 12:00 PM
PyCharm是一款强大的集成开发环境（IDE），而PyTorch是深度学习领域备受欢迎的开源框架。在机器学习和深度学习领域，使用PyCharm和PyTorch进行开发可以极大地提高开发效率和代码质量。本文将详细介绍如何在PyCharm中安装配置PyTorch，并附上具体的代码示例，帮助读者更好地利用这两者的强大功能。第一步：安装PyCharm和Python
Linux系统中GDM的工作原理及配置方法
Mar 01, 2024 pm 06:36 PM
标题：Linux系统中GDM的工作原理及配置方法在Linux操作系统中，GDM（GNOMEDisplayManager）是一种常见的显示管理器，用于控制图形用户界面（GUI）登录和用户会话管理。本文将介绍GDM的工作原理和配置方法，以及提供具体的代码示例。一、GDM的工作原理GDM是GNOME桌面环境下的显示管理器，负责启动X服务器并提供登录界面，用户输
在PyCharm中如何设置Git配置
Feb 20, 2024 am 09:47 AM
标题：如何在PyCharm中正确配置Git在现代软件开发中，版本控制系统是非常重要的工具，而Git作为其中一种流行的版本控制系统，为开发者提供了强大的功能和灵活的操作。而PyCharm作为一款功能强大的Python集成开发环境，自带了对Git的支持，能够让开发者更加便捷地管理代码版本。本文将介绍如何在PyCharm中正确地配置Git，以便于在开发过程中更好地
了解Linux Bashrc：功能、配置与使用方法
Mar 20, 2024 pm 03:30 PM
了解LinuxBashrc：功能、配置与使用方法在Linux系统中，Bashrc（BourneAgainShellruncommands）是一个非常重要的配置文件，其中包含了系统启动时自动运行的各种命令和设置。Bashrc文件通常位于用户的家目录下，是一个隐藏文件，它的作用是为用户自定义设置Bashshell的环境。一、Bashrc的功能设置环境
简单易懂的PyCharm配置Git教程
Feb 20, 2024 am 08:28 AM
PyCharm是一款常用的集成开发环境（IDE），在日常开发中，使用Git管理代码是必不可少的。本文将介绍如何在PyCharm中配置Git，并使用Git进行代码管理，并附带具体代码示例。第一步：安装Git首先，确保在你的计算机上已经安装了Git。如果没有安装，可以前往[Git官网](https://git-scm.com/)下载并安装最新版本的Git
win11系统如何配置工作组
Feb 22, 2024 pm 09:50 PM
Win11系统如何配置工作组工作组是一种在局域网中连接多台计算机的方式，它允许计算机之间共享文件、打印机和其他资源。在Win11系统中，配置工作组非常简单，只需按照以下步骤操作即可。步骤1：打开“设置”应用程序首先，点击Win11系统的“开始”按钮，然后在弹出的菜单中选择“设置”应用程序。你也可以使用快捷键“Win+I”打开“设置”。步骤2：选择“系统”在“设置”应用程序中，你会看到多个选项。请点击“系统”选项，进入系统设置页面。步骤3：选择“关于”在“系统”设置页面中，你会看到多个子选项。请点
MyBatis Generator配置参数解读及最佳实践
Feb 23, 2024 am 09:51 AM
MyBatisGenerator是MyBatis官方提供的一个代码生成工具，可以帮助开发人员快速生成符合数据库表结构的JavaBean、Mapper接口以及XML映射文件。在使用MyBatisGenerator进行代码生成的过程中，配置参数的设置是至关重要的。本文将从配置参数的角度出发，深入探讨MyBatisGenerator的
Linux系统中如何配置和安装FTPS
Mar 20, 2024 pm 02:03 PM
标题：Linux系统中如何配置和安装FTPS，需要具体代码示例在Linux系统中，FTPS是一种安全的文件传输协议，与FTP相比，FTPS通过TLS/SSL协议对传输的数据进行加密，提高了数据传输的安全性。在本文中，将介绍如何在Linux系统中配置和安装FTPS，并提供具体的代码示例。步骤一：安装vsftpd打开终端，输入以下命令安装vsftpd：sudo
