Table of Contents
1. 配置YARN
2. ?配置MapReduce2
Home Database Mysql Tutorial 配置Hadoop2.0的内存资源

配置Hadoop2.0的内存资源

Jun 07, 2016 pm 04:34 PM
y Memory resource Configuration

在Hadoop2.0中, YARN负责管理MapReduce中的资源(内存, CPU等)并且将其打包成Container. 这样可以精简MapReduce, 使之专注于其擅长的数据处理任务, 将无需考虑资源调度. 如下图所示 YARN会管理集群中所有机器的可用计算资源. 基于这些资源YARN会调度应用(比如


在Hadoop2.0中, YARN负责管理MapReduce中的资源(内存, CPU等)并且将其打包成Container. 这样可以精简MapReduce, 使之专注于其擅长的数据处理任务, 将无需考虑资源调度. 如下图所示

Screen Shot 2014-05-24 at 上午11.43.46YARN会管理集群中所有机器的可用计算资源. 基于这些资源YARN会调度应用(比如MapReduce)发来的资源请求, 然后YARN会通过分配Container来给每个应用提供处理能力, Container是YARN中处理能力的基本单元, 是对内存, CPU等的封装.

本文中假设集群中每个节点的配置为48G内存, 12个硬盘, 2个hex core CPU(12 核).

1. 配置YARN


在Hadoop集群中, 平衡内存, CPU等的使用很重要, 这样才能避免整个集群的计算能力不会因为某种资源而受限. 根据Hortonworks的推荐, 每个硬盘和核1-2个Container能够达到最好的集群使用平衡. 如果集群的每个节点有12个硬盘和12个核, 那么每个节点上最好最多20个Container.

因为每个节点有48G内存, 我们为操作系统保留部分内存, 所以分配40G内存给YARN, 8G留给操作系统. 下面配置的是每个节点上YARN可以使用的最大内存.

在yarn-site.xml中
yarn.nodemanager.resource.memory-mb
40960
Copy after login

然后需要配置如何把这些资源分配给Container, 可以配置分配给Container的最小内存, 因为我们允许每个节点最多20个Container, 所以每个Container的内存为40G / 20 = 2G

在yarn-site.xml中
yarn.scheduler.minimum-allocation-mb
2048
Copy after login


2. ?配置MapReduce2


MapReduce2构建在YARN的基础之上, 使用YARN的Container来调度和运行其map和reduce任务.

在配置YARN上的MapReduce资源使用时, 需要考虑:
  1. 每个Map和Reduce任务的物理内存限制
  2. 每个任务的JVM堆栈大小
  3. 每个任务的虚拟内存

可以设置每个map和reduce任务的最大内存, 该值应该大于等于Container的最小内存. 比如前面我们设置每个Container的最小内存(yarn.scheduler.minimum-allocation-mb)为2GB, 所以我们可以设置map任务的内存为4GB, reduce任务的内存为8GB:

在mapred-site.xml中
mapreduce.map.memory.mb
4096
mapreduce.reduce.memory.mb
8192
Copy after login

每个Container会为每个map和reduce任务运行一个JVM, JVM的堆栈大小应该小于map和reduce的内存大小:

在mapred-site.xml中
mapreduce.map.java.opts
-Xmx3072m
mapreduce.reduce.java.opts
-Xmx6144m
Copy after login

前面设置的是map和reduce任务可以使用的物理内存, 而虚拟内存(物理内存+paged memory)的上限是由每个Container的虚拟内存比例决定的, 默认值为2.1:

在yarn-site.xml中:
yarn.nodemanager.vmem-pmem-ratio
2.1
Copy after login

根据之前的所有设置, 每个map任务的内存分配为
  • 物理内存 = 4GB
  • map任务的Container的JVM堆栈 = 3GB
  • 虚拟内存大小 = 4 * 2.1 = 8.4GB

在YARN和MapReduce2中, 除此之外没有其他的map和reduce任务的资源预配置. 整个集群可以根据作业的需要动态的分配map和reduce, 比如在本例中, YARN会配置最多10(40/4)个mapper或者5(40/8)个reducer, 或者是其他合适的组合.

 

参考文献:

[1].?How to Plan and Configure YARN and MapReduce 2 in HDP 2.0

 
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
1 months ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Large memory optimization, what should I do if the computer upgrades to 16g/32g memory speed and there is no change? Large memory optimization, what should I do if the computer upgrades to 16g/32g memory speed and there is no change? Jun 18, 2024 pm 06:51 PM

For mechanical hard drives or SATA solid-state drives, you will feel the increase in software running speed. If it is an NVME hard drive, you may not feel it. 1. Import the registry into the desktop and create a new text document, copy and paste the following content, save it as 1.reg, then right-click to merge and restart the computer. WindowsRegistryEditorVersion5.00[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SessionManager\MemoryManagement]"DisablePagingExecutive"=d

How to check memory usage on Xiaomi Mi 14Pro? How to check memory usage on Xiaomi Mi 14Pro? Mar 18, 2024 pm 02:19 PM

Recently, Xiaomi released a powerful high-end smartphone Xiaomi 14Pro, which not only has a stylish design, but also has internal and external black technology. The phone has top performance and excellent multitasking capabilities, allowing users to enjoy a fast and smooth mobile phone experience. However, performance will also be affected by memory. Many users want to know how to check the memory usage of Xiaomi 14Pro, so let’s take a look. How to check memory usage on Xiaomi Mi 14Pro? Introduction to how to check the memory usage of Xiaomi 14Pro. Open the [Application Management] button in [Settings] of Xiaomi 14Pro phone. To view the list of all installed apps, browse the list and find the app you want to view, click on it to enter the app details page. In the application details page

Is there a big difference between 8g and 16g memory in computers? (Choose 8g or 16g of computer memory) Is there a big difference between 8g and 16g memory in computers? (Choose 8g or 16g of computer memory) Mar 13, 2024 pm 06:10 PM

When novice users buy a computer, they will be curious about the difference between 8g and 16g computer memory? Should I choose 8g or 16g? In response to this problem, today the editor will explain it to you in detail. Is there a big difference between 8g and 16g of computer memory? 1. For ordinary families or ordinary work, 8G running memory can meet the requirements, so there is not much difference between 8g and 16g during use. 2. When used by game enthusiasts, currently large-scale games basically start at 6g, and 8g is the minimum standard. Currently, when the screen is 2k, higher resolution will not bring higher frame rate performance, so there is no big difference between 8g and 16g. 3. For audio and video editing users, there will be obvious differences between 8g and 16g.

Samsung announced the completion of 16-layer hybrid bonding stacking process technology verification, which is expected to be widely used in HBM4 memory Samsung announced the completion of 16-layer hybrid bonding stacking process technology verification, which is expected to be widely used in HBM4 memory Apr 07, 2024 pm 09:19 PM

According to the report, Samsung Electronics executive Dae Woo Kim said that at the 2024 Korean Microelectronics and Packaging Society Annual Meeting, Samsung Electronics will complete the verification of the 16-layer hybrid bonding HBM memory technology. It is reported that this technology has passed technical verification. The report also stated that this technical verification will lay the foundation for the development of the memory market in the next few years. DaeWooKim said that Samsung Electronics has successfully manufactured a 16-layer stacked HBM3 memory based on hybrid bonding technology. The memory sample works normally. In the future, the 16-layer stacked hybrid bonding technology will be used for mass production of HBM4 memory. ▲Image source TheElec, same as below. Compared with the existing bonding process, hybrid bonding does not need to add bumps between DRAM memory layers, but directly connects the upper and lower layers copper to copper.

Micron: HBM memory consumes 3 times the wafer volume, and production capacity is basically booked for next year Micron: HBM memory consumes 3 times the wafer volume, and production capacity is basically booked for next year Mar 22, 2024 pm 08:16 PM

This site reported on March 21 that Micron held a conference call after releasing its quarterly financial report. At the conference, Micron CEO Sanjay Mehrotra said that compared to traditional memory, HBM consumes significantly more wafers. Micron said that when producing the same capacity at the same node, the current most advanced HBM3E memory consumes three times more wafers than standard DDR5, and it is expected that as performance improves and packaging complexity intensifies, in the future HBM4 This ratio will further increase. Referring to previous reports on this site, this high ratio is partly due to HBM’s low yield rate. HBM memory is stacked with multi-layer DRAM memory TSV connections. A problem with one layer means that the entire

Understand Linux Bashrc: functions, configuration and usage Understand Linux Bashrc: functions, configuration and usage Mar 20, 2024 pm 03:30 PM

Understanding Linux Bashrc: Function, Configuration and Usage In Linux systems, Bashrc (BourneAgainShellruncommands) is a very important configuration file, which contains various commands and settings that are automatically run when the system starts. The Bashrc file is usually located in the user's home directory and is a hidden file. Its function is to customize the Bashshell environment for the user. 1. Bashrc function setting environment

Sources say Samsung Electronics and SK Hynix will commercialize stacked mobile memory after 2026 Sources say Samsung Electronics and SK Hynix will commercialize stacked mobile memory after 2026 Sep 03, 2024 pm 02:15 PM

According to news from this website on September 3, Korean media etnews reported yesterday (local time) that Samsung Electronics and SK Hynix’s “HBM-like” stacked structure mobile memory products will be commercialized after 2026. Sources said that the two Korean memory giants regard stacked mobile memory as an important source of future revenue and plan to expand "HBM-like memory" to smartphones, tablets and laptops to provide power for end-side AI. According to previous reports on this site, Samsung Electronics’ product is called LPWide I/O memory, and SK Hynix calls this technology VFO. The two companies have used roughly the same technical route, which is to combine fan-out packaging and vertical channels. Samsung Electronics’ LPWide I/O memory has a bit width of 512

Lexar launches Ares Wings of War DDR5 7600 16GB x2 memory kit: Hynix A-die particles, 1,299 yuan Lexar launches Ares Wings of War DDR5 7600 16GB x2 memory kit: Hynix A-die particles, 1,299 yuan May 07, 2024 am 08:13 AM

According to news from this website on May 6, Lexar launched the Ares Wings of War series DDR57600CL36 overclocking memory. The 16GBx2 set will be available for pre-sale at 0:00 on May 7 with a deposit of 50 yuan, and the price is 1,299 yuan. Lexar Wings of War memory uses Hynix A-die memory chips, supports Intel XMP3.0, and provides the following two overclocking presets: 7600MT/s: CL36-46-46-961.4V8000MT/s: CL38-48-49 -1001.45V In terms of heat dissipation, this memory set is equipped with a 1.8mm thick all-aluminum heat dissipation vest and is equipped with PMIC's exclusive thermal conductive silicone grease pad. The memory uses 8 high-brightness LED beads and supports 13 RGB lighting modes.

See all articles