配置Hadoop2.0的内存资源
在Hadoop2.0中, YARN负责管理MapReduce中的资源(内存, CPU等)并且将其打包成Container. 这样可以精简MapReduce, 使之专注于其擅长的数据处理任务, 将无需考虑资源调度. 如下图所示 YARN会管理集群中所有机器的可用计算资源. 基于这些资源YARN会调度应用(比如
在Hadoop2.0中, YARN负责管理MapReduce中的资源(内存, CPU等)并且将其打包成Container. 这样可以精简MapReduce, 使之专注于其擅长的数据处理任务, 将无需考虑资源调度. 如下图所示

本文中假设集群中每个节点的配置为48G内存, 12个硬盘, 2个hex core CPU(12 核).
1. 配置YARN
在Hadoop集群中, 平衡内存, CPU等的使用很重要, 这样才能避免整个集群的计算能力不会因为某种资源而受限. 根据Hortonworks的推荐, 每个硬盘和核1-2个Container能够达到最好的集群使用平衡. 如果集群的每个节点有12个硬盘和12个核, 那么每个节点上最好最多20个Container.
因为每个节点有48G内存, 我们为操作系统保留部分内存, 所以分配40G内存给YARN, 8G留给操作系统. 下面配置的是每个节点上YARN可以使用的最大内存.
在yarn-site.xml中
yarn.nodemanager.resource.memory-mb 40960
然后需要配置如何把这些资源分配给Container, 可以配置分配给Container的最小内存, 因为我们允许每个节点最多20个Container, 所以每个Container的内存为40G / 20 = 2G
在yarn-site.xml中
yarn.scheduler.minimum-allocation-mb 2048
2. ?配置MapReduce2
MapReduce2构建在YARN的基础之上, 使用YARN的Container来调度和运行其map和reduce任务.
在配置YARN上的MapReduce资源使用时, 需要考虑:
- 每个Map和Reduce任务的物理内存限制
- 每个任务的JVM堆栈大小
- 每个任务的虚拟内存
可以设置每个map和reduce任务的最大内存, 该值应该大于等于Container的最小内存. 比如前面我们设置每个Container的最小内存(yarn.scheduler.minimum-allocation-mb)为2GB, 所以我们可以设置map任务的内存为4GB, reduce任务的内存为8GB:
在mapred-site.xml中
mapreduce.map.memory.mb 4096 mapreduce.reduce.memory.mb 8192
每个Container会为每个map和reduce任务运行一个JVM, JVM的堆栈大小应该小于map和reduce的内存大小:
在mapred-site.xml中
mapreduce.map.java.opts -Xmx3072m mapreduce.reduce.java.opts -Xmx6144m
前面设置的是map和reduce任务可以使用的物理内存, 而虚拟内存(物理内存+paged memory)的上限是由每个Container的虚拟内存比例决定的, 默认值为2.1:
在yarn-site.xml中:
yarn.nodemanager.vmem-pmem-ratio 2.1
根据之前的所有设置, 每个map任务的内存分配为
- 物理内存 = 4GB
- map任务的Container的JVM堆栈 = 3GB
- 虚拟内存大小 = 4 * 2.1 = 8.4GB
在YARN和MapReduce2中, 除此之外没有其他的map和reduce任务的资源预配置. 整个集群可以根据作业的需要动态的分配map和reduce, 比如在本例中, YARN会配置最多10(40/4)个mapper或者5(40/8)个reducer, 或者是其他合适的组合.
参考文献:
[1].?How to Plan and Configure YARN and MapReduce 2 in HDP 2.0
原文地址:配置Hadoop2.0的内存资源, 感谢原作者分享。

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

For mechanical hard drives or SATA solid-state drives, you will feel the increase in software running speed. If it is an NVME hard drive, you may not feel it. 1. Import the registry into the desktop and create a new text document, copy and paste the following content, save it as 1.reg, then right-click to merge and restart the computer. WindowsRegistryEditorVersion5.00[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SessionManager\MemoryManagement]"DisablePagingExecutive"=d

Recently, Xiaomi released a powerful high-end smartphone Xiaomi 14Pro, which not only has a stylish design, but also has internal and external black technology. The phone has top performance and excellent multitasking capabilities, allowing users to enjoy a fast and smooth mobile phone experience. However, performance will also be affected by memory. Many users want to know how to check the memory usage of Xiaomi 14Pro, so let’s take a look. How to check memory usage on Xiaomi Mi 14Pro? Introduction to how to check the memory usage of Xiaomi 14Pro. Open the [Application Management] button in [Settings] of Xiaomi 14Pro phone. To view the list of all installed apps, browse the list and find the app you want to view, click on it to enter the app details page. In the application details page

When novice users buy a computer, they will be curious about the difference between 8g and 16g computer memory? Should I choose 8g or 16g? In response to this problem, today the editor will explain it to you in detail. Is there a big difference between 8g and 16g of computer memory? 1. For ordinary families or ordinary work, 8G running memory can meet the requirements, so there is not much difference between 8g and 16g during use. 2. When used by game enthusiasts, currently large-scale games basically start at 6g, and 8g is the minimum standard. Currently, when the screen is 2k, higher resolution will not bring higher frame rate performance, so there is no big difference between 8g and 16g. 3. For audio and video editing users, there will be obvious differences between 8g and 16g.

According to the report, Samsung Electronics executive Dae Woo Kim said that at the 2024 Korean Microelectronics and Packaging Society Annual Meeting, Samsung Electronics will complete the verification of the 16-layer hybrid bonding HBM memory technology. It is reported that this technology has passed technical verification. The report also stated that this technical verification will lay the foundation for the development of the memory market in the next few years. DaeWooKim said that Samsung Electronics has successfully manufactured a 16-layer stacked HBM3 memory based on hybrid bonding technology. The memory sample works normally. In the future, the 16-layer stacked hybrid bonding technology will be used for mass production of HBM4 memory. ▲Image source TheElec, same as below. Compared with the existing bonding process, hybrid bonding does not need to add bumps between DRAM memory layers, but directly connects the upper and lower layers copper to copper.

This site reported on March 21 that Micron held a conference call after releasing its quarterly financial report. At the conference, Micron CEO Sanjay Mehrotra said that compared to traditional memory, HBM consumes significantly more wafers. Micron said that when producing the same capacity at the same node, the current most advanced HBM3E memory consumes three times more wafers than standard DDR5, and it is expected that as performance improves and packaging complexity intensifies, in the future HBM4 This ratio will further increase. Referring to previous reports on this site, this high ratio is partly due to HBM’s low yield rate. HBM memory is stacked with multi-layer DRAM memory TSV connections. A problem with one layer means that the entire

Understanding Linux Bashrc: Function, Configuration and Usage In Linux systems, Bashrc (BourneAgainShellruncommands) is a very important configuration file, which contains various commands and settings that are automatically run when the system starts. The Bashrc file is usually located in the user's home directory and is a hidden file. Its function is to customize the Bashshell environment for the user. 1. Bashrc function setting environment

According to news from this website on September 3, Korean media etnews reported yesterday (local time) that Samsung Electronics and SK Hynix’s “HBM-like” stacked structure mobile memory products will be commercialized after 2026. Sources said that the two Korean memory giants regard stacked mobile memory as an important source of future revenue and plan to expand "HBM-like memory" to smartphones, tablets and laptops to provide power for end-side AI. According to previous reports on this site, Samsung Electronics’ product is called LPWide I/O memory, and SK Hynix calls this technology VFO. The two companies have used roughly the same technical route, which is to combine fan-out packaging and vertical channels. Samsung Electronics’ LPWide I/O memory has a bit width of 512

According to news from this website on May 6, Lexar launched the Ares Wings of War series DDR57600CL36 overclocking memory. The 16GBx2 set will be available for pre-sale at 0:00 on May 7 with a deposit of 50 yuan, and the price is 1,299 yuan. Lexar Wings of War memory uses Hynix A-die memory chips, supports Intel XMP3.0, and provides the following two overclocking presets: 7600MT/s: CL36-46-46-961.4V8000MT/s: CL38-48-49 -1001.45V In terms of heat dissipation, this memory set is equipped with a 1.8mm thick all-aluminum heat dissipation vest and is equipped with PMIC's exclusive thermal conductive silicone grease pad. The memory uses 8 high-brightness LED beads and supports 13 RGB lighting modes.
