Hadoop 2.0基本架构和发展趋势
作者: Dong | 新浪微博： 西成懂 | 可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明 网址:http://dongxicheng.org/mapreduce-nextgen/hadoop-2-0-yarn-now-and-future/ 本博客的文章集合:http://dongxicheng.org/recommend/ 重大消息：我
作者:Dong | 新浪微博：西成懂 | 可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明
网址:http://dongxicheng.org/mapreduce-nextgen/hadoop-2-0-yarn-now-and-future/
本博客的文章集合:http://dongxicheng.org/recommend/
重大消息：我的Hadoop新书《Hadoop技术内幕：深入解析MapReduce架构设计与实现原理》已经开始在各大网站销售了，购书链接地址： 当当购书网址，京东购书网址，卓越购书网址。新书官方宣传主页： http://hadoop123.com/。
这是昨天下午我在清华 ChinaHadoop 夏季沙龙上关于“Hadoop 2.0基本架构和发展趋势”演讲的PPT，大家感受一下。
在这个PPT中，我尝试解答以下几个关于Hadoop的问题：
1. 什么是Hadoop 2.0？
2. 什么是YARN？
3. YARN的现状？
4. YARN发展趋势？
5. MapReduce与YARN的关系？
【PPT阅读和下载链接】：
微盘下载链接：Hadoop 2.0基本架构和发展趋势
百度网盘下载链接：Hadoop 2.0基本架构和发展趋势
YARN直接衍生于MapReduce但完全不同于MapReduce，MapReduce是一个低效的批处理计算框架，一般用来做一些离线的计算，不适合迭代计算、交互式计算、实时计算等场景，随着这些新的场景的产生和迫切要求，势必出现一个统一的平台管理这些计算框架，这就是YARN，用户可在YARN之上，跑各种计算任务，包括MapReduce计算、迭代计算（比如pagerank和一些数据挖掘算法）、交互式计算（比如SQL查询）、实时计算（比如storm等），这样看来，MapReduce这种批处理框架将越来越少的被使用到，而由其他新型更高效的计算框架取代，前几年之所以一直使用MR，是因为没有更高效成熟的计算框架出现，大家没有别的选择不得不使用它，而YARN的出现打破了这种局面，随着YARN的成熟（目前是alpha版，这个月可能发布beta版，9月份之前可能会发布稳定版），越来越多的公司，为了提高大数据处理效率，将采用新的计算框架，这时候，必须使用YARN，因为很多计算框架只能运行在YARN上，不能单独部署使用。当前比较明智的做法是，好好使用和学习MapReduce（YARN的很多实现源码级重用了MapReduce实现），同时关注YARN的发展动态和实验性使用它。没办法，互联网就是这样，技术更新速度超快，当你还在学习MapReduce的时候，新的计算框架出现了一箩筐，当Hadoop用了8年左右时间推出 1.0 稳定版后，2.0稳定版只用了2年的时间就要发布了。
原创文章，转载请注明： 转载自董的博客
本文链接地址: http://dongxicheng.org/mapreduce-nextgen/hadoop-2-0-yarn-now-and-future/
作者：Dong，作者介绍：http://dongxicheng.org/about/
本博客的文章集合:http://dongxicheng.org/recommend/
Copyright © 2013
This feed is for personal, non-commercial use only.
The use of this feed on other websites breaches copyright. If this content is not in your news reader, it makes the page you are viewing an infringement of the copyright. (Digital Fingerprint:
)
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1382
52
What is the architecture and working principle of Spring Data JPA?
Apr 17, 2024 pm 02:48 PM
SpringDataJPA is based on the JPA architecture and interacts with the database through mapping, ORM and transaction management. Its repository provides CRUD operations, and derived queries simplify database access. Additionally, it uses lazy loading to only retrieve data when necessary, thus improving performance.
1.3ms takes 1.3ms! Tsinghua's latest open source mobile neural network architecture RepViT
Mar 11, 2024 pm 12:07 PM
Paper address: https://arxiv.org/abs/2307.09283 Code address: https://github.com/THU-MIG/RepViTRepViT performs well in the mobile ViT architecture and shows significant advantages. Next, we explore the contributions of this study. It is mentioned in the article that lightweight ViTs generally perform better than lightweight CNNs on visual tasks, mainly due to their multi-head self-attention module (MSHA) that allows the model to learn global representations. However, the architectural differences between lightweight ViTs and lightweight CNNs have not been fully studied. In this study, the authors integrated lightweight ViTs into the effective
How steep is the learning curve of golang framework architecture?
Jun 05, 2024 pm 06:59 PM
The learning curve of the Go framework architecture depends on familiarity with the Go language and back-end development and the complexity of the chosen framework: a good understanding of the basics of the Go language. It helps to have backend development experience. Frameworks that differ in complexity lead to differences in learning curves.
Hand-tearing Llama3 layer 1: Implementing llama3 from scratch
Jun 01, 2024 pm 05:45 PM
1. Architecture of Llama3 In this series of articles, we implement llama3 from scratch. The overall architecture of Llama3: Picture the model parameters of Llama3: Let's take a look at the actual values of these parameters in the Llama3 model. Picture [1] Context window (context-window) When instantiating the LlaMa class, the variable max_seq_len defines context-window. There are other parameters in the class, but this parameter is most directly related to the transformer model. The max_seq_len here is 8K. Picture [2] Vocabulary-size and AttentionL
Review! Comprehensively summarize the important role of basic models in promoting autonomous driving
Jun 11, 2024 pm 05:29 PM
Written above & the author’s personal understanding: Recently, with the development and breakthroughs of deep learning technology, large-scale foundation models (Foundation Models) have achieved significant results in the fields of natural language processing and computer vision. The application of basic models in autonomous driving also has great development prospects, which can improve the understanding and reasoning of scenarios. Through pre-training on rich language and visual data, the basic model can understand and interpret various elements in autonomous driving scenarios and perform reasoning, providing language and action commands for driving decision-making and planning. The base model can be data augmented with an understanding of the driving scenario to provide those rare feasible features in long-tail distributions that are unlikely to be encountered during routine driving and data collection.
In-depth understanding of the architecture and working principles of the Spring framework
Jan 24, 2024 am 09:41 AM
An in-depth analysis of the architecture and working principles of the Spring framework Introduction: Spring is one of the most popular open source frameworks in the Java ecosystem. It not only provides a powerful set of container management and dependency injection functions, but also provides many other functions, such as transactions. Management, AOP, data access, etc. This article will provide an in-depth analysis of the architecture and working principles of the Spring framework, and explain related concepts through specific code examples. 1. Core concepts of the Spring framework 1.1IoC (Inversion of Control) Core of Spring
Research on the impact and role of Golang on the development of blockchain
Feb 26, 2024 pm 04:24 PM
Golang (Go language for short) as a programming language has gradually emerged in the blockchain field in recent years. Its efficient concurrent processing capabilities and concise syntax features make it a favored choice in blockchain development. This article will explore how Golang helps the development of blockchain and demonstrate its superiority in blockchain applications through specific code examples. 1. Golang’s advantages in the blockchain field: Efficient concurrent processing capabilities: Nodes in the blockchain system need to process a large amount of transactions and data at the same time, and Gola
A deep dive into the architecture and features of Oracle RAC
Mar 07, 2024 pm 03:18 PM
OracleRAC (RealApplicationClusters) is a scalability solution provided by Oracle Corporation that allows Oracle database instances to be run on multiple servers and multiple servers are combined together to provide high availability and scalability. The architecture and features of OracleRAC are very unique. This article will deeply explore the architecture, features and specific code examples of OracleRAC. 1. OracleRAC architecture Oracle
