Home Database Mysql Tutorial 重新认识Mesos的设计架构

重新认识Mesos的设计架构

Jun 07, 2016 pm 04:29 PM
mesos author Weibo Sina Architecture design

作者: Dong | 新浪微博: 西成懂 | 可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明 网址:http://dongxicheng.org/apache-mesos/study-mesos-architecture-in-deep/ Mesos中包含四类主要的服务(实际上是一个socket server),它们分别是

Mesos中包含四类主要的服务(实际上是一个socket server),它们分别是Mesos Master,Mesos Slave,SchedulerProcess和ExecutorProcess,它们之间通过Protocal Buffer消息进行通信,每种服务内部注册了若干种Protocal Buffer消息处理器,一旦收到某种消息,则会调用相应的消息处理器进行处理。除了以上四种服务之外,Mesos还对外提供了三种可编程组件,分别是Alloctor、Framework Scheduler和Framework Executor,编写这几个组件必须按照要求实现了几个接口,而这些接口将分别被下图中相邻的服务调用。

大部分人看到以上Mesos架构后,均会认为Framework必须是一个通用的框架,比如MapReduce、Storm、Spark等,而Mesos Master负责将资源分配给各个框架,而各个框架的Scheduler进一步将资源分配给其内部的各个应用程序。这种观念是错误的,是对Mesos架构的一种错误解读。

事实上,Framework不仅可以是通用的框架,也可以是像Hadoop的Job或者YARN的Application那样的简单计算任务,也就是说,Framework并需要一定是一个“Framework”,或者一个长时间运行的服务(比如JobTracker等),也可以是一个短生命周期的Job或者Application。如果让Framework对应一个Hadoop Job,则可以这样设计Framework Scheduler和Framework Executor:

(1)Framework Scheduler功能

Framework Scheduler负责按照作业的输入数据量,将之分解成若干任务,并为这些任务申请资源、监控这些任务的运行状态,一旦发现某个任务运行失败则重新为之申请资源。

(2)Framework Executor功能

为一个节点上的Map Task或者Reduce Task准备运行环境,包括准备各种jar包、二进制文件,设置必要的环境变量,进行必要的资源隔离,启动Jetty Shuffle以为Reduce Task提供远程数据拷贝服务等,接收来自Framework Scheduler的命令(启动任务、杀死任务等),并执行。

通过上面的介绍可以知道,Framework Scheduler只负责运行一个Hadoop Job,而如果你对YARN比较熟悉,便会发现者正是YARN中的MapReduce ApplicationMaster做的事情,没错,Mesos与YARN的设计架构如此的相近,以至于我们很容易通过修改YARN 的任何一个ApplicationMaster,让它作为一个Framework Scheduler运行在Mesos中。

最近Mesos提供了一个mesos-submit工具(https://github.com/apache/mesos/blob/trunk/docs/Using-the-mesos-submit-tool.md,注意,该工具尚不完善),该工具可以让用户的Framework Scheduler运行在任何一个Mesos Slave上,以防止客户端运行过多的Framework Scheduler,这样,Mesos的整个架构和工作流程已经变得与YARN相差无几了。

为了让大家更容易理解Mesos和YARN在架构上的相似性,下面给出了Mesos和YARN的组件对应表:

Mesos中的组件 YARN中的组件 功能
Mesos Master Resource Manager 整个集群的资源管理和调度
Mesos Slave Node Manager 单个节点的资源管理(资源隔离、汇报等)、任务启动等
Framework Executor
Framework Scheduler ApplicationMaster 单个应用程序的管理和资源二次调度,基本操作均包括注册、资源申请/获取、资源分配(给内部的任务)等。

既然Mesos和YARN如此的相近,那么我们到底应该使用哪一个呢?或者说,哪一个系统更有前景?

就目前看来,YARN在以下几个方面存在明显优势:(1)人力投入大。目前YARN有专门的公司(hortonwork)维护和开发 (2)知名度高。YARN之前从Hadoop 1.0中演化而来,继承了Hadoop的知名度,且有大量公司和开发人员共享patch。然而,Mesos最大优点的设计简单、容易上手使用,它不像YARN那样,一个资源的分配过程要涉及到若干个状态机,且每种状态机十几种状态,十几种事件。但稳定性看,两个系统都处于研发和测试阶段,离稳定可用还有一段距离。

 

原创文章,转载请注明: 转载自董的博客

本文链接地址: http://dongxicheng.org/apache-mesos/study-mesos-architecture-in-deep/

作者:Dong,作者介绍:http://dongxicheng.org/about/


Copyright © 2013
This feed is for personal, non-commercial use only.
The use of this feed on other websites breaches copyright. If this content is not in your news reader, it makes the page you are viewing an infringement of the copyright. (Digital Fingerprint:
)
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to search previous Weibo by date on Weibo_How to search previous Weibo by date on Weibo How to search previous Weibo by date on Weibo_How to search previous Weibo by date on Weibo Mar 30, 2024 pm 07:26 PM

1. First open the mobile web browser, search for the Weibo web version, and click the avatar button in the upper left corner after entering. 2. Then click Settings in the upper right corner. 3. Click the version switching option in settings. 4. Then select the color version option in the version switch. 5. Click Search to enter the search page. 6. After entering the keywords, click Find People. 7. When the search completion interface appears, click Filter. 8. Finally, enter the specific date in the release time column and click Filter.

How to turn off Weibo live mode_How to turn off Weibo live mode How to turn off Weibo live mode_How to turn off Weibo live mode Mar 30, 2024 am 10:16 AM

1. First open Weibo, click Search, and click More Hot Searches. 2. Then select a hot search to enter. 3. Then enter the hot search comment live mode, find and click the standard version in the upper right corner. 4. Finally, you can return to the hot search standard comment area. If you want to continue viewing the live comment area, click the live version in the upper right corner. Note: Only when there are a lot of real-time comments will it automatically enter the live mode. Some hot searches will automatically enter the live mode. Generally, hot searches are in standard mode.

How to post videos on Weibo without compressing the image quality_How to post videos on Weibo without compressing the image quality How to post videos on Weibo without compressing the image quality_How to post videos on Weibo without compressing the image quality Mar 30, 2024 pm 12:26 PM

1. First open Weibo on your mobile phone and click [Me] in the lower right corner (as shown in the picture). 2. Then click [Gear] in the upper right corner to open settings (as shown in the picture). 3. Then find and open [General Settings] (as shown in the picture). 4. Then enter the [Video Follow] option (as shown in the picture). 5. Then open the [Video Upload Resolution] setting (as shown in the picture). 6. Finally, select [Original Image Quality] to avoid compression (as shown in the picture).

Retro trend! HMD and Heineken jointly launch flip phone: transparent shell design Retro trend! HMD and Heineken jointly launch flip phone: transparent shell design Apr 17, 2024 pm 06:50 PM

According to news on April 17, HMD teamed up with the well-known beer brand Heineken and the creative company Bodega to launch a unique flip phone - The Boring Phone. This phone is not only full of innovation in design, but also returns to nature in terms of functionality, aiming to lead people back to real interpersonal interactions and enjoy the pure time of drinking with friends. Boring mobile phone adopts a unique transparent flip design, showing a simple yet elegant aesthetic. It is equipped with a 2.8-inch QVGA display inside and a 1.77-inch display outside, providing users with a basic visual interaction experience. In terms of photography, although it is only equipped with a 30-megapixel camera, it is enough to handle simple daily tasks.

How to delete the draft box of 'Weibo'_Tutorial on deleting the draft box of 'Weibo' How to delete the draft box of 'Weibo'_Tutorial on deleting the draft box of 'Weibo' Mar 30, 2024 pm 12:46 PM

1. First open Weibo, click on My, and click on the draft box. 2. Then long press any draft and click to clear the draft box or delete the draft. 3. Finally click OK.

ZTE 5G portable Wi-Fi U50S goes on sale for NT$899 at first launch: top speed 500Mbps ZTE 5G portable Wi-Fi U50S goes on sale for NT$899 at first launch: top speed 500Mbps Apr 26, 2024 pm 03:46 PM

According to news on April 26, ZTE’s 5G portable Wi-Fi U50S is now officially on sale, starting at 899 yuan. In terms of appearance design, ZTE U50S Portable Wi-Fi is simple and stylish, easy to hold and pack. Its size is 159/73/18mm and is easy to carry, allowing you to enjoy 5G high-speed network anytime and anywhere, achieving an unimpeded mobile office and entertainment experience. ZTE 5G portable Wi-Fi U50S supports the advanced Wi-Fi 6 protocol with a peak rate of up to 1800Mbps. It relies on the Snapdragon X55 high-performance 5G platform to provide users with an extremely fast network experience. Not only does it support the 5G dual-mode SA+NSA network environment and Sub-6GHz frequency band, the measured network speed can even reach an astonishing 500Mbps, which is easily satisfactory.

If Weibo svip deletes visitor records, can the other party still see it? If Weibo svip deletes visitor records, can the other party still see it? Apr 02, 2024 am 08:04 AM

Once Weibo users activate the SVIP function, they can manage visitor records, and if they visit a stranger's homepage, they can delete it before displaying it, so that they do not know that they are browsing the homepage. ?Can the other party still see it after deleting the visitor record on Weibo svip? Can the other party still see it after deleting the visitor record on Weibo svip? 1. After deleting the visitor record on svip, the other party cannot see it. It will only see the increase in the number of visitors, but it will not show the visitors. The user's ID and avatar. 2. Ordinary users and VIP users cannot delete or view access records. They can only see how many people have visited them. 3. Weibo access data will be updated at 8:00 every day. If you want to delete access records, you need to

Teclast M50 Mini tablet is here: 8.7-inch IPS screen, 5000mAh battery Teclast M50 Mini tablet is here: 8.7-inch IPS screen, 5000mAh battery Apr 04, 2024 am 08:31 AM

According to news on April 3, Taipower’s upcoming M50 Mini tablet computer is a device with rich functions and powerful performance. This new 8-inch small tablet is equipped with an 8.7-inch IPS screen, providing users with an excellent visual experience. Its metal body design is not only beautiful but also enhances the durability of the device. In terms of performance, the M50Mini is equipped with the Unisoc T606 eight-core processor, which has two A75 cores and six A55 cores, ensuring a smooth and efficient running experience. At the same time, the tablet is also equipped with a 6GB+128GB storage solution and supports 8GB memory expansion, which meets users’ needs for storage and multi-tasking. In terms of battery life, M50Mini is equipped with a 5000mAh battery and supports Ty

See all articles