Home Database Mysql Tutorial 多集群下资源共享方案介绍

多集群下资源共享方案介绍

Jun 07, 2016 pm 04:29 PM
introduce author Weibo Sina plan Resource Sharing cluster

作者: Dong | 新浪微博: 西成懂 | 可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明 网址:http://dongxicheng.org/mapreduce-nextgen/multi-clusters-sharing-resources/ 本博客的文章集合: http://dongxicheng.org/recommend/ 1. 集群共

1. 集群共享带来的好处

在一些公司中,所有服务器按照业务逻辑被拆分成若干个集群,这些集群各自运行着相对独立的业务,并由专门的运维人员管理。

然而,由于集群中应用特点不同,它们的利用率可能不一致,同一时间段,有的集群业务繁忙,资源利用率高,甚至可能出现排队等待资源的情况,而另外一些集群则业务空闲,资源利用率很低。此外,多个集群也可能导致数据冗余度增加,比如一个集群中的业务需要另外一个集群中的数据,可能需要跨集群拷贝数据,进而占用更多的磁盘空间。

为了解决多集群带来的问题,可将多个集群统一为一个集群,这将带来以下几个好处:

(1)提高集群整体利用率。所有业务运行在一个大的集群中,并按需分配给各个应用程序,可达到资源错峰交谷的目的,提高系统整体利用率和均衡率。

(2)减少管理成本。多个集群合并成一个大集群后,便于统一管理和统一运维,可减少管理成本。

2. 多集群共享方案

有些公司中,同时存在多个Hadoop集群,这些集群运行着不同的业务线,并对实时性、资源需求、优先级等有不同要求。为了实现集群资源共享,可将多个Hadoop集群合并成一个,并由Hadoop作业调度器统负责资源管理和任务分配。为了对应原先的业务逻辑,作业调度器通常将用户分成若干个队列(每个业务线一个队列),并给每个队列分配一定比例的资源,作业调度器中包含资源分配逻辑,能够按照一定的策略将资源分配各这些队列,并灵活实现资源按需分配,以达到资源最高利用。




3. 多服务共享方案

Hadoop只是当前比较流行的开源软件之一,除了Hadoop,很多公司还有其他一些服务,比如Web Server,Thrift Server、DataBase Service等,同样,通常考虑到服务的重要性和将来的发展,会将这些服务单独部署到独立的几台机器上,在大部分情况下,很多机器的资源利用率很低,为了提高机器整体利用率和支持服务自动化部署,可将这些服务混搭地部署到各个服务器上,于是资源统一管理系统诞生了,业界典型代表是Google的Borg和腾讯的Torca,开源界代表是Mesos和YARN。

资源统一管理系统的另外一个值得一提的动机是数据中心编程,也就是说,可以把数据中心看成一台超大的服务器,由一个资源管理系统对所有的计算资源,包括CPU、内存、硬盘、端口号等,进行统一管理和调度,涉及到的关键技术包括资源分配、资源隔离、资源划分等,具体可参考腾讯Torca的设计文档。




4. 参考资料

(1)Torca:http://djt.qq.com/article-329-1.html

(2)Mesos:http://www.mesosproject.org/

(3)YARN:http://hadoop.apache.org/docs/r0.23.6/

原创文章,转载请注明: 转载自董的博客

本文链接地址: http://dongxicheng.org/mapreduce-nextgen/multi-clusters-sharing-resources/

作者:Dong,作者介绍:http://dongxicheng.org/about/

本博客的文章集合:http://dongxicheng.org/recommend/


Copyright © 2013
This feed is for personal, non-commercial use only.
The use of this feed on other websites breaches copyright. If this content is not in your news reader, it makes the page you are viewing an infringement of the copyright. (Digital Fingerprint:
)
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to search previous Weibo by date on Weibo_How to search previous Weibo by date on Weibo How to search previous Weibo by date on Weibo_How to search previous Weibo by date on Weibo Mar 30, 2024 pm 07:26 PM

1. First open the mobile web browser, search for the Weibo web version, and click the avatar button in the upper left corner after entering. 2. Then click Settings in the upper right corner. 3. Click the version switching option in settings. 4. Then select the color version option in the version switch. 5. Click Search to enter the search page. 6. After entering the keywords, click Find People. 7. When the search completion interface appears, click Filter. 8. Finally, enter the specific date in the release time column and click Filter.

How to release someone blocked on Weibo_How to unblock Weibo How to release someone blocked on Weibo_How to unblock Weibo Mar 30, 2024 am 09:51 AM

1. Click [My] in the lower right corner of the Weibo app to enter your personal homepage. 2. Then, click the gear icon in the upper right corner. 3. At this time, we enter the settings page, select [Block Settings] here to enter. 4. In the blocking settings, we can see the friends we have blocked, tap it, and click [Unblock] in the pop-up options. 5. Or users can directly search Weibo users to enter the blocked user homepage, and click [...] on the upper right to remove the blacklist.

How to set up real-time notifications for Weibo's special attention_How to set up real-time notifications for Weibo's special attention How to set up real-time notifications for Weibo's special attention_How to set up real-time notifications for Weibo's special attention Mar 29, 2024 pm 10:01 PM

1. Open Weibo, click My, and then click the Settings option in the upper right corner. 2. After opening the settings, find and click on the push notification settings inside. 3. After entering the push notification settings, find Special Interest and click Real-time Notification.

How to turn off Weibo live mode_How to turn off Weibo live mode How to turn off Weibo live mode_How to turn off Weibo live mode Mar 30, 2024 am 10:16 AM

1. First open Weibo, click Search, and click More Hot Searches. 2. Then select a hot search to enter. 3. Then enter the hot search comment live mode, find and click the standard version in the upper right corner. 4. Finally, you can return to the hot search standard comment area. If you want to continue viewing the live comment area, click the live version in the upper right corner. Note: Only when there are a lot of real-time comments will it automatically enter the live mode. Some hot searches will automatically enter the live mode. Generally, hot searches are in standard mode.

How to post videos on Weibo without compressing the image quality_How to post videos on Weibo without compressing the image quality How to post videos on Weibo without compressing the image quality_How to post videos on Weibo without compressing the image quality Mar 30, 2024 pm 12:26 PM

1. First open Weibo on your mobile phone and click [Me] in the lower right corner (as shown in the picture). 2. Then click [Gear] in the upper right corner to open settings (as shown in the picture). 3. Then find and open [General Settings] (as shown in the picture). 4. Then enter the [Video Follow] option (as shown in the picture). 5. Then open the [Video Upload Resolution] setting (as shown in the picture). 6. Finally, select [Original Image Quality] to avoid compression (as shown in the picture).

How to change the video cover on Weibo_How to change the video cover on Weibo How to change the video cover on Weibo_How to change the video cover on Weibo Mar 29, 2024 pm 10:31 PM

1. First open Weibo, click on My, click on the avatar, and enter the personal homepage. 2. Then click on the background image. 3. Then click to change the video cover. 4. Finally, select the video in the album, click Next, and publish and upload it.

How to delete the draft box of 'Weibo'_Tutorial on deleting the draft box of 'Weibo' How to delete the draft box of 'Weibo'_Tutorial on deleting the draft box of 'Weibo' Mar 30, 2024 pm 12:46 PM

1. First open Weibo, click on My, and click on the draft box. 2. Then long press any draft and click to clear the draft box or delete the draft. 3. Finally click OK.

If Weibo svip deletes visitor records, can the other party still see it? If Weibo svip deletes visitor records, can the other party still see it? Apr 02, 2024 am 08:04 AM

Once Weibo users activate the SVIP function, they can manage visitor records, and if they visit a stranger's homepage, they can delete it before displaying it, so that they do not know that they are browsing the homepage. ?Can the other party still see it after deleting the visitor record on Weibo svip? Can the other party still see it after deleting the visitor record on Weibo svip? 1. After deleting the visitor record on svip, the other party cannot see it. It will only see the increase in the number of visitors, but it will not show the visitors. The user's ID and avatar. 2. Ordinary users and VIP users cannot delete or view access records. They can only see how many people have visited them. 3. Weibo access data will be updated at 8:00 every day. If you want to delete access records, you need to

See all articles