多集群下资源共享方案介绍
作者: Dong | 新浪微博: 西成懂 | 可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明 网址:http://dongxicheng.org/mapreduce-nextgen/multi-clusters-sharing-resources/ 本博客的文章集合: http://dongxicheng.org/recommend/ 1. 集群共
作者:Dong | 新浪微博:西成懂 | 可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明
网址:http://dongxicheng.org/mapreduce-nextgen/multi-clusters-sharing-resources/
本博客的文章集合:http://dongxicheng.org/recommend/
1. 集群共享带来的好处
在一些公司中,所有服务器按照业务逻辑被拆分成若干个集群,这些集群各自运行着相对独立的业务,并由专门的运维人员管理。
然而,由于集群中应用特点不同,它们的利用率可能不一致,同一时间段,有的集群业务繁忙,资源利用率高,甚至可能出现排队等待资源的情况,而另外一些集群则业务空闲,资源利用率很低。此外,多个集群也可能导致数据冗余度增加,比如一个集群中的业务需要另外一个集群中的数据,可能需要跨集群拷贝数据,进而占用更多的磁盘空间。
为了解决多集群带来的问题,可将多个集群统一为一个集群,这将带来以下几个好处:
(1)提高集群整体利用率。所有业务运行在一个大的集群中,并按需分配给各个应用程序,可达到资源错峰交谷的目的,提高系统整体利用率和均衡率。
(2)减少管理成本。多个集群合并成一个大集群后,便于统一管理和统一运维,可减少管理成本。
2. 多集群共享方案
有些公司中,同时存在多个Hadoop集群,这些集群运行着不同的业务线,并对实时性、资源需求、优先级等有不同要求。为了实现集群资源共享,可将多个Hadoop集群合并成一个,并由Hadoop作业调度器统负责资源管理和任务分配。为了对应原先的业务逻辑,作业调度器通常将用户分成若干个队列(每个业务线一个队列),并给每个队列分配一定比例的资源,作业调度器中包含资源分配逻辑,能够按照一定的策略将资源分配各这些队列,并灵活实现资源按需分配,以达到资源最高利用。
3. 多服务共享方案
Hadoop只是当前比较流行的开源软件之一,除了Hadoop,很多公司还有其他一些服务,比如Web Server,Thrift Server、DataBase Service等,同样,通常考虑到服务的重要性和将来的发展,会将这些服务单独部署到独立的几台机器上,在大部分情况下,很多机器的资源利用率很低,为了提高机器整体利用率和支持服务自动化部署,可将这些服务混搭地部署到各个服务器上,于是资源统一管理系统诞生了,业界典型代表是Google的Borg和腾讯的Torca,开源界代表是Mesos和YARN。
资源统一管理系统的另外一个值得一提的动机是数据中心编程,也就是说,可以把数据中心看成一台超大的服务器,由一个资源管理系统对所有的计算资源,包括CPU、内存、硬盘、端口号等,进行统一管理和调度,涉及到的关键技术包括资源分配、资源隔离、资源划分等,具体可参考腾讯Torca的设计文档。
4. 参考资料
(1)Torca:http://djt.qq.com/article-329-1.html
(2)Mesos:http://www.mesosproject.org/
(3)YARN:http://hadoop.apache.org/docs/r0.23.6/
原创文章,转载请注明: 转载自董的博客
本文链接地址: http://dongxicheng.org/mapreduce-nextgen/multi-clusters-sharing-resources/
作者:Dong,作者介绍:http://dongxicheng.org/about/
本博客的文章集合:http://dongxicheng.org/recommend/
Copyright © 2013
This feed is for personal, non-commercial use only.
The use of this feed on other websites breaches copyright. If this content is not in your news reader, it makes the page you are viewing an infringement of the copyright. (Digital Fingerprint:
)

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

1. First open the mobile web browser, search for the Weibo web version, and click the avatar button in the upper left corner after entering. 2. Then click Settings in the upper right corner. 3. Click the version switching option in settings. 4. Then select the color version option in the version switch. 5. Click Search to enter the search page. 6. After entering the keywords, click Find People. 7. When the search completion interface appears, click Filter. 8. Finally, enter the specific date in the release time column and click Filter.

1. Click [My] in the lower right corner of the Weibo app to enter your personal homepage. 2. Then, click the gear icon in the upper right corner. 3. At this time, we enter the settings page, select [Block Settings] here to enter. 4. In the blocking settings, we can see the friends we have blocked, tap it, and click [Unblock] in the pop-up options. 5. Or users can directly search Weibo users to enter the blocked user homepage, and click [...] on the upper right to remove the blacklist.

1. Open Weibo, click My, and then click the Settings option in the upper right corner. 2. After opening the settings, find and click on the push notification settings inside. 3. After entering the push notification settings, find Special Interest and click Real-time Notification.

1. First open Weibo, click Search, and click More Hot Searches. 2. Then select a hot search to enter. 3. Then enter the hot search comment live mode, find and click the standard version in the upper right corner. 4. Finally, you can return to the hot search standard comment area. If you want to continue viewing the live comment area, click the live version in the upper right corner. Note: Only when there are a lot of real-time comments will it automatically enter the live mode. Some hot searches will automatically enter the live mode. Generally, hot searches are in standard mode.

1. First open Weibo on your mobile phone and click [Me] in the lower right corner (as shown in the picture). 2. Then click [Gear] in the upper right corner to open settings (as shown in the picture). 3. Then find and open [General Settings] (as shown in the picture). 4. Then enter the [Video Follow] option (as shown in the picture). 5. Then open the [Video Upload Resolution] setting (as shown in the picture). 6. Finally, select [Original Image Quality] to avoid compression (as shown in the picture).

1. First open Weibo, click on My, click on the avatar, and enter the personal homepage. 2. Then click on the background image. 3. Then click to change the video cover. 4. Finally, select the video in the album, click Next, and publish and upload it.

1. First open Weibo, click on My, and click on the draft box. 2. Then long press any draft and click to clear the draft box or delete the draft. 3. Finally click OK.

Once Weibo users activate the SVIP function, they can manage visitor records, and if they visit a stranger's homepage, they can delete it before displaying it, so that they do not know that they are browsing the homepage. ?Can the other party still see it after deleting the visitor record on Weibo svip? Can the other party still see it after deleting the visitor record on Weibo svip? 1. After deleting the visitor record on svip, the other party cannot see it. It will only see the increase in the number of visitors, but it will not show the visitors. The user's ID and avatar. 2. Ordinary users and VIP users cannot delete or view access records. They can only see how many people have visited them. 3. Weibo access data will be updated at 8:00 every day. If you want to delete access records, you need to
