Cloudera Manager 5 和 CDH5离线安装参考-Mysql Tutorial-php.cn

Cloudera Manager 5 和 CDH5离线安装参考

WBOY

Release： 2016-06-07 16:38:15

Original

1857 people have browsed it

0. ?主机规划 Cloudera Manager需要一台单独的主机 debugo01 Cloudera Manager, yum repo debugo02 NameNode, DataNode, Yarn RM, Spark Master, Spark Worker debugo03 DataNode, Spark Worker, Secondery NameNode 1. 准备工作(需要在所有节点操作) a). 网

0. ?主机规划

Cloudera Manager需要一台单独的主机
debugo01 Cloudera Manager, yum repo
debugo02 NameNode, DataNode, Yarn RM, Spark Master, Spark Worker …
debugo03 DataNode, Spark Worker, Secondery NameNode …

1. 准备工作(需要在所有节点操作)

a). 网络配置、 /etc/hosts或DNS 、hostname配置（略）
b). 时区和ntp

cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
chkconfig ntpd on

Copy after login

c). 关闭selinux和iptables

chkconfig iptables off
chkconfig ip6tables off
sed -i 's/^SELINUX.*/SELINUX=disabled/' /etc/selinux/config

Copy after login

d). 配置ISO源

/etc/yum.repos.d/CentOS-ISO.repo
[base]
name=CentOS-$releasever - ISO
baseurl=http://debugo01/CentOS_6.5_Final
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
# 在debugo01上挂载镜像文件到http目录下
yum install -y httpd
mount -o loop CentOS*.iso /var/www/html

Copy after login

e). 安装包
所有主机都需要安装mysql-connector、jdk和scala（需要去Oracle和Scala官方下载1.7最新版）

yum install -y mysql-connector-java
rpm -qa | grep java    #（需要卸载openjdk-1.6)
rpm -ivh jdk-1.7.0_67-fcs.x86_64.rpm
rpm -ivh scala-2.10.4-400.noarch.rpm

Copy after login

f). 重启主机

2. CM Server配置

离线安装有两种方式，官方推荐使用parcel来进行离线安装。但是一直尝试失败，下面使用yum repo进行离线安装。
a). 安装软件包

yum install mysql-server mysql createrepo
chkconfig mysqld on

Copy after login

b). 下载Cloudera Manager

wget http://archive-primary.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.1.1_x86_64.tar.gz
tar xvzf cloudera-manager-el6-cm5.1.1_x86_64.tar.gz -C /opt

Copy after login

c). 创建MySQL用户

mysql -uroot -e "CREATE DATABASE cm DEFAULT CHARSET utf8 COLLATE utf8_general_ci"
mysql -uroot -e "CREATE DATABASE cdh DEFAULT CHARSET utf8 COLLATE utf8_general_ci"
mysql -uroot -e 'GRANT all on cm.* TO cm@"%" IDENTIFIED BY "cm"'
mysql -uroot -e 'GRANT all on cdh.* TO cdh@"%" IDENTIFIED BY "cdh"'
mysqladmin flush-privileges

Copy after login

d).同步yum repo
最新的repo请在http://archive.cloudera.com/ 中下载

vim /etc/yum.repo.d/cloudera-cdh5.repo 
[cloudera-cdh5]
# Packages for Cloudera's Distribution for Hadoop, Version 5, on RedHat	or CentO
S 6 x86_64
name=Cloudera's Distribution for Hadoop, Version 5
baseurl=http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5/
gpgkey = http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloude
ra    
gpgcheck = 1
[cloudera-manager]
name = Cloudera Manager, Version 5.1.1
baseurl = http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.1.1/
gpgkey = http://archive.cloudera.com/redhat/cdh/RPM-GPG-KEY-cloudera
gpgcheck = 1
enabled=1
cd /var/www/html
mkdir cdh5 && cd cdh5
reposync -r cloudera-cdh5
createrepo .
ln -s /var/www/html/cdh5/cloudera-cdh5/RPMS /var/www/html/cdh5/RPMS
# 漫长得等待...
mkdir ../cm && cd cm
reposync -r cloudera-manager
createrepo .

Copy after login

在debugo02,debugo03上测试http访问，并配置下面的repo信息。
[cloudera-cdh5]
# Packages for Cloudera’s Distribution for Hadoop, Version 5, on RedHat or CentOS 6 x86_64
name=Cloudera’s Distribution for Hadoop, Version 5
baseurl=http://debugo01/cdh5
gpgcheck = 0
enabled = 1

[cloudera-manager]
name = Cloudera Manager, Version 5.1.1
baseurl = http://debugo01/cm
gpgcheck = 0
enabled=1
e). 配置cm

sed -i 's/^server_host.*/server_host=debugo01/' /opt/cm-5.1.1/etc/cloudera-scm-agent/config.ini
/opt/cm-5.1.1/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p --scm-host localhost scm scm scm
Enter database password: 
Verifying that we can write to /opt/cm-5.1.1/etc/cloudera-scm-server
Creating SCM configuration file in /opt/cm-5.1.1/etc/cloudera-scm-server
Executing:  /usr/java/latest/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.1.1/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cm-5.1.1/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

Copy after login

f). 启动cm

/opt/cm-5.1.1/etc/init.d/cloudera-scm-agent start
/opt/cm-5.1.1/etc/init.d/cloudera-scm-server start

Copy after login

3. 安装CDH

a). 通过浏览器访问debugo01，使用admin/admin登录
b). 选择Cloudera Express来创建cluster。express是cloudera的免费版，相比企业版只是少了升级和一些高级特性（http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installation-Guide/cm5ig_feature_differences.html?scroll=cmfeig_topic_5_1）
c). 接下来Specify hosts for your CDH cluster installation，添加新主机debugo02,并将Currently Managed Hosts中的cm-server主机debugo01也勾选上。
d). Select Repository – Use Parcel， CDH – CDH-5.1.0-1.cdh5.1.0.p0.53
e). Provide SSH login credentials. 使用root并输入root密码
f). Cluster Installation
这里会遇到一个DNS反向解析错误，不能正确解析Cloudera Manager Server主机名的问题，解决方法将host工具改名，安装成功后再修改回来。
mv /usr/bin/host /usr/bin/host.bak
为了加快安装，可以在安装开始后手工删除debugo02、debugo03中/etc/yum.repos.d/新生成的repo文件，让cluster的直接使用本地源。
g). 安装后进入集群配置界面，输入前面创建的cdh数据库。根据需求配置角色。完成查看host和集群状态正常,安装完成。很多细节的配置警告可以根据提示来调整。