Home > Database > Mysql Tutorial > Hadoop集群(CDH4)实践之 (3) Hive搭建

Hadoop集群(CDH4)实践之 (3) Hive搭建

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB
Release: 2016-06-07 16:32:53
Original
1174 people have browsed it

目录结构 Hadoop集群(CDH4)实践之 (0) 前言 Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建 Hadoop集群(CDH4)实践之 (2) HBaseZookeeper搭建 Hadoop集群(CDH4)实践之 (3) Hive搭建 Hadoop集群(CHD4)实践之 (4) Oozie搭建 Hadoop集群(CHD4)实践之 (5) Sqoop安

目录结构
Hadoop集群(CDH4)实践之 (0) 前言
Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建
Hadoop集群(CDH4)实践之 (3) Hive搭建
Hadoop集群(CHD4)实践之 (4) Oozie搭建
Hadoop集群(CHD4)实践之 (5) Sqoop安装

本文内容
Hadoop集群(CDH4)实践之 (3) Hive搭建

参考资料
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/CDH4-Installation-Guide.html

环境准备
OS: CentOS 6.4 x86_64
Servers:
hadoop-master: 172.17.20.230 内存10G
- namenode
- hbase-master

hadoop-secondary: 172.17.20.234 内存10G
- secondarybackupnamenode,jobtracker
- hive-server,hive-metastore

hadoop-node-1: 172.17.20.231 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

hadoop-node-2: 172.17.20.232 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

hadoop-node-3: 172.17.20.233 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

对以上角色做一些简单的介绍:
namenode - 整个HDFS的命名空间管理服务
secondarynamenode - 可以看做是namenode的冗余服务
jobtracker - 并行计算的job管理服务
datanode - HDFS的节点服务
tasktracker - 并行计算的job执行服务
hbase-master - Hbase的管理服务
hbase-regionServer - 对Client端插入,删除,查询数据等提供服务
zookeeper-server - Zookeeper协作与配置管理服务
hive-server - Hive的管理服务
hive-metastore - Hive的元存储,用于对元数据进行类型检查与语法分析

本文定义的规范,避免在配置多台服务器上产生理解上的混乱:
以下操作都只需要在 Hive 所在主机,即 hadoop-secondary 上执行。

1. 安装前的准备
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建

2. 安装Hive
$ sudo yum install hive hive-metastore hive-server
$ sudo yum install hive-jdbc hive-hbase

3. 安装MySQL JDBC Connector
$ sudo yum install mysql-connector-java
$ sudo ln -s /usr/share/java/mysql-connector-java.jar /usr/lib/hive/lib/mysql-connector-java.jar

4. 安装MySQL
$ sudo yum install mysql-server
$ sudo /etc/init.d/mysqld start

$ sudo /usr/bin/mysql_secure_installation

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

[...]

Enter current password for root (enter for none):

OK, successfully used password, moving on...

[...]

Set root password? [Y/n] y

New password: hiveserver

Re-enter new password: hiverserver

Remove anonymous users? [Y/n] Y

[...]

Disallow root login remotely? [Y/n] N

[...]

Remove test database and access to it [Y/n] Y

[...]

Reload privilege tables now? [Y/n] Y

All done!

Copy after login

5. 创建数据库并授权
$ mysql -u root -phiveserver

1

2

3

4

5

6

7

8

9

10

11

12

mysql> CREATE DATABASE metastore;

mysql> USE metastore;

mysql> SOURCE /usr/lib/hive/scripts/metastore/upgrade/mysql/hive-schema-0.10.0.mysql.sql;

mysql> CREATE USER 'hive'@'%' IDENTIFIED BY 'hiveserver';

mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'%';

mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'%';

mysql> CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hiveserver';

mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'localhost';

mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'localhost';

mysql> CREATE USER 'hive'@'127.0.0.1' IDENTIFIED BY 'hiveserver';

mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'127.0.0.1';

mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'127.0.0.1';

Copy after login

6. 配置hive-site.xml
$ sudo vim /etc/hive/conf/hive-site.xml

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="http://heylinux.com/archives/configuration.xsl"?>

  javax.jdo.option.ConnectionURL

  jdbc:mysql://hadoop-secondary/metastore

  the URL of the MySQL database

  javax.jdo.option.ConnectionDriverName

  com.mysql.jdbc.Driver

  javax.jdo.option.ConnectionUserName

  hive

  javax.jdo.option.ConnectionPassword

  hiveserver

  datanucleus.autoCreateSchema

  false

  datanucleus.fixedDatastore

  true

  datanucleus.autoStartMechanism

  SchemaTable

  hive.metastore.uris

  thrift://hadoop-secondary:9083

  IP address (or fully-qualified domain name) and port of the metastore host

  hive.aux.jars.path

  file:////usr/lib/hive/lib/hbase.jar,file:///usr/lib/hive/lib/zookeeper.jar,file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.5.0.jar,file:///usr/lib/hive/lib/guava-11.0.2.jar

  hbase.zookeeper.quorum

  hadoop-node-1,hadoop-node-2,hadoop-node-3

Copy after login

7. 启动Hive
$ /etc/init.d/hive-metastore start
$ /etc/init.d/hive-server start

8. 创建Hive所需的HDFS目录
$ sudo -u hdfs hadoop fs -mkdir /user/hive
$ sudo -u hdfs hadoop fs -mkdir /user/hive/warehouse
$ sudo -u hdfs hadoop fs -ls -R /user
$ sudo -u hdfs hadoop fs -chown -R hive /user/hive
$ sudo -u hdfs hadoop fs -chmod -R 1777 /user/hive/warehouse

$ sudo -u hdfs hadoop fs -chmod -R 777 /tmp/hadoop-mapred
$ sudo -u hdfs hadoop fs -chmod -R 777 /tmp/hive-hive
$ sudo chown -R hive:hive /var/lib/hive/.hivehistory

9. 至此,Hive的搭建就已经完成。

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template