Home Database Mysql Tutorial Apache Hive入门2

Apache Hive入门2

Jun 07, 2016 pm 04:25 PM
apache hive internet getting Started company

我的偏见 : 对于互联网公司来说提炼挖掘生产中用户产生的大量日志是个有价值的工作,在这些看似垃圾东西中蕴含着大量的商业机会和用户的需求,如果可以简单的实现当然皆大欢喜,但是面对庞大的输出日志需要去提炼的时候只好束手无则无视他的存在,就拿我们最

我的偏见:
  对于互联网公司来说提炼挖掘生产中用户产生的大量日志是个有价值的工作,在这些看似垃圾东西中蕴含着大量的商业机会和用户的需求,如果可以简单的实现当然皆大欢喜,但是面对庞大的输出日志需要去提炼的时候只好束手无则无视他的存在,就拿我们最熟悉的Log4J来说,如果是上TB的海量文本类型输出的格式根本无法查询,更谈不上数据挖掘。
   当然我们也知道存放在数据库里面当然是不错的选择,对抗海量的日志数据存储与查询坚持做下去的话最后换来的是高昂的代价,Hive借助MapReduce的计算+HDFS的海量存储的功能,对与海量的日志存储、查询,数据挖掘Hive也是个不错的选择,在功能上和总体成本上高于刚刚所说的前两者。
   但是对与存储在其他类型的NOSQL产品,Hive投入正式生产的环境我们还需要进一步的分析与比较,例如:MongoDB vs Hive,这个2个产品对于海量日志数据挖掘的性能与系统的扩展性来说谁将会比谁更胜一筹喃?因为 MongoDB 中也是支持海量级分布式存储,并且MongoDB也支持了MapReduce算法,这样我就需要为此得出一个结论,或者说我需要一个折中的方案,但不是现在。
 
   Hive 中不支持对数据的改写和添加,所有的数据都是在加载的时候中确定好的。Hive中你添加了数据就无法删除的,SQL的fans们是不是很费解?我是这样认为的就拿Google的Google Analytics分析工具为例子,在Google Analytics分析工具 会有对日志进行删除的操作吗?答案是“NO!”,也许在Hive设计的时候Facebook的工程师就认为对需要进行挖掘的数据删除是一项没有必要的工作,Hive在Facebook能成功,并且运行在上千台的节点上 或许就是Facebook工程师们明确了Hive不去做的事情,让Hive只做好那一部分事情。

以上是我对 Hive Why(为什么使用Hive)片面的观点。
 

集群中的Hive
Hive的失效转发可以设计成这样的,将元数据和存储的数据分别保存在同一个位置,如图所示:
Hive Cluster

多个Hive的物理节点连接到相同的数据库和HDFS环境,防止Hive Thrift Server单点失效(SPFO)问题。

将Hive的MetaData 存储在 MySQL中,MySQL的运行环境支持双向同步或者集群,这样至少2台数据库服务器上热备份着Hive的元数据存储,例如:

  javax.jdo.option.ConnectionURL
  jdbc:mysql://192.168.1.203:3306/hive?createDatabaseIfNotExist=true

多个物理Hive节点的数据内容保存在HDFS上,通过修改 hive-default.xml 配置 文件,指向NameNode节点即可,例如:

  hive.metastore.warehouse.dir
   hdfs://serv1:9000/user/hive/warehouse
   /user/hive/warehouse –>
  location of default database for the warehouse


Hive 与 Log4J

在Hive中建一张表叫user_log,里面含有4个字段,以\t划分,一行一条数据,建表的脚本如下:
CREATE TABLE USER_LOG(DateInfo STRING,LogName STRING,LogLevel STRING,MSG STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' STORED AS TEXTFILE;

在Log4J的日子输出格式中每个栏目以Tab相隔开,中间不是空格而是Tab,例如:
log4j.appender.A1.layout.ConversionPattern=%d    %-6p    %l %m  %n

输入数据文件格式如下:
Hive Cluster
 
在Hive的控制台下将Log4J输出的日志文件导入Hive中,操作结果如下,如图所示:
Hive Cluster

在建表语句中还可以加入正则表达式,的自定义功能都在contrib里面, 例如:将WEB日志先用正则表达式进行组合,再按需要的条件进行组合输入到表中
add jar ../build/contrib/hive_contrib.jar;
CREATE TABLE apachelog (
host STRING,
identity STRING,
user STRING,
time STRING,
request STRING,
status STRING,
size STRING,
referer STRING,
agent STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
"input.regex" = "([^ ]*) ([^ ]*) ([^ ]*) (-|\\[[^\\]]*\\]) ([^ \"]*|\"[^\"]*\") (-|[0-9]*) (-|[0-9]*)(?: ([^ \"]*|\"[^\"]*\") ([^ \"]*|\"[^\"]*\"))?",
"output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s"
)STORED AS TEXTFILE;


对于Hive的学习方向:

1、hive 的 hive-default.xml 配置文件中的参数细节,主要针对Hive的性能优化。
2、Hive 是如何使用MapReduce来完成工作的整套机制。
 

我的口水,目前使用Hadoop产品的国内外公司有如下一些:
Facebook
    处理内部产生的日志和数据挖掘、展示以及机器学习
IBM
    用Hadoop MapReduce 分析billions of lines of GPS data 并产生交通路线资讯.
Krugle
    用Hadoop and Nutch 建构原始码搜寻引擎
SEDNS – Security Enhanced DNS Group
    收集全世界的DNS 以探索网路分散式内容.
Technical analysis and Stock Research
    分析股票资讯
University of Nebraska Lincoln, Research Computing Facility
    用Hadoop跑约200TB的Compact Muon Solenoid经验分析紧凑渺子线圈,该机构是为瑞士欧洲核子研究组织CERN的大型强子对撞器计划的两大通用型粒子侦测器中的一个。
Yahoo!
    使用Hadoop平台来发现发送垃圾邮件的僵尸网络
趋势科技
    过滤像是钓鱼网站或恶意连结的网页内容
还有以前提到过的:A9.com、ImageShack、Last.fm、Powerset、纽约时报
 
国内使用Hadoop的公司有:
淘宝、中国移动研究院、英特尔研究院、金山软件、百度、腾讯、新浪、搜狐。

相关文章:
Apache Hive入门1

–end–

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to set the cgi directory in apache How to set the cgi directory in apache Apr 13, 2025 pm 01:18 PM

To set up a CGI directory in Apache, you need to perform the following steps: Create a CGI directory such as "cgi-bin", and grant Apache write permissions. Add the "ScriptAlias" directive block in the Apache configuration file to map the CGI directory to the "/cgi-bin" URL. Restart Apache.

How to start apache How to start apache Apr 13, 2025 pm 01:06 PM

The steps to start Apache are as follows: Install Apache (command: sudo apt-get install apache2 or download it from the official website) Start Apache (Linux: sudo systemctl start apache2; Windows: Right-click the "Apache2.4" service and select "Start") Check whether it has been started (Linux: sudo systemctl status apache2; Windows: Check the status of the "Apache2.4" service in the service manager) Enable boot automatically (optional, Linux: sudo systemctl

How to delete more than server names of apache How to delete more than server names of apache Apr 13, 2025 pm 01:09 PM

To delete an extra ServerName directive from Apache, you can take the following steps: Identify and delete the extra ServerName directive. Restart Apache to make the changes take effect. Check the configuration file to verify changes. Test the server to make sure the problem is resolved.

How to check Debian OpenSSL configuration How to check Debian OpenSSL configuration Apr 12, 2025 pm 11:57 PM

This article introduces several methods to check the OpenSSL configuration of the Debian system to help you quickly grasp the security status of the system. 1. Confirm the OpenSSL version First, verify whether OpenSSL has been installed and version information. Enter the following command in the terminal: If opensslversion is not installed, the system will prompt an error. 2. View the configuration file. The main configuration file of OpenSSL is usually located in /etc/ssl/openssl.cnf. You can use a text editor (such as nano) to view: sudonano/etc/ssl/openssl.cnf This file contains important configuration information such as key, certificate path, and encryption algorithm. 3. Utilize OPE

How to use Debian Apache logs to improve website performance How to use Debian Apache logs to improve website performance Apr 12, 2025 pm 11:36 PM

This article will explain how to improve website performance by analyzing Apache logs under the Debian system. 1. Log Analysis Basics Apache log records the detailed information of all HTTP requests, including IP address, timestamp, request URL, HTTP method and response code. In Debian systems, these logs are usually located in the /var/log/apache2/access.log and /var/log/apache2/error.log directories. Understanding the log structure is the first step in effective analysis. 2. Log analysis tool You can use a variety of tools to analyze Apache logs: Command line tools: grep, awk, sed and other command line tools.

How to view your apache version How to view your apache version Apr 13, 2025 pm 01:15 PM

There are 3 ways to view the version on the Apache server: via the command line (apachectl -v or apache2ctl -v), check the server status page (http://<server IP or domain name>/server-status), or view the Apache configuration file (ServerVersion: Apache/<version number>).

How to connect to the database of apache How to connect to the database of apache Apr 13, 2025 pm 01:03 PM

Apache connects to a database requires the following steps: Install the database driver. Configure the web.xml file to create a connection pool. Create a JDBC data source and specify the connection settings. Use the JDBC API to access the database from Java code, including getting connections, creating statements, binding parameters, executing queries or updates, and processing results.

What to do if the apache80 port is occupied What to do if the apache80 port is occupied Apr 13, 2025 pm 01:24 PM

When the Apache 80 port is occupied, the solution is as follows: find out the process that occupies the port and close it. Check the firewall settings to make sure Apache is not blocked. If the above method does not work, please reconfigure Apache to use a different port. Restart the Apache service.

See all articles