Apache Hive入门2
我的偏见 : 对于互联网公司来说提炼挖掘生产中用户产生的大量日志是个有价值的工作,在这些看似垃圾东西中蕴含着大量的商业机会和用户的需求,如果可以简单的实现当然皆大欢喜,但是面对庞大的输出日志需要去提炼的时候只好束手无则无视他的存在,就拿我们最
我的偏见:
对于互联网公司来说提炼挖掘生产中用户产生的大量日志是个有价值的工作,在这些看似垃圾东西中蕴含着大量的商业机会和用户的需求,如果可以简单的实现当然皆大欢喜,但是面对庞大的输出日志需要去提炼的时候只好束手无则无视他的存在,就拿我们最熟悉的Log4J来说,如果是上TB的海量文本类型输出的格式根本无法查询,更谈不上数据挖掘。
当然我们也知道存放在数据库里面当然是不错的选择,对抗海量的日志数据存储与查询坚持做下去的话最后换来的是高昂的代价,Hive借助MapReduce的计算+HDFS的海量存储的功能,对与海量的日志存储、查询,数据挖掘Hive也是个不错的选择,在功能上和总体成本上高于刚刚所说的前两者。
但是对与存储在其他类型的NOSQL产品,Hive投入正式生产的环境我们还需要进一步的分析与比较,例如:MongoDB vs Hive,这个2个产品对于海量日志数据挖掘的性能与系统的扩展性来说谁将会比谁更胜一筹喃?因为 MongoDB 中也是支持海量级分布式存储,并且MongoDB也支持了MapReduce算法,这样我就需要为此得出一个结论,或者说我需要一个折中的方案,但不是现在。
Hive 中不支持对数据的改写和添加,所有的数据都是在加载的时候中确定好的。Hive中你添加了数据就无法删除的,SQL的fans们是不是很费解?我是这样认为的就拿Google的Google Analytics分析工具为例子,在Google Analytics分析工具 会有对日志进行删除的操作吗?答案是“NO!”,也许在Hive设计的时候Facebook的工程师就认为对需要进行挖掘的数据删除是一项没有必要的工作,Hive在Facebook能成功,并且运行在上千台的节点上 或许就是Facebook工程师们明确了Hive不去做的事情,让Hive只做好那一部分事情。
以上是我对 Hive Why(为什么使用Hive)片面的观点。
集群中的Hive
Hive的失效转发可以设计成这样的,将元数据和存储的数据分别保存在同一个位置,如图所示:
多个Hive的物理节点连接到相同的数据库和HDFS环境,防止Hive Thrift Server单点失效(SPFO)问题。
将Hive的MetaData 存储在 MySQL中,MySQL的运行环境支持双向同步或者集群,这样至少2台数据库服务器上热备份着Hive的元数据存储,例如:
多个物理Hive节点的数据内容保存在HDFS上,通过修改 hive-default.xml 配置 文件,指向NameNode节点即可,例如:
/user/hive/warehouse –>
Hive 与 Log4J
在Hive中建一张表叫user_log,里面含有4个字段,以\t划分,一行一条数据,建表的脚本如下:
CREATE TABLE USER_LOG(DateInfo STRING,LogName STRING,LogLevel STRING,MSG STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' STORED AS TEXTFILE;
在Log4J的日子输出格式中每个栏目以Tab相隔开,中间不是空格而是Tab,例如:
log4j.appender.A1.layout.ConversionPattern=%d %-6p %l %m %n
输入数据文件格式如下:
在Hive的控制台下将Log4J输出的日志文件导入Hive中,操作结果如下,如图所示:
在建表语句中还可以加入正则表达式,的自定义功能都在contrib里面, 例如:将WEB日志先用正则表达式进行组合,再按需要的条件进行组合输入到表中
add jar ../build/contrib/hive_contrib.jar;
CREATE TABLE apachelog (
host STRING,
identity STRING,
user STRING,
time STRING,
request STRING,
status STRING,
size STRING,
referer STRING,
agent STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
"input.regex" = "([^ ]*) ([^ ]*) ([^ ]*) (-|\\[[^\\]]*\\]) ([^ \"]*|\"[^\"]*\") (-|[0-9]*) (-|[0-9]*)(?: ([^ \"]*|\"[^\"]*\") ([^ \"]*|\"[^\"]*\"))?",
"output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s"
)STORED AS TEXTFILE;
对于Hive的学习方向:
1、hive 的 hive-default.xml 配置文件中的参数细节,主要针对Hive的性能优化。
2、Hive 是如何使用MapReduce来完成工作的整套机制。
我的口水,目前使用Hadoop产品的国内外公司有如下一些:
Facebook
处理内部产生的日志和数据挖掘、展示以及机器学习
IBM
用Hadoop MapReduce 分析billions of lines of GPS data 并产生交通路线资讯.
Krugle
用Hadoop and Nutch 建构原始码搜寻引擎
SEDNS – Security Enhanced DNS Group
收集全世界的DNS 以探索网路分散式内容.
Technical analysis and Stock Research
分析股票资讯
University of Nebraska Lincoln, Research Computing Facility
用Hadoop跑约200TB的Compact Muon Solenoid经验分析紧凑渺子线圈,该机构是为瑞士欧洲核子研究组织CERN的大型强子对撞器计划的两大通用型粒子侦测器中的一个。
Yahoo!
使用Hadoop平台来发现发送垃圾邮件的僵尸网络
趋势科技
过滤像是钓鱼网站或恶意连结的网页内容
还有以前提到过的:A9.com、ImageShack、Last.fm、Powerset、纽约时报
在国内使用Hadoop的公司有:
淘宝、中国移动研究院、英特尔研究院、金山软件、百度、腾讯、新浪、搜狐。
相关文章:
Apache Hive入门1
–end–
原文地址:Apache Hive入门2, 感谢原作者分享。

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

Kimi: In just one sentence, in just ten seconds, a PPT will be ready. PPT is so annoying! To hold a meeting, you need to have a PPT; to write a weekly report, you need to have a PPT; to make an investment, you need to show a PPT; even when you accuse someone of cheating, you have to send a PPT. College is more like studying a PPT major. You watch PPT in class and do PPT after class. Perhaps, when Dennis Austin invented PPT 37 years ago, he did not expect that one day PPT would become so widespread. Talking about our hard experience of making PPT brings tears to our eyes. "It took three months to make a PPT of more than 20 pages, and I revised it dozens of times. I felt like vomiting when I saw the PPT." "At my peak, I did five PPTs a day, and even my breathing was PPT." If you have an impromptu meeting, you should do it

In the early morning of June 20th, Beijing time, CVPR2024, the top international computer vision conference held in Seattle, officially announced the best paper and other awards. This year, a total of 10 papers won awards, including 2 best papers and 2 best student papers. In addition, there were 2 best paper nominations and 4 best student paper nominations. The top conference in the field of computer vision (CV) is CVPR, which attracts a large number of research institutions and universities every year. According to statistics, a total of 11,532 papers were submitted this year, and 2,719 were accepted, with an acceptance rate of 23.6%. According to Georgia Institute of Technology’s statistical analysis of CVPR2024 data, from the perspective of research topics, the largest number of papers is image and video synthesis and generation (Imageandvideosyn

We know that LLM is trained on large-scale computer clusters using massive data. This site has introduced many methods and technologies used to assist and improve the LLM training process. Today, what we want to share is an article that goes deep into the underlying technology and introduces how to turn a bunch of "bare metals" without even an operating system into a computer cluster for training LLM. This article comes from Imbue, an AI startup that strives to achieve general intelligence by understanding how machines think. Of course, turning a bunch of "bare metal" without an operating system into a computer cluster for training LLM is not an easy process, full of exploration and trial and error, but Imbue finally successfully trained an LLM with 70 billion parameters. and in the process accumulate

Editor of the Machine Power Report: Yang Wen The wave of artificial intelligence represented by large models and AIGC has been quietly changing the way we live and work, but most people still don’t know how to use it. Therefore, we have launched the "AI in Use" column to introduce in detail how to use AI through intuitive, interesting and concise artificial intelligence use cases and stimulate everyone's thinking. We also welcome readers to submit innovative, hands-on use cases. Video link: https://mp.weixin.qq.com/s/2hX_i7li3RqdE4u016yGhQ Recently, the life vlog of a girl living alone became popular on Xiaohongshu. An illustration-style animation, coupled with a few healing words, can be easily picked up in just a few days.

Retrieval-augmented generation (RAG) is a technique that uses retrieval to boost language models. Specifically, before a language model generates an answer, it retrieves relevant information from an extensive document database and then uses this information to guide the generation process. This technology can greatly improve the accuracy and relevance of content, effectively alleviate the problem of hallucinations, increase the speed of knowledge update, and enhance the traceability of content generation. RAG is undoubtedly one of the most exciting areas of artificial intelligence research. For more details about RAG, please refer to the column article on this site "What are the new developments in RAG, which specializes in making up for the shortcomings of large models?" This review explains it clearly." But RAG is not perfect, and users often encounter some "pain points" when using it. Recently, NVIDIA’s advanced generative AI solution

Concurrency testing and debugging Concurrency testing and debugging in Java concurrent programming are crucial and the following techniques are available: Concurrency testing: Unit testing: Isolate and test a single concurrent task. Integration testing: testing the interaction between multiple concurrent tasks. Load testing: Evaluate an application's performance and scalability under heavy load. Concurrency Debugging: Breakpoints: Pause thread execution and inspect variables or execute code. Logging: Record thread events and status. Stack trace: Identify the source of the exception. Visualization tools: Monitor thread activity and resource usage.

To add a server to Eclipse, follow these steps: Create a server runtime environment Configure the server Create a server instance Select the server runtime environment Configure the server instance Start the server deployment project

On July 24, Kuaishou video generation large model Keling AI announced that the basic model has been upgraded again and is fully open for internal testing. Kuaishou said that in order to allow more users to use Keling AI and better meet the different levels of usage needs of creators, from now on, on the basis of fully open internal testing, it will also officially launch a membership system for different categories of members. Provide corresponding exclusive functional services. At the same time, the basic model of Keling AI has also been upgraded again to further enhance the user experience. The basic model effect has been upgraded to further improve the user experience. Since its release more than a month ago, Keling AI has been upgraded and iterated many times. With the launch of this membership system, the basic model effect of Keling AI has once again undergone transformation. The first is that the picture quality has been significantly improved. The visual quality generated through the upgraded basic model
