大数据分析:结合Hadoop或ElasticMapReduce使用Hunk
作者 Jonathan Allen ，译者 张晓鹏 Hunk是Splunk公司一款比较新的产品，用来对Hadoop和其它NoSQL数据存储进行探测和可视化，它的新版本将会支持亚马逊的Elastic MapReduce。 结合Hadoop使用Hunk Hadoop由两个单元组成，首先是被称为HDFS的存储单元，HDFS可
作者 Jonathan Allen ，译者 张晓鹏
Hunk是Splunk公司一款比较新的产品，用来对Hadoop和其它NoSQL数据存储进行探测和可视化，它的新版本将会支持亚马逊的Elastic MapReduce。
结合Hadoop使用Hunk
Hadoop由两个单元组成，首先是被称为HDFS的存储单元，HDFS可以分布在成千上万个复制的节点上。接下来是MapReduce单元，它负责跟踪和管理被命名为map-reduce jobs的作业。
之前，开发者会用到Splunk Hadoop Connect (SHC)连接器。SHC通过常用的推模型（push model）来输出数据到Hadoop中，这块处理相当地好，但相反方向的处理却可能会有问题。当通过Splunk来探测数据时，原始的数据会被吸收到Splunk Server来检索和处理。就像人们猜想的那样，这个过程并没有发挥出Hadoop计算能力的优势。
Hunk通过提供与Hadoop MapReduce节点协同工作的适配器来解决这个问题。Splunk的查询被转化成Hadoop MapReduce的作业，这些作业在Hadoop集群中处理，最终只有结果被取回到Splunk 服务器中进行分析和可视化。
通过这种方式，Hunk提供了抽象层，以便用户和开发者不需要关心怎么去写Hadoop MapReduce的作业。Hunk还能在MapReduce作业启动前就提供结果预览，以减少无用搜索的数量。
结合Elastic MapReduce使用Hunk
亚马逊的Elastic MapReduce可以看做是对Hadoop的补充，同时也是Hadoop的竞争者。EMR既可以运行在Hadoop HDFS集群上，也可以直接运行在AWS S3上。亚马逊宣称使用AWS S3的优势在于比HDFS集群更易于管理。
当运行Elastic MapReduce时，Hunk提供了相同的抽象层和预览功能，就像它在Hadoop上做的一样。所以从用户的观点来看，在Hadoop和EMR之间切换不会造成什么变化。
云上的Hunk
在云上托管Hunk的传统方法是买一个标准版的许可证，然后部署到虚拟机中，这和你现场安装一样简单。接下来是对Hunk的运行实例进行手动配置以让它对应到正确的Hadoop或AWS集群上。
这个月的新版本里，Hunk的运行实例可以在AWS上进行自动化配置，这包括自动发现EMR数据源，这样Hunk实例可以在几分钟内上线使用。为了充分利用这个优势，Hunk运行实例是按小时来计费。
虚索引（Virtual Indexes）
Hunk中的有个关键概念是“虚索引（Virtual Indexes）”。这些索引已不同原本的意义，变成只是由Hunk来体现Hadoop和EMR集群处理的一种方式。从Splunk的用户界面上看，它们像是真正的索引，即使其数据处理是在map-reduce作业中完成的。并且，由于看起来像索引，你可以在它们之上创建持久的二级索引（persistent secondary indexes）。当你要处理部分数据，然后进一步检查或在多个方面可视化时，这个持久的二级索引会非常有用。
查看英文原文：Big Data Analytics: Using Hunk with Hadoop and Elastic MapReduce
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1386
52
Read CSV files and perform data analysis using pandas
Jan 09, 2024 am 09:26 AM
Pandas is a powerful data analysis tool that can easily read and process various types of data files. Among them, CSV files are one of the most common and commonly used data file formats. This article will introduce how to use Pandas to read CSV files and perform data analysis, and provide specific code examples. 1. Import the necessary libraries First, we need to import the Pandas library and other related libraries that may be needed, as shown below: importpandasaspd 2. Read the CSV file using Pan
Introduction to data analysis methods
Jan 08, 2024 am 10:22 AM
Common data analysis methods: 1. Comparative analysis method; 2. Structural analysis method; 3. Cross analysis method; 4. Trend analysis method; 5. Cause and effect analysis method; 6. Association analysis method; 7. Cluster analysis method; 8 , Principal component analysis method; 9. Scatter analysis method; 10. Matrix analysis method. Detailed introduction: 1. Comparative analysis method: Comparative analysis of two or more data to find the differences and patterns; 2. Structural analysis method: A method of comparative analysis between each part of the whole and the whole. ; 3. Cross analysis method, etc.
11 basic distributions that data scientists use 95% of the time
Dec 15, 2023 am 08:21 AM
Following the last inventory of "11 Basic Charts Data Scientists Use 95% of the Time", today we will bring you 11 basic distributions that data scientists use 95% of the time. Mastering these distributions helps us understand the nature of the data more deeply and make more accurate inferences and predictions during data analysis and decision-making. 1. Normal Distribution Normal Distribution, also known as Gaussian Distribution, is a continuous probability distribution. It has a symmetrical bell-shaped curve with the mean (μ) as the center and the standard deviation (σ) as the width. The normal distribution has important application value in many fields such as statistics, probability theory, and engineering.
11 Advanced Visualizations for Data Analysis and Machine Learning
Oct 25, 2023 am 08:13 AM
Visualization is a powerful tool for communicating complex data patterns and relationships in an intuitive and understandable way. They play a vital role in data analysis, providing insights that are often difficult to discern from raw data or traditional numerical representations. Visualization is crucial for understanding complex data patterns and relationships, and we will introduce the 11 most important and must-know charts that help reveal the information in the data and make complex data more understandable and meaningful. 1. KSPlotKSPlot is used to evaluate distribution differences. The core idea is to measure the maximum distance between the cumulative distribution functions (CDF) of two distributions. The smaller the maximum distance, the more likely they belong to the same distribution. Therefore, it is mainly interpreted as a "system" for determining distribution differences.
Machine learning and data analysis using Go language
Nov 30, 2023 am 08:44 AM
In today's intelligent society, machine learning and data analysis are indispensable tools that can help people better understand and utilize large amounts of data. In these fields, Go language has also become a programming language that has attracted much attention. Its speed and efficiency make it the choice of many programmers. This article introduces how to use Go language for machine learning and data analysis. 1. The ecosystem of machine learning Go language is not as rich as Python and R. However, as more and more people start to use it, some machine learning libraries and frameworks
How to use ECharts and php interfaces to implement data analysis and prediction of statistical charts
Dec 17, 2023 am 10:26 AM
How to use ECharts and PHP interfaces to implement data analysis and prediction of statistical charts. Data analysis and prediction play an important role in various fields. They can help us understand the trends and patterns of data and provide references for future decisions. ECharts is an open source data visualization library that provides rich and flexible chart components that can dynamically load and process data by using the PHP interface. This article will introduce the implementation method of statistical chart data analysis and prediction based on ECharts and php interface, and provide
Explore the perfect combination of PHP and blockchain
Mar 27, 2024 pm 02:12 PM
Exploring the perfect combination of PHP and blockchain As blockchain technology matures and its application scope continues to expand, people begin to try to combine traditional programming languages with blockchain to achieve more types of applications. In this context, PHP language, as a widely used network programming language, has also begun to explore its combination with blockchain technology. This article will focus on the combination of PHP language and blockchain technology, which will involve specific code examples for readers to better understand. What is blockchain? First, let's
Integrated Excel data analysis
Mar 21, 2024 am 08:21 AM
1. In this lesson, we will explain integrated Excel data analysis. We will complete it through a case. Open the course material and click on cell E2 to enter the formula. 2. We then select cell E53 to calculate all the following data. 3. Then we click on cell F2, and then we enter the formula to calculate it. Similarly, dragging down can calculate the value we want. 4. We select cell G2, click the Data tab, click Data Validation, select and confirm. 5. Let’s use the same method to automatically fill in the cells below that need to be calculated. 6. Next, we calculate the actual wages and select cell H2 to enter the formula. 7. Then we click on the value drop-down menu to click on other numbers.
