What are hive built-in functions?

hive built-in functions: 1. User-defined functions to process data; 2. Used to solve the need to input one line and output multiple lines [(On-to-many mapping)]; 3. User-defined aggregation Function, operates on multiple data rows and produces one data row.

#The operating environment of this article: Windows 7 system, Dell G3 computer.

hive built-in function:

Definition:

UDF (User-Defined-Function), user-defined function pair The data is processed.

UDTF (User-Defined Table-Generating Functions) is used to solve the requirement of inputting one line and outputting multiple lines (On-to-many mapping).

UDAF (User Defined Aggregation Function) is a user-defined aggregation function that operates on multiple data rows and generates one data row.

Usage:

1. The UDF function can be directly applied to the select statement, format the query structure, and then output the content.

2. When writing UDF functions, you need to pay attention to the following points:

a) Custom UDF needs to inherit org.apache.hadoop.hive.ql.UDF.

b) Need to implement the evaluate function.

c) The evaluate function supports overloading.

hive’s local mode:

Most Hadoop jobs require the complete scalability provided by hadoop to process big data. However, sometimes the amount of input data to hive is very small. In this case, the time consumed to execute the task for the query may be much longer than the actual job execution time. For most of these situations, Hive can handle all tasks on a single machine through local mode. For small data sets, the execution time is significantly reduced.

In this way, operations with a relatively small amount of data can be executed locally, which is much faster than submitting tasks to the cluster for execution.

Configure the following parameters to enable Hive’s local mode:

hive> set hive.exec.mode.local.auto=true;(默认为false)

Only when a job meets the following conditions can it truly use local mode:

1. The input data size of the job must be smaller than the parameter: hive.exec.mode.local.auto.inputbytes.max (default 128MB)

2. The number of maps of the job must be smaller than the parameter: hive.exec.mode .local.auto.tasks.max (default 4)

　 3. The reduce number of job must be 0 or 1

Related free learning recommendations: php programming(Video)

The above is the detailed content of What are hive built-in functions?. For more information, please follow other related articles on the PHP Chinese website!