Sqoop配置安装
Sqoop是一个用来将Hadoop和关系型数据库中的数据相互转移的工具，可以将一个关系型数据库（例如 ： MySQL ,Oracle ,Postgres等）
Sqoop是一个用来将Hadoop和关系型数据库中的数据相互转移的工具，可以将一个关系型数据库（例如 ： MySQL ,Oracle ,Postgres等）中的数据导入到Hadoop的HDFS中，也可以将HDFS的数据导入到关系型数据库中。
Sqoop的User Guide地址：
1:tar zxvf sqoop-1.1.0.tar.gz
2:修改配置文件 /home/hadoopuser/sqoop-1.1.0/conf/sqoop-site.xml
一般只需要修改如下几个项：
sqoop.metastore.client.enable.autoconnect
sqoop.metastore.client.autoconnect.url
sqoop.metastore.client.autoconnect.username
sqoop.metastore.client.autoconnect.password
sqoop.metastore.server.location
sqoop.metastore.server.port
3：
bin/sqoop help
bin/sqoop help import
4:
[hadoopuser@master sqoop-1.1.0]$ bin/sqoop import --connect jdbc:mysql://localhost/ppc --table data_ip --username kwps -P
Enter password:
11/02/18 10:51:58 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not find appropriate Hadoop shim for 0.20.2
java.lang.RuntimeException: Could not find appropriate Hadoop shim for 0.20.2
at com.cloudera.sqoop.shims.ShimLoader.loadShim(ShimLoader.java:190)
at com.cloudera.sqoop.shims.ShimLoader.getHadoopShim(ShimLoader.java:109)
at com.cloudera.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:173)
at com.cloudera.sqoop.tool.ImportTool.init(ImportTool.java:81)
at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:411)
at com.cloudera.sqoop.Sqoop.run(Sqoop.java:134)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:170)
at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:196)
at com.cloudera.sqoop.Sqoop.main(Sqoop.java:205)
解决办法:
默认情况下:
./hadoop-0.20.2/conf/hadoop-env.sh
# Extra Java runtime options. Empty by default.
# export HADOOP_OPTS=-server
需要更改成:
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsqoop.shim.jar.dir=/home/hadoopuser/sqoop-1.1.0/shims"
特别需要注意的是：
Sqoop目前在Apache 版本的Hadoop 0.20.2上是无法使用的。
目前只支持CDH 3 beta 2版本。所以如果想使用的话，得升级到 CDH 3 beta 2版本了。
“Sqoop does not run with Apache Hadoop 0.20.2. The only supported platform is CDH 3 beta 2. It requires features of MapReduce not available in the Apache 0.20.2 release of Hadoop. You should upgrade to CDH 3 beta 2 if you want to run Sqoop 1.0.0.”
这个问题 已经被Cloudera 标记为 Major Bug，希望能尽快解决吧。
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1390
52
Java Errors: Hadoop Errors, How to Handle and Avoid
Jun 24, 2023 pm 01:06 PM
Java Errors: Hadoop Errors, How to Handle and Avoid When using Hadoop to process big data, you often encounter some Java exception errors, which may affect the execution of tasks and cause data processing to fail. This article will introduce some common Hadoop errors and provide ways to deal with and avoid them. Java.lang.OutOfMemoryErrorOutOfMemoryError is an error caused by insufficient memory of the Java virtual machine. When Hadoop is
Using Hadoop and HBase in Beego for big data storage and querying
Jun 22, 2023 am 10:21 AM
With the advent of the big data era, data processing and storage have become more and more important, and how to efficiently manage and analyze large amounts of data has become a challenge for enterprises. Hadoop and HBase, two projects of the Apache Foundation, provide a solution for big data storage and analysis. This article will introduce how to use Hadoop and HBase in Beego for big data storage and query. 1. Introduction to Hadoop and HBase Hadoop is an open source distributed storage and computing system that can
How to use PHP and Hadoop for big data processing
Jun 19, 2023 pm 02:24 PM
As the amount of data continues to increase, traditional data processing methods can no longer handle the challenges brought by the big data era. Hadoop is an open source distributed computing framework that solves the performance bottleneck problem caused by single-node servers in big data processing through distributed storage and processing of large amounts of data. PHP is a scripting language that is widely used in web development and has the advantages of rapid development and easy maintenance. This article will introduce how to use PHP and Hadoop for big data processing. What is HadoopHadoop is
Explore the application of Java in the field of big data: understanding of Hadoop, Spark, Kafka and other technology stacks
Dec 26, 2023 pm 02:57 PM
Java big data technology stack: Understand the application of Java in the field of big data, such as Hadoop, Spark, Kafka, etc. As the amount of data continues to increase, big data technology has become a hot topic in today's Internet era. In the field of big data, we often hear the names of Hadoop, Spark, Kafka and other technologies. These technologies play a vital role, and Java, as a widely used programming language, also plays a huge role in the field of big data. This article will focus on the application of Java in large
How to install Hadoop in linux
May 18, 2023 pm 08:19 PM
1: Install JDK1. Execute the following command to download the JDK1.8 installation package. wget--no-check-certificatehttps://repo.huaweicloud.com/java/jdk/8u151-b12/jdk-8u151-linux-x64.tar.gz2. Execute the following command to decompress the downloaded JDK1.8 installation package. tar-zxvfjdk-8u151-linux-x64.tar.gz3. Move and rename the JDK package. mvjdk1.8.0_151//usr/java84. Configure Java environment variables. echo'
Use PHP to achieve large-scale data processing: Hadoop, Spark, Flink, etc.
May 11, 2023 pm 04:13 PM
As the amount of data continues to increase, large-scale data processing has become a problem that enterprises must face and solve. Traditional relational databases can no longer meet this demand. For the storage and analysis of large-scale data, distributed computing platforms such as Hadoop, Spark, and Flink have become the best choices. In the selection process of data processing tools, PHP is becoming more and more popular among developers as a language that is easy to develop and maintain. In this article, we will explore how to leverage PHP for large-scale data processing and how
Data processing engines in PHP (Spark, Hadoop, etc.)
Jun 23, 2023 am 09:43 AM
In the current Internet era, the processing of massive data is a problem that every enterprise and institution needs to face. As a widely used programming language, PHP also needs to keep up with the times in data processing. In order to process massive data more efficiently, PHP development has introduced some big data processing tools, such as Spark and Hadoop. Spark is an open source data processing engine that can be used for distributed processing of large data sets. The biggest feature of Spark is its fast data processing speed and efficient data storage.
Comparison and application scenarios of Redis and Hadoop
Jun 21, 2023 am 08:28 AM
Redis and Hadoop are both commonly used distributed data storage and processing systems. However, there are obvious differences between the two in terms of design, performance, usage scenarios, etc. In this article, we will compare the differences between Redis and Hadoop in detail and explore their applicable scenarios. Redis Overview Redis is an open source memory-based data storage system that supports multiple data structures and efficient read and write operations. The main features of Redis include: Memory storage: Redis
