hadoop第一个程序WordCount.java的编译运行过程
java是hadoop开发的标准官方语言，本文下载了官方的WordCount.java并对其进行了编译和打包，然后使用测试数据运行了该hadoop程序。 这里假定已经装好了hadoop的环境，在Linux下运行hadoop命令能够正常执行； 下载java版本的WordCount.java程序。 将WordCount
java是hadoop开发的标准官方语言，本文下载了官方的WordCount.java并对其进行了编译和打包，然后使用测试数据运行了该hadoop程序。
这里假定已经装好了hadoop的环境，在Linux下运行hadoop命令能够正常执行；
下载java版本的WordCount.java程序。
将WordCount.java复制到linux下的一个目录，这里我复制到/home/crazyant/hadoop_wordcount
[crazyant@dev.mechine hadoop_wordcount]$ ll
total 4
-rwxr–r–? 1 crazyant crazyant 1921 Aug 16 20:03 WordCount.java
在该目录（/home/crazyant/hadoop_wordcount）下创建wordcount_classes目录，用于存放编译WordCount.java生成的class文件。
[crazyant@dev.mechine hadoop_wordcount]$ mkdir wordcount_classes
[crazyant@dev.mechine hadoop_wordcount]$ ll
total 8
drwxrwxr-x? 2 crazyant crazyant 4096 Aug 16 20:07 wordcount_classes
-rwxr–r–? 1 crazyant crazyant 1921 Aug 16 20:03 WordCount.java
编译WordCount.java文件，其中-classpath选项表示要引用hadoop官方的包，-d选项表示要将编译后的class文件生成的目标目录。
[crazyant@dev.mechine hadoop_wordcount]$ javac -classpath /home/crazyant/app/hadoop/hadoop-2-core.jar -d wordcount_classes WordCount.java
[crazyant@dev.mechine hadoop_wordcount]$ ll -R
.:
total 8
drwxrwxr-x? 3 crazyant crazyant 4096 Aug 16 20:09 wordcount_classes
-rwxr–r–? 1 crazyant crazyant 1921 Aug 16 20:03 WordCount.java
./wordcount_classes:
total 4
drwxrwxr-x? 3 crazyant crazyant 4096 Aug 16 20:09 org
./wordcount_classes/org:
total 4
drwxrwxr-x? 2 crazyant crazyant 4096 Aug 16 20:09 myorg
./wordcount_classes/org/myorg:
total 12
-rw-rw-r–? 1 crazyant crazyant 1546 Aug 16 20:09 WordCount.class
-rw-rw-r–? 1 crazyant crazyant 1938 Aug 16 20:09 WordCount$Map.class
-rw-rw-r–? 1 crazyant crazyant 1611 Aug 16 20:09 WordCount$Reduce.class
然后将编译后的class文件打包：
[crazyant@dev.mechine hadoop_wordcount]$ jar -cvf wordcount.jar -C wordcount_classes/ .
added manifest
adding: org/(in = 0) (out= 0)(stored 0%)
adding: org/myorg/(in = 0) (out= 0)(stored 0%)
adding: org/myorg/WordCount$Map.class(in = 1938) (out= 798)(deflated 58%)
adding: org/myorg/WordCount$Reduce.class(in = 1611) (out= 649)(deflated 59%)
adding: org/myorg/WordCount.class(in = 1546) (out= 749)(deflated 51%)
[crazyant@dev.mechine hadoop_wordcount]$ ll
total 12
drwxrwxr-x? 3 crazyant crazyant 4096 Aug 16 20:09 wordcount_classes
-rw-rw-r–? 1 crazyant crazyant 3169 Aug 16 20:11 wordcount.jar
-rwxr–r–? 1 crazyant crazyant 1921 Aug 16 20:03 WordCount.java
在本地用echo生成一个文件，用于输入数据：
[crazyant@dev.mechine hadoop_wordcount]$ echo “hello world, hello crazyant, i am the ant, i am your brother” > inputfile
[crazyant@dev.mechine hadoop_wordcount]$ more inputfile
hello world, hello crazyant, i am the ant, i am your brother
在hadoop上建立一个目录，里面建立输入文件的目录
[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -mkdir /app/word_count/input
[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -ls /app/word_count
Found 1 items
drwxr-xr-x?? 3 czt czt????????? 0 2013-08-16 20:16 /app/word_count/input
将本地刚刚写的的inputfile上传到hadoop上的input目录
[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -put inputfile /app/word_count/input
[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -ls /app/word_count/input
Found 1 items
-rw-r–r–?? 3 czt czt???????? 61 2013-08-16 20:18 /app/word_count/input/inputfile
运行jar，以建立的Input目录作为输入参数
[crazyant@dev.mechine hadoop_wordcount]$ hadoop jar wordcount.jar org.myorg.WordCount /app/word_count/input /app/word_count/output
13/08/16 20:19:38 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/08/16 20:19:40 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/08/16 20:19:40 INFO compress.LzoCodec: Successfully loaded & initialized native-lzo library
13/08/16 20:19:40 INFO compress.LzmaCodec: Successfully loaded & initialized native-lzma library
13/08/16 20:19:40 INFO compress.QuickLzCodec: Successfully loaded & initialized native-quicklz library
13/08/16 20:19:40 INFO mapred.FileInputFormat: Total input paths to process : 1
13/08/16 20:19:41 INFO mapred.JobClient: splits size : 61
13/08/16 20:19:41 INFO mapred.JobClient: Running job: job_20130813122541_105844
13/08/16 20:19:43 INFO mapred.JobClient:? map 0% reduce 0%
13/08/16 20:19:57 INFO mapred.JobClient:? map 24% reduce 0%
13/08/16 20:20:07 INFO mapred.JobClient:? map 93% reduce 0%
13/08/16 20:20:16 INFO mapred.JobClient:? map 100% reduce 1%
13/08/16 20:20:26 INFO mapred.JobClient:? map 100% reduce 61%
13/08/16 20:20:36 INFO mapred.JobClient:? map 100% reduce 89%
13/08/16 20:20:47 INFO mapred.JobClient:? map 100% reduce 96%
13/08/16 20:20:57 INFO mapred.JobClient:? map 100% reduce 98%
13/08/16 20:21:00 INFO mapred.JobClient: Updating completed job! Ignoring …
13/08/16 20:21:00 INFO mapred.JobClient: Updating completed job! Ignoring …
13/08/16 20:21:00 INFO mapred.JobClient: Job complete: job_20130813122541_105844
13/08/16 20:21:00 INFO mapred.JobClient: Counters: 19
13/08/16 20:21:00 INFO mapred.JobClient:?? File Systems
13/08/16 20:21:00 INFO mapred.JobClient:???? HDFS bytes read=1951
13/08/16 20:21:00 INFO mapred.JobClient:???? HDFS bytes written=68
13/08/16 20:21:00 INFO mapred.JobClient:???? Local bytes read=5174715
13/08/16 20:21:00 INFO mapred.JobClient:???? Local bytes written=256814
13/08/16 20:21:00 INFO mapred.JobClient:?? Job Counters
13/08/16 20:21:00 INFO mapred.JobClient:???? Launched reduce tasks=100
13/08/16 20:21:00 INFO mapred.JobClient:???? Rack-local map tasks=61
13/08/16 20:21:00 INFO mapred.JobClient:???? ORIGINAL_REDUCES=100
13/08/16 20:21:00 INFO mapred.JobClient:???? Launched map tasks=61
13/08/16 20:21:00 INFO mapred.JobClient:???? MISS_SCHEDULED_REDUCES=15
13/08/16 20:21:00 INFO mapred.JobClient:?? TASK_STATISTICS
13/08/16 20:21:00 INFO mapred.JobClient:???? Total Map Slot Time=34
13/08/16 20:21:00 INFO mapred.JobClient:???? Attempt_0 Map Task Count=61
13/08/16 20:21:00 INFO mapred.JobClient:???? Total Reduce Slot Time=892
13/08/16 20:21:00 INFO mapred.JobClient:?? Map-Reduce Framework
13/08/16 20:21:00 INFO mapred.JobClient:???? Reduce input groups=9
13/08/16 20:21:00 INFO mapred.JobClient:???? Combine output records=0
13/08/16 20:21:00 INFO mapred.JobClient:???? Map input records=1
13/08/16 20:21:00 INFO mapred.JobClient:???? Reduce output records=9
13/08/16 20:21:00 INFO mapred.JobClient:???? Map input bytes=61
13/08/16 20:21:00 INFO mapred.JobClient:???? Combine input records=0
13/08/16 20:21:00 INFO mapred.JobClient:???? Reduce input records=9
查看output目录是否有结果
[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -ls /app/word_count/output??????????????????????????????????????????????????? Found 100 items
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00000
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00001
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00002
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00003
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00004
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00005
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00006
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00007
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00008
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00009
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00010
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00011
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00012
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00013
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00014
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00015
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00016
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00017
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00018
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00019
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00020
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00021
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00022
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00023
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00024
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00025
将该目录下所有文本文件合并后下载到本地
[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -getmerge /app/word_count/output wordcount_result
[crazyant@dev.mechine hadoop_wordcount]$ ls
inputfile? wordcount_classes? wordcount.jar? WordCount.java? wordcount_result
查看一下下载下来的计算结果
[crazyant@dev.mechine hadoop_wordcount]$ more wordcount_result
i?????? 2
your??? 1
crazyant,?????? 1
brother 1
hello?? 2
am????? 2
world,? 1
the???? 1
ant,??? 1
统计结果正确；
参考文章：http://hadoop.apache.org/docs/r0.18.3/mapred_tutorial.html#Example%3A+WordCount+v1.0
原文地址：hadoop第一个程序WordCount.java的编译运行过程, 感谢原作者分享。
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1377
52
How to make Google Maps the default map in iPhone
Apr 17, 2024 pm 07:34 PM
The default map on the iPhone is Maps, Apple's proprietary geolocation provider. Although the map is getting better, it doesn't work well outside the United States. It has nothing to offer compared to Google Maps. In this article, we discuss the feasible steps to use Google Maps to become the default map on your iPhone. How to Make Google Maps the Default Map in iPhone Setting Google Maps as the default map app on your phone is easier than you think. Follow the steps below – Prerequisite steps – You must have Gmail installed on your phone. Step 1 – Open the AppStore. Step 2 – Search for “Gmail”. Step 3 – Click next to Gmail app
Fix: Operator denied request error in Windows Task Scheduler
Aug 01, 2023 pm 08:43 PM
To automate tasks and manage multiple systems, mission planning software is a valuable tool in your arsenal, especially as a system administrator. Windows Task Scheduler does the job perfectly, but lately many people have reported operator rejected request errors. This problem exists in all iterations of the operating system, and even though it has been widely reported and covered, there is no effective solution. Keep reading to find out what might actually work for other people! What is the request in Task Scheduler 0x800710e0 that was denied by the operator or administrator? Task Scheduler allows automating various tasks and applications without user input. You can use it to schedule and organize specific applications, configure automatic notifications, help deliver messages, and more. it
How to sort photos by face on Windows 10 and 11
Aug 08, 2023 pm 10:41 PM
The operation of Windows is getting better and better with every version, with attractive features to improve the user experience. One feature users will want to explore on Windows 10 and 11 is the ability to sort photos by faces. This feature allows you to group photos of friends and family using facial recognition. Sounds fun, right? Read on to learn how to take advantage of this feature. Can I group photos by faces on Windows? Yes, you can use the Photos app to group pictures by faces on Windows 10 and 11. However, this feature is not available on the Photos app version. Additionally, you can link these photos to contacts using the People tab. Therefore, using this function you can
How to write a simple countdown program in C++?
Nov 03, 2023 pm 01:39 PM
C++ is a widely used programming language that is very convenient and practical in writing countdown programs. Countdown program is a common application that can provide us with very precise time calculation and countdown functions. This article will introduce how to use C++ to write a simple countdown program. The key to implementing a countdown program is to use a timer to calculate the passage of time. In C++, we can use the functions in the time.h header file to implement the timer function. The following is the code for a simple countdown program
Clock app missing in iPhone: How to fix it
May 03, 2024 pm 09:19 PM
Is the clock app missing from your phone? The date and time will still appear on your iPhone's status bar. However, without the Clock app, you won’t be able to use world clock, stopwatch, alarm clock, and many other features. Therefore, fixing missing clock app should be at the top of your to-do list. These solutions can help you resolve this issue. Fix 1 – Place the Clock App If you mistakenly removed the Clock app from your home screen, you can put the Clock app back in its place. Step 1 – Unlock your iPhone and start swiping to the left until you reach the App Library page. Step 2 – Next, search for “clock” in the search box. Step 3 – When you see “Clock” below in the search results, press and hold it and
How to open a website using Task Scheduler
Oct 02, 2023 pm 11:13 PM
Do you frequently visit the same website at about the same time every day? This can lead to spending a lot of time with multiple browser tabs open and cluttering the browser while performing daily tasks. Well, how about opening it without having to launch the browser manually? It's very simple and doesn't require you to download any third-party apps, as shown below. How do I set up Task Scheduler to open a website? Press the key, type Task Scheduler in the search box, and then click Open. Windows On the right sidebar, click on the Create Basic Task option. In the Name field, enter the name of the website you want to open and click Next. Next, under Triggers, click Time Frequency and click Next. Select how long you want the event to repeat and click Next. Select enable
Can't allow access to camera and microphone in iPhone
Apr 23, 2024 am 11:13 AM
Are you getting "Unable to allow access to camera and microphone" when trying to use the app? Typically, you grant camera and microphone permissions to specific people on a need-to-provide basis. However, if you deny permission, the camera and microphone will not work and will display this error message instead. Solving this problem is very basic and you can do it in a minute or two. Fix 1 – Provide Camera, Microphone Permissions You can provide the necessary camera and microphone permissions directly in settings. Step 1 – Go to the Settings tab. Step 2 – Open the Privacy & Security panel. Step 3 – Turn on the “Camera” permission there. Step 4 – Inside, you will find a list of apps that have requested permission for your phone’s camera. Step 5 – Open the “Camera” of the specified app
iOS 17: How to organize iMessage apps in Messages
Sep 18, 2023 pm 05:25 PM
In iOS 17, Apple not only added several new messaging features, but also tweaked the design of the Messages app to give it a cleaner look. All iMessage apps and tools, such as the camera and photo options, can now be accessed by tapping the "+" button above the keyboard and to the left of the text input field. Clicking the "+" button brings up a menu column with a default order of options. Starting from the top, there's camera, photos, stickers, cash (if available), audio, and location. At the very bottom is a "More" button, which when tapped will reveal any other installed messaging apps (you can also swipe up to reveal this hidden list). How to reorganize your iMessage app You can do this below
