Hadoop自己编写程序WordCount.java 手动编译程序
假设：Hadoop已安装好。Hadoop安装步骤：https://blog.csdn.net/LiuHuan_study/article/details/84347262
第一步：配置环境变量
export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64/
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
export CLASSPATH=.:${HADOOP_HOME}/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.7.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-2.7.7.jar:${CLASSPATH} 可以以上配置放在cat ~/.bashrc中，每次启动服务器时，环境变量都有效。
或者放在一个启动文件vi /root/hadoop-env.sh中，每次服务器重启都需要，先执行这个脚本文件
或者，每次启动Hadoop时，手动执行一遍上诉命令
第二步：编写WordCount.java程序
WordCount.java
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class WordCount {
public static class TokenizerMapper
extends Mapper{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
public static class IntSumReducer
extends Reducer {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable values,
Context context
) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
第三步：借助Xshell工具上传到服务器，然后进行编译
上传命令：rz
我上传的路径为：/home/
编译：
#编译，打包jar文件。因为编译后可能会生成多个.class文件，所以需要加星号
javac WordCount.java
jar cvf WordCount.jar WordCount*class
第四步：使用WordCount.jar包，计算某一文档或者某一路径下（该路径下只有文档，没有文件夹）所有文档中单词的个数
/LiuHuan统计的路径；/output统计结果输出路径。
hadoop jar /home/WordCount.jar WordCount /LiuHuan /output
#或者
hadoop jar /home/WordCount.jar WordCount /LiuHuan/test.txt /output
注意：保证路径/output在HDFS中不存在，否则会报错。
第五步：查看统计结果
[root@localhost home]# hdfs dfs -cat /output/part-r-00000
Huan 2
Liu 2
hello 1
world! 1