尚硅谷大数据技术之Oozie（新）第4章 Oozie的使用
4.3 案例三：Oozie调度MapReduce任务
目标：使用Oozie调度MapReduce任务
分步执行：
1）找到一个可以运行的mapreduce任务的jar包（可以用官方的，也可以是自己写的）
2）拷贝官方模板到oozie-apps
[atguigu@hadoop102 oozie-4.0.0-cdh5.3.6]$ cp -r /opt/module/cdh/ oozie-4.0.0-cdh5.3.6/examples/apps/map-reduce/ oozie-apps/
- 测试一下wordcount在yarn中的运行
[atguigu@hadoop102 oozie-4.0.0-cdh5.3.6]$ /opt/module/cdh/hadoop-2.5.0-cdh5.3.6/bin/yarn jar /opt/module/cdh/hadoop-2.5.0-cdh5.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.6.jar wordcount /input/ /output/
4) 配置map-reduce任务的job.properties以及workflow.xml
job.properties
|
nameNode=hdfs://hadoop102:8020 jobTracker=hadoop103:8032 queueName=default examplesRoot=oozie-apps #hdfs://hadoop102:8020/user/admin/oozie-apps/map-reduce/workflow.xml oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/map-reduce/workflow.xml outputDir=map-reduce |
workflow.xml
|
<workflow-app xmlns="uri:oozie:workflow:0.2" name="map-reduce-wf"> <start to="mr-node"/> <action name="mr-node"> <map-reduce> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <prepare> <delete path="${nameNode}/output/"/> </prepare> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> <!-- 配置调度MR任务时，使用新的API --> <property> <name>mapred.mapper.new-api</name> <value>true</value> </property>
<property> <name>mapred.reducer.new-api</name> <value>true</value> </property>
<!-- 指定Job Key输出类型 --> <property> <name>mapreduce.job.output.key.class</name> <value>org.apache.hadoop.io.Text</value> </property>
<!-- 指定Job Value输出类型 --> <property> <name>mapreduce.job.output.value.class</name> <value>org.apache.hadoop.io.IntWritable</value> </property>
<!-- 指定输入路径 --> <property> <name>mapred.input.dir</name> <value>/input/</value> </property>
<!-- 指定输出路径 --> <property> <name>mapred.output.dir</name> <value>/output/</value> </property>
<!-- 指定Map类 --> <property> <name>mapreduce.job.map.class</name> <value>org.apache.hadoop.examples.WordCount$TokenizerMapper</value> </property>
<!-- 指定Reduce类 --> <property> <name>mapreduce.job.reduce.class</name> <value>org.apache.hadoop.examples.WordCount$IntSumReducer</value> </property>
<property> <name>mapred.map.tasks</name> <value>1</value> </property> </configuration> </map-reduce> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app> |
5）拷贝待执行的jar包到map-reduce的lib目录下
[atguigu@hadoop102 oozie-4.0.0-cdh5.3.6]$ cp -a /opt /module/cdh/hadoop-2.5.0-cdh5.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.6.jar oozie-apps/map-reduce/lib
6）上传配置好的app文件夹到HDFS
[atguigu@hadoop102 oozie-4.0.0-cdh5.3.6]$ /opt/module/cdh/hadoop-2.5.0-cdh5.3.6/bin/hdfs dfs -put oozie-apps/map-reduce/ /user/admin/oozie-apps
7）执行任务
[atguigu@hadoop102 oozie-4.0.0-cdh5.3.6]$ bin/oozie job -oozie http://hadoop102:11000/oozie -config oozie-apps/map-reduce/job.properties -run
