編寫map函數(shù)
wordcount_mapper.py
#!/usr/bin/env python
# ---------------------------------------------------------------
#This mapper code will input a line of text and output <word, 1>
#
# ---------------------------------------------------------------
import sys
for line in sys.stdin:
line = line.strip()
keys = line.split()
for key in keys:
value = 1
print('{0}\t{1}'.format(key, value) ) #the {} is replaced by 0th,1st items in format list
reduce函數(shù)
word count_reducer.py
#!/usr/bin/env python
# ---------------------------------------------------------------
#This reducer code will input a line of text and
# output <word, total-count>
# ---------------------------------------------------------------
import sys
last_key = None
running_total = 0
# -----------------------------------
# 使用循環(huán)讀取輸入并計(jì)數(shù)
# --------------------------------
for input_line in sys.stdin:
input_line = input_line.strip()
this_key, value = input_line.split("\t", 1)
value = int(value)
if last_key == this_key:
running_total += value # add value to running total
else:
if last_key:
print( "{0}\t{1}".format(last_key, running_total) )
running_total = value #reset values
last_key = this_key
if last_key == this_key:
print( "{0}\t{1}".format(last_key, running_total))
```
如果你是Yarn的話，需要另外下載streaming的jar包[參考地址](http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-streaming/2.7.3). input 事先準(zhǔn)備一些文件。
streaming 使用絕對(duì)地址，output 不能是已經(jīng)存在的目錄 mapper 和reducer使用絕對(duì)地址
hadoop jar /Download/hadoop-streaming-2.7.3.jar
-input /hello \
-output /output
-mapper /usr/local/yarn/hadoop-2.7.3/wordcount/wordcount_mapper.py
-reducer /usr/local/yarn/hadoop-2.7.3/wordcount/wordcount_reducer.py
然后查看/output就可以看到結(jié)果。