摘自：http://staticor.io/post/hadoop/2016-01-23hadoop-definitive-guide-note#toc_7

Developing a MapReduce ApplicationRunning on a ClusterLaunching a Job
MapReduce Web UI

Tuning a JobMapReduce Workflows

How MapReduce WorksJob Submission
Job Intialization
Task Assignment
Task ExecutionStreaming

Progress and Status UpdatesJOB Completion

Failures

Shuffle and Sort

MapReduce Types and FormatsTypes
Default MapReduce Jobstreaming

Input FormatsInputSplitFileInputFormat
split size 公式
Small files and CombineFileInputFormat
Preventing splitting
Processing a whole file as a record

Text Input
Multiple Inputs

Output FormatsMultiple Outputs

MapReduce FeaturesCounterTask counters
Job counters

sortingPREPARATION
PARITIAL SORT
TOTAL SORT
Secondary SortSTREAMING

joining datasets

Developing a MapReduce Application
Writing a program in MapReduce follows a certain pattern. Before we start writing a MapReduce program, however, we need to set up and configure the development.
Configuration class ( org.apache.hadoop.conf package)

Running on a Cluster
In a distributed setting, things are a little more complex. For a start, a jobs' classes must be packaged into a jobJAR file to send to the cluster. Hadoop will find the job JAR automactically by searching for the JAR on the drivers' classpath that contains the class set in the setJarByClass() method ( on JobConf or Job).
The client class pathuser's client-side classpath set by hadoop jar is made up of:

• The job Jar file- Any JAR files in the lib directory of the job JAR file, and the classes direcotry ( if present)- The classpath defined by HADOOP_CLASSPATH, if set.

Launching a Job
$ hadoop jar hadoop-examples.jar v2.MaxTemperatureDriver \ -conf conf/hadoop-cluster.xml input/ncdc/all max-temp

Job, Task, and Task Attempt IDs

job ID 根據 YARN application ID 生成. YARN application ID 由 YARN RM 生成 (與 RM 的composed time, counter maintained, 以及后面添加的app唯一標識序號 有關), app ID 一般長得樣子是application_141041231231_0003
, 對應的jobID 就是把這前的application 替換為 job_. ==> job_141041231231_0003一個Job再細分為幾個Task, 它們的ID是 將job_前綴替換為 task ==> task_141041231231_0003_m_00003. 前后綴加上任務編號, 以區別不同的task.
MapReduce Web UI
YARN Page: resource-manager-host:8088
resource manager page
job history page
mapreduce job page

Tuning a Job
" Can I Make it run faster" may come out of our minds, after our job is working.
You should think about these profile in checklist before you start trying to optimize at the task level.
Number of mappers 如果每個mapper只運行幾秒就停了, 你應該讓他們運行更長時間, 1分鐘甚至更長. 對于small files 去看看 CombineFileInputFormat (本博客中有單獨文章提及)
Number of reducers 每個reducer的建議時長至少是5分鐘, 產出數據大小應是blocksize水平的. 書中后面會單獨講解.

Combiners 在 mapper-reducer 之間的shuffle過程中能否使用某Combiner 提升效率

Intermidiate compression map輸出進行壓縮可能提升效率

Custom serialization

Shuffle tweaks

MapReduce Workflows
這里繼 求當年最高溫度之后再嘗試尋找一個新的例子 -- 求一年366天中, 歷年在多種天氣狀態下的平均最大溫度. 1.1號為例, 先取下雨天氣的天氣, 然后求1901年至2000年 每年1.1號的下雨天氣的最大溫度, 找到最大值, 再對各類天氣狀態的最大求均值.
這個分析需要進行分解:
計算 (日期-無年, 天氣狀態) 為key下的最大溫度
求上面輸出的key下平均值

結合之前的分析, 能看到這里的任務要分2步或以上的MapReduce完成.
How MapReduce Works
MapReduce Job 的Lifetime:
hadoop客戶端, 用戶提交MapReduce Job
YARN RM(resource manager) 來分配cluster上的資源
YARN NM(NameNode) 準備用于運算的containers
MapRededuce application master, 在job運行過程中的"協調人".
HDFS share job間的文件

application master 在 OutputCommitter上 執行 setupJob() 方法, 為task的輸出創建臨時woking space及輸出目錄. 詳情還要查詢 Output Committers
Task Assignment
non-uber task. application master 開始向 RM 為他創建好的 mapper/reducer申請container 資源完成任務. mapper的請求優先級會高于reducer -- 這是因為在執行sort, reducer 任務之前 所有mapper結果要完成. 對于 reduce的請求, 至少要等 5% 的map任務完成 才會開始接受受理. Reduce 任務可以在集群的任意位置執行, 但map task 受到數據局部性(data locality)制約. map, reduce task默認的分配內存是 1024MB. 該值的properties: mapreduce.map.memory.mb mapreduce.reduce.memory.mb, mapreduce.map.cpo.vcores, mapreduce.reduce.cpu.vcores
data locality , 在quora中找到了一個答案中的解釋:
Data locality is a core concept of Hadoop, based on several assumptions around the use of MapReduce. In short, keep data on disks that are close to the CPU and RAM that will be used to process and store it. If you had a cluster of 100 machines, and needed to read a selection of records, the records should be adjacent on disk, fit into RAM and be processable (e.g sorted or computed) using that machine's CPU.

Task Execution
Streaming

另, 對于一個failed的task, app master 會再給他4次重試的機會(4 可進行修改, 參數為:mapreduce.map.maxattempts) app master 會在重新調度的時候盡可能的使用與先前不同的Node來執行這個失敗的任務 4次不行的話 則整個Job標記為失敗.
如果失敗的task 超過一定個數, 則會激活 job failure的改變. Profiles: mapreudce.map.failures.maxpercent, mapreduce.reduce.failures.maxpercent (也就是說失敗的個數小于這個百分比, 那么 appication將會繼續執行其它的task.
application master failure

node manager failure 當node manager fail RM上看不到其返回的heartbeat. (比如10分鐘內看不到, 時間配置:yearn.resourcemanager.nm.liveness-monitor.expiry-interval-ms)

resource manager failure

這個問題就比較嚴重了. 暫時先不想了解.
Shuffle and Sort

注, 上圖中能看出, map的輸出結果不是直接寫到disk中, 而是先到一個內存中的緩存區(memory buffer), 這個默認大小是100MB(可通過 mapreduce.task.io.sort.mb 參數配置), 當緩存區的空間達到一定水平(默認是 80%mapreduce.map.sort.splill.percent 0.80) , 將會啟動spill寫到disk. (map持續向buffer中寫 , 不會停止) , 要是 buffer達到100%了, 則map的輸出則會暫停, 直到spill完成. 不過還要注意的就是, 寫向disk之前, data也會按照reducer的要求進行partition. 按照給的或默認的方式, 在每個partition內執行combine.
compress map output通過來說, 對map的結果進行壓縮處理, 將會提高Job的效率. 因為這樣會節約磁盤空間, 減少向reducer轉移的數據量大小. 默認, 是不進行壓縮. 但可通過mapreduce.map.output.compress = true.

reducer side

MapReduce Types and Formats
Types
map: (K1, V1) -> list(K2, V2)reduce: (K2, list(V2)) -> list(K3, V3)
若增加了combiner
map: (K1, V1) -> list(K2, V2)combiner: (K2, list(V2)) -> list(K2, V2)reduce: (K2, list(V2)) -> list(K3, V3)
Input types: 輸入的格式被 input format 指定, 例如: TextInputFormat 指定 是

Default MapReduce Job
如果MapReduce 沒有 mapper, reducer會是怎樣?

public MinimalMapReduce extends Configured implements Tool{ @Override public int run(String[] args)throws Exception { if (args.length != 2) { System.err.printf("Usage: %s [generic options] <input> <output> \n", getClass().getSimpleName()); ToolRunner.printGenericCommandUsage(System.err); return -1; } Job job = new Job(getConf()); job.setJarByClass(getClass()); FileInputFormat.addInputPath(job, new Path(args[0]); FileoutputFormat.addOutputPath(job, new Path(args[1]); return job.waitForCompletion(true) ? 0 : 1 ; } public static void main(String[] args ) throws Exception { int exitCode = ToolRunner.run(new MinimalMapReduce(), args); System.exit(exitCode); }}

每行是一個record(key,value分別是 line's offset + line ) 組成. 下面是MapReduce 的 driver程序, 使用精確的配置參數:
public class MinimalMapReduceWithDefaults extends Configured implements Tool { @Override public int run(String[] args) throws Exception{ Job job = JobBuilder.parseInputAndOuput(this, getConf(), args); if (job == null) { return -1; } job.setInputFormatClass(TextInputFormat.class); job.setMapperClass(Mapper.Class); job.setMapOutputKeyClass(LongWritable.class); job.setMapOutputValueClass(Text.class); job.setPartitionerClass(HashPartitioner.class); job.setNumReduceTasks(1); job.setReducerClass(Reducer.class); job.setOutputKeyClass(LongWritable.class); job.setOutputValueClass(Text.class); job.setOutputFormatClass(TextOutputFormat.class); return job.waitForCompletion(true)? :0 : 1; } public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new MinimalMapReduceWithDefaults(), args); System.exit(exitCode); }}

以上就是MapReduce 一個Job的框架代碼. 默認的input format 是 TextInputFormat , 提供了
keys the offset of the beginning of the line in the file.

values the line of text

mapper的泛用型定義
public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT> { protected void map(KEYIN key, VALUEIN value, Context context) throws IOException, InterruptedException { context.write((KEYOUT) key, (VALUEOUT) value); }}

默認partitioner : HashPartitioner 根據 map keyout 的key -hash值 & Integer.Max_VALUE, mod % reduce個數, 進行partition. 原理是讓每個reducer處理的 mapOutput key的種類數是even的. (特殊情況當然也能想象: 某幾種output key的個數非常多, 則就造成 個別reducer 處理的數據量非常非常大)
public class HashPartitioner<K, V> extends Partitioner<K, V> { public int getPartition(K key, V value, int numReduceTasks){ return (key.hashCode() & Integer.Max_VALUE ) % numReduceTasks; } }

Choose the number of reducers
很多新人(說我呢) 都覺得reducer個數越多越好 -- 這樣對于map/reduce之間的數據流有好處. 選擇合適的reducer個數也不是一件容易的事. 使用多的reducer固然增大并行化, 讓每個reducer處理的數據量減少. 然而, 這樣你會得到很多的小文件 -- 相對來說, 把這種文件控制在一定水平內才是最優的策略. 通常來說, 一個reducer的處理時間控制在5分鐘左右, 產出的數據量大小應該是和HDFS的blocksize相當的.

reducer generic type
public class Reducer<KEYIN, VALUEIN, KEYOUT, VALUEOUT> { protected void reduce(KEYIN key, Iterable<VALUEIN> values, Context context) throws IOException, InterruptedException { for (VALUEIN value: values) { context.write((KEYOUT) key, (VALUEOUT) value); } }}

在reducer處理之前數據會先經過shuffle的排序.
streaming

這里2個方法功能也非常簡單, 目的是為了獲得輸入的字節長度大小和位置信息. 位置用來分割時將map task處理的數據更接近, 大小信息方便將較大的先被處理, 這樣方便減少job的運行時間. (貪心算法)
MapReduce里用InputFormat完成對InputSplits的創建. 也是一個抽象類:
public abstract class InputFormat<K, V> { public abstract List<InputSplit> getSplits(JobContext context) throws IOException, InterruptedException; public abstract RecordReader<K, V> createRecordReader(InputSplit split, TaskAttemptContext context) throws IOException, InterruptedException; }

這里的getSplits()方法對輸入進行分割, 然后把結果發送到application master, am分配給mapper. mapper將會使用 createRecordReader() 實現的結果, 獲得Split的結果 -- RecordReader, 原來輸入中record的數據流. 下面是mapper類中run()方法, 完成對 record流的解析處理.
public void run(Context context) throws IOException, InterruptedException{ setup(context) while (context.nextKeyValue()){ map(context.getCurrentKey(), context.getCurrentValue(), context); cleanup(context); }}

FileInputFormat
FileInputFormat 是對 InputFormat類的繼承, 用于對文件輸入的指定, 它能干兩件事: 1. 定義文件在Job中的輸入格式; 2. 完成文件的Split. 不過一般的, 在實際工作是使用的它的子類. 見下圖:

input paths

FileInputFormat 首先對輸入的路徑定義了幾個常用方法:
public static void addInputPath(Job job, Path path)public static void addInputPaths(Job job, String commaSeparatedPaths)public static void setInputPath(Job job, Path ... input Paths)public static void setInputPaths(Job job, String commaSeparatedPaths)

add~方法用來將一個路徑或一組路徑添加到輸入的列表中. set~方法則是用新的參數替換掉原來已有的輸入列表. (比如add 加了3個, set之后加的3個將 不再)
注, 這里路徑支持 glob pattern
注, 雖然用戶沒有自己指定, 但FileInputFormat將自動過濾那些隱藏文件(以 . 或者 _ 開頭的文件)
FileInputFormat input splits

FileInputFormat 如何產生input splits> ? 它只能split那些"large" files. 這里大小的定義是和HDFS block 大小有關. 當然這里也有知道有這幾個property

• mapreduce.input.fileinputformat.split.minsize 最小分割值, 默認為1- mapreduce.input.fileinputformat.split.maxsize 最大分割值, 默認為 Long.MAX_VALUE- dfs.blocksize long 一般是128 MB(即為 134217728)

最小分割常常是1個byte, 有時有的format 會定義一個下界. Application 可能也會對minSplitSize進行設置, 把它設為一個大于HDFS block 的值, 但這未必是一個好方式. 最大分割, 只有當這個值小于 block size才會起作用.
split size 公式
max(minimumSize, min(maximumSize, blockSize))
默認情況: minimumSize < blockSize < maximumSize
Small files and CombineFileInputFormat
HDFS 是更擅長于處理個數少, 而塊頭大的數據文件, 相較于個數多,而卻很小的小文件簇來說.
如果是面臨著非常多的小文件, 比如10000個大小均小于10MB的輸入文件群, 再直接用FileInputFormat就不合適了 -- 它是對每個文件進行分割.
為什么說小文件太多 HDFS反而應付不好>>?MapReduce工作模式對集群上磁盤間的文件轉移要求很高, 小文件過多, 無形之中加大了對文件的seek工作. 再者, 太多的小文件加大了namenode的管理mapping的工作. 一個可取的策略是將這些小文件通過合并形成sequencedfiles -- key作為文件名, value為文件內容. 但要是目前已經有這些小文件了, 這時應考慮使用 CombineFileInputFormat.

Preventing splitting
有時我們想對一整個文件作為一個Input SPlit, 而不想切分. 方法1: 將minimum split size 設置的非常大. 方法2: 對FileInputFormat實現的子類 重寫它的 isSplittable()方法. 例子
import org.apache.hadoop.fs.Path;import org.apache.hadoop.mapreduce.JobContext;import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;public class NonSplittableTextInputFormat extends TextInputFormat{ @Override protected boolean isSplitable(JobContext context, Path, file){ return false; }}

Processing a whole file as a record
有些特殊情況, 要求我們以多個目錄中各個文件作為record. (比如某個時間戳下產生的數據, 以時間戳命名) 對于這樣的場景, 可使用 WholeFileInputFormat
public class WholeFileInputFormat extends FileInputFormat<NullWritable, BytesWritable> { @Override protected boolean isSplitable(JobContext context, Path file){ return false; } @Override publick RecordReader<NullWrtable, BytesWritable> createRecordReader( InputSplit split, TaskAttemptContext context) throws IOException, InterruptedException { WholeFileRecordReader reader = new WholeFileRecordReader(); reader.initialize(split, context); return reader; }}

注, 由于整個文件內容作為這里record的value, 因此肯定是不可分的. 同之前, 重寫isSplitable()方法.
The RecordReader used by WholeFileInputFormat for reading a whole file as a record

class WholeFileRecordReader extends RecordReader<NullWritable, BytesWritable> {private FileSplit fileSplit;private Configuration conf;private BytesWritable value = new BytesWritable();private boolean processed = false; @Overridepublic void initialize(InputSplit split, TaskAttemptContext context) throws IOException, InterruptedException { this.fileSplit = (FileSplit) split; this.conf = context.getConfiguration();}@Overridepublic boolean nextKeyValue() throws IOException, InterruptedException { if (!processed) { byte[] contents = new byte[(int) fileSplit.getLength()]; Path file = fileSplit.getPath(); FileSystem fs = file.getFileSystem(conf); FSDataInputStream in = null; try { in = fs.open(file); IOUtils.readFully(in, contents, 0, contents.length); value.set(contents, 0, contents.length); }finally{ IOUtils.closeStream(in); } processed = true; return true } return false;}@Overridepublic NullWritable getCurrentKey() throws IOException, InterruptedException { return NullWritable.get();}@Overridepublic BytesWritable getCurrentValue() throws IOException, InterruptedException { return value;}@Overridepublic float getProgress() throws IOException{ return processed ? 1.0f : 0.0f;}@Override public void close() throws IOException { //do nothing}}

Text Input
TextInputFormat 示例文本 共四行文本

On the top of the Crumpetty Tree The Quangle Wangle sat, But his face you could not see, On account of his Bea ver Hat.

經TextInputFormat, 得到的 對是這樣的對:

(0, On the top of the Crumpetty Tree) (33, The Quangle Wangle sat,) (57, But his face you could not see,) (89, On account of his Bea ver Hat.)

The Relationship Between Input Splits and HDFS Blocks
以這里的record形式(一行一行的文本)為例, HDFS中的block并不會考慮太多 - 不會強制要求一行文本全部放在同一個block中. 但Split不會把同一行文本分配在2個不同的Input Split中.

幾點要注意的地方:

• Controlling the maximum line length 如果使用 TextInputFormat, User可以對每行的長度進行控制. -- mapreduce.input.linerecordreader.line.maxlength - KeyValueTextInputFormat 若每一行已經具備了明文的<Key,Value> 結構, 可使用這個子類方便的實現Input的格式指定. 利用參數 mapreduce.input.keyvaluelinerecordreader.key.value.separator 指定每行的分隔符. 默認是tab鍵. - 若將多行指定為一條 record, 請研究 NLineInputFormat .

Multiple Inputs
MultipleInputs.addInputPath(job, path1, TextInputFormat.class, MaxTemperatureMapper.class);MultipleInputs.addInputPath(job, path2, TextInputFormat.class, MaxTemperatureMapper.class);

Output Formats
OutputFormat 的類結構圖如下所示:

還是之前 天氣數據的例子, 根據天氣狀態(station)進行partition.
方法: 在reducer處理過程中, 對station進行處理 -- 1. 把擁有相同station的map輸出 partition到同一個; 2. 設置reducer個數等于station的種類數. partitioner的長相:
public class StationPartitioner extends Partitioner<LongWritable, Text> { private NcdcRecordParse parser = new NcdcRecordParser(); @Override pubic int getPartition(LongWritable key, Text value, int numPartitions) { parser.parse(value); } private int getPartition(String stationId) { /// }}

這里省略掉的 getPartition 方法, 簡單的說是把已有的stationID 轉化成 partition 的索引.
思考: 這樣做的方式存在哪些不足?
實現partition需要我們已經對輸出信息掌握, 比如這里 要先知道都有哪些station, 才能進行處理. 即使事先給你一個參考字典, 但是若出現了未知的情況, 難免會有麻煩.
Partition的個數被人為的指定了, 這樣極可能導致uneven-sized partitions -- 絕大多數reducer處理非常少量的數據, 這決不是一種高效的思路. 如果極個別的reducer 消耗時間明顯長于其它的reducer, 那么這樣reducer將直接決定job的執行時間.

為了讓Job盡快完成, 默認使用 HashPartitioner 完成Partition (盡量以免 unevenly-sized partition)
multipleoutputs 例子
public class PartitionByStationUsingMultipleOutputs extends Configured implement Tool { static class StationMapper extends Mapper<LongWritable, Text, Text, Text> { private NcdcRecord parser = new NcdcRecordParser(); @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { parser.parse(value); context.write(new Text(parser.getStationId()), value); } } static class MultipleOutputsReducer extends Reducers<Text, Text, NullWritable, Text> { private MultipleOutputs<NullWritable, Text> multipleOutputs; @Override protected void setup(Context context) throws IOException, InterruptedException { multipleOutputs = new MultipleOutputs<NullWritable, Text>(context); } @Override protected void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException { for (Text value: values) { multipleOutputs.write(NullWritable.get(), value, key.toString()); } } @Override protected void cleanup(Context context) throws IOException, InterruptedException { multipleOutputs.close(); } @Override public int run(String[] args) throws Exception { Job job = JobBuilder.parseInputANdOutput(this, getConf(), args); if (job == null) return -1; job.setMapperClass(StationMapper.class); job.setMapOutputKeyClass(Text.class); job.setReducerClass(MultipleOutputReducer.class); job.setOutputKeyClass(NullWritable.class); return job.waitForCompletion(true) ? 0 : 1; } public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new PartitionByStationUsingMultipleOutputs(), args); System.exit(exitCode); } }}

注, 注意 MultipleOutputs.write() 寫的key, value.
我想目標輸出創建任意的子目錄>? 沒問題, 看這段代碼吧
@Override protected void reduce(Text key, Iterable<Text>values, Context context) throws IOException, InterruptedException { for (Text value : values) { parser.parse(value); String basePath = String.format("%s/%s/part", parser.getStationId(), parser.getYear()); multipleOutputs.write(NullWritable.get(), value, basePath); }}

注: LazeOutput
FileOutputFormat 創建的 part-r-nnnnn files 即使這個文件是空的(大小為0) , 也會同樣創建. 如果不希望這樣, 可使用 LazyOutputFormat. 使用 Streaming的話則是 -lazyOutput 參數.
MapReduce Features
這一章, 討論 counter, joining, 和 sorting
Counter
在任務執行過程中的一些任務信息的反饋, 有些內容我們希望看到稍加統計的結果, 由于統計本身并不復雜, 也不會用到太多高深的技巧, 大多是對job, task等的計數類反饋, 所以這里先來簡單了解一下Counter -- 根據所有計數對象的類型, 分參悟了以下幾種:
MapReduce task counters -- org.apache.hadoop.mapreduce.TaskCounter
Filesystem counters -- org.apache.hadoop.mapreduce.FileSYstemCounter
FileInputFormat counters -- org.apahce.hadoop.mapreduce.lib.input.FileInputFormatCounter
FileOutputFormat counters -- org.apahce.hadoop.mapreduce.lib.output.FileOutputFormatCounter
Job counters -- org.apache.hadoop.mapreduce.JobCounter

Task counters
顯然, 這是一類針對task而收集信息的工具. 例如 MAP_INPUT_RECORDS 等, 有很多我們在MapReduce WebUI 上看到的信息, 其實都是出自于它們的返回結果. 下面來見表:
MAP_INPUT_RECORDS 統計每個map處理的records個數. 最后聚合, 得到整個Job的輸入record個數.
SPLIT_RAW_BYTES input-split 對象的bytes, 由于是在原來輸入數據又增加了分割的offset, 因此會大于真正的 total input size.
MAP_OUTPUT_RECORDS map output 產出的record個數. 通過每個map的OutputCollector()調用其 collect()方法來完成.

MAO_OUTPUT_BYTES map output產出的非壓縮類bytes大小, 通過每個map的OutputCollector()調用其 collect()方法來完成.

MAP_OUTPUT_MATERIALIZE_BYTES map output 直接向Disk產出的bytes大小(包括壓縮類的文件的大小)

COMBINE_INPUT_RECORDS 被所有combiners處理過的 input records個數.

COMBINE_OUTPUT_RECORDS 被所有combiners處理過的 output records個數.

REDUCE_INPUT_GROUPS 所有reducer處理的key種類數, 對reducer執行reduce() 方法累增得到.

REDUCE_INPUT_RECORDS 所有reducer處理的 input records 個數.

REDUCE_OUTPUT_RECORDS 所有reducer處理的 output records 個數.

REDUCE_SHUFFLE_BYTES map output 到 reducer過程中 shuffle用到的bytes大小

SPILLED_RECORDS 所有map/reduce過程中spill到磁盤中的records個數

CPU_MILLISECONDS CPU 對該任務的消耗 毫秒

PHYSICAL_MEMORY_BYTES 任務消耗的內存大小

VIRTUAL_MEMORY_BYTES 任務消耗的虛擬內存大小

COMMITTED_HEAP_BYTES JVM可用的內存大小

GC_TIME_MILLIS GC消耗時間 毫秒

SHUFFLED_MAPS map output 產生的文件數, (被shuffle之后再由reducer處理)

FAILED_SHUFFLE shuffle過程中 map output copy失敗的個數

MERGED_MAP_OUTPUTS map output 被合并的個數 (在Shuffle端處理)

BYTES_READ filesystem task counter, map/reduce 讀入的bytes

BYTES_WRITTEN filesystem task counter, map/reduce 寫入的bytes

READ_OPS filesystem task counter, map/reduce 讀的操作個數

LARGE_READ_OPS filesystem task counter, map/reduce 讀的操作個數, 限定于 large read(如對于大型目錄列表的讀入)

WRITE_OPS filesystem task counter, map/reduce 寫的操作個數

Job counters
job counter 與其它類的counter 有所不同, 全由application master操縱. 它們用來對Job進行統計匯總, 如:
TOTAL_LAUNCHED_MAPS mapper 啟動個數

TOTAL_LAUNCHED_REDUCES reducer 啟動個數

TOTAL_LAUNCHED_UBERTASKS uber task的個數

NUM_FAILED_MAPS mapper失敗個數

NUM_FAILED_REDUCES reducer失敗個數

NUM_KILLED_MAPS mapper killed 個數

NUM_KILLED_REDUCES reducer killed 個數

其它內容暫時略過.
sorting
The ability to sort data is at the heart of MapReduce.

這一部分將會接觸到MapReduce中如何使用Sort來重新組織數據流, 以及不同的sort 方式.
PREPARATION
以之前的溫度數據為例, 由于要求某些溫度數據的最大值, 而原始數據是TEXT結構, 顯然不能應照數值型進行排序, 那么 TEXT -> INT(或其它的FLOAT, DOUBLE) 轉化過程就要考慮是否有invalid data. 在map端要對不合理的數據過濾處理. 下面看一個實現 數值轉型的 程序
public class SortDataPreprocessor extends Configured implements Toll { static class CleannerMapper extends Mapper<LongWritable, Text, IntWritable, Text> { private NcdcRecordParser parser =new NcdcRecordParser(); @Override protected void map(LongWritable key, Text value, Context context) { parser.parse(value); if (parser.isValidTemprature)) { context.write(new IntWritable(parser.getAirTemperature()), value); } } } @Override public int run(String[] args) throws Exception{ Job job = JobBuilder.parseInputAndOutput(this, getConf(), args); if (job == null) { return -1 } job.setMapperClass(CleannerMapper.class); job.setOutputKeyClass(IntWritable.class); job.setOutputValueKeyClass(Text.class); job.setNumReduceTasks(0); job.setOutputFormatClass(SequenceFileOutputFormat.class); SequenceFileOutputFormat.setCompressOutput(job, true); SequenceFileOutputFormat.setCompressorClass(job.GzipCodec.class); SequenceFileOutputFormat.setOutputCompressionType(job, CompressionType.BLOCK); return job.waitForCompletion(true) ? 0 : 1; } public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new SortDataPreprocessor(), args); System.exit(exitCode); }}

PARITIAL SORT
默認情況下, MapReduce會根據input records的key進行排序.
Example 9-4. A MapReduce program for sorting a SequenceFile with IntWritable keys using the default HashPartitioner

public class SortByTemperatureUsingHashPartitioner extends Configured implements Tool{ @Override public int run (String[] args) throws Exception{ Job job = JobBuilder.parseInputAndOutput(this, getConf(), args); if (job == null) return -1; job.setInputFormatClass(SequenceFileInputFormat.class); job.setOutputKeyClass(IntWrtable.class); job.setOutputFormatClass(SequenceFileOutputFormat.class); SequenceFileOutputFormat.setCompressOutput(job, true); SequenceFileOutputFormat.setOutputCompressorClass(job, GzipCodec.class); SequenceFileOutputFormat.setOutputCompressionType(job, CompressionType.Block); return job.waitForCompletion(true) ? 0 : 1; } public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new SortByTemperatureUsingHashPartitioner(), args); System.exit(exitCode); } }

w假定使用30個reducer來執行程序:
$ hadoop jar hadoop-examples.jar SortByTemperatureUsingHashPartitioner \ -D mapreduce.job.reduces=30 input/folder output/folder

TOTAL SORT
How can you produce a globally sorted file using Hadoop? 如果只有一個partition 可能答案就解決了, 但我們面臨問題是多個. 要如何做到全局性排序呢. -- 把構造partitioner時與要排序的值結合到一起. 比如我們有4個partition, 然后把 < -10度的放在第一個, [-10, 0) 放在第2個, [0, 10) 放在第3個, 其余是第4個. 然后在每個partition中對溫度進行排序.

MapReduce program for sorting a SequenceFile with IntWritable keys using the TotalOrderPartitioner to globally sort the day
public class SortByTemperatureUsingTotalOrderPartitioner extends Configured implements Tool {@Override public int run(String[] args) throws Exception{ Job job = JobBuilder.parseInputAndOutput(this, getConf(), args); if (job == null) return -1 ; }job.setInputFormatClass(SequenceFileInputFormat.class);job.setOutputKeyClass(IntWritable.class);job.setOutputFormatClass(SequenceFileOutputFormat.class);SequenceFileOutputFormat.setCompressOutput(job, true); SequenceFileOutputFormat.setOutputCompressorClass(job, GzipCodec.classs); SequenceFileOutputFormat.setOutputCompressionType(job, CompressionType.BLOCK); job.setPartitionerClass(TotalORderPartitioner.class); InputSampler.Sampler<IntWrtable, Text> sampler = new InputSampler.RandomSample<IntWritable, Text>(0.1, 10000, 10); InputSampler.writePartitionFile(job, sampler); // Add to DistributedCacheConfiguration conf = job.getConfiguration(); String partitionFile = TotalOrderPartitioner.getPartitionFile(conf); URI partitionUri = new URI(partitionFile); job.addCacheFile(partitionUri); return job.waitForCompletion(true) ? 0 : 1; }public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new SortByTemperatureUsingTotalOrderPartitioner(), args); System.exit(exitCode); }}

Secondary Sort
Application to find the maximum temperature by sorting temperatures in the key

public class MaxTemperatureUsingSecondarySort extends Tool{ static class MaxTemperatureMapper extends Mapper<LongWritable, Text, IntPair, NullWritable> { private NcdcRecordParser parser = new NcdcRecordParser(); @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { parser.parse(value); if (parser.isValidTemperature()){ context.write(new IntPair(parser.getYearInt(), parser.getAirTemperture()), NullWritable.get()); } } } static class MaxTemperatureReducer extends Reducer<IntPair, NullWritalbe, IntPair, NullWritalbe> { @Override protected void reduce(IntPair key, Iterable<NullWritable> values, Context context throws IOException, InterruptedException{ context.write(key, NullWritable.get()); } } public static class FirstPartitioner extends Partitioner<IntPair, NullWritable> { @Override public int getPartition(IntPair key, NullWritable value, int numPartitions){ // multiply by 127 to perform some mixing return Math.abs(key.getFirst() * 127 ) % numPartitions; } } public static class KeyComparator extends WritableComparator { protected KeyComparator() { super(IntPair.class, true); } @Override public int compare(WritableComparable w1, WritableComparable w2) { IntPair ip1 = (IntPair) w1; IntPair ip2 = (IntPair) w2; int cmp = IntPair.compare(ip1.getFirst(), ip2.getFirst()); if (cmp != 0) return cpm; return - IntPair.compare(ip1.getSecond(), ip2.getSecond()); } } public static class GroupComparator extends WritableComparator { protected GroupComparator() { super(IntPair.class, true); } @Override public int compare(WritableComparable w1, WritableComparable w2) { IntPair ip1 = (IntPair) w1; IntPair ip2 = (IntPair) w2; int cmp = IntPair.compare(ip1.getFirst(), ip2.getFirst()); return cmp; } @Override public int run(String[] args) throws Exception { Job job = JobBuilder.parseInputAndOutput(this, getConf(), args); if (job==null) return -1; job.setMapperClass(MaxTemperatureMapper.class); job.setSortComparatorClass(KeyComparator.class); job.setGroupingComparatorClass(GroupComparator.class); job.setReducerClass(MaxTemperatureReducer.class); job.setOutputKeyClass(IntPair.class); job.setOutputValueClass(NullWritable.class); return job.waitForCompletion(true) ? 0 : 1; } public static void main(String[] args ) throws Exception { int exitCode = ToolRunner.run(new MaxTemperatureUsingSecondarySort(), args); System.exit(exitCode); }}

STREAMING
to do a secondary sort in Streaming, we can take advantage of a couple of library classes that Hadoop provides.
$ hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-*.jar \ -D stream.num.map.output.key.fields=2 \ -D mapreduce.partition.keypartitioner.options=-k1, 1 \ -D mapreduce.job.output.key.comparator.class=org.apache.hadoop.mapred.lib.KeyFieldBasedComparator \ -D mapreduce.partition.keycomparator.options="-k1n k2nr" \ -input input/ncdc/all \ -output output-secondarysport-streaming \ -mapper ... -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner \ -reducer ch...

joining datasets