以官方Hadoop中的 WordCount案例分析 ，Job作業的提交過程：
public static void main(String[] args) throws Exception {
// Create a new Job
Configuration conf=new Configuration(true);
Job job = Job.getInstance(conf);
job.setJarByClass(MyWorkCountJob.class);
// Specify various job-specific parameters
job.setJobName("myWorkCountjob");
//設置輸入文件路徑
FileInputFormat.addInputPath(job, new Path("/user/root/hello.txt"));
//設置輸出文件路徑
Path outPath=new Path("/sxt/mr/output");
if(FileSystem.get(conf).exists(outPath))
FileSystem.get(conf).delete(outPath);
FileOutputFormat.setOutputPath(job, outPath);
job.setMapperClass(MyMapper.class);
job.setReducerClass(MyReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
// Submit the job, then poll for progress until the job is complete
job.waitForCompletion(true);//job 提交的入口
}
waitForCompletion方法
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();// 任務提交1.1
}
if (verbose) {
monitorAndPrintJob();//實時監控Job任務并打印相關的日志
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
1.1 submit 方法
public void submit()
throws IOException, InterruptedException, ClassNotFoundException {
ensureState(JobState.DEFINE);//確定job狀態
setUseNewAPI();//默認使用新的API
connect();//獲得與集群的連接
final JobSubmitter submitter =
getJobSubmitter(cluster.getFileSystem(), cluster.getClient());
status = ugi.doAs(new PrivilegedExceptionAction<JobStatus>() {
public JobStatus run() throws IOException, InterruptedException,
ClassNotFoundException {
//異步調用submitJobInternal方法提交任務 1.2
return submitter.submitJobInternal(Job.this, cluster);
}
});
state = JobState.RUNNING;
LOG.info("The url to track the job: " + getTrackingURL());
}
submit方法首先創建了JobSubmitter實例，然后異步調用了JobSubmitter的submitJobInternal方法
1.2 submitJobInternal 方法
JobStatus submitJobInternal(Job job, Cluster cluster)
throws ClassNotFoundException, InterruptedException, IOException {
//檢查job的輸出路徑是否存在，如果存在則拋出異常
checkSpecs(job);
Configuration conf = job.getConfiguration();
addMRFrameworkToDistributedCache(conf);
//初始化臨時目錄和返回的輸出路徑。
Path jobStagingArea = JobSubmissionFiles.getStagingDir(cluster, conf);
//configure the command line options correctly on the submitting dfs
InetAddress ip = InetAddress.getLocalHost();
if (ip != null) {
submitHostAddress = ip.getHostAddress();
submitHostName = ip.getHostName();
conf.set(MRJobConfig.JOB_SUBMITHOST,submitHostName);
conf.set(MRJobConfig.JOB_SUBMITHOSTADDR,submitHostAddress);
}
//獲取新的JobId
JobID jobId = submitClient.getNewJobID();
job.setJobID(jobId);
// 獲取提交目錄
Path submitJobDir = new Path(jobStagingArea, jobId.toString());
......
//把作業上傳到集群中去
copyAndConfigureFiles(job, submitJobDir);
Path submitJobFile = JobSubmissionFiles.getJobConfPath(submitJobDir);
// 創建切片列表 找出每個文件的切片列表 合并切片列表的數量就是Map任務個數 客戶端統計
int maps = writeSplits(job, submitJobDir); //2.1核心方法
conf.setInt(MRJobConfig.NUM_MAPS, maps);//文件分片的大小 就是Map任務數量
......
// Write job file to submit dir 相關配置寫入到job.xml中
writeConf(conf, submitJobFile);
// Now, actually submit the job (using the submit name) 真正的提交作業
status = submitClient.submitJob( //2.3 提交job到RecourceManager
jobId, submitJobDir.toString(), job.getCredentials());
...
}
2.1 文件切片操作 writeSplits -> writeNewSplits 計算向數據移動模型的核心
private <T extends InputSplit>
int writeNewSplits(JobContext job, Path jobSubmitDir) throws IOException,
InterruptedException, ClassNotFoundException {
Configuration conf = job.getConfiguration();
InputFormat<?, ?> input =
ReflectionUtils.newInstance(job.getInputFormatClass(), conf);
List<InputSplit> splits = input.getSplits(job);
T[] array = (T[]) splits.toArray(new InputSplit[splits.size()]);
// sort the splits into order based on size, so that the biggest
// go first
Arrays.sort(array, new SplitComparator());
//將split信息和SplitMetaInfo都寫入HDFS中
JobSplitWriter.createSplitFiles(jobSubmitDir, conf,
jobSubmitDir.getFileSystem(conf), array);
return array.length;
}
writeNewSplits方法中，劃分任務數量最關鍵的代碼即為InputFormat的getSplits方法（InputFormat有不同實現類 框架默認的是TextInputFormat）。此時的Input即為TextInputFormat的父類FileInputFormat，其getSplits方法的實現如下：
public List<InputSplit> getSplits(JobContext job) throws IOException {
Stopwatch sw = new Stopwatch().start();
long minSize = Math.max(getFormatMinSplitSize(), getMinSplitSize(job));//默認最小值 1
long maxSize = getMaxSplitSize(job);//默認最大值 Long類型的最大值
// generate splits
List<InputSplit> splits = new ArrayList<InputSplit>();
List<FileStatus> files = listStatus(job);//獲取源文件的源信息列表
for (FileStatus file: files) {
Path path = file.getPath();
long length = file.getLen();
if (length != 0) {
BlockLocation[] blkLocations;
//獲取文件的block塊列表
if (file instanceof LocatedFileStatus) {
blkLocations = ((LocatedFileStatus) file).getBlockLocations();
} else {
FileSystem fs = path.getFileSystem(job.getConfiguration());
blkLocations = fs.getFileBlockLocations(file, 0, length);
}
if (isSplitable(job, path))
long blockSize = file.getBlockSize();
long splitSize = computeSplitSize(blockSize, minSize, maxSize);
//核心代碼塊
long bytesRemaining = length;
while (((double) bytesRemaining)/splitSize > SPLIT_SLOP) {
int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
splits.add(makeSplit(path, length-bytesRemaining, splitSize,
blkLocations[blkIndex].getHosts(),
blkLocations[blkIndex].getCachedHosts()));
bytesRemaining -= splitSize;
}
//核心代碼塊結束
if (bytesRemaining != 0) {
int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
splits.add(makeSplit(path, length-bytesRemaining, bytesRemaining,
blkLocations[blkIndex].getHosts(),
blkLocations[blkIndex].getCachedHosts()));
}
} else { // not splitable
splits.add(makeSplit(path, 0, length, blkLocations[0].getHosts(),
blkLocations[0].getCachedHosts()));
}
} else {
//Create empty hosts array for zero length files
splits.add(makeSplit(path, 0, length, new String[0]));
}
}
// Save the number of input files for metrics/loadgen
job.getConfiguration().setLong(NUM_INPUT_FILES, files.size());
sw.stop();
if (LOG.isDebugEnabled()) {
LOG.debug("Total # of splits generated by getSplits: " + splits.size()
+ ", TimeTaken: " + sw.elapsedMillis());
}
return splits;
}
2.2 核心代碼塊分析
對每個輸入文件進行split劃分。注意這只是個邏輯的劃分 因此執行的是FileInputFormat類中的getSplits方法。只有非壓縮的文件和幾種特定壓縮方式壓縮后的文件才分片。分片的大小由如下幾個參數決定：mapreduce.input.fileinputformat.split.maxsize、mapreduce.input.fileinputformat.split.minsize、文件的blocksize大小確定。
具體計算方式為：
Math.max(minSize, Math.min(maxSize, blockSize))
分片的大小有可能比默認塊大小64M要大，當然也有可能小于它，默認情況下分片大小為當前HDFS的塊大小，64M
第一步 將bytesRemaining(剩余未分片字節數)初始化設置為整個文件的長度
第二步 如果bytesRemaining超過分片大小splitSize一定量才會將文件分成多個InputSplit，SPLIT_SLOP(默認1.1)。接著就會執行如下方法獲取block的索引，其中第二個參數是這個block在整個文件中的偏移量
protected int getBlockIndex(BlockLocation[] blkLocations,
long offset) {
for (int i = 0 ; i < blkLocations.length; i++) {
// is the offset inside this block? 核心代碼塊 判斷當前的偏移量是否在某個block中 是就返回當前index 位置信息
if ((blkLocations[i].getOffset() <= offset) &&
(offset < blkLocations[i].getOffset() + blkLocations[i].getLength())){
return i;
}
}
BlockLocation last = blkLocations[blkLocations.length -1];
long fileLength = last.getOffset() + last.getLength() -1;
throw new IllegalArgumentException("Offset " + offset +
" is outside of file (0.." +
fileLength + ")");
}
第三步 將符合條件的塊的索引對應的block信息的主機節點以及文件的路徑名、開始的偏移量、分片大小splitSize封裝到一個InputSplit中加入List<InputSplit> splits 列表。
第四步 bytesRemaining -= splitSize修改剩余字節大小 循環以上操作 直到不滿足條件 剩余bytesRemaining還不為0，表示還有未分配的數據，將剩余的數據及最后一個block加入splits列表
以上是 整個getSplits獲取切片的過程。當使用基于FileInputFormat實現InputFormat時，為了提高MapTask的數據本地化，應盡量使InputSplit大小與block大小相同
2.3 submitter 實現了ClientProtocol接口的類 在1.1中connect()連接集群時 調用init初始化方法 由框架讀取 HDFS的配置文件中配置了mapreduce.framework.name屬性為“yarn”的話，會創建一個YARNRunner對象 submitter 就是YARNRunner 對象
submitter.submitJobInternal(Job.this, cluster)
YARNRunner的構造方法：
public YARNRunner(Configuration conf, ResourceMgrDelegate resMgrDelegate,
ClientCache clientCache) {
this.conf = conf;
try {
this.resMgrDelegate = resMgrDelegate;
this.clientCache = clientCache;
this.defaultFileContext = FileContext.getFileContext(this.conf);
} catch (UnsupportedFileSystemException ufe) {
throw new RuntimeException("Error in instantiating YarnClient", ufe);
}
}
ResourceMgrDelegate實際上ResourceManager的代理類，其實現了YarnClient接口，通過ApplicationClientProtocol代理直接向RM提交Job，殺死Job，查詢Job運行狀態等操作。
YarnRunner 類的submitJob方法
public JobStatus submitJob(JobID jobId, String jobSubmitDir, Credentials ts)
throws IOException, InterruptedException {
addHistoryToken(ts);
// Construct necessary information to start the MR AM
//Client構造ASC。ASC中包括了調度隊列，優先級，用戶認證信息，除了這些基本的信息之外，還包括用來啟動AM的CLC信息，一個CLC中包括jar包、依賴文件、安全token，以及運行任務過程中需要的其他文件
ApplicationSubmissionContext appContext =
createApplicationSubmissionContext(conf, jobSubmitDir, ts);
// Submit to ResourceManager
try {
ApplicationId applicationId =
resMgrDelegate.submitApplication(appContext); // 2.4 提交ASC到RecoureManeger
ApplicationReport appMaster = resMgrDelegate
.getApplicationReport(applicationId);
String diagnostics =
(appMaster == null ?
"application report is null" : appMaster.getDiagnostics());
if (appMaster == null
|| appMaster.getYarnApplicationState() == YarnApplicationState.FAILED
|| appMaster.getYarnApplicationState() == YarnApplicationState.KILLED) {
throw new IOException("Failed to run job : " +
diagnostics);
}
return clientCache.getClient(jobId).getJobStatus(jobId);
} catch (YarnException e) {
throw new IOException(e);
}
}
2.4 到這里一個Client就完成了一次Job任務的提交
2.5 YARN 框架 統一的資源管理 任務調度
相關的角色
**ResourceManager **
集群節點資源的統一管理
**NodeManager ** 每個DN上都會對應一個NM進程
- 與RM匯報資源的使用情況
- 管理運行的Container生命周期
Container：【節點NM上CPU,MEM,I/O大小等資源的虛擬描述】
MR-ApplicationMaster-Container
每個Job作業對應一個AM，避免單點故障，負載到不同的節點
創建Task時需要和RM申請資源（Container），然后向存放具體資源的DN通信，由DN創建Container并且啟動進程同時下發任務（這里就實現了計算向數據移動）
Task-Container 任務執行進程
DN上執行的JVM進程，接收到AM下發的任務后，通過反射機制創建具體的任務對象后 執行具體的任務
** 執行流程**
1 RM 在空閑的DN 上啟動AM
2 AM向RM申請資源 ,RM將資源分配信息給AM
3 AM在和數據所在的NM節點通信,創建Container并且通知NM啟動Container(JVM進程),分發具體任務到NM上，Container通過反射調起具體的任務類執行
4 如果是MapReduce框架 則進入到MapTask流程 具體分析見 http://liujiacai.net/blog/2014/09/07/yarn-intro/
