以官方Hadoop中的 WordCount案例分析 ，Job作業(yè)的提交過程：

public static void main(String[] args) throws Exception {
          // Create a new Job
          Configuration conf=new Configuration(true);
          Job job = Job.getInstance(conf);
          job.setJarByClass(MyWorkCountJob.class);
         // Specify various job-specific parameters    
          job.setJobName("myWorkCountjob");
          //設(shè)置輸入文件路徑
          FileInputFormat.addInputPath(job, new Path("/user/root/hello.txt"));
          //設(shè)置輸出文件路徑
          Path outPath=new Path("/sxt/mr/output");
          if(FileSystem.get(conf).exists(outPath))
           FileSystem.get(conf).delete(outPath);
          FileOutputFormat.setOutputPath(job, outPath);
         job.setMapperClass(MyMapper.class);
          
          job.setReducerClass(MyReducer.class);
          job.setOutputKeyClass(Text.class);
          job.setOutputValueClass(IntWritable.class);
          // Submit the job, then poll for progress until the job is complete
          job.waitForCompletion(true);//job 提交的入口
     }

waitForCompletion方法

public boolean waitForCompletion(boolean verbose
                                   ) throws IOException, InterruptedException,
                                            ClassNotFoundException {
    if (state == JobState.DEFINE) {
      submit();// 任務(wù)提交1.1
    }
    if (verbose) {
      monitorAndPrintJob();//實(shí)時監(jiān)控Job任務(wù)并打印相關(guān)的日志
    } else {
      // get the completion poll interval from the client.
      int completionPollIntervalMillis =
        Job.getCompletionPollInterval(cluster.getConf());
      while (!isComplete()) {
        try {
          Thread.sleep(completionPollIntervalMillis);
        } catch (InterruptedException ie) {
        }
      }
    }
    return isSuccessful();
  }

1.1 submit 方法

public void submit()
         throws IOException, InterruptedException, ClassNotFoundException {
    ensureState(JobState.DEFINE);//確定job狀態(tài)
    setUseNewAPI();//默認(rèn)使用新的API
    connect();//獲得與集群的連接
    final JobSubmitter submitter =
        getJobSubmitter(cluster.getFileSystem(), cluster.getClient());
    status = ugi.doAs(new PrivilegedExceptionAction<JobStatus>() {
      public JobStatus run() throws IOException, InterruptedException,
      ClassNotFoundException {
            //異步調(diào)用submitJobInternal方法提交任務(wù) 1.2
        return submitter.submitJobInternal(Job.this, cluster);
      }
    });
    state = JobState.RUNNING;
    LOG.info("The url to track the job: " + getTrackingURL());
   }

submit方法首先創(chuàng)建了JobSubmitter實(shí)例，然后異步調(diào)用了JobSubmitter的submitJobInternal方法

1.2 submitJobInternal 方法

JobStatus submitJobInternal(Job job, Cluster cluster)
  throws ClassNotFoundException, InterruptedException, IOException {

    //檢查job的輸出路徑是否存在，如果存在則拋出異常
    checkSpecs(job);
    Configuration conf = job.getConfiguration();
    addMRFrameworkToDistributedCache(conf);

      //初始化臨時目錄和返回的輸出路徑。
    Path jobStagingArea = JobSubmissionFiles.getStagingDir(cluster, conf);
    //configure the command line options correctly on the submitting dfs
    InetAddress ip = InetAddress.getLocalHost();
    if (ip != null) {
      submitHostAddress = ip.getHostAddress();
      submitHostName = ip.getHostName();
      conf.set(MRJobConfig.JOB_SUBMITHOST,submitHostName);
      conf.set(MRJobConfig.JOB_SUBMITHOSTADDR,submitHostAddress);
    }
      //獲取新的JobId 
    JobID jobId = submitClient.getNewJobID();
    job.setJobID(jobId);
      // 獲取提交目錄
    Path submitJobDir = new Path(jobStagingArea, jobId.toString());
    ......
        //把作業(yè)上傳到集群中去
      copyAndConfigureFiles(job, submitJobDir);

      Path submitJobFile = JobSubmissionFiles.getJobConfPath(submitJobDir);
     
      // 創(chuàng)建切片列表 找出每個文件的切片列表 合并切片列表的數(shù)量就是Map任務(wù)個數(shù)  客戶端統(tǒng)計
      int maps = writeSplits(job, submitJobDir); //2.1核心方法
      conf.setInt(MRJobConfig.NUM_MAPS, maps);//文件分片的大小 就是Map任務(wù)數(shù)量
      ......

      // Write job file to submit dir    相關(guān)配置寫入到j(luò)ob.xml中
      writeConf(conf, submitJobFile);
     
      // Now, actually submit the job (using the submit name) 真正的提交作業(yè)
      status = submitClient.submitJob( //2.3 提交job到RecourceManager
          jobId, submitJobDir.toString(), job.getCredentials());
     ...
  }

2.1 文件切片操作 writeSplits -> writeNewSplits 計算向數(shù)據(jù)移動模型的核心

private <T extends InputSplit>
  int writeNewSplits(JobContext job, Path jobSubmitDir) throws IOException,
      InterruptedException, ClassNotFoundException {
    Configuration conf = job.getConfiguration();
    InputFormat<?, ?> input =
      ReflectionUtils.newInstance(job.getInputFormatClass(), conf);

    List<InputSplit> splits = input.getSplits(job);
    T[] array = (T[]) splits.toArray(new InputSplit[splits.size()]);

    // sort the splits into order based on size, so that the biggest
    // go first
    Arrays.sort(array, new SplitComparator());
      //將split信息和SplitMetaInfo都寫入HDFS中
    JobSplitWriter.createSplitFiles(jobSubmitDir, conf,
        jobSubmitDir.getFileSystem(conf), array);
    return array.length;
  }

writeNewSplits方法中，劃分任務(wù)數(shù)量最關(guān)鍵的代碼即為InputFormat的getSplits方法（InputFormat有不同實(shí)現(xiàn)類 框架默認(rèn)的是TextInputFormat）。此時的Input即為TextInputFormat的父類FileInputFormat，其getSplits方法的實(shí)現(xiàn)如下：

 public List<InputSplit> getSplits(JobContext job) throws IOException {
    Stopwatch sw = new Stopwatch().start();
    long minSize = Math.max(getFormatMinSplitSize(), getMinSplitSize(job));//默認(rèn)最小值 1
    long maxSize = getMaxSplitSize(job);//默認(rèn)最大值 Long類型的最大值

    // generate splits
    List<InputSplit> splits = new ArrayList<InputSplit>();
    List<FileStatus> files = listStatus(job);//獲取源文件的源信息列表
    for (FileStatus file: files) {
      Path path = file.getPath();
      long length = file.getLen();
      if (length != 0) {
        BlockLocation[] blkLocations;
            //獲取文件的block塊列表
        if (file instanceof LocatedFileStatus) {
          blkLocations = ((LocatedFileStatus) file).getBlockLocations();
        } else {
          FileSystem fs = path.getFileSystem(job.getConfiguration());
          blkLocations = fs.getFileBlockLocations(file, 0, length);
        }
        if (isSplitable(job, path))
          long blockSize = file.getBlockSize();
          long splitSize = computeSplitSize(blockSize, minSize, maxSize);
        //核心代碼塊
          long bytesRemaining = length;
          while (((double) bytesRemaining)/splitSize > SPLIT_SLOP) {
            int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
            splits.add(makeSplit(path, length-bytesRemaining, splitSize,
                        blkLocations[blkIndex].getHosts(),
                        blkLocations[blkIndex].getCachedHosts()));
            bytesRemaining -= splitSize;
          }
//核心代碼塊結(jié)束
          if (bytesRemaining != 0) { 
            int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
            splits.add(makeSplit(path, length-bytesRemaining, bytesRemaining,
                       blkLocations[blkIndex].getHosts(),
                       blkLocations[blkIndex].getCachedHosts()));
          }
        } else { // not splitable
          splits.add(makeSplit(path, 0, length, blkLocations[0].getHosts(),
                      blkLocations[0].getCachedHosts()));
        }
      } else {
        //Create empty hosts array for zero length files
        splits.add(makeSplit(path, 0, length, new String[0]));
      }
    }
    // Save the number of input files for metrics/loadgen
    job.getConfiguration().setLong(NUM_INPUT_FILES, files.size());
    sw.stop();
    if (LOG.isDebugEnabled()) {
      LOG.debug("Total # of splits generated by getSplits: " + splits.size()
          + ", TimeTaken: " + sw.elapsedMillis());
    }
    return splits;
  }

2.2 核心代碼塊分析
對每個輸入文件進(jìn)行split劃分。注意這只是個邏輯的劃分 因此執(zhí)行的是FileInputFormat類中的getSplits方法。只有非壓縮的文件和幾種特定壓縮方式壓縮后的文件才分片。分片的大小由如下幾個參數(shù)決定：mapreduce.input.fileinputformat.split.maxsize、mapreduce.input.fileinputformat.split.minsize、文件的blocksize大小確定。
具體計算方式為：
Math.max(minSize, Math.min(maxSize, blockSize))
分片的大小有可能比默認(rèn)塊大小64M要大，當(dāng)然也有可能小于它，默認(rèn)情況下分片大小為當(dāng)前HDFS的塊大小，64M

第一步 將bytesRemaining(剩余未分片字節(jié)數(shù))初始化設(shè)置為整個文件的長度
第二步 如果bytesRemaining超過分片大小splitSize一定量才會將文件分成多個InputSplit，SPLIT_SLOP(默認(rèn)1.1)。接著就會執(zhí)行如下方法獲取block的索引，其中第二個參數(shù)是這個block在整個文件中的偏移量

protected int getBlockIndex(BlockLocation[] blkLocations,
                              long offset) {
    for (int i = 0 ; i < blkLocations.length; i++) {
      // is the offset inside this block? 核心代碼塊 判斷當(dāng)前的偏移量是否在某個block中 是就返回當(dāng)前index 位置信息
      if ((blkLocations[i].getOffset() <= offset) &&
          (offset < blkLocations[i].getOffset() + blkLocations[i].getLength())){
        return i;
      }
    }
    BlockLocation last = blkLocations[blkLocations.length -1];
    long fileLength = last.getOffset() + last.getLength() -1;
    throw new IllegalArgumentException("Offset " + offset +
                                       " is outside of file (0.." +
                                       fileLength + ")");
  }

第三步 將符合條件的塊的索引對應(yīng)的block信息的主機(jī)節(jié)點(diǎn)以及文件的路徑名、開始的偏移量、分片大小splitSize封裝到一個InputSplit中加入List<InputSplit> splits 列表。

第四步 bytesRemaining -= splitSize修改剩余字節(jié)大小 循環(huán)以上操作 直到不滿足條件 剩余bytesRemaining還不為0，表示還有未分配的數(shù)據(jù)，將剩余的數(shù)據(jù)及最后一個block加入splits列表

以上是 整個getSplits獲取切片的過程。當(dāng)使用基于FileInputFormat實(shí)現(xiàn)InputFormat時，為了提高M(jìn)apTask的數(shù)據(jù)本地化，應(yīng)盡量使InputSplit大小與block大小相同

2.3 submitter 實(shí)現(xiàn)了ClientProtocol接口的類 在1.1中connect()連接集群時 調(diào)用init初始化方法 由框架讀取 HDFS的配置文件中配置了mapreduce.framework.name屬性為“yarn”的話，會創(chuàng)建一個YARNRunner對象 submitter 就是YARNRunner 對象
submitter.submitJobInternal(Job.this, cluster)

YARNRunner的構(gòu)造方法：

public YARNRunner(Configuration conf, ResourceMgrDelegate resMgrDelegate,
     ClientCache clientCache) {
   this.conf = conf;
   try {
     this.resMgrDelegate = resMgrDelegate;
     this.clientCache = clientCache;
     this.defaultFileContext = FileContext.getFileContext(this.conf);
   } catch (UnsupportedFileSystemException ufe) {
     throw new RuntimeException("Error in instantiating YarnClient", ufe);
   }
 }

ResourceMgrDelegate實(shí)際上ResourceManager的代理類，其實(shí)現(xiàn)了YarnClient接口，通過ApplicationClientProtocol代理直接向RM提交Job，殺死Job，查詢Job運(yùn)行狀態(tài)等操作。
YarnRunner 類的submitJob方法

public JobStatus submitJob(JobID jobId, String jobSubmitDir, Credentials ts)
  throws IOException, InterruptedException {
    addHistoryToken(ts);
    // Construct necessary information to start the MR AM
       //Client構(gòu)造ASC。ASC中包括了調(diào)度隊(duì)列，優(yōu)先級，用戶認(rèn)證信息，除了這些基本的信息之外，還包括用來啟動AM的CLC信息，一個CLC中包括jar包、依賴文件、安全token，以及運(yùn)行任務(wù)過程中需要的其他文件
    ApplicationSubmissionContext appContext =
      createApplicationSubmissionContext(conf, jobSubmitDir, ts);
    // Submit to ResourceManager
    try {
      ApplicationId applicationId =
          resMgrDelegate.submitApplication(appContext); // 2.4 提交ASC到RecoureManeger 

      ApplicationReport appMaster = resMgrDelegate
          .getApplicationReport(applicationId);
      String diagnostics =
          (appMaster == null ?
              "application report is null" : appMaster.getDiagnostics());
      if (appMaster == null
          || appMaster.getYarnApplicationState() == YarnApplicationState.FAILED
          || appMaster.getYarnApplicationState() == YarnApplicationState.KILLED) {
        throw new IOException("Failed to run job : " +
            diagnostics);
      }
      return clientCache.getClient(jobId).getJobStatus(jobId);
    } catch (YarnException e) {
      throw new IOException(e);
    }
  }

2.4 到這里一個Client就完成了一次Job任務(wù)的提交

2.5 YARN 框架 統(tǒng)一的資源管理 任務(wù)調(diào)度

yarn.png

**NodeManager ** 每個DN上都會對應(yīng)一個NM進(jìn)程

• 與RM匯報資源的使用情況
• 管理運(yùn)行的Container生命周期
  Container：【節(jié)點(diǎn)NM上CPU,MEM,I/O大小等資源的虛擬描述】

MR-ApplicationMaster-Container
每個Job作業(yè)對應(yīng)一個AM，避免單點(diǎn)故障，負(fù)載到不同的節(jié)點(diǎn)
創(chuàng)建Task時需要和RM申請資源（Container），然后向存放具體資源的DN通信，由DN創(chuàng)建Container并且啟動進(jìn)程同時下發(fā)任務(wù)（這里就實(shí)現(xiàn)了計算向數(shù)據(jù)移動）

Task-Container 任務(wù)執(zhí)行進(jìn)程
DN上執(zhí)行的JVM進(jìn)程，接收到AM下發(fā)的任務(wù)后，通過反射機(jī)制創(chuàng)建具體的任務(wù)對象后 執(zhí)行具體的任務(wù)

** 執(zhí)行流程**
1 RM 在空閑的DN 上啟動AM
2 AM向RM申請資源 ,RM將資源分配信息給AM
3 AM在和數(shù)據(jù)所在的NM節(jié)點(diǎn)通信,創(chuàng)建Container并且通知NM啟動Container(JVM進(jìn)程),分發(fā)具體任務(wù)到NM上，Container通過反射調(diào)起具體的任務(wù)類執(zhí)行
4 如果是MapReduce框架 則進(jìn)入到MapTask流程 具體分析見 http://liujiacai.net/blog/2014/09/07/yarn-intro/