Flink Checkpoint配置

范例:

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

// start a checkpoint every 1000 ms
env.enableCheckpointing(1000);

// advanced options:
// set mode to exactly-once (this is the default)
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);

// checkpoints have to complete within one minute, or are discarded
env.getCheckpointConfig().setCheckpointTimeout(60000);

// make sure 500 ms of progress happen between checkpoints
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(500);

// allow only one checkpoint to be in progress at the same time
env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);

// enable externalized checkpoints which are retained after job cancellation
env.getCheckpointConfig().enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);

// This determines if a task will be failed if an error occurs in the execution of the task’s checkpoint procedure.
env.getCheckpointConfig().setFailOnCheckpointingErrors(true);
  • 使用StreamExecutionEnvironment.enableCheckpointing方法來設置開啟checkpoint
    (具體可以使用StreamExecutionEnvironment.getCheckpointConfig.enableCheckpointing(long interval)),
    或者StreamExecutionEnvironment.getCheckpointConfig.enableCheckpointing(long interval, CheckpointingMode mode)。
    interval用于指定checkpoint的觸發間隔(單位milliseconds)
    CheckpointingMode默認是CheckpointingMode.EXACTLY_ONCE
    也可以指定為CheckpointingMode.AT_LEAST_ONCE

  • 也可以通過StreamExecutionEnvironment.getCheckpointConfig().setCheckpointingMode來設置CheckpointingMode,
    一般對于超低延遲的應用(大概幾毫秒)可以使用CheckpointingMode.AT_LEAST_ONCE,其他大部分應用使用CheckpointingMode.EXACTLY_ONCE就可以
    checkpointTimeout用于指定checkpoint執行的超時時間(單位milliseconds),超時沒完成就會被abort掉。

  • minPauseBetweenCheckpoints用于指定checkpoint距上一個checkpoint完成之后最少等多久可以出發另一個checkpoint,
    當指定這個參數時,maxConcurrentCheckpoints的值為1

  • maxConcurrentCheckpoints用于指定運行中的checkpoint最多可以有多少個;
    如果有設置了minPauseBetweenCheckpoints,則maxConcurrentCheckpoints這個參數就不起作用了(大于1的值不起作用)

  • enableExternalizedCheckpoints用于開啟checkpoints的外部持久化,但是在job失敗的時候不會自動清理,需要自己手工清理stateExternalizedCheckpointCleanup用于指定當job canceled的時候externalized checkpoint該如何清理,DELETE_ON_CANCELLATION的話,在job canceled的時候會自動刪除externalized state,但是如果是FAILED的狀態則會保留;RETAIN_ON_CANCELLATION則在job canceled的時候會保留externalized checkpoint state

  • failOnCheckpointingErrors用于指定在checkpoint發生異常的時候,是否應該failtask,默認為true,如果設置為false,則task會拒絕checkpoint然后繼續運行

flink-conf.yaml相關配置:

#==============================================================================
# Fault tolerance and checkpointing
#==============================================================================

# The backend that will be used to store operator state checkpoints if
# checkpointing is enabled.
#
# Supported backends are 'jobmanager', 'filesystem', 'rocksdb', or the
# <class-name-of-factory>.
#
# state.backend: filesystem

# Directory for checkpoints filesystem, when using any of the default bundled
# state backends.
#
# state.checkpoints.dir: hdfs://namenode-host:port/flink-checkpoints

# Default target directory for savepoints, optional.
#
# state.savepoints.dir: hdfs://namenode-host:port/flink-checkpoints

# Flag to enable/disable incremental checkpoints for backends that
# support incremental checkpoints (like the RocksDB state backend). 
#
# state.backend.incremental: false
  • state.backend用于指定checkpoint state存儲的backend,默認為none
  • state.backend.async用于指定backend是否使用異步snapshot(默認為true),有些不支持async或者只支持asyncstate backend可能會忽略這個參數
  • state.backend.fs.memory-threshold,默認為1024,用于指定存儲于filesstate大小閾[yù]值,如果小于該值則會存儲在root checkpoint metadata file
  • state.backend.incremental,默認為false,用于指定是否采用增量checkpoint,有些不支持增量checkpointbackend會忽略該配置
  • state.backend.local-recovery,默認為false
  • state.checkpoints.dir,默認為none,用于指定checkpointdata filesmeta data存儲的目錄,該目錄必須對所有參與的TaskManagersJobManagers可見
  • state.checkpoints.num-retained,默認為1,用于指定保留的已完成的checkpoints個數
  • state.savepoints.dir,默認為none,用于指定savepoints的默認目錄
  • taskmanager.state.local.root-dirs,默認為none

小結:

  • 可以通過使用StreamExecutionEnvironment.enableCheckpointing方法來設置開啟checkpoint;具體可以使用enableCheckpointing(long interval),或者enableCheckpointing(long interval, CheckpointingMode mode)

  • checkpoint的高級配置可以配置enableExternalizedCheckpoints(用于開啟checkpoints的外部持久化,在job failed的時候externalized checkpoint state無法自動清理,但是在job canceled的時候可以配置是刪除還是保留state)

  • flink-conf.yaml里頭也有checkpoint的相關配置,主要是state backend的配置,比如state.backend.async、state.backend.incremental、state.checkpoints.dir、state.savepoints.dir

Java 配置實例:

/**
 * 是否重啟標識flag
 */
private static boolean replayFlag = true;

/**
 * 重啟次數
 */
private static Integer replayTimes;

/**
 * 重啟時間間隔
 */
private static Integer replaySeconds;

private static Long checkPointTime;

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

if (replayFlag) {
    env.setRestartStrategy(RestartStrategies.fixedDelayRestart(
            replayTimes,
            Time.of(replaySeconds, TimeUnit.SECONDS)
    ));
    CheckpointConfig config = env.getCheckpointConfig();

    //env.setStateBackend(new FsStateBackend(checkPointDir));
    // 任務流取消和故障時會保留Checkpoint數據,以便根據實際需要恢復到指定的Checkpoint
    config.enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
    
    // 設置checkpoint的周期, 每隔1000 ms進行啟動一個檢查點
    config.setCheckpointInterval(checkPointTime);
    
    // 設置模式為exactly-once
    config.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
    
    // 確保檢查點之間有至少500 ms的間隔【checkpoint最小間隔】
    config.setMinPauseBetweenCheckpoints(500);
    
    // 檢查點必須在一分鐘內完成,或者被丟棄【checkpoint的超時時間】
    config.setCheckpointTimeout(checkPointTime);
    
    // 同一時間只允許進行一個檢查點
    config.setMaxConcurrentCheckpoints(1);
}
最后編輯于
?著作權歸作者所有,轉載或內容合作請聯系作者
平臺聲明:文章內容(如有圖片或視頻亦包括在內)由作者上傳并發布,文章內容僅代表作者本人觀點,簡書系信息發布平臺,僅提供信息存儲服務。

推薦閱讀更多精彩內容