原文：https://developer.android.com/reference/android/media/MediaCodec.html

MediaCodec class can be used to access low-level media codecs, i.e. encoder/decoder components. It is part of the Android low-level multimedia support infrastructure (normally used together with MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface, and AudioTrack.)

MediaCodec 類可以處理低層級的多媒體編解碼器，如：encoder/decoder。它是Android低層級的多媒體支持基礎的一部分（通常和MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface, and AudioTrack 一起使用）

Screen Shot 2017-05-23 at 10.55.42 AM.png

In broad terms, a codec processes input data to generate output data. It processes data asynchronously and uses a set of input and output buffers. At a simplistic level, you request (or receive) an empty input buffer, fill it up with data and send it to the codec for processing. The codec uses up the data and transforms it into one of its empty output buffers. Finally, you request (or receive) a filled output buffer, consume its contents and release it back to the codec.
總的來說，codec 處理輸入的數據，生成輸出數據。異步處理數據，并且使用一系列的輸入輸出緩沖(buffer). 在一個簡化的層次， 你請求（或者接收）一個空的輸入buffer, 填充數據后把它交給codec進行處理。codec用完數據后把buffer轉換成它的里面的眾多空的輸出buffer中的一個。最后，你請求（或者接收）一個充滿數據的輸出buffer，提取出它里面的數據，然后把它釋放回codec。

Data Types

Codecs operate on three kinds of data: compressed data, raw audio data and raw video data. All three kinds of data can be processed using ByteBuffers, but you should use a Surface for raw video data to improve codec performance. Surface uses native video buffers without mapping or copying them to ByteBuffers; thus, it is much more efficient. You normally cannot access the raw video data when using a Surface, but you can use the ImageReader class to access unsecured decoded (raw) video frames. This may still be more efficient than using ByteBuffers, as some native buffers may be mapped into direct ByteBuffers. When using ByteBuffer mode, you can access raw video frames using the Image class and getInput/OutputImage(int).

數據類型

Codec對三種數據進行處理：壓縮數據，音頻原始數據和視頻原始數據。三種數據都能夠用ByteBuffers進行處理，但是為了提高codec處理能力，在處理視頻原始數據時，你應該使用Surface。Surface使用native的視頻buffer，不需要映射或都復制到ByteBuffers;因此，效率更高。通常使用Surface時，你無法接獲取原始視頻數據，但是你可以使用ImageReader類來獲取未加密的解碼后的（原始）視頻幀。即使這樣也比使用ByteBuffers的效率更高，因為一些native buffers可以被映射到ByteBuffers. 當使用ByteBuffer模式時，你可以用Image， getInput/OutputImage(int)獲取到原始視頻幀。

Compressed Buffers

Input buffers (for decoders) and output buffers (for encoders) contain compressed data according to the format's type. For video types this is a single compressed video frame. For audio data this is normally a single access unit (an encoded audio segment typically containing a few milliseconds of audio as dictated by the format type), but this requirement is slightly relaxed in that a buffer may contain multiple encoded access units of audio. In either case, buffers do not start or end on arbitrary byte boundaries, but rather on frame/access unit boundaries.

壓縮 Buffers

輸入buffers(解碼器使用）和輸出buffers(編碼器使用）包含根據格式壓縮的數據。如果是視頻，那么就是壓縮后的一幀。如果是音頻，通常是一個access單位（一個編碼過的音頻片段，一般來說包含幾毫秒的音頻），但也可能幾個access單位。不論是哪種情況，buffers 不會在隨意的byte邊界開始或結束，而是以frame/acess為單位來劃分起始和結束位置。

Raw Audio Buffers

Raw audio buffers contain entire frames of PCM audio data, which is one sample for each channel in channel order. Each sample is a 16-bit signed integer in native byte order.

原始音頻buffers

原始音頻buffers包含所有PCM音頻數據幀。PCM音頻數據是一個按通道的順序對所有音頻通道的采樣結果。每個采樣是一個16位的帶符號的整數。

 short[] getSamplesForChannel(MediaCodec codec, int bufferId, int channelIx) {
   ByteBuffer outputBuffer = codec.getOutputBuffer(bufferId);
   MediaFormat format = codec.getOutputFormat(bufferId);
   ShortBuffer samples = outputBuffer.order(ByteOrder.nativeOrder()).asShortBuffer();
   int numChannels = formet.getInteger(MediaFormat.KEY_CHANNEL_COUNT);
   if (channelIx < 0 || channelIx >= numChannels) {
     return null;
   }
   short[] res = new short[samples.remaining() / numChannels];
   for (int i = 0; i < res.length; ++i) {
     res[i] = samples.get(i * numChannels + channelIx);
   }
   return res;
 }

Raw Video Buffers

In ByteBuffer mode video buffers are laid out according to their color format. You can get the supported color formats as an array from getCodecInfo().getCapabilitiesForType(…).colorFormats. Video codecs may support three kinds of color formats:

原始視頻buffers

在ByteBuffer模式下，視頻buffers根據它們的顏色格式進行布置的。調用getCodecInfo().getCapabilitiesForType(…).colorFormats可以得到一個包含所有支持的顏色格式的數組。視頻codecs可能支持三種顏色格式：

• ****native raw video format:**** This is marked by COLOR_FormatSurface and it can be used with an input or output Surface.

• ****flexible YUV buffers (such as COLOR_FormatYUV420Flexible):**** These can be used with an input/output Surface, as well as in ByteBuffer mode, by using getInput/OutputImage(int).

• ****other, specific formats:**** These are normally only supported in ByteBuffer mode. Some color formats are vendor specific. Others are defined in MediaCodecInfo.CodecCapabilities. For color formats that are equivalent to a flexible format, you can still use getInput/OutputImage(int).
  All video codecs support flexible YUV 4:2:0 buffers since LOLLIPOP_MR1.

• ****native原始視頻格式**** 由COLOR_FormatSurface進行標記，可以在輸入或者輸出的Surface中進行使用。

• ****可變的YUV buffers(例如COLOR_FormatYUV420Flexible) ：****這些buffers可以在輸入/輸出Surface中使用，同樣在ByteBuffer模式下通過getInput/OutputImage(int)也可以使用.

• ****其他，特定的格式：****這些格式通常只在ByteBuffer模式下支持。一些顏色格式是廠商定制的。其他的格式在MediaCodecInfo.CodecCapabilities中進行定義。對于和可變的格式相同的顏色格式，你仍然可以使用getInput/OutputImage(int).從LOLLIPOP_MR1(API Level 22)開始所有的視頻codecs支持可變的YUV 4:2:0 buffers。

Accessing Raw Video ByteBuffers on Older Devices

Prior to LOLLIPOP and Image support, you need to use the KEY_STRIDE and KEY_SLICE_HEIGHT output format values to understand the layout of the raw output buffers.

在LOLLIPOP之前的設備上獲取原始視頻ByteBuffers

在LOLLIPOP之前的或不支持Image的系統版本中，你需要用KEY_STRIDE和KEY_SLICE_HEIGHT輸出格式值來理解原始輸出buffers的布置。

Note that on some devices the slice-height is advertised as 0. This could mean either that the slice-height is the same as the frame height, or that the slice-height is the frame height aligned to some value (usually a power of 2). Unfortunately, there is no standard and simple way to tell the actual slice height in this case. Furthermore, the vertical stride of the U plane in planar formats is also not specified or defined, though usually it is half of the slice height.

注意在一些設備上，slice-height值是0。這是說，要么 slice-height和幀高度一致，要么是幀高度與某個值（通常是2的指數）對齊之后的值。不幸的是，在這種情況下，沒有一個標準簡單的方法來告實際的slice height。Furthermore, the vertical stride of the U plane in planar formats is also not specified or defined, though usually it is half of the slice height.（不知道這句什么意思）

The KEY_WIDTH and KEY_HEIGHT keys specify the size of the video frames; however, for most encodings the video (picture) only occupies a portion of the video frame. This is represented by the 'crop rectangle'.
KEY_WIDTH和KEY_HEIGHT指定了視頻的幀寬高；但是，對于大多數的encodings，視頻（圖片）只占了視頻幀的一部分。這部分就是'crop rectangle'.

You need to use the following keys to get the crop rectangle of raw output images from the output format. If these keys are not present, the video occupies the entire video frame.The crop rectangle is understood in the context of the output frame before applying any rotation.
獲取輸出格式的原始輸出圖片的crop rectangle，需要用到以下的keys。如果沒有提供這些keys，視頻占據全部的視頻幀。在旋轉之前，crop rectangle在輸出幀的上下文中進行‘理解’。

The right and bottom coordinates can be understood as the coordinates of the right-most valid column/bottom-most valid row of the cropped output image.
右邊和底部的坐標可以理解為裁剪后的圖片最右面的列和最下邊的行的坐標。

The size of the video frame (before rotation) can be calculated as such:
旋轉前的視頻幀的大小可以通過以下方式進行計算：

 MediaFormat format = decoder.getOutputFormat(…);
 int width = format.getInteger(MediaFormat.KEY_WIDTH);
 if (format.containsKey("crop-left") && format.containsKey("crop-right")) {
     width = format.getInteger("crop-right") + 1 - format.getInteger("crop-left");
 }
 int height = format.getInteger(MediaFormat.KEY_HEIGHT);
 if (format.containsKey("crop-top") && format.containsKey("crop-bottom")) {
     height = format.getInteger("crop-bottom") + 1 - format.getInteger("crop-top");
 }

Also note that the meaning of BufferInfo.offset was not consistent across devices. On some devices the offset pointed to the top-left pixel of the crop rectangle, while on most devices it pointed to the top-left pixel of the entire frame.
另外需要注意的是，BufferInfo.offset 代表的意義在不同的設備上并不一樣。在一些設備上，offset指crop rectangle最左上方的像素值，而在另一些設備上是指整個幀的最左上方的像素值。

States

During its life a codec conceptually exists in one of three states: Stopped, Executing or Released. The Stopped collective state is actually the conglomeration of three states: Uninitialized, Configured and Error, whereas the Executing state conceptually progresses through three sub-states: Flushed, Running and End-of-Stream.

狀態

在codec的生命周期中，邏輯上只處于三種狀態中的一種：停止，執行和釋放。停止狀態包含三種狀態：Uninitialized, Configured和Error；執行狀態也含三種狀態：Flushed, Running和End-of-Stream.

Screen Shot 2017-05-23 at 11.07.16 AM.png

When you create a codec using one of the factory methods, the codec is in the Uninitialized state. First, you need to configure it via configure(…), which brings it to the Configured state, then call start() to move it to the Executing state. In this state you can process data through the buffer queue manipulation described above.
當你用其中一個工廠方法創建一個codec時，codec處于Uninitialized狀態。首先，你需要用configure(…)去配置它，使它進入Configured狀態，然后調用start()方法使codec進入Executing狀態。在Executing狀態，你可以操作buffer隊列進行數據處理。

The Executing state has three sub-states: Flushed, Running and End-of-Stream. Immediately after start() the codec is in the Flushed sub-state, where it holds all the buffers. As soon as the first input buffer is dequeued, the codec moves to the Running sub-state, where it spends most of its life. When you queue an input buffer with the end-of-stream marker, the codec transitions to the End-of-Stream sub-state. In this state the codec no longer accepts further input buffers, but still generates output buffers until the end-of-stream is reached on the output. You can move back to the Flushed sub-state at any time while in the Executing state using flush().
執行狀態含三種狀態：Flushed, Running和End-of-Stream.調用start()方法后，code立即處于Flushed sub-state, 持有所有buffers。當第一個輸入buffer被dequeued，codec進入Running sub-state, 這個state會占據大部分生命周期。當你用一個end-of-stream標志queue一個輸入buffer時，codec進入End-of-Stream狀態。在這個狀態，codec不再接收輸入buffers，但是仍然在生成輸出buffers，直到輸出至end-of-stream。在Executing狀態 ，任何時候你都可以調用flush()返回Flushed sub-state.

Call stop() to return the codec to the Uninitialized state, whereupon it may be configured again. When you are done using a codec, you must release it by calling release().
調用stop()方法使codec返回Uninitialized狀態,然后它可以重新被配置。使用完codec，必須調用release()方法.

On rare occasions the codec may encounter an error and move to the Error state. This is communicated using an invalid return value from a queuing operation, or sometimes via an exception. Call reset() to make the codec usable again. You can call it from any state to move the codec back to the Uninitialized state. Otherwise, call release() to move to the terminal Released state.
極少數的情況下，codec會因為錯誤而進入Error狀態 。這種況情下，queue操作會返回一個非法值，或者有時候拋出異常。調用reset()方法重置codec。在任何狀態下都可以調用reset()方法使codec回到Uninitialized狀態。

Creation

Use MediaCodecList to create a MediaCodec for a specific MediaFormat. When decoding a file or a stream, you can get the desired format from MediaExtractor.getTrackFormat. Inject any specific features that you want to add using MediaFormat.setFeatureEnabled, then call MediaCodecList.findDecoderForFormat to get the name of a codec that can handle that specific media format. Finally, create the codec using createByCodecName(String).

創建

使用MediaCodecList來創建一個特定MediaFormat格式的MediaCodec。當解碼一個文件或者流的時候，你可以用MediaExtractor.getTrackFormat來獲取相應的格式。插入任何指定的特性，使用MediaFormat.setFeatureEnabled，然后調用MediaCodecList.findDecoderForFormat來獲取可以處理這種多媒體格式的codec。最后，使用createByCodecName(String)來創建codec.

Note: On LOLLIPOP, the format to MediaCodecList.findDecoder/EncoderForFormat must not contain a frame rate. Use format.setString(MediaFormat.KEY_FRAME_RATE, null) to clear any existing frame rate setting in the format.
注意：在LOLLIPOP, MediaCodecList.findDecoder/EncoderForFormat中的format不包含frame rate. 使用format.setString(MediaFormat.KEY_FRAME_RATE, null)來清空已經format存在的frame rate.

You can also create the preferred codec for a specific MIME type using createDecoder/EncoderByType(String). This, however, cannot be used to inject features, and may create a codec that cannot handle the specific desired media format.
你也可以使用createDecoder/EncoderByType(String)來創建針對特定MIME類型的codec。然后，這種codec不能用來插入特性，然后可能不能處理特定的多媒體格式。

Creating secure decoders

On versions KITKAT_WATCH and earlier, secure codecs might not be listed in MediaCodecList, but may still be available on the system. Secure codecs that exist can be instantiated by name only, by appending ".secure" to the name of a regular codec (the name of all secure codecs must end in ".secure".) createByCodecName(String) will throw an IOException if the codec is not present on the system.

創建一個加密的解碼器

在KITKAT_WATCH（API Level 20)和之前，加密的codecs可能不在MediaCodecList里，但是仍然是可用的。在正常的codec名字后面添加".secure", 加密的codec只可以通過名字來初始化。createByCodecName(String)會拋出異常，如果codec在系統中不存在。

From LOLLIPOP onwards, you should use the FEATURE_SecurePlayback feature in the media format to create a secure decoder.
LOLLIPOP之后，你應該用通過media format 的FEATURE_SecurePlayback來創建一個加密的解碼器。

Initialization

After creating the codec, you can set a callback using setCallback if you want to process data asynchronously. Then, configure the codec using the specific media format. This is when you can specify the output Surface for video producers – codecs that generate raw video data (e.g. video decoders). This is also when you can set the decryption parameters for secure codecs (see MediaCrypto). Finally, since some codecs can operate in multiple modes, you must specify whether you want it to work as a decoder or an encoder.

初始化

創建codec之后，如果你想異步處理數據，你可以設置一個callback。然后，使用特定的media format來配置codec。這時你可以為video生產者（生生原始視頻數據的codec, 例如：視頻解碼器）指定輸出Surface。在這時也可以為加密的codec設置解密參數(查看：MediaCrypto).最后，因為一些codec能在幾種模式下工作，你必須指定它是做為解碼器還是編碼器。

Since LOLLIPOP, you can query the resulting input and output format in the Configured state. You can use this to verify the resulting configuration, e.g. color formats, before starting the codec.
從LOLLIPOP開始，在Configured狀態，你可以查詢輸入輸出格式。你可以使用這個功能來驗證配置結果，例如：顏色格式，在還沒有starting codec之前。

If you want to process raw input video buffers natively with a video consumer – a codec that processes raw video input, such as a video encoder – create a destination Surface for your input data using createInputSurface() after configuration. Alternately, set up the codec to use a previously created persistent input surface by calling setInputSurface(Surface).
如果你想用一個codec（比如一個視頻編碼器）來處理原始的視頻輸入buffer，在配置之后，調用createInputSurface() 來為輸入數據創建一個目的地Surface。或者，調用setInputSurface(Surface)來設置一個之前已經創建好的輸入surface.

Codec-specific Data

Some formats, notably AAC audio and MPEG4, H.264 and H.265 video formats require the actual data to be prefixed by a number of buffers containing setup data, or codec specific data. When processing such compressed formats, this data must be submitted to the codec after start() and before any frame data. Such data must be marked using the flag BUFFER_FLAG_CODEC_CONFIG in a call to queueInputBuffer.

指定編碼的數據

一些格式，如AAC音頻和MPEG4, H.264, H.265視頻格式需要在實際數據之前添加一些含設置數據或者指定編碼的數據的buffers做為前綴。在處理這種壓縮格式時，這些數據必須在start()之后、在任何幀數據之前提交給codec。在queueInputBuffer時，這種數據必須用BUFFER_FLAG_CODEC_CONFIG標志。

Codec-specific data can also be included in the format passed to configure in ByteBuffer entries with keys "csd-0", "csd-1", etc. These keys are always included in the track MediaFormat obtained from the MediaExtractor. Codec-specific data in the format is automatically submitted to the codec upon start(); you MUST NOT submit this data explicitly. If the format did not contain codec specific data, you can choose to submit it using the specified number of buffers in the correct order, according to the format requirements. In case of H.264 AVC, you can also concatenate all codec-specific data and submit it as a single codec-config buffer.

Android uses the following codec-specific data buffers. These are also required to be set in the track format for proper MediaMuxer track configuration. Each parameter set and the codec-specific-data sections marked with (*) must start with a start code of "\x00\x00\x00\x01".

Screen Shot 2017-05-23 at 11.09.26 AM.png

Note: care must be taken if the codec is flushed immediately or shortly after start, before any output buffer or output format change has been returned, as the codec specific data may be lost during the flush. You must resubmit the data using buffers marked with BUFFER_FLAG_CODEC_CONFIG after such flush to ensure proper codec operation.
注意：如果codec馬上或者start之后就flushed，在任何輸出buffer或者輸出格式變化被返回之前，指定編碼的數據可能在flush的過程中丟失。在flush之后，你必須重新用BUFFER_FLAG_CODEC_CONFIG標記的buffers提交這些數據來確保正常的codec操作。

Encoders (or codecs that generate compressed data) will create and return the codec specific data before any valid output buffer in output buffers marked with the codec-config flag. Buffers containing codec-specific-data have no meaningful timestamps.

Data Processing

Each codec maintains a set of input and output buffers that are referred to by a buffer-ID in API calls. After a successful call to start() the client "owns" neither input nor output buffers. In synchronous mode, call dequeueInput/OutputBuffer(…) to obtain (get ownership of) an input or output buffer from the codec. In asynchronous mode, you will automatically receive available buffers via the MediaCodec.Callback.onInput/OutputBufferAvailable(…) callbacks.

數據處理

每個codec維護著一批輸入和輸出buffers，在API請求中， 這些buffer可以通過buffer-ID來指向。在成功調用start()方法后，客戶端既沒有‘擁有’輸出也沒有‘擁有’輸入buffers。在同步模式中，調用dequeueInput/OutputBuffer(…)從codec中獲取輸入或者輸出buffer。在異步模式中，通過MediaCodec.Callback.onInput/OutputBufferAvailable(…) 回調，你會自動收到可用的buffers。

Upon obtaining an input buffer, fill it with data and submit it to the codec using queueInputBuffer – or queueSecureInputBuffer if using decryption. Do not submit multiple input buffers with the same timestamp (unless it is codec-specific data marked as such).
收到輸入buffer時，填入數據后用queueInputBuffer交給codec， 或者，如果是加密的，用queueSecureInputBuffer。不要同時提交多個有同要的時間戳的輸入buffers（除非是指定codec的data).

The codec in turn will return a read-only output buffer via the onOutputBufferAvailable callback in asynchronous mode, or in response to a dequeuOutputBuffer call in synchronous mode. After the output buffer has been processed, call one of the releaseOutputBuffer methods to return the buffer to the codec.
在異步模式下，通過onOutputBufferAvailable 回調codec會返回一個只讀的輸出buffer，或者在同步模式下，響應dequeuOutputBuffer。輸出buffer被處理后，調用releaseOutputBuffer來釋放buffer給codec。

While you are not required to resubmit/release buffers immediately to the codec, holding onto input and/or output buffers may stall the codec, and this behavior is device dependent. Specifically, it is possible that a codec may hold off on generating output buffers until all outstanding buffers have been released/resubmitted. Therefore, try to hold onto to available buffers as little as possible.
當你沒有被要求馬上提交或者釋放buffers給codec， 持有輸入或者輸出buffers可能使codec停止工作，最終結果可能與設備有關。特別地，codec可能取消生成輸出buffers直到所有未完成的buffers被釋放/提交。所以，盡可能地少持有buffers.

Depending on the API version, you can process data in three ways:
根據不同的API版本，你可以用以下三種方式來處理數據。

Screen Shot 2017-05-23 at 11.11.21 AM.png

Asynchronous Processing using Buffers

Since LOLLIPOP, the preferred method is to process data asynchronously by setting a callback before calling configure. Asynchronous mode changes the state transitions slightly, because you must call start() after flush() to transition the codec to the Running sub-state and start receiving input buffers. Similarly, upon an initial call to start the codec will move directly to the Running sub-state and start passing available input buffers via the callback.

異步處理中的buffers使用

從LOLLIPOP開始，較好的方式是在配置之前設置回調，異步處理數據。異步模式稍微改變了狀態的變換，因為你必須在flush()之后調用start()，使codec處于Running 狀態，接收輸入buffers.同樣， codec初始化開始之后會直接進入Running狀態， 并通過回調傳遞可用的輸入buffers.

Screen Shot 2017-05-23 at 11.11.59 AM.png

MediaCodec is typically used like this in asynchronous mode:
異步模式下，MediaCodec一般是這樣使用的:

MediaCodec codec = MediaCodec.createByCodecName(name);
 MediaFormat mOutputFormat; // member variable
 codec.setCallback(new MediaCodec.Callback() {
   @Override
   void onInputBufferAvailable(MediaCodec mc, int inputBufferId) {
     ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferId);
     // fill inputBuffer with valid data
     …
     codec.queueInputBuffer(inputBufferId, …);
   }

   @Override
   void onOutputBufferAvailable(MediaCodec mc, int outputBufferId, …) {
     ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
     MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
     // bufferFormat is equivalent to mOutputFormat
     // outputBuffer is ready to be processed or rendered.
     …
     codec.releaseOutputBuffer(outputBufferId, …);
   }

   @Override
   void onOutputFormatChanged(MediaCodec mc, MediaFormat format) {
     // Subsequent data will conform to new format.
     // Can ignore if using getOutputFormat(outputBufferId)
     mOutputFormat = format; // option B
   }

   @Override
   void onError(…) {
     …
   }
 });
 codec.configure(format, …);
 mOutputFormat = codec.getOutputFormat(); // option B
 codec.start();
 // wait for processing to complete
 codec.stop();
 codec.release();

Synchronous Processing using Buffers

Since LOLLIPOP, you should retrieve input and output buffers using getInput/OutputBuffer(int) and/or getInput/OutputImage(int) even when using the codec in synchronous mode. This allows certain optimizations by the framework, e.g. when processing dynamic content. This optimization is disabled if you call getInput/OutputBuffers().

同步處理Buffers的使用

從LOLLIPOP開始，你應該使用getInput/OutputBuffer(int)和getInput/OutputImage(int)來獲取輸入和輸出buffers，即使是在同步模式下使用codec.這樣做，framework會有一些優化，比如在處理動態內容。你調用getInput/OutputBuffers()時就不會有優化。

Note: do not mix the methods of using buffers and buffer arrays at the same time. Specifically, only call getInput/OutputBuffers directly after start() or after having dequeued an output buffer ID with the value of INFO_OUTPUT_FORMAT_CHANGED.

注意：不要在同時使用buffers和buffer arrays的方法。特別是在start()之后馬上調用getInput/OutputBuffers，或者是dequeued一個ID為INFO_OUTPUT_FORMAT_CHANGED的輸出buffer.

MediaCodec is typically used like this in synchronous mode:
同步模式下，MediaCodec一般是這樣使用的:

MediaCodec codec = MediaCodec.createByCodecName(name);
 codec.configure(format, …);
 MediaFormat outputFormat = codec.getOutputFormat(); // option B
 codec.start();
 for (;;) {
   int inputBufferId = codec.dequeueInputBuffer(timeoutUs);
   if (inputBufferId >= 0) {
     ByteBuffer inputBuffer = codec.getInputBuffer(…);
     // fill inputBuffer with valid data
     …
     codec.queueInputBuffer(inputBufferId, …);
   }
   int outputBufferId = codec.dequeueOutputBuffer(…);
   if (outputBufferId >= 0) {
     ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
     MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
     // bufferFormat is identical to outputFormat
     // outputBuffer is ready to be processed or rendered.
     …
     codec.releaseOutputBuffer(outputBufferId, …);
   } else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
     // Subsequent data will conform to new format.
     // Can ignore if using getOutputFormat(outputBufferId)
     outputFormat = codec.getOutputFormat(); // option B
   }
 }
 codec.stop();
 codec.release();

Synchronous Processing using Buffer Arrays (deprecated)

In versions KITKAT_WATCH and before, the set of input and output buffers are represented by the ByteBuffer[] arrays. After a successful call to start(), retrieve the buffer arrays using getInput/OutputBuffers(). Use the buffer ID-s as indices into these arrays (when non-negative), as demonstrated in the sample below. Note that there is no inherent correlation between the size of the arrays and the number of input and output buffers used by the system, although the array size provides an upper bound.

同步模式下Buffer Arrays的使用（deprecated)

在KITKAT_WATCH（API Level 20)和之前，輸入和輸出buffers都是用ByteBuffer[] 表示的。成功調用start()之后，通過getInput/OutputBuffers()來獲取Buffer數組。像下面的例子，采用buffer IDs來做為數組索引。注意，數組長度和系統中使用的輸入和輸入buffers數量并沒有內在關聯，雖然數組有一個上限。

 MediaCodec codec = MediaCodec.createByCodecName(name);
 codec.configure(format, …);
 codec.start();
 ByteBuffer[] inputBuffers = codec.getInputBuffers();
 ByteBuffer[] outputBuffers = codec.getOutputBuffers();
 for (;;) {
   int inputBufferId = codec.dequeueInputBuffer(…);
   if (inputBufferId >= 0) {
     // fill inputBuffers[inputBufferId] with valid data
     …
     codec.queueInputBuffer(inputBufferId, …);
   }
   int outputBufferId = codec.dequeueOutputBuffer(…);
   if (outputBufferId >= 0) {
     // outputBuffers[outputBufferId] is ready to be processed or rendered.
     …
     codec.releaseOutputBuffer(outputBufferId, …);
   } else if (outputBufferId == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
     outputBuffers = codec.getOutputBuffers();
   } else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
     // Subsequent data will conform to new format.
     MediaFormat format = codec.getOutputFormat();
   }
 }
 codec.stop();
 codec.release();

End-of-stream Handling

When you reach the end of the input data, you must signal it to the codec by specifying the BUFFER_FLAG_END_OF_STREAM flag in the call to queueInputBuffer. You can do this on the last valid input buffer, or by submitting an additional empty input buffer with the end-of-stream flag set. If using an empty buffer, the timestamp will be ignored.

結束stream

當輸入數據結束時，你必須通知codec, 在調用queueInputBuffer時標志BUFFER_FLAG_END_OF_STREAM.你可以在最后一個輸入buffer進行標志，或者增加一個設置了end-of-stream標志的額外的空buffer。如果使用空buffer，時間戳會被忽略。

The codec will continue to return output buffers until it eventually signals the end of the output stream by specifying the same end-of-stream flag in the MediaCodec.BufferInfo set in dequeueOutputBuffer or returned via onOutputBufferAvailable. This can be set on the last valid output buffer, or on an empty buffer after the last valid output buffer. The timestamp of such empty buffer should be ignored.
Codec會持續返回輸入buffers直到最后在dequeueOutputBuffer 或者onOutputBufferAvailable 返回的MediaCodec.BufferInfo中標記同樣的end-of-stream來說明輸出流結束。可能在最后一個輸出buffer中標記或者在最后一個buffer后面添加一個空的buffer進記標記。空的buffer上面的時間戳會被忽略。

Do not submit additional input buffers after signaling the end of the input stream, unless the codec has been flushed, or stopped and restarted.
標記輸入流結束后，不要再繼續添加輸入buffers，除非codec已經被flushed，停止或者重啟了。

Using an Output Surface

The data processing is nearly identical to the ByteBuffer mode when using an output Surface; however, the output buffers will not be accessible, and are represented as null values. E.g. getOutputBuffer/Image(int) will return null and getOutputBuffers() will return an array containing only null-s.

使用一個輸出Surface

使用一個輸出Surface的數據處理與ByteBuffer模式下的數據處理幾乎是一樣的；但是，在Surface模式下，不能獲取輸出buffers（值都為null).例如：getOutputBuffer/Image(int) 會返回null, getOutputBuffers()會返回一個只包含null的數組。

When using an output Surface, you can select whether or not to render each output buffer on the surface. You have three choices:
當使用輸出Surface時，你可以選擇是否在surface上繪制每個輸出buffer.你有三個選擇：

• Do not render the buffer: Call releaseOutputBuffer(bufferId, false).

• 不繪制buffer:調用releaseOutputBuffer(bufferId, false).

• Render the buffer with the default timestamp: Call releaseOutputBuffer(bufferId, true).

• 繪制有缺省時間戳的buffer:調用releaseOutputBuffer(bufferId, true)

• Render the buffer with a specific timestamp: Call releaseOutputBuffer(bufferId, timestamp).
• 繪制指定時間戳的buffer: 調用releaseOutputBuffer(bufferId, timestamp).

Since M, the default timestamp is the presentation timestamp of the buffer (converted to nanoseconds). It was not defined prior to that.
從M(API Level23)開始，缺省時間戳是presentation時間戳（轉換成nanoseconds）。在M之前是沒有定義的。

Also since M, you can change the output Surface dynamically using setOutputSurface.
同樣，從M開始，你可以使用setOutputSurface動態地改變輸出Surface.

Transformations When Rendering onto Surface

If the codec is configured into Surface mode, any crop rectangle, rotation and video scaling mode will be automatically applied with one exception:

在Surface繪制時進行轉換

如果Codec設置為Surface模式，任何矩形的剪裁，旋轉和video縮放模式會自動地進行，但有一個例外：

Prior to the M release, software decoders may not have applied the rotation when being rendered onto a Surface. Unfortunately, there is no standard and simple way to identify software decoders, or if they apply the rotation other than by trying it out.
在M之前，在Surface繪制時，軟解碼可能不會進行旋轉變換。不幸的是，沒有標準而簡單的方法來對軟解碼進行判斷是否進行了旋轉變換。

There are also some caveats.
下面是一些警告：

Note that the pixel aspect ratio is not considered when displaying the output onto the Surface. This means that if you are using VIDEO_SCALING_MODE_SCALE_TO_FIT mode, you must position the output Surface so that it has the proper final display aspect ratio. Conversely, you can only use VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING mode for content with square pixels (pixel aspect ratio or 1:1).
注意，當在Surface上繪制輸出時，像素的寬高比是不被考慮的。這意味著，如果你正在使用VIDEO_SCALING_MODE_SCALE_TO_FIT模式，你必須確保輸入Surface最終有一個合適的寬高比。相反地，當顯示的內容是正方形的，你只能使用VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING模式。

Note also that as of N release, VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING mode may not work correctly for videos rotated by 90 or 270 degrees.
注意，當N(API Level 24)發布 ，videos旋轉了90或者270度時，VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING模式可能不能正常工作。

When setting the video scaling mode, note that it must be reset after each time the output buffers change. Since the INFO_OUTPUT_BUFFERS_CHANGED event is deprecated, you can do this after each time the output format changes.
每當輸出buffers改變時，必須重新設置video縮放模式。因為INFO_OUTPUT_BUFFERS_CHANGED事件已經deprecated，每次輸出格式改變時，你可以進行重置。

Using an Input Surface

When using an input Surface, there are no accessible input buffers, as buffers are automatically passed from the input surface to the codec. Calling dequeueInputBuffer will throw an IllegalStateException, and getInputBuffers() returns a bogus ByteBuffer[] array that MUST NOT be written into.

使用一個輸入Surface

當使用一個輸入Surface時，不能獲取輸入buffers，因為它們被自動從輸入Surface傳遞給了Codec。調用dequeueInputBuffer會拋出IllegalStateException，調用getInputBuffers()會返回一個假的ByteBuffer[]數組，而且這個數組不能進行賦值。

Call signalEndOfInputStream() to signal end-of-stream. The input surface will stop submitting data to the codec immediately after this call.
調用signalEndOfInputStream()來通知結束。調用這個方法之后，輸入Surface就馬上停止提交數據給Codec了。

Seeking & Adaptive Playback Support

Video decoders (and in general codecs that consume compressed video data) behave differently regarding seek and format change whether or not they support and are configured for adaptive playback. You can check if a decoder supports adaptive playback via CodecCapabilities.isFeatureSupported(String). Adaptive playback support for video decoders is only activated if you configure the codec to decode onto a Surface.

快進和適應播放的支持

視頻解碼器（或者一般的消耗壓縮視頻數據的codecs)在快進和格式變化上表現不一致，不論它們是否支持，并且被配置進行適應性播放。你可以通過CodecCapabilities.isFeatureSupported(String)來檢查一個解碼器是否支持適應性播放。只有你設置codec解碼到Surface上，視頻解碼器的適應性播放才會被激活。

Stream Boundary and Key Frames

It is important that the input data after start() or flush() starts at a suitable stream boundary: the first frame must a key frame. A key frame can be decoded completely on its own (for most codecs this means an I-frame), and no frames that are to be displayed after a key frame refer to frames before the key frame.

Stream邊界和關鍵幀

start()和flush()之后，輸入數據從一個合適的stream邊界開始是非常重要的:第一個幀必須是關鍵幀。一個關鍵幀可以被完全的解碼（對于大多數的codecs來說是一個I-frame), 而且關鍵幀之后顯示的幀不能指向關鍵幀之前的幀。

The following table summarizes suitable key frames for various video formats.
下表的列表概括了不同的視頻格式的合適的關鍵幀。

Screen Shot 2017-05-23 at 11.17.57 AM.png

For decoders that do not support adaptive playback (including when not decoding onto a Surface)

In order to start decoding data that is not adjacent to previously submitted data (i.e. after a seek) you MUST flush the decoder. Since all output buffers are immediately revoked at the point of the flush, you may want to first signal then wait for the end-of-stream before you call flush. It is important that the input data after a flush starts at a suitable stream boundary/key frame.

不支持適應性播放的（包括沒有解碼到Surface的）解碼器

為了解碼和之前提交的數據（比如，快進后）不相臨的數據，你必須flsuh解碼器。因為所有的輸出buffers在flush時立刻被廢除了，所以在你調用flush前，你可能需要先通知，然后等待end-of-stream 。flush之后，輸入數據從一個合適的stream邊界/關鍵幀開始是非常重要的。

Note: the format of the data submitted after a flush must not change; flush() does not support format discontinuities; for that, a full stop() - configure(…) - start() cycle is necessary.
注意：flush之后提交的數據不能更改格式；flush()不支持格式的不連續性；對于不連續的格式，需要stop()-configure(...)-start()。

Also note: if you flush the codec too soon after start() – generally, before the first output buffer or output format change is received – you will need to resubmit the codec-specific-data to the codec. See the codec-specific-data section for more info.
還要注意的是：當你在start()之后馬上進行flush--一般來說，在第一個輸出buffer或者輸出格式變換通知收到以前--你必須重新提交批定編碼的數據給codec。更多信息，請查看codec-specific-data這一節。

For decoders that support and are configured for adaptive playback

In order to start decoding data that is not adjacent to previously submitted data (i.e. after a seek) it is not necessary to flush the decoder; however, input data after the discontinuity must start at a suitable stream boundary/key frame.

支持并配置適應性播放的解碼器

開始解碼一個和上一次提交的數據不相鄰的數據（比如，快進），沒有必要flush解碼器。然而，非連續性的輸入數據必須從一個合適的stream邊界/關鍵幀開始。

For some video formats - namely H.264, H.265, VP8 and VP9 - it is also possible to change the picture size or configuration mid-stream. To do this you must package the entire new codec-specific configuration data together with the key frame into a single buffer (including any start codes), and submit it as a regular input buffer.
對于一些視頻格式，比如：H.264, H.265, VP8 and VP9，是可能改變圖片大小或者mid-stream的結構。要實現這一步，你必須對整個新的特定編碼的配置數據和關鍵幀進行打包成一個新buffer(包括任何start codes),然作為一個普通的輸入buffer進行提交。

You will receive an INFO_OUTPUT_FORMAT_CHANGED return value from dequeueOutputBuffer or a onOutputFormatChanged callback just after the picture-size change takes place and before any frames with the new size have been returned.
在圖片大小改變后，并且在任何新的大小的幀被返回之前，dequeueOutputBuffer或者onOutputFormatChanged回調馬上就會收到一個INFO_OUTPUT_FORMAT_CHANGED返回值。

Note: just as the case for codec-specific data, be careful when calling flush() shortly after you have changed the picture size. If you have not received confirmation of the picture size change, you will need to repeat the request for the new picture size.
注意：對于指定codec的數據，改變圖片大小后馬上調用flush()要非常小心。如果沒有收到圖片大小改變的確認，你需要重新提交新的圖片大小請求。

Error handling

The factory methods createByCodecName and createDecoder/EncoderByType throw IOException on failure which you must catch or declare to pass up. MediaCodec methods throw IllegalStateException when the method is called from a codec state that does not allow it; this is typically due to incorrect application API usage. Methods involving secure buffers may throw MediaCodec.CryptoException, which has further error information obtainable from getErrorCode().

應對錯誤的情況

工廠方法createByCodecName和createDecoder/EncoderByType 在遇錯的時候會拋出IOException，你必須進行catch或者聲明異常。當codec在一個不允許的狀態被調用時會拋出IllegalStateException；這是典型的API錯誤引起的。與加密的buffers有關的方法可能會拋出MediaCodec.CryptoException，使用getErrorCode()方法可以獲取進一步的錯誤信息。

Internal codec errors result in a MediaCodec.CodecException, which may be due to media content corruption, hardware failure, resource exhaustion, and so forth, even when the application is correctly using the API. The recommended action when receiving a CodecException can be determined by calling isRecoverable() and isTransient():
內部的codec錯誤會拋出MediaCodec.CodecException，可能是因多媒體文件損壞，硬件錯誤，沒有資源等，即使是正確地使用了API也是如此。遇到CodecException時，調用isRecoverable()和isTransient()可以決定合適的方法。

• recoverable errors: If isRecoverable() returns true, then call stop(), configure(…), and start() to recover.

• 能恢復的錯誤: 如果isRecoverable() 返回true, 那么調用stop(),configure(…), 和start() 來恢復。

• transient errors: If isTransient() returns true, then resources are temporarily unavailable and the method may be retried at a later time.

• 短暫的錯誤:如果isTransient()返回true,那么暫時資源不可用，可以晚點再試這個方法。

• fatal errors: If both isRecoverable() and isTransient() return false, then the CodecException is fatal and the codec must be reset or released.

• 致命的錯誤:如果isRecoverable()和isTransient()都返回false，那么CodecException是致命的，codec必須被reset或者released.

Both isRecoverable() and isTransient() do not return true at the same time.
isRecoverable()和isTransient()不可能同時返回true;

Valid API Calls and API History

This sections summarizes the valid API calls in each state and the API history of the MediaCodec class. For API version numbers, see Build.VERSION_CODES.

正確的API請求和API歷史記錄

這小節概括了在每個狀態下正確的API請求，以及MediaCodec類的API歷史。查看API版本號，請看Build.VERSION_CODES。

Screen Shot 2017-05-25 at 11.14.34 AM.png