media/v4l/dev-decoder.rst

1 .. SPDX-License-Identifier: GPL-2.0
6 Memory-to-Memory Stateful Video Decoder Interface
9 A stateful video decoder takes complete chunks of the bytestream (e.g. Annex-B
34 5. Single-planar API (see :ref:`planar-apis`) and applicable structures may be
35    used interchangeably with multi-planar API, unless specified otherwise,
36    depending on decoder capabilities and following the general V4L2 guidelines.
41 7. Given an ``OUTPUT`` buffer A, then A’ represents a buffer on the ``CAPTURE``
44 .. _decoder-glossary:
97    Instantaneous Decoder Refresh; a type of a keyframe in an H.264/HEVC-encoded
102    can be decoded fully on its own.
105    a processing unit in image and video compression formats based on linear
106    block transforms (e.g. H.264, VP8, VP9); codec-specific, but for most of
122 resume point
125    SPS/PPS/IDR sequence (H.264/HEVC); a resume point is required to start decode
126    of a new stream, or to resume decoding after a seek.
146    additional (non-visual) information contained inside encoded bytestream;
163 .. kernel-render:: DOT
179        qi -> Initialization [ label = "open()" ];
181        Initialization -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ];
183        CaptureSetup -> Stopped [ label = "CAPTURE\nbuffers\nready" ];
185        Decoding -> ResChange [ label = "Stream\nresolution\nchange" ];
186        Decoding -> Drain [ label = "V4L2_DEC_CMD_STOP" ];
187        Decoding -> EoS [ label = "EoS mark\nin the stream" ];
188        Decoding -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
189        Decoding -> Stopped [ label = "VIDIOC_STREAMOFF(CAPTURE)" ];
190        Decoding -> Decoding;
192        ResChange -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ];
193        ResChange -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
195        EoS -> Drain [ label = "Implicit\ndrain" ];
197        Drain -> Stopped [ label = "All CAPTURE\nbuffers dequeued\nor\nVIDIOC_STREAMOFF(CAPTURE)" ];
198        Drain -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
200        Seek -> Decoding [ label = "VIDIOC_STREAMON(OUTPUT)" ];
201        Seek -> Initialization [ label = "VIDIOC_REQBUFS(OUTPUT, 0)" ];
203        Stopped -> Decoding [ label = "V4L2_DEC_CMD_START\nor\nVIDIOC_STREAMON(CAPTURE)" ];
204        Stopped -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
211    client may call :c:func:`VIDIOC_ENUM_FMT` on ``OUTPUT``.
214      format set on ``CAPTURE``.
217      In particular whether or not the decoder has a full-fledged bytestream
221    :c:func:`VIDIOC_ENUM_FMT` on ``CAPTURE``.
223    * Only the formats supported for the format currently active on ``OUTPUT``
227      the client must first set that coded format on ``OUTPUT`` and then
228      enumerate formats on ``CAPTURE``.
240      decoder for given raw pixel format and the coded format currently set on
243 4. Supported profiles and levels for the coded format currently set on
250 1. Set the coded format on ``OUTPUT`` via :c:func:`VIDIOC_S_FMT`.
279      resolution instantly based on the width and height returned by
291       and the client must ensure it matches its needs afterwards.
293 2.  Allocate source (bytestream) buffers via :c:func:`VIDIOC_REQBUFS` on
318     Alternatively, :c:func:`VIDIOC_CREATE_BUFS` on the ``OUTPUT`` queue can be
346 3.  Start streaming on the ``OUTPUT`` queue via :c:func:`VIDIOC_STREAMON`.
364     * If the client has not set the coded resolution of the stream on its own,
366       :c:func:`VIDIOC_TRY_FMT` or :c:func:`VIDIOC_REQBUFS` on the ``CAPTURE``
379        A client capable of acquiring stream parameters from the bytestream on
381        to non-zero values matching the coded size of the stream, skip this step
383        rely on any driver queries regarding stream parameters, such as
398 1.  Call :c:func:`VIDIOC_G_FMT` on the ``CAPTURE`` queue to get format for the
443           returned by :c:func:`VIDIOC_G_FMT` on ``CAPTURE``.
445     * The following selection targets are supported on ``CAPTURE``:
457           ``CAPTURE``; defaults to ``V4L2_SEL_TGT_CROP_DEFAULT``; read-only on
471           read-only on hardware without additional compose/scaling capabilities.
481        successfully parses the stream metadata. The client must not rely on the
484 3.  **Optional.** Enumerate ``CAPTURE`` formats via :c:func:`VIDIOC_ENUM_FMT` on
510 4.  **Optional.** Set the ``CAPTURE`` format via :c:func:`VIDIOC_S_FMT` on the
527    * Setting the ``CAPTURE`` format will reset the compose selection rectangles
528      to their default values, based on the new resolution, as described in the
531 5. **Optional.** Set the compose rectangle via :c:func:`VIDIOC_S_SELECTION` on
546          read-only on hardware without additional compose/scaling capabilities.
552          returned by :c:func:`VIDIOC_G_FMT` on ``CAPTURE``.
557       supported one to meet codec and hardware requirements. The client needs
560 6.  If all the following conditions are met, the client may resume the decoding
571     In that case, the remaining steps do not apply and the client may resume
578       on the ``CAPTURE`` queue.
585     buffers on the ``CAPTURE`` queue until a buffer marked with the
589     on the ``CAPTURE`` queue to stop streaming.
594        :c:func:`VIDIOC_STREAMOFF` on it would abort the sequence and trigger a
611 10. Allocate ``CAPTURE`` buffers via :c:func:`VIDIOC_REQBUFS` on the
644     Alternatively, :c:func:`VIDIOC_CREATE_BUFS` on the ``CAPTURE`` queue can be
690 11. Call :c:func:`VIDIOC_STREAMON` on the ``CAPTURE`` queue to start decoding
701 The content of the source ``OUTPUT`` buffers depends on the active coded pixel
702 format and may be affected by codec-specific extended controls, as stated in
706 buffer queues and memory-to-memory devices. In addition, the order of decoded
716   on ``CAPTURE`` (e.g. if it does not contain encoded data, or if only
720   on ``CAPTURE`` (if the encoded data contained more than one frame, or if
724 * a buffer queued to ``OUTPUT`` may result in a buffer being produced on
729 * buffers may become available on the ``CAPTURE`` queue without additional
752      buffers are out-of-order compared to the ``OUTPUT`` buffers): ``CAPTURE``
764   Change` sequence needs to be followed,
766 * if a ``V4L2_EVENT_EOS`` event is pending, the `End of Stream` sequence needs
773 of details depending on the decoder capabilities. Specifically:
783 further operations on corresponding decoder file handle will return the -EIO
785 alternatively reinitialize the instance by stopping streaming on both queues,
792 The seek does not require any specific operation on the ``CAPTURE`` queue, but
817    queue until a suitable resume point is found.
822       from a resume point (e.g. SPS or a keyframe). Any queued ``OUTPUT``
824       resume point is found.  While looking for a resume point, the decoder
827       Some hardware is known to mishandle seeks to a non-resume point. Such an
829       being made available on the ``CAPTURE`` queue. Drivers must ensure that
847 4. After a resume point is found, the decoder will start returning ``CAPTURE``
870    produced.  For example, given the sequence of operations on the
875    any of the following results on the ``CAPTURE`` queue is allowed:
886    To achieve instantaneous seek, the client may restart streaming on the
911 * bit-depth of the bitstream has been changed.
940        with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from
953        :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the
962    streaming. Calling :c:func:`VIDIOC_STREAMOFF` on the ``OUTPUT`` queue would
1023         marked with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from
1035       dequeued. It is a deprecated behavior and the client must not rely on it.
1043    * ``V4L2_DEC_CMD_START`` - the decoder will not be reset and will resume
1046    * a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the
1047      ``CAPTURE`` queue - the decoder will resume the operation normally,
1051    * a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the
1052      ``OUTPUT`` queue - any pending source buffers will be returned to the
1057    Once the drain sequence is initiated, the client needs to drive it to
1059    issuing :c:func:`VIDIOC_STREAMOFF` on any of the ``OUTPUT`` or ``CAPTURE``
1062    will fail with -EBUSY error code if attempted.
1080 1. Setting the format on the ``OUTPUT`` queue may change the set of formats
1081    supported/advertised on the ``CAPTURE`` queue. In particular, it also means
1082    that the ``CAPTURE`` format may be reset and the client must not rely on the
1085 2. Enumerating formats on the ``CAPTURE`` queue always returns only formats
1088 3. Setting the format on the ``CAPTURE`` queue does not change the list of
1089    formats available on the ``OUTPUT`` queue. An attempt to set a ``CAPTURE``
1094 4. Enumerating formats on the ``OUTPUT`` queue always returns the full set of
1097 5. While buffers are allocated on any of the ``OUTPUT`` or ``CAPTURE`` queues,
1098    the client must not change the format on the ``OUTPUT`` queue. Drivers will
1099    return the -EBUSY error code for any such format change attempt.