include/linux/zstd.h

2  * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
5  * This source code is licensed under the BSD-style license found in the
12  * Free Software Foundation. This program is dual-licensed; you may select
24 /*-*****************************************************************************
28  * targeting real-time compression scenarios at zlib-level and better
29  * compression ratios. The zstd compression library provides in-memory
34  *  - a single step, reusing a context (described as Explicit memory management)
35  *  - unbounded multiple steps (described as Streaming compression)
38  *  - a single step (described as Simple dictionary API)
39  *  - a single step, reusing a dictionary (described as Fast dictionary API)
45  * enum ZSTD_ErrorCode - zstd error codes
77  * ZSTD_maxCLevel() - maximum compression level available
83  * ZSTD_compressBound() - maximum compressed size in worst case scenario
90  * ZSTD_isError() - tells if a size_t function result is an error code
93  * Return: Non-zero iff the code is an error.
97 	return code > (size_t)-ZSTD_error_maxCode;  in ZSTD_isError()
100  * ZSTD_getErrorCode() - translates an error function result to a ZSTD_ErrorCode
111 	return (ZSTD_ErrorCode)(0 - functionResult);  in ZSTD_getErrorCode()
115  * enum ZSTD_strategy - zstd compression search strategy
131  * struct ZSTD_compressionParameters - zstd compression parameters
156  * struct ZSTD_frameParameters - zstd frame parameters
157  * @contentSizeFlag: Controls whether content size will be present in the frame
159  * @checksumFlag:    Controls whether a 32-bit checksum is generated at the end
160  *                   of the frame for error detection.
161  * @noDictIDFlag:    Controls whether dictID will be saved into the frame header
173  * struct ZSTD_parameters - zstd parameters
175  * @fParams: The frame parameters.
183  * ZSTD_getCParams() - returns ZSTD_compressionParameters for selected level
194  * ZSTD_getParams() - returns ZSTD_parameters for selected level
199  * The same as ZSTD_getCParams() except also selects the default frame
207 /*-*************************************
212  * ZSTD_CCtxWorkspaceBound() - amount of memory needed to initialize a ZSTD_CCtx
225  * struct ZSTD_CCtx - the zstd compression context
232  * ZSTD_initCCtx() - initialize a zstd compression context
243  * ZSTD_compressCCtx() - compress src into dst
246  * @dst:         The buffer to compress src into.
247  * @dstCapacity: The size of the destination buffer. May be any size, but
260  * ZSTD_DCtxWorkspaceBound() - amount of memory needed to initialize a ZSTD_DCtx
268  * struct ZSTD_DCtx - the zstd decompression context
275  * ZSTD_initDCtx() - initialize a zstd decompression context
286  * ZSTD_decompressDCtx() - decompress zstd compressed src into dst
288  * @dst:         The buffer to decompress src into.
289  * @dstCapacity: The size of the destination buffer. Must be at least as large
302 /*-************************
307  * ZSTD_compress_usingDict() - compress src into dst using a dictionary
310  * @dst:         The buffer to compress src into.
311  * @dstCapacity: The size of the destination buffer. May be any size, but
330  * ZSTD_decompress_usingDict() - decompress src into dst using a dictionary
332  * @dst:         The buffer to decompress src into.
333  * @dstCapacity: The size of the destination buffer. Must be at least as large
349 /*-**************************
354  * ZSTD_CDictWorkspaceBound() - memory needed to initialize a ZSTD_CDict
363  * struct ZSTD_CDict - a digested dictionary to be used for compression
368  * ZSTD_initCDict() - initialize a digested dictionary for compression
369  * @dictBuffer:    The dictionary to digest. The buffer is referenced by the
387  * ZSTD_compress_usingCDict() - compress src into dst using a ZSTD_CDict
392  * @dst:         The buffer to compress src into.
393  * @dstCapacity: The size of the destination buffer. May be any size, but
411  * ZSTD_DDictWorkspaceBound() - memory needed to initialize a ZSTD_DDict
419  * struct ZSTD_DDict - a digested dictionary to be used for decompression
424  * ZSTD_initDDict() - initialize a digested dictionary for decompression
425  * @dictBuffer:    The dictionary to digest. The buffer is referenced by the
442  * ZSTD_decompress_usingDDict() - decompress src into dst using a ZSTD_DDict
444  * @dst:         The buffer to decompress src into.
445  * @dstCapacity: The size of the destination buffer. Must be at least as large
462 /*-**************************
467  * struct ZSTD_inBuffer - input buffer for streaming
468  * @src:  Start of the input buffer.
469  * @size: Size of the input buffer.
480  * struct ZSTD_outBuffer - output buffer for streaming
481  * @dst:  Start of the output buffer.
482  * @size: Size of the output buffer.
494 /*-*****************************************************************************
495  * Streaming compression - HowTo
500  * operations. It is recommended to re-use ZSTD_CStream in situations where many
512  * buffer, using ZSTD_flushStream(). `output->pos` will be updated. There might
513  * still be some content left within the internal buffer if `output->size` is
514  * too small. It returns the number of bytes left in the internal buffer and
517  * ZSTD_endStream() instructs to finish a frame. It will perform a flush and
518  * write frame epilogue. The epilogue is required for decoders to consider a
519  * frame completed. Similar to ZSTD_flushStream(), it may not be able to flush
520  * the full content if `output->size` is too small. In which case, call again
522  * in the internal buffer and must be called until it returns 0.
526  * ZSTD_CStreamWorkspaceBound() - memory needed to initialize a ZSTD_CStream
535  * struct ZSTD_CStream - the zstd streaming compression context
541  * ZSTD_initCStream() - initialize a zstd streaming compression context
560  * ZSTD_initCStream_usingCDict() - initialize a streaming compression context
577  * ZSTD_resetCStream() - reset the context using parameters from creation
582  * loading, since it can be reused. If `pledgedSrcSize` is non-zero the frame
583  * content size is always written into the frame header.
589  * ZSTD_compressStream() - streaming compress some of input into output
591  * @output: Destination buffer. `output->pos` is updated to indicate how much
593  * @input:  Source buffer. `input->pos` is updated to indicate how much data was
595  *          `input->pos < input->size`, and it's up to the caller to present
608  * ZSTD_flushStream() - flush internal buffers into output
610  * @output: Destination buffer. `output->pos` is updated to indicate how much
622  * ZSTD_endStream() - flush internal buffers into output and end the frame
624  * @output: Destination buffer. `output->pos` is updated to indicate how much
628  * been flushed and the frame epilogue has been written.
636  * ZSTD_CStreamInSize() - recommended size for the input buffer
638  * Return: The recommended size for the input buffer.
642  * ZSTD_CStreamOutSize() - recommended size for the output buffer
644  * When the output buffer is at least this large, it is guaranteed to be large
647  * Return: The recommended size for the output buffer.
653 /*-*****************************************************************************
654  * Streaming decompression - HowTo
658  * ZSTD_DStream objects can be re-used multiple times.
662  * If `input->pos < input->size`, some input has not been consumed.
664  * If `output->pos < output->size`, decoder has flushed everything it could.
665  * Returns 0 iff a frame is completely decoded and fully flushed.
667  * than the current frame.
671  * ZSTD_DStreamWorkspaceBound() - memory needed to initialize a ZSTD_DStream
680  * struct ZSTD_DStream - the zstd streaming decompression context
685  * ZSTD_initDStream() - initialize a zstd streaming decompression context
698  * ZSTD_initDStream_usingDDict() - initialize streaming decompression context
714  * ZSTD_resetDStream() - reset the context using parameters from creation
724  * ZSTD_decompressStream() - streaming decompress some of input into output
726  * @output: Destination buffer. `output.pos` is updated to indicate how much
728  * @input:  Source buffer. `input.pos` is updated to indicate how much data was
735  * ZSTD_decompressStream() will not consume the last byte of the frame until
736  * the entire frame is flushed.
738  * Return:  Returns 0 iff a frame is completely decoded and fully flushed.
741  *          ZSTD_isError(). The size hint will never load more than the frame.
747  * ZSTD_DStreamInSize() - recommended size for the input buffer
749  * Return: The recommended size for the input buffer.
753  * ZSTD_DStreamOutSize() - recommended size for the output buffer
755  * When the output buffer is at least this large, it is guaranteed to be large
758  * Return: The recommended size for the output buffer.
763 /* --- Constants ---*/
767 #define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
768 #define ZSTD_CONTENTSIZE_ERROR   (0ULL - 2)
782 #define ZSTD_SEARCHLOG_MAX    (ZSTD_WINDOWLOG_MAX-1)
797 /* magic number + skippable frame length */
801 /*-*************************************
806  * ZSTD_findFrameCompressedSize() - returns the size of a compressed frame
807  * @src:     Source buffer. It should point to the start of a zstd encoded frame
808  *           or a skippable frame.
809  * @srcSize: The size of the source buffer. It must be at least as large as the
810  *           size of the frame.
812  * Return:   The compressed size of the frame pointed to by `src` or an error,
818 /*-*************************************
822  * ZSTD_getFrameContentSize() - returns the content size in a zstd frame header
823  * @src:     It should point to the start of a zstd encoded frame.
824  * @srcSize: The size of the source buffer. It must be at least as large as the
825  *           frame header. `ZSTD_frameHeaderSize_max` is always large enough.
827  * Return:   The frame content size stored in the frame header if known.
829  *           frame header. `ZSTD_CONTENTSIZE_ERROR` on invalid input.
834  * ZSTD_findDecompressedSize() - returns decompressed size of a series of frames
839  * If any zstd encoded frame in the series doesn't have the frame content size
840  * set, `ZSTD_CONTENTSIZE_UNKNOWN` is returned. But frame content size is always
845  * frames, and so it must traverse the input to read each frame header. This is
846  * efficient as most of the data is skipped, however it does mean that all frame
855 /*-*************************************
859  * ZSTD_checkCParams() - ensure parameter values remain within authorized range
867  * ZSTD_adjustCParams() - optimize parameters for a given srcSize and dictSize
877 /*--- Advanced decompression functions ---*/
880  * ZSTD_isFrame() - returns true iff the buffer starts with a valid frame
881  * @buffer: The source buffer to check.
882  * @size:   The size of the source buffer, must be at least 4 bytes.
884  * Return: True iff the buffer starts with a zstd or skippable frame identifier.
886 unsigned int ZSTD_isFrame(const void *buffer, size_t size);
889  * ZSTD_getDictID_fromDict() - returns the dictionary id stored in a dictionary
890  * @dict:     The dictionary buffer.
891  * @dictSize: The size of the dictionary buffer.
895  *            dictionary can still be loaded as a content-only dictionary.
900  * ZSTD_getDictID_fromDDict() - returns the dictionary id stored in a ZSTD_DDict
905  *         content-only dictionary.
910  * ZSTD_getDictID_fromFrame() - returns the dictionary id stored in a zstd frame
911  * @src:     Source buffer. It must be a zstd encoded frame.
912  * @srcSize: The size of the source buffer. It must be at least as large as the
913  *           frame header. `ZSTD_frameHeaderSize_max` is always large enough.
915  * Return:   The dictionary id required to decompress the frame stored within
917  *           0 if the frame does not require a dictionary, the dictionary id
918  *           wasn't stored in the frame, `src` is not a zstd frame, or `srcSize`
924  * struct ZSTD_frameParams - zstd frame parameters stored in the frame header
925  * @frameContentSize: The frame content size, or 0 if not present.
926  * @windowSize:       The window size, or 0 if the frame is a skippable frame.
938  * ZSTD_getFrameParams() - extracts parameters from a zstd or skippable frame
939  * @fparamsPtr: On success the frame parameters are written here.
940  * @src:        The source buffer. It must point to a zstd or skippable frame.
941  * @srcSize:    The size of the source buffer. `ZSTD_frameHeaderSize_max` is
951 /*-*****************************************************************************
952  * Buffer-less and synchronous inner streaming functions
954  * This is an advanced API, giving full control over buffer management, for
960 /*-*****************************************************************************
961  * Buffer-less streaming compression (synchronous mode)
965  * ZSTD_CCtx object can be re-used multiple times within successive compression
978  * - ZSTD_compressContinue() has no internal buffer. It uses externally provided
979  *   buffer only.
980  * - Interface is synchronous : input is consumed entirely and produce 1+
982  * - Caller must ensure there is enough space in `dst` to store compressed data
987  * - ZSTD_compressContinue() presumes prior input ***is still accessible and
991  * - ZSTD_compressContinue() detects that prior input has been overwritten when
992  *   `src` buffer overlaps. In which case, it will "discard" the relevant memory
995  * Finish a frame with ZSTD_compressEnd(), which will write the last block(s)
997  * will write a final empty block to end the frame. Without last block mark,
1000  * `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress some new
1001  * frame.
1004 /*=====   Buffer-less streaming compression functions  =====*/
1022 /*-*****************************************************************************
1023  * Buffer-less streaming decompression (synchronous mode)
1027  * A ZSTD_DCtx object can be re-used multiple times.
1029  * First typical operation is to retrieve frame parameters, using
1031  * important information to correctly decode the frame, such as the minimum
1032  * rolling buffer size to allocate to decompress data (`windowSize`), and the
1041  * Frame parameters are extracted from the beginning of the compressed frame.
1066  * current block. Alternatively, a round buffer of sufficient size is also
1067  * possible. Sufficient size is determined by frame parameters.
1071  * properly handle maximum back-reference.
1073  * A frame is fully decoded when ZSTD_nextSrcSizeToDecompress() returns zero.
1078  * decode a frame.
1082  * Skippable frames allow integration of user-defined data into a flow of
1085  * a) Skippable frame ID - 4 Bytes, Little endian format, any value from
1087  * b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
1088  * c) Frame Content - any content (User Data) of length equal to Frame Size
1090  * For skippable frames ZSTD_getFrameParams() returns fparamsPtr->windowLog==0
1091  * what means that a frame is skippable.
1092  * Note: If fparamsPtr->frameContentSize==0, it is ambiguous: the frame might
1093  *       actually be a zstd encoded frame with no content. For purposes of
1094  *       decompression, it is valid in both cases to skip the frame using
1096  * It also returns frame size as fparamsPtr->frameContentSize.
1099 /*=====   Buffer-less streaming decompression functions  =====*/
1117 /*-*****************************************************************************
1120  * Block functions produce and decode raw zstd blocks, without frame metadata.
1121  * Frame metadata cost is typically ~18 bytes, which can be non-negligible for
1126  * - Compressing and decompressing require a context structure
1128  * - It is necessary to init context before starting
1133  * - Block size is limited, it must be <= ZSTD_getBlockSizeMax()
1135  *   + Consider using the regular ZSTD_compress() instead, as frame metadata
1137  * - When a block is considered not compressible enough, ZSTD_compressBlock()