Lines Matching +full:parameter +full:- +full:less
1 /* SPDX-License-Identifier: GPL-2.0+ OR BSD-3-Clause */
6 * This source code is licensed under both the BSD-style license (found in the
9 * You may select, at your option, one of the above-listed licenses.
40 * typically with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual.
61 real-time compression scenarios at zlib-level and better compression ratios.
62 The zstd compression library provides in-memory compression and decompression
66 which is currently 22. Levels >= 20, labeled `--ultra`, should be used with
72 - a single step (described as Simple API)
73 - a single step, reusing a context (described as Explicit context)
74 - unbounded multiple steps (described as Streaming compression)
78 - a single step (described as Simple dictionary API)
79 - a single step, reusing a dictionary (described as Bulk-processing
85 Advanced experimental APIs should never be used with a dynamically-linked
90 /*------ Version ------*/
120 /* All magic numbers are supposed read/written to/from files/memory using little-endian convention …
162 * @return : - decompressed size of `src` frame content, if known
163 * - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
164 …* - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too …
173 …* note 3 : decompressed size is always present when compression is completed using single-pass fu…
175 * note 4 : decompressed size can be very large (64-bits value),
182 #define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
183 #define ZSTD_CONTENTSIZE_ERROR (0ULL - 2)
212 * maximum compressed size in worst case single-pass scenario.
213 * When invoking `ZSTD_compress()`, or any other one-pass compression function,
228 …) ? 0 : (srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /*…
229 …D_compressBound(size_t srcSize); /*!< maximum compressed size in worst case single-pass scenario */
254 * Note : re-using context is just a speed / resource optimization.
256 * Note 2: For parallel execution in multi-threaded environments,
268 * If any advanced parameter was set using the advanced API,
305 * __They do not apply to one-shot variants such as ZSTD_compressCCtx()__ .
333 * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */
334 … ZSTD_c_compressionLevel=100, /* Set compression parameters according to pre-defined cLevel table.
347 ZSTD_c_windowLog=101, /* Maximum allowed back-reference distance, expressed as power of 2.
361 ZSTD_c_chainLog=103, /* Size of the multi-probe search table, as a power of 2.
365 * This parameter is useless for "fast" strategy.
371 * This parameter is useless for "fast" and "dFast" strategies.
399 … * This is helpful in low bandwidth streaming environments to improve end-to-end latency,
401 * Note: this parameter is stable since v1.5.6.
402 * It was present as an experimental parameter in earlier versions,
408 * This parameter is designed to improve compression ratio
411 … * Note: enabling this parameter increases default ZSTD_c_windowLog to 128 MB
419 * default: windowlog - 7.
430 … * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN).
431 … * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage.
441 …ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */
444 /* multi-threading parameters */
445 …/* These parameters are only active if multi-threading is enabled (compiled with build macro ZSTD_…
446 … * Otherwise, trying to set any other value than default (0) will be a no-op and return an error.
447 * In a situation where it's unknown if the linked library supports multi-threading or not,
457 … * Default value is `0`, aka "single-threaded mode" : no worker is spawned,
470 … * - 0 means "default" : value will be determined by the library, depending on strategy
471 * - 1 means "no overlap"
472 * - 9 means "full overlap", using a full window size.
533 * - an error status field, which must be tested using ZSTD_isError()
534 * - lower and upper bounds, both inclusive
539 * Set one compression parameter, selected by enum ZSTD_cParameter.
541 … Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter).
542 …* Setting a parameter is generally only possible during frame initialization (before starting com…
543 * Exception : when using multi-threading mode (nbWorkers >= 1),
576 * - The session : will stop compressing current frame, and make CCtx ready to start a new one.
582 * - The parameters : changes all parameters back to "default".
586 * - Both : similar to resetting the session, followed by resetting parameters.
592 * (note that this entry point doesn't even expose a compression level parameter).
595 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
596 * - The function is always blocking, returns when compression is completed.
624 …* This parameter is only useful in streaming mode, since no internal buffer is allocated in single…
653 * - an error status field, which must be tested using ZSTD_isError()
654 * - both lower and upper bounds, inclusive
659 * Set one compression parameter, selected by enum ZSTD_dParameter.
661 … Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter).
662 …* Setting a parameter is only possible during frame initialization (before starting decompression…
694 /*-***********************************************************************
695 * Streaming compression - HowTo
700 * It is recommended to reuse ZSTD_CStream since it will play nicer with system's memory, by re-usi…
730 * using ZSTD_compressStream2() with ZSTD_e_flush. `output->pos` will be updated.
731 * Note that, if `output->size` is too small, a single invocation with ZSTD_e_flush might not be en…
777 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
778 …* - Compression parameters cannot be changed once compression is started (save a list of exceptio…
779 * - output->pos must be <= dstCapacity, input->pos must be <= srcSize
780 …* - output->pos and input->pos will be updated. They are guaranteed to remain below their respect…
781 * - endOp must be a valid directive
782 …* - When nbWorkers==0 (default), function is blocking : it completes its job before returning to …
783 …* - When nbWorkers>=1, function is non-blocking : it copies a portion of input, distributes jobs …
786 …* - Exception : if the first call requests a ZSTD_e_end directive and provides enough dstCapacity…
787 * - @return provides a minimum amount of data remaining to be flushed from internal buffers
792 * - after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0),
796 * - note: if an operation ends with an error, it may leave @cctx in an undefined state.
798 * In order to be re-employed after an error, a state must be reset,
845 * the next read size (if non-zero and not an error). ZSTD_compressStream2()
846 * returns the minimum nb of bytes left to flush (if non-zero and not an error).
855 /*-***************************************************************************
856 * Streaming decompression - HowTo
860 * ZSTD_DStream objects can be re-employed multiple times.
909 * - `input.pos < input.size`, some input remaining and caller should provide remaining input
911 * - `output.pos < output.size`, decoder flushed internal output buffer.
912 * - `output.pos == output.size`, unflushed data potentially present in the internal buffers,
923 * In order to re-use such a state, it must be first reset,
971 …Dict can be created once and shared by multiple threads concurrently, since its usage is read-only.
977 …* expecting a ZSTD_CDict parameter with any data, including those without a known dictionary.…
1025 * It can still be loaded, but as a content-only dictionary. */
1031 * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
1037 * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
1044 * - The frame does not require a dictionary to be decoded (most common case).
1045 …* - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a …
1046 * Note : this use case also happens when using a non-conformant dictionary.
1047 …* - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible…
1048 * - This is not a Zstandard frame.
1061 * In contrast, Prefixes are single-use.
1069 * Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary,
1070 * meaning "return to no-dictionary mode".
1075 * It's also a CPU consuming operation, with non-negligible impact on latency.
1092 * and supersede any compression parameter previously set within CCtx.
1093 * The parameters ignored are labelled as "superseded-by-cdict" in the ZSTD_cParameter enum docs.
1094 * The ignored parameters will be used again if the CCtx is returned to no-dictionary mode.
1097 * Special : Referencing a NULL CDict means "return to no-dictionary mode".
1104 * Reference a prefix (single-usage dictionary) for next compressed frame.
1118 * It's a CPU consuming operation, with non-negligible impact on latency.
1130 * Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
1131 * meaning "return to no-dictionary mode".
1133 * which has a non-negligible impact on CPU usage and latency.
1156 * Special: referencing a NULL DDict means "return to no-dictionary mode".
1162 * Reference a prefix (single-usage dictionary) to decompress next frame.
1229 /* compression parameter bounds */
1240 #define ZSTD_SEARCHLOG_MAX (ZSTD_WINDOWLOG_MAX-1)
1258 …* The limit does not apply for one-pass decoders (such as ZSTD_decompress()), since no additional …
1261 /* LDM parameter bounds */
1269 #define ZSTD_LDM_HASHRATELOG_MAX (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN)
1271 /* Advanced parameter bounds */
1278 /* --- Advanced types --- */
1304 * rep == 1 --> offset == repeat_offset_1
1305 * rep == 2 --> offset == repeat_offset_2
1306 * rep == 3 --> offset == repeat_offset_3
1308 * rep == 1 --> offset == repeat_offset_2
1309 * rep == 2 --> offset == repeat_offset_3
1310 * rep == 3 --> offset == repeat_offset_1 - 1
1324 …h; /*< match length searched : larger == faster decompression, sometimes less compression */
1331 …int checksumFlag; /*< 1: generate a 32-bits checksum using XXH64 algorithm at end of frame, for…
1348 …ZSTD_dlm_byRef = 1 /*< Reference dictionary content -- the dictionary buffer must outlive its u…
1353 … ZSTD_f_zstd1_magicless = 1 /* Variant of zstd frame format, without initial 4-bytes magic number.
1378 * - The contents of the CDict can be copied into the working context. This
1381 * the compression faster per-byte of input. However, the initial copy of
1386 * - The CDict's tables can be used in-place. In this model, compression is
1388 * tables. However, this model incurs no start-up cost (as long as the
1392 * - The CDict's tables are not used at all, and instead we use the working
1424 ZSTD_ps_enable = 1, /* Force-enable the feature */
1437 * @return : - decompressed size of all data in all successive frames
1438 * - if the decompressed size cannot be determined: ZSTD_CONTENTSIZE_UNKNOWN
1439 * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1445 * note 3 : decompressed size can be very large (64-bits value),
1460 * @return : - upper-bound for the decompressed size of all data in all successive frames
1461 * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1464 …* note 2 : the upper-bound is exact when the decompressed size field is available in every ZSTD …
1466 …* note 3 : when the decompressed size field isn't available, the upper-bound for that frame is c…
1467 * upper-bound = # blocks * min(128 KB, Window_Size)
1485 …tID; /* for ZSTD_skippableFrame, contains the skippable magic variant [0-15] */
1504 * Zstd supports in-place decompression, where the input and output buffers overlap.
1513 * |---------------------------------------|-----------|----------|
1518 * NOTE: This applies only to single-pass decompression through ZSTD_decompress() or
1520 * NOTE: This function supports multi-frame input.
1533 * WARNING: This macro does not support multi-frame input, the input must be a single
1544 …((originalSize) == 0 ? 0 : 3 * (((originalSize) + (blockSize) - 1) / blockSize)) /* 3 bytes per bl…
1556 * @return : upper-bound for the number of sequences that can be generated
1559 * note : returns number of sequences - to get bytes, multiply by sizeof(ZSTD_Sequence).
1587 * @returns The number of sequences generated, necessarily less than
1626 …* using the advanced parameter ZSTD_c_repcodeResolution. Repcodes will improve compression rati…
1636 …* - ZSTD_c_minMatch MUST be set as less than or equal to the smallest match generated by the ma…
1637 …* - ZSTD_c_compressionLevel accordingly adjusts the strength of the entropy coder, as it would …
1638 …* - ZSTD_c_windowLog affects offset validation: this function will return an error at higher de…
1641 …* Note: Repcodes are, as of now, always re-calculated within this function, ZSTD_Sequence.rep is e…
1655 * instead of receiving (src,srcSize) as input parameter, receives (literals,litSize),
1661 * - Only supports explicit delimiter mode
1662 * - Currently does not support Sequences validation (so input Sequences are trusted)
1663 * - Not compatible with frame checksum, which must be disabled
1664 * - If any block is incompressible, will fail and return an error
1665 …* - @litSize must be == sum of all @.litLength fields in @inSeqs. Any discrepancy will generate an…
1666 …* - @litBufCapacity is the size of the underlying buffer into which literals are written, starting…
1668 …* - @decompressedSize must be correct, and correspond to the sum of all Sequences. Any discrepancy…
1682 * Skippable frames begin with a 4-byte magic number. There are 16 possible choices of magic number,
1684 * As such, the parameter magicVariant controls the exact skippable frame magic number variant used,
1688 …* with a 4-byte unsigned int, or if the parameter magicVariant is greater than 15 (and therefore i…
1699 …* The parameter @magicVariant will receive the magicVariant that was supplied when the frame was w…
1700 * i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START.
1729 * to compress data of any size using one-shot compression ZSTD_compressCCtx() or ZSTD_compress2()
1734 * Note that the size estimation is specific for one-shot compression,
1740 * because the ZSTD_CCtx* state will need less memory for small inputs.
1746 * Note : only single-threaded compression is supported.
1760 …ams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression …
1761 * Note : CStream size estimation is only correct for single-threaded compression.
1763 …* Note 2 : ZSTD_estimateCStreamSize* functions are not compatible with the Block-Level Sequence P…
1790 * Initialize an object using a pre-allocated fixed-size buffer.
1792 * Provided pointer *must be 8-bytes aligned*.
1807 * Limitation 2 : static cctx currently not compatible with multi-threading.
1843 #pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
1864 * a threaded compression mode (via ZSTD_c_nbWorkers parameter).
1912 …* same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-component…
1932 * note that changes to the .windowLog parameter will be ignored.
1952 …aced by ZSTD_compress2(), in combination with ZSTD_CCtx_setParameter() and other parameter setters.
1964 …ed by ZSTD_compress2(), in combination with ZSTD_CCtx_loadDictionary() and other parameter setters.
2018 /* Force back-reference distances to remain < windowSize,
2035 * literals compression based on the compression parameters - specifically,
2076 * structure for the dictionary that is read-optimized.
2096 * In general, you should expect compression to be faster--sometimes very much
2097 * so--and CDict creation to be slightly slower. Eventually, we will probably
2103 * Experimental parameter.
2124 * this flag is ALWAYS memory safe, and will never access out-of-bounds
2136 * Experimental parameter.
2140 * calls. Specifically: (out.size - out.pos) will never grow. This gives the
2150 * Zstd will check that (out.size - out.pos) never grows and return an error
2173 * When Sequence validation is disabled (default), Sequences are compressed as-is,
2184 * note: this parameter only influences the first splitter stage,
2204 * While the post-splitter does overlap with the pre-splitter,
2221 * Set to ZSTD_ps_disable to never use row-based matchfinder.
2222 * Set to ZSTD_ps_enable to force usage of row-based matchfinder.
2225 * the row-based matchfinder based on support for SIMD instructions and the window log.
2237 * results than when the two buffers are non-contiguous. This flag forces zstd
2238 * to always load the prefix in non-contiguous mode, even if it happens to be
2253 * In some situations, zstd uses CDict tables in-place rather than copying them
2256 * "cold" (outside CPU cache). This parameter instructs zstd to prefetch CDict tables
2257 * when they are used in-place.
2261 * into the working context, so there is no need to prefetch. This parameter is
2266 * Note: for this parameter, ZSTD_ps_auto is currently equivalent to ZSTD_ps_disable,
2267 * but in the future zstd may conditionally enable this feature via an auto-detection
2278 * is block-by-block: the internal sequence producer will only be called for blocks
2281 * normal (fully-internal) compression operation.
2283 * The user is strongly encouraged to read the full Block-Level Sequence Producer API
2284 * documentation (below) before setting this parameter. */
2291 * This parameter can be used to set an upper bound on the blocksize
2299 * This parameter only has an effect if ZSTD_c_blockDelimiters is
2302 * This parameter affects how zstd parses external sequences,
2304 * or from an external block-level sequence producer.
2325 * Get the requested compression parameter value, selected by enum ZSTD_cParameter,
2334 * - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure
2335 * - ZSTD_CCtxParams_setParameter() : Push parameters one by one into
2339 * - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to
2343 * - ZSTD_compressStream2() : Do compression using the CCtx.
2344 * - ZSTD_freeCCtxParams() : Free the memory, accept NULL pointer.
2347 * for static allocation of CCtx for single-threaded compression.
2371 * Set one compression parameter, selected by enum ZSTD_cParameter.
2381 * Get the requested value of one compression parameter, selected by enum ZSTD_cParameter.
2449 …* This parameter is only useful in streaming mode, since no internal buffer is allocated in singl…
2456 * Get the requested decompression parameter value, selected by enum ZSTD_dParameter,
2463 * experimental parameter,
2468 * Experimental parameter.
2484 * If you need to avoid the input buffer allocation use the buffer-less
2488 * this flag is ALWAYS memory safe, and will never access out-of-bounds
2500 * Experimental parameter.
2511 * Experimental parameter.
2524 * WARNING: Enabling this parameter and calling ZSTD_DCtx_refDDict(), will trigger memory
2538 * This parameter can be used to disable Huffman assembly at runtime.
2564 * This instruction is mandatory to decode data without a fully-formed header,
2687 * This is typically useful to skip dictionary loading stage, since it will reuse it in-place.
2713 * Note : (ingested - consumed) is amount of input data buffered internally, not yet compressed.
2769 /* ********************* BLOCK-LEVEL SEQUENCE PRODUCER API *********************
2772 * The Block-Level Sequence Producer API allows users to provide their own custom
2774 * of sequences (literals and matches) is then post-processed by libzstd to produce
2777 * This block-level offload API is a more granular complement of the existing
2778 * frame-level offload API compressSequences() (introduced in v1.5.1). It offers
2788 * Block-Level Sequence Producer API.
2793 * arguments to the user-provided function:
2795 * - sequenceProducerState: a pointer to a user-managed state for the sequence
2798 * - outSeqs, outSeqsCapacity: an output buffer for the sequence producer.
2802 * - src, srcSize: an input buffer for the sequence producer to parse.
2805 * - dict, dictSize: a history buffer, which may be empty, which the sequence
2810 * - compressionLevel: a signed integer representing the zstd compression level
2816 * - windowSize: a size_t representing the maximum allowed offset for external
2821 * The user-provided function shall return a size_t representing the number of
2823 * code if it is greater than outSeqsCapacity. The return value must be non-zero
2824 * if srcSize is non-zero. The ZSTD_SEQUENCE_PRODUCER_ERROR macro is provided
2828 * If the user-provided function does not return an error code, the sequences
2832 * - The sum of matchLengths and literalLengths must equal srcSize.
2833 * - All sequences in the parse, except for the final sequence, must have
2836 * - All offsets must respect the windowSize parameter as specified in
2838 * - If the final sequence has matchLength == 0, it must also have offset == 0.
2844 * If the user-provided function returns an error, zstd will either fall back
2855 * This setting will persist until the next parameter reset of the CCtx.
2866 * COMPRESSION WILL FAIL if it is enabled and the user tries to compress with a block-level
2868 * - Note that ZSTD_c_enableLongDistanceMatching is auto-enabled by default in some
2872 * - As of this writing, ZSTD_c_enableLongDistanceMatching is disabled by default
2874 * check the docs on ZSTD_c_enableLongDistanceMatching whenever the Block-Level Sequence
2879 * - Dictionaries are not currently supported. Compression will *not* fail if the user
2881 * - Stream history is not currently supported. All advanced compression APIs, including
2885 * Third, multi-threading within a single compression is not currently supported. In other words,
2887 * Multi-threading across compressions is fine: simply create one CCtx per thread.
2889 * Long-term, we plan to overcome all three limitations. There is no technical blocker to
2893 #define ZSTD_SEQUENCE_PRODUCER_ERROR ((size_t)(-1))
2905 * Instruct zstd to use a block-level external sequence producer function.
2908 * responsible for managing its lifetime. This parameter is sticky across
2912 * Sequence producer registration is considered to be an "advanced parameter",
2948 * Buffer-less and synchronous inner streaming functions (DEPRECATED)
2962 Buffer-less streaming compression (synchronous mode)
2973 - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only.
2974 - Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks.
2975 …- Caller must ensure there is enough space in `dst` to store compressed data under worst case scen…
2978 …- ZSTD_compressContinue() presumes prior input ***is still accessible and unmodified*** (up to max…
2980 …- ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps.
2990 /*===== Buffer-less streaming compression functions =====*/
2991 ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.")
2993 ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.")
2995 ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.")
3002 ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.")
3004 ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.")
3015 Buffer-less streaming decompression (synchronous mode)
3030 such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`).
3040 …or that previous contiguous segment is large enough to properly handle maximum back-reference dist…
3045 …which can return an error code if required value is too large for current system (in 32-bits mode).
3055 as long as the encoder and decoder progress in "lock-step",
3077 Skippable frames allow integration of user-defined data into a flow of concatenated frames.
3080 a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
3081 b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
3082 c) Frame Content - any content (User Data) of length equal to Frame Size
3083 For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame.
3087 /*===== Buffer-less streaming decompression functions =====*/
3115 - ZSTD_c_format = ZSTD_f_zstd1_magicless
3116 - ZSTD_c_contentSizeFlag = 0
3117 - ZSTD_c_checksumFlag = 0
3118 - ZSTD_c_dictIDFlag = 0
3125 …Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 1…
3129 - Compressing and decompressing require a context structure
3131 - It is necessary to init context before starting
3134 - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
3138 …- When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero)…
3157 …kSize); /*< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression.…