API Usage

To interface with Zstandard, simply import the zstandard module:

import zstandard

It is a popular convention to alias the module as a different name for brevity:

import zstandard as zstd

This module attempts to import and use either the C extension or CFFI implementation. On Python platforms known to support C extensions (like CPython), it raises an ImportError if the C extension cannot be imported. On Python platforms known to not support C extensions (like PyPy), it only attempts to import the CFFI implementation and raises ImportError if that can’t be done. On other platforms, it first tries to import the C extension then falls back to CFFI if that fails and raises ImportError if CFFI fails.

To change the module import behavior, a PYTHON_ZSTANDARD_IMPORT_POLICY environment variable can be set. The following values are accepted:

default

The behavior described above.

cffi_fallback

Always try to import the C extension then fall back to CFFI if that fails.

cext

Only attempt to import the C extension.

cffi

Only attempt to import the CFFI implementation.

In addition, the zstandard module exports a backend attribute containing the string name of the backend being used. It will be one of cext or cffi (for C extension and cffi, respectively).

Note

The documentation in this section makes references to various zstd concepts and functionality. See Concepts for more details.

Choosing an API

There are multiple APIs for performing compression and decompression. This is because different applications have different needs and this library wants to facilitate optimal use in as many use cases as possible.

From a high-level, APIs are divided into one-shot and streaming: either you are operating on all data at once or you operate on it piecemeal.

The one-shot APIs are useful for small data, where the input or output size is known. (The size can come from a buffer length, file size, or stored in the zstd frame header.) A limitation of the one-shot APIs is that input and output must fit in memory simultaneously. For say a 4 GB input, this is often not feasible.

The one-shot APIs also perform all work as a single operation. So, if you feed it large input, it could take a long time for the function to return.

The streaming APIs do not have the limitations of the simple API. But the price you pay for this flexibility is that they are more complex than a single function call.

The streaming APIs put the caller in control of compression and decompression behavior by allowing them to directly control either the input or output side of the operation.

With the streaming input, compressor, and decompressor APIs, the caller has full control over the input to the compression or decompression stream. They can directly choose when new data is operated on.

With the streaming ouput APIs, the caller has full control over the output of the compression or decompression stream. It can choose when to receive new data.

When using the streaming APIs that operate on file-like or stream objects, it is important to consider what happens in that object when I/O is requested. There is potential for long pauses as data is read or written from the underlying stream (say from interacting with a filesystem or network). This could add considerable overhead.

Thread and Object Reuse Safety

Unless stated otherwise, ZstdCompressor and ZstdDecompressor instances cannot be used for temporally overlapping (de)compression operations. i.e. if you start a (de)compression operation on an instance or a helper object derived from it, it isn’t safe to start another (de)compression operation from the same instance until the first one has finished.

ZstdCompressor and ZstdDecompressor instances have no guarantees about thread safety. Do not operate on the same ZstdCompressor and ZstdDecompressor instance simultaneously from different threads. It is fine to have different threads call into a single instance, just not at the same time.

Objects derived from ZstdCompressor and ZstdDecompressor that perform (de)compression operations (such as ZstdCompressionReader and ZstdDecompressionWriter) are bound to the ZstdCompressor or ZstdDecompressor from which they came and are therefore not thread safe by extension.

Some operations require multiple function calls to complete. e.g. streaming operations. A single ZstdCompressor or ZstdDecompressor cannot be used for simultaneously active operations. e.g. you must not start a streaming operation when another streaming operation is already active.

If you need to perform multiple compression or decompression operations in parallel, you MUST construct multiple ZstdCompressor or ZstdDecompressor instances so each independent operation has its own ZstdCompressor or ZstdDecompressor instance.

The C extension releases the GIL during non-trivial calls into the zstd C API. Non-trivial calls are notably compression and decompression. Trivial calls are things like parsing frame parameters. Where the GIL is released is considered an implementation detail and can change in any release.

APIs that accept bytes-like objects don’t enforce that the underlying object is read-only. However, it is assumed that the passed object is read-only for the duration of the function call. It is possible to pass a mutable object (like a bytearray) to e.g. ZstdCompressor.compress(), have the GIL released, and mutate the object from another thread. Such a race condition is a bug in the consumer of python-zstandard. Most Python data types are immutable, so unless you are doing something fancy, you don’t need to worry about this.

Performance Considerations

The ZstdCompressor and ZstdDecompressor types maintain state to a persistent compression or decompression context. Reusing a ZstdCompressor or ZstdDecompressor instance for multiple operations is faster than instantiating a new ZstdCompressor or ZstdDecompressor for each operation. The differences are magnified as the size of data decreases. For example, the difference between context reuse and non-reuse for 100,000 100 byte inputs will be significant (possibly over 10x faster to reuse contexts) whereas 10 100,000,000 byte inputs will be more similar in speed (because the time spent doing compression dwarfs time spent creating new contexts).