ZST (Zstandard) Format Guide

Zstandard (often abbreviated as zstd) is a modern, high-performance lossless compression algorithm and format developed by Yann Collet at Facebook (now Meta) in 2015. Designed as a next-generation replacement for gzip, Zstandard provides an exceptional balance between compression ratio and speed — it typically compresses and decompresses 2-5x faster than gzip while achieving equal or better compression ratios. The algorithm is published as RFC 8878 and is freely available under a BSD license.

History of Zstandard

Yann Collet, who also created the LZ4 compression algorithm in 2011, developed Zstandard to bridge the gap between LZ4's extreme speed and traditional algorithms' compression efficiency. Released as open source in 2015, Zstandard quickly gained adoption due to its remarkable performance characteristics. Facebook deployed it across their production infrastructure to handle petabytes of data compression daily. In 2021, Zstandard was formalized as RFC 8878 by the IETF, establishing it as an internet standard. The algorithm's adoption accelerated rapidly: Arch Linux switched its package format from .tar.xz to .tar.zst in 2020, and the Linux kernel added zstd support for btrfs filesystem compression, squashfs, and initramfs. Major databases including PostgreSQL, MySQL, and RocksDB added native zstd support for data compression.

Key Features and Uses

Zstandard's key innovation is its configurable compression levels ranging from ultra-fast negative levels (faster than LZ4 but with some compression) to level 22 (approaching LZMA/xz ratios). The default level 3 provides gzip-like ratios at 3x the speed. Unique features include trainable dictionary compression for optimizing small data compression, native multi-threaded compression, a seekable frame format for random access, and long distance matching for improved ratios on large files. The format uses xxHash64 for integrity verification and supports both streaming and frame-based compression.

Common Applications

Zstandard is used extensively in modern infrastructure: Arch Linux and Fedora use .pkg.tar.zst for package distribution; the Linux kernel uses zstd for btrfs, squashfs, and initramfs compression; databases (PostgreSQL 15+, MySQL 8.0, RocksDB, ClickHouse) use zstd for data and WAL compression; Facebook/Meta compresses petabytes of production data with zstd; container registries use zstd for image layer compression (OCI Image Spec); and modern web browsers (Chrome 123+, Firefox 126+) support Content-Encoding: zstd for HTTP compression. CI/CD systems increasingly use zstd for artifact compression due to its superior speed.

Advantages and Disadvantages

Advantages

Exceptional Speed: 2-5x faster decompression than gzip
Better Ratios: Outperforms gzip at comparable speed settings
Flexible Levels: Ultra-fast negative to maximum level 22
Multi-threaded: Native parallel compression built-in
Dictionary Mode: Trainable dictionaries for small data optimization
RFC Standard: Formalized as RFC 8878 internet standard
Seekable Format: Optional random access within compressed files
Wide Adoption: Linux kernel, databases, package managers
Open Source: BSD licensed, freely available

Disadvantages

Newer Format: Not as universally available as gzip (yet)
Single File Only: Cannot archive directories — must combine with tar
No Encryption: No built-in password protection
Windows Support: Not natively supported, requires 7-Zip 23+
Not HTTP Universal: Not all CDNs/browsers support zstd encoding yet
Tool Installation: Requires zstd package on older Linux systems
Memory Usage: Higher memory at very high compression levels
No Recovery: No built-in error correction mechanism