There's always a niche: rawzip and the cost of ergonomics

I recently released rawzip 0.5, a zip archive file parser that focuses on minimalism in terms of dependencies (there are none) and overhead.

Looking at benchmarks: rawzip has anywhere from 15x to 15000x less overhead than the next fastest zip library.

I’m using the overhead terminology deliberately. Most use cases should see compression dominate profiles. Most.

Working with 1 GiB+ zip files with 100 thousand entries forced my hand to create a zip library. Existing solutions were so slow that I thought I was hitting an infinite loop. I filed a few github issues and made a few comments, one of them even made it in a fasterthanlime video covering the sans-io nature of his zip file parser:

I dug into the libraries and found APIs with performance hamstrung in the name of ergonomics. Hiding allocations can make an API nicer to use, but making 100 thousand allocations isn’t free. I found out the hard way that targeting musl exacerbated this issue as the default musl allocator is harmful (to performance) and was causing a further 7x slowdown.

So began the journey of writing a zip file library.

Zero Zero Zero Zero

Early on I decided on:

• Zero allocations
• Zero dependencies
• Zero copy
• Zero unsafe

Zero allocations. The genesis of the library. By reading into user provided buffers, rawzip gives a lending iterator through the entries of the zip file. We can get a comparison of the ergonomic hit by comparing rawzip to zip-rs in how to dump all the entries to stdout. First zip-rs:

let mut zip = zip::ZipArchive::new(reader)?;

for i in 0..zip.len() {
    let mut file = zip.by_index(i)?;
    println!("Filename: {}", file.name());
    std::io::copy(&mut file, &mut std::io::stdout())?;
}

The rawzip equivalent version is substantially longer:

let mut buf = vec![0u8; rawzip::RECOMMENDED_BUFFER_SIZE];
let archive = rawzip::ZipArchive::from_file(file, &mut buf)?;

let mut entries = archive.entries(&mut buf);
while let Some(entry) = entries.next_entry()? {
    println!("Filename: {}", entry.file_path().try_normalize()?.as_str());

    let zip_entry = archive.get_entry(entry.wayfinder())?;
    let reader = zip_entry.reader();
    match entry.compression_method() {
        rawzip::CompressionMethod::STORE => {
            let mut reader = zip_entry.verifying_reader(reader);
            std::io::copy(&mut reader, &mut std::io::stdout())?;
        }
        rawzip::CompressionMethod::DEFLATE => {
            let inflater = flate2::read::DeflateDecoder::new(reader);
            let mut reader = zip_entry.verifying_reader(inflater);
            std::io::copy(&mut reader, &mut std::io::stdout())?;
        }
        method => panic!("add your compression method here: {method:?}"),
    }
}

I’m not too concerned with the amount of code. To me it follows the spirit of the Rust API Guidelines (C-INTERMEDIATE) (functions expose intermediate results to avoid duplicate work), and each stage of zip file parsing is exposed orthogonally.

• Open a ZipArchive and orient around the end of the central directory
• Iterate through the central directory entries for authoritative metadata and get a “wayfinder” to the local entry.
• “Seek” to the local entry and get the Read for the compressed data
• Pipe that data through a decompressor
• Pipe the output to an entry integrity check (uncompressed size and CRC).

Rawzip APIs are opt in and composable. Don’t want entry integrity checks? Well, then don’t wrap the decompressed Read in a verifying_reader! Zip-rs doesn’t expose these knobs, and it shouldn’t, most users aren’t looking to customize how they read a zip file.

Rawzip accepts non-growable &mut [u8] instead of a &mut Vec<u8> and relies on the user to trap for the BufferTooSmall error to grow it themselves when appropriate. In practice this dynamic growing mechanism isn’t required as we know from the Zip spec that each central directory entry should be smaller than 64 KiB and pathologically can’t be bigger than 192 KiB (+ 46 bytes). These are buffer sizes reasonable enough to allocate upfront.

Zero Dependencies

There are a half dozen quality DEFLATE implementations in Rust. Use the best one for the task. That might be libdeflater, a specialized implementation where all the input is required up front and output pre-allocated, but the performance is unrivaled. See a cute, new zstd implementation in Rust? Be my guest! A library with dozens of feature flags gating which backends get compiled in? That’s a nightmare of flag combinations and perpetual version bumps.

My general philosophy is to provide an implementation of a feature within rawzip when feasible, and provide hooks for user provided implementations without resorting to a trait soup to support them. CRC, ZipCrypto (weak), datetime, and IO fall into this category. Take datetime, zip archives have UTC or local datetimes that are easy enough to decipher that no library is needed, and these can be encapsulated by the user with another datetime library. As part of rawzip’s tests, the datetime implementation is property checked against jiff.

Other features are complex enough that a rawzip implementation isn’t feasible, but the hooks can still be exposed. Compression and WinZIP AES are the two features that embody this. If WinZIP AES support is desired, pull in the RustCrypto ecosystem or openssl as desired.

Zero dependencies isn’t without tradeoffs. Rawzip’s built-in CRC implementation is no slouch, reaching 5 GB/s and is the fastest CRC implementation on Wasm¹. However, without dependencies and unsafe code, rawzip is not privy to certain hardware intrinsics and will lose to libraries such as crc32fast on those platforms. The great news is that rawzip, again is composable and swapping the CRC implementation is relatively painless:

let zip_reader = entry.reader();
let expected = zip_reader.claim_verifier();
let inflater = flate2::read::DeflateDecoder::new(zip_reader);

// n + 1 detects oversized output while bounding work on invalid input.
let mut reader = flate2::CrcReader::new(inflater).take(expected.uncompressed_size + 1);
let mut plaintext = Vec::new();
let uncompressed_size = std::io::copy(&mut reader, &mut plaintext)?;

// call expected.valid for entry integrity

Users are also empowered to simply skip integrity checks either for forensics or for performance (not that this is a recommendation, but sometimes the inner data has its own integrity checks and you don’t want to “duplicate” this work).

Specialized dependencies don’t always bring performance benefits. For instance, I thought I was leaving performance on the table by not depending on memchr::memmem::rfind to search for the end of the central directory. Benchmarks disabused me, and my SWAR based implementation is 2-10x faster. Even the naive rfind implementation is 4x faster than memmem::rfind when the data window nearly matches the needle:

fn rfind(haystack: &[u8], needle: &[u8]) -> Option<usize> {
    haystack
        .windows(needle.len())
        .rposition(|window| window == needle)
}

The APIs that can or do allocate, like normalizing file paths to guard against zip slip, are off the hot path and can be amortized by the caller.

Zero Copy

When your zip file is in memory (eg: mmap’ed), reading into user provided buffers is a needless ergonomic hurdle and the next bottleneck in performance. An entire class of errors is eliminated (IO / BufferTooSmall). Hence, the creation of a mirror API that operates over in memory data (see the benchmark where the in memory API smokes the reader API).

It is strictly more work as a library maintainer to have two sets of APIs, but having a no std, zero copy API is generally useful. With 50:1 compression ratios, it is common to have the zip file in memory and stream the uncompressed data.

Zero Unsafe

Ok this one is just a vanity metric.

Other libraries of mine leverage unsafe appropriately, but something is just rewarding when you can create a fast and composable API without any unsafe.

Moreso, considering the domain of zip file parsing, using unsafe doesn’t sit well.

I feel the same way writing code that provably can’t panic, but that’s a whole lot harder.

Positioned IO

Rawzip doesn’t do IO itself, it relies on a positioned IO ReaderAt trait that encapsulates “give me bytes at an offset, immutably and synchronously”. I have a previous post on how well positioned positioned io is for zip files (pardon the pun). Decompression is embarrassingly parallel and positioned IO lets one share a rawzip archive across many threads seamlessly. No cloning, arcs, or mutexes here.

Positioned IO is different from sans-io where the same parser can be driven through synchronous or asynchronous means. Fasterthanlime breaks that down in the video mentioned at the start of the post, and they talk about their own rc-zip crate for parsing zip files.

It may seem limiting to impose a synchronous interface in this async world, especially with all the hype around io_uring, but I’m not convinced. Parallelized decompression can still maximize drive throughput synchronously. Even networked reads work well enough under synchronous I/O. For example, crater has a custom ReaderAt for byte-offset reads of the crates.io zip database dump directly from S3.

LLM Usage

Was AI used in creating rawzip?

Of course. It would be irresponsible to create a parsing library and not have agents go through the spec line by line to ensure conformance. And when I had questions, it was incredibly useful to have agents consult well-established implementations to understand precedent.

I’d ask questions like:

Given this list of zip file repos […], how does each implementation determine if the data descriptor contains 4 or 8 byte sized fields?

Or

What is the behavior when values in the data descriptor don’t match those in the central directory

In several cases there was no consensus on what was the “right” way to parse, which security researchers love. There’s a paper on exploiting these gaps² alongside ZipDiff, the differential fuzzer.

There’s not enough space in this article to talk about zip file’s checkered past when it comes to security. The rawzip readme touches on what rawzip provides and what a hardened extractor needs to worry about (hint: it’s a lot).

The community was equally important in reporting edge cases that were genuinely missed by LLMs and hadn’t shown up in production. I’m very thankful for those reports.

Zip Insanity

For those not in the know and want a taste of the insanity of the zip file format (and this example has been simplified):

• A classic zip can only hold up to 65,535 entries. So Zip64 was developed to support larger archives.
• Zip64 needs to be backwards compatible so if “65,535” is recorded, the spec says one should expect additional Zip64 metadata to be present that contains the true count of entries.
• Except classic zips that contain exactly 65,535 entries. They won’t have Zip64 metadata
• Except Zip64 metadata can be present even if 65,535 wasn’t recorded as the number of entries

So one ends up unconditionally searching out this Zip64 metadata, and writing code that filters out false positives, as a preceding entry could contain a comment, file name, or extra data filled with arbitrary data that matches Zip64 metadata signature.

Conclusion

Since writing rawzip, I’ve realized how often I have the need to parse zip files, and have ended up processing millions of user supplied zips in production with rawzip.

Where does rawzip go from here? Without dependencies and a slow moving spec, rawzip should be able to reach a steady state, and the 0.5 release is the product of putting rawzip through its paces over the last 18 months. I like to wait a year without any changes / releases before re-releasing the latest version as 1.0

I’m happy with the read side, there’s just a couple use cases that I wished the write side catered to more easily:

• Recompress zip entries with better compression methods but keep headers identical. I’ve written before about the unreasonable effectiveness of repacking poorly deflated zips with zstd³ and currently it’s too much effort to losslessly encode an entry’s local and central directory metadata.
• Backpatching writer that doesn’t rely on data descriptors for sequential archive parsing

I’ve tried to setup the API such that these can be added in a non-breaking way, I just want to be cognizant of the sisyphean task of being everything to everyone.

And I think that is the right way to view any existing ecosystem (not just rust crates dealing with zip files). What niche is perennially underserved?

For rawzip, it was the realization that ergonomics have a cost. That cost is invisible to most, and I wanted to know what a zip library looks like when you refuse to pay it and zip files scale into obscene territory. rawzip is a niche for those bringing their own dependencies or chasing overhead, but it’s not a replacement.