scpcap/STREAMING_ZSTD_FORMAT.md

Streaming .pcap.zst Format

Status: implemented (Rust reference: scpcap) Audience: an implementer writing a C++ reader that streams pcap records out of a .pcap.zst file while it is still being written.

---

1. What this format is and why it exists

A pcap-stream .pcap.zst is a plain concatenation of independent zstd frames:

[skippable marker frame]   identity + version   (carries no pcap bytes)
[zstd frame]               compress(24-byte pcap global header)
[zstd frame]               compress(record-aligned chunk of pcap records)   \
[zstd frame]               compress(record-aligned chunk of pcap records)    |  repeats
[zstd frame]               ...                                              /

Two properties fall out of that, and both are load-bearing:

1. It is a valid .zst file. zstd -d file.pcap.zst concatenates the frame outputs and reproduces the original .pcap byte for byte. The marker is a zstd skippable frame, so compliant decoders ignore it. Nothing about this format requires a special tool to read offline.
2. It is streamable and seekable at frame granularity. Every data frame is cut on a pcap record boundary and declares its decompressed size in its header. A reader can therefore walk the file frame by frame, know exactly how many uncompressed pcap bytes each frame contributes, and decompress any single frame independently — without decompressing anything before it.

The point of the format is that a file written this way can be tailed as it grows and served (or relayed) without recompression: each stored frame is already exactly the payload you'd put on the wire.

What the marker frame buys us

The marker is the only thing that distinguishes our file from an arbitrary zstd some.pcap produced by the CLI. Both have the extension .pcap.zst; both decode to the same pcap. But a CLI-produced file is typically one enormous frame whose boundaries have nothing to do with pcap records, so none of the streaming properties hold. A reader that assumed frame boundaries were record-aligned would hand out torn records.

So: a file without a valid leading marker frame must be rejected, not "best-efforted".

---

2. The marker frame (byte level)

A zstd skippable frame is magic (4 LE) | payload_size (4 LE) | payload. Skippable magics occupy the range 0x184D2A50 .. 0x184D2A5F; we write the base value.

Offset	Size	Value	Meaning
0	4	0x184D2A50 (LE)	zstd skippable-frame magic
4	4	6 (LE)	payload size
8	4	"PSZ1" (0x50 0x53 0x5A 0x31)	pcap-stream zst marker magic
12	1	1	format version
13	1	0	flags (reserved; must be written as 0)

Total: 14 bytes, always at physical offset 0.

Hex, in full:

50 2A 4D 18   06 00 00 00   50 53 5A 31   01   00

Reader rules:

• The first frame in the file must be a skippable frame whose payload is a valid marker (magic PSZ1, version 1). Anything else → reject the file (NotMarked).
• Validation reads only the first 4 payload bytes (magic) and the version byte. Ignore the flags byte and any trailing payload bytes — a future version may extend the payload, and the payload_size field lets you skip whatever you don't understand.
• An unknown version is a hard reject, not a warning. Version is bumped only on an incompatible container change.
• Skippable frames may in principle appear elsewhere in the file. Treat any skippable frame as contributing zero source bytes and step over it. (Today only the leading marker exists.)

---

3. Frame invariants a reader may rely on

These are guaranteed by every writer of this format. A file that violates them is malformed and should be rejected rather than reinterpreted.

1. Frame 0 (physical offset 0) is the marker. See above.
2. Frame 1 is the header frame: it decompresses to exactly the 24-byte pcap global header, and it covers source range [0, 24). If its declared content size is not 24, the file is malformed. Parse it as a classic pcap global header (any of the four magics — LE/BE × µs/ns; pcapng is rejected upstream).
3. Every subsequent frame is a data frame whose decompressed bytes are a whole number of complete pcap records — never a partial record header, never a partial payload. Concatenating all data frames' outputs yields the pcap body (everything after byte 24).
4. Every frame declares its decompressed content size in its zstd frame header (the writer compresses with the input size known, so the field is always present). A frame with an undeclared content size is malformed. This is what lets a reader map physical → source offsets without decompressing anything.
5. Every frame carries a zstd content checksum. Decompression verifies it, so single-bit corruption surfaces as a decode error rather than as corrupt packets.
6. Frames are contiguous in source coordinates. Frame n's source range starts exactly where frame n−1's ended. There are no gaps and no overlaps.
7. Frames are append-only and immutable once written. Bytes already on disk are never rewritten in place. The only exception is truncation of a torn trailing frame at writer restart (§7), which happens before any new bytes are appended and always cuts at a frame boundary.
8. Data frames target ~256 KiB of uncompressed source bytes (frame_target_bytes, default 262144) but may be smaller: a flush timer (default 100 ms) cuts a short frame to bound latency, and the last frame of a file is whatever is left. Do not assume any frame size. Size the read window generously (1 MiB is comfortable) and grow it if a single frame ever exceeds it.

Note that packet counts are not carried in the container. If you need them, count records while decompressing.

---

4. Two coordinate systems

Keep these strictly separate; conflating them is the single easiest way to get this wrong.

• Source offset — a byte offset into the decompressed pcap. The global header occupies [0, 24); the first data frame starts at 24. This is the coordinate space of the uncompressed pcap.
• Physical offset — a byte offset into the .pcap.zst on disk. Includes the 14-byte marker and all compressed frame bytes.

A frame table entry is therefore four numbers:

struct ZstFrame {
    uint64_t src_start;   // source (uncompressed pcap) offset of first byte
    uint64_t src_end;     // exclusive; src_end - src_start == declared content size
    uint64_t phys_start;  // physical offset of the compressed frame
    uint64_t phys_len;    // compressed frame size on disk
};

src_end - src_start comes from ZSTD_getFrameContentSize; phys_len comes from ZSTD_findFrameCompressedSize. Neither requires decompressing the frame — which is the whole trick: you can build the frame table for a multi-GB file by reading only frame headers, and you only decompress the frames a consumer actually asks for.

The marker frame (and any future skippable frame) advances phys and not src.

---

5. Walking the frames

5.1 The two zstd calls you need

Purpose	libzstd (C)
Skippable magic test	(magic & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START
Compressed size of the frame at p	ZSTD_findFrameCompressedSize(p, n)
Declared decompressed size	ZSTD_getFrameContentSize(p, n)
Decompress one frame	ZSTD_decompress(dst, dstCap, p, csize)

Both size calls are header-only reads. Important subtleties:

• ZSTD_findFrameCompressedSize returns an error code (test with ZSTD_isError) when the buffer doesn't contain the whole frame — but it returns the same kind of error for a genuinely corrupt frame. You cannot distinguish "need more bytes" from "corrupt" by the error alone. Disambiguate with position: if you have read to end-of-file, treat it as a torn trailing frame (stop, retry later); if not, grow your window and retry. This is exactly what the reference implementation does.
• ZSTD_findFrameCompressedSize may also return a size larger than the buffer you passed. Guard with csize <= n and treat a larger value as "need more bytes".
• ZSTD_getFrameContentSize returns ZSTD_CONTENTSIZE_UNKNOWN or ZSTD_CONTENTSIZE_ERROR as sentinels. For this format, either is malformed (invariant 4) — do not fall back to decompressing to learn the size.
• Skippable frames have no content size. Compute their length yourself: 8 + payload_size, where payload_size is the LE uint32 at offset 4.

5.2 The walk

State: phys_pos (next physical byte to parse — always a frame boundary), src_pos (next source offset — the source frontier), plus saw_marker, header, file_id.

loop until phys_pos == file_size (fstat each poll; the file grows under you):
    read a window at phys_pos  (1 MiB; short reads are normal — retry/accumulate)
    at_eof := (bytes_read == file_size - phys_pos)

    if window starts with a skippable magic:
        if the full skippable frame isn't in the window:
            if !at_eof: grow window, retry
            else:       stop (torn trailing skippable) — retry on a later poll
        if phys_pos == 0: it must be a valid marker, else reject NotMarked
        phys_pos += 8 + payload_size          # src_pos unchanged
        continue

    if !saw_marker: reject NotMarked          # a data frame before any marker

    csize := ZSTD_findFrameCompressedSize(window)
    if error or csize > bytes_read:
        if at_eof: stop (torn trailing frame) — retry on a later poll
        else:      grow window, retry

    content := ZSTD_getFrameContentSize(window[0..csize])
    if unknown or error: reject Malformed

    if header not yet seen:                   # this is the header frame
        if content != 24: reject Malformed
        header := parse_pcap_global_header(decompress(window[0..csize]))
        src_pos  += 24                        # -> 24
        phys_pos += csize
        continue

    emit ZstFrame{ src_start: src_pos, src_end: src_pos + content,
                   phys_start: phys_pos, phys_len: csize }
    src_pos  += content
    phys_pos += csize

Key points:

• A torn trailing frame is normal, not an error. While the writer is mid-append you will routinely see a partial frame at EOF. Stop the walk before it, leave phys_pos pointing at its start, and pick it up on a later poll. Never surface a frame you haven't fully read.
• phys_pos only ever advances across complete frames, so it is always a valid resume point.
• Cap the work per poll (the reference consumes ~8 MiB of physical bytes per call, rounded up to the frame boundary crossed) so that catching up on a large backlog doesn't stall your event loop.
• Everything above is pread-based on a held-open fd — see §7.3 for why that matters.

To read the packets for a frame: pread phys_len bytes at phys_start, ZSTD_decompress into a buffer of src_end - src_start bytes, and you have a run of complete pcap records. To relay the frame to a downstream zstd-speaking consumer: send those phys_len bytes verbatim. No decompress, no recompress.

---

6. File identity (FileId)

Filenames are not unique — a capture restart can recreate the same name with different content — so files are identified by content, not by path.

FileId = { first_packet_ts : u64,   // nanoseconds
           header_crc32    : u32 }

• header_crc32 = CRC-32 (IEEE, the zlib/crc32fast polynomial, init 0xFFFFFFFF, reflected, final xor) over the 40 bytes global_header[0..24] || first_record_header[0..16] — i.e. over the decompressed source bytes [0, 40).
• first_packet_ts = the first record's timestamp normalized to nanoseconds: ts_sec * 1e9 + ts_frac * 1000 for a µs-resolution file (magic 0xA1B2C3D4 / 0xD4C3B2A1), or ts_sec * 1e9 + ts_frac for a ns-resolution file (magic 0xA1B23C4D / 0x4D3CB2A1). Fields are read in the file's endianness.
• All-zeros is the provisional id: the file exists and has a header but no first packet yet. Recompute once the first data frame lands.

For a .pcap.zst, this means: decompress the header frame (24 bytes) and just enough of the first data frame to get its first 16 bytes, concatenate, CRC.

---

7. Detecting completion: the .partial rename

This is the part that took the most iteration. Read this section carefully — the wrong approach looks like it works and then loses data or reports the wrong file size in production.

7.1 The scheme

A file being written exists on disk under a name with an in-progress suffix, and is atomically renamed to its final name when — and only when — it is complete:

20260714/cap-20260714-14:30:00.pcap.zst.partial     ← still growing
                    │
                    │  rename(2)  — atomic, same directory
                    ▼
20260714/cap-20260714-14:30:00.pcap.zst             ← complete, immutable

The suffix is configurable (partial_suffix, default .partial) and matches SolarCapture's own convention. The four physical forms a reader must recognize are:

<name>.pcap                <name>.pcap.partial
<name>.pcap.zst            <name>.pcap.zst.partial

Strip the partial suffix first, then a .zst suffix, to recover the logical name (<name>.pcap). The logical name is what appears in the protocol, in resume tokens, and in your own bookkeeping — it is stable across the rename, which is the whole point. Two flags fall out of the stripping: partial (is it still growing?) and zst (is it compressed?).

7.2 The rule

The rename is the completion signal. Nothing else is.

Concretely, a reader declares a file complete when both of these hold:

1. The rename has been observed — the final path exists.
2. The file is fully drained — you have walked every byte of it: phys_pos == file_size (fresh fstat), i.e. no torn trailing frame remains.

Only then do you know the final size, which for a .pcap.zst is the source frontier (src_pos), not the physical file size. (For a raw .pcap the two coincide, which is a trap if you generalize from the raw path.)

(This is strictly a hazard for a raw .pcap reader, not for this format: a writer that preallocates space ahead of its real cursor can make even fstat's reported size diverge from real content, which requires content-level record validation to detect — not just the phys/src split above. The .pcap.zst writer here never preallocates, and nothing that preallocates writes .pcap.zst directly, so this doesn't apply to the format described in this document.)

If (1) holds but (2) doesn't — the rename landed but there's still an unframed tail — wait. Do not declare completion with a size you'd have to revise. Poll again; the tail is already fully written, so this resolves on the next tick.

7.3 What NOT to do (each of these was tried and removed)

• ❌ Do not use a quiet period / size-stability heuristic ("no growth for N seconds ⇒ done"). A capture stall is indistinguishable from a finished file, and it fires early under load. This was ripped out in favor of the rename signal alone.
• ❌ Do not treat the appearance of the next file as the completion signal for the current one. Same commit, same reason: it couples two files' lifecycles and it's wrong at a capture restart.
• ❌ Do not re-open the file by path when you notice the rename. The reader must keep serving from the fd it already holds. A rename does not invalidate an open fd — it still points at the same inode, and reads continue to work through the rename and even through an unlink. If you close and reopen, you introduce a window where the .partial path is gone and the reader ENOENTs, and you break continuity for any consumer mid-file.
• ❌ Do not assume the on-disk file size is the source size for a .zst. It's the compressed size. Track the source frontier separately.
• ❌ Do not re-scan and re-register the renamed file as a new file. A directory scan after the rename sees a new directory entry (the final name). Because the logical name is suffix-stripped, it maps to the file you already know — dedupe on the logical name, not the physical path.

7.4 The one subtlety that bites: the ENOENT window

There is a gap between "the rename happened" and "we have finished draining and finalized the file". That gap can be seconds (framing the tail, serving a backlog). During that gap:

• the .partial path no longer exists (rename is atomic — the old name is gone the instant the new one appears), but
• your in-memory state may still be pointing at the .partial path for any new open.

Anything that opens the file by path in that window — a lazily-rebuilt frame, a new consumer attaching, a cache miss — gets ENOENT, on a file that is perfectly healthy.

The fix (commit 4c5a0e8) is: the moment you observe the rename, repoint your stored serve path at the final path, immediately — long before you finalize the file. Keep draining through the fd you already hold (which is unaffected), but make sure any new open uses the final name.

on scan tick, for each growing file:
    if is_partial && !complete_signal && exists(final_path):
        complete_signal = true
        serve_path = final_path        # ← do this NOW, not at finalization
        # keep tailing via the already-open fd; finalize once drained (§7.2)

That ordering — signal and repoint first, drain second, finalize third — is the whole rename protocol. Get it wrong and you get intermittent ENOENTs under exactly the conditions that are hardest to reproduce (a consumer reconnecting during the rename).

7.5 Detecting the rename

Poll: on each scan, for every file you are tailing that is still partial, stat its final path. Existence of the final path = the rename happened. This is cheap (one stat per growing file, and there is normally exactly one) and needs no inotify. The reference implementation drives this on a 1 s timer; a caller-driven scan loop can poll as often as it likes.

If you prefer inotify, watch for IN_MOVED_TO in the capture directory — but keep the stat fallback: inotify queues overflow, and missing a rename means never completing a file.

7.6 Files discovered already-final

A file discovered with no partial suffix (final name, no .partial sibling) is born complete — it finished before you ever saw it. Register it as complete immediately. Don't wait for a rename that already happened.

A reader that builds a frame table (§4) to serve random-access requests should build it lazily, the first time a consumer actually asks for the file — walking/framing every already-final file at discovery reads every byte of every file in a directory of a day's captures for nothing. A reader that just streams packets out sequentially (walking §5.2 once, start to finish, with no frame table) doesn't have this concern either way.

7.7 Writer restart and truncation (reader's perspective)

You never write these files, but the process that does can restart mid-file. On restart it walks its own .partial, and truncates it back to a frame boundary — never mid-frame — before resuming appends.

The guarantee that matters to a reader: content at a given source offset never changes. A restart can only shrink the file back to an earlier frame boundary and re-append from there; the capture data being re-appended is the same data, so whatever lands at a given source offset is byte-for-byte identical to what would have been there without the restart. Only the physical framing (where frame boundaries fall) can differ across a restart — never the decompressed content.

Two cases follow from that:

• Ordinary case — the truncated bytes were never surfaced. A writer only ever truncates a torn trailing frame (invariant 7 in §3), and a reader following §5.2 never surfaces a torn trailing frame in the first place — it stops before it and waits. So phys_pos already sits at or before the truncation point. There is nothing to undo: the next poll simply finds the file has grown again (with corrected content) from the same boundary you were already parked at, and the walk continues exactly as if nothing happened.
• Defensive case — the file shrinks below phys_pos. Check this on every poll with a fresh fstat, the same as you already do to detect EOF: if file_size < phys_pos, trust the frames you have already read — they decoded and checksummed cleanly, so they're not in question — and simply stop advancing. Wait, polling as usual, until file_size grows back past phys_pos again, then resume the walk from phys_pos exactly as before. Because content per source offset is guaranteed stable, whatever now occupies the bytes at and beyond phys_pos is the correct continuation, so no rewind or re-parse of already-consumed frames is needed.

---

8. Error handling

Condition	Action
First frame is not a valid marker	Reject the file (NotMarked). It's a foreign .zst whose frames don't align to records. Log the path; do not attempt to stream it.
Marker version unknown	Reject.
Header frame's content size ≠ 24	Reject (Malformed).
Any frame with undeclared/error content size	Reject (Malformed).
Torn frame at EOF	Not an error. Stop before it; retry on the next poll.
zstd decode error / checksum failure	Reject and alarm. Never resync by scanning forward for a plausible frame magic — you'd hand out garbage records. Stop tailing this file.
incl_len > 256 KiB (MAX_INCL_LEN) or > snaplen	Treat as corruption; stop and alarm, don't resync. (Relevant when validating decompressed records.)

The consistent principle: fail loudly, never resync. A file that violates an invariant is either corrupt or not ours, and silently guessing at frame boundaries is worse than stopping.

---

9. Constants

Name	Value
SKIPPABLE_MAGIC	0x184D2A50 (range 0x184D2A50..=0x184D2A5F)
MARKER_MAGIC	"PSZ1"
MARKER_VERSION	1
Marker frame length	14 bytes
GLOBAL_HEADER_LEN	24
RECORD_HEADER_LEN	16
MAX_INCL_LEN	262144 (256 KiB)
frame_target_bytes (default)	262144 (~256 KiB of source bytes)
frame_flush_ms (default)	100 (may cut a short frame)
zstd_level (default)	2 (writer-side; irrelevant to a reader)
partial_suffix (default)	.partial
Read window	1 MiB, grown (×2) if a single frame exceeds it
Scan interval (reference default; caller-driven elsewhere)	1000 ms
Poll interval (reference default; caller-driven elsewhere)	20 ms

---

10. Conformance checklist

A C++ reader is done when it passes these.

• Walks a marked file (marker + header frame + N data frames) and produces a frame table whose first entry starts at source offset 24 and whose entries are contiguous.
• Rejects an unmarked file — e.g. one produced by plain zstd file.pcap — with a distinct "not our format" error.
• Rejects a marker with a bad version byte.
• Handles a truncated marker (fewer than 14 bytes on disk) as "need more", not as an error.
• Tails a growing file: a frame appended mid-poll is surfaced on a subsequent poll, and a torn trailing frame is not surfaced until complete.
• Whole-file zstd -d output equals the source .pcap byte for byte.
• Decompressing frame n alone yields exactly src_end - src_start bytes, and those bytes are a whole number of pcap records.
• A corrupted byte inside a frame surfaces as a decode error (checksum), not as packets.
• FileId computed from a .pcap.zst equals the FileId computed from the same capture stored as a raw .pcap.
• Rename: a .partial renamed mid-stream is detected; the file is finalized with the source size (not the compressed size); a consumer attached across the rename sees no interruption and no ENOENT; a new consumer attaching during the drain window opens the final path successfully.
• A file discovered already-final (no .partial) is treated as complete without waiting for a rename.