scpcap/STREAMING_ZSTD_FORMAT.md

# Streaming `.pcap.zst` Format

**Status:** implemented (Rust reference: `pcapstream-core`, `pcapstream-server`, `pcapstream-client`)
**Audience:** an implementer writing a C++ reader that streams pcap records out of a `.pcap.zst`
file *while it is still being written*.

---

## 1. What this format is and why it exists

A pcap-stream `.pcap.zst` is a **plain concatenation of independent zstd frames**:

```text
[skippable marker frame]   identity + version   (carries no pcap bytes)
[zstd frame]               compress(24-byte pcap global header)
[zstd frame]               compress(record-aligned chunk of pcap records)   \
[zstd frame]               compress(record-aligned chunk of pcap records)    |  repeats
[zstd frame]               ...                                              /
```

Two properties fall out of that, and both are load-bearing:

1. **It is a valid `.zst` file.** `zstd -d file.pcap.zst` concatenates the frame outputs and
   reproduces the original `.pcap` byte for byte. The marker is a zstd *skippable* frame, so
   compliant decoders ignore it. Nothing about this format requires a special tool to read
   offline.
2. **It is streamable and seekable at frame granularity.** Every data frame is cut on a pcap
   *record boundary* and declares its decompressed size in its header. A reader can therefore
   walk the file frame by frame, know exactly how many uncompressed pcap bytes each frame
   contributes, and decompress any single frame independently — without decompressing anything
   before it.

The point of the format is that a file written this way can be tailed as it grows and served (or
relayed) **without recompression**: each stored frame is already exactly the payload you'd put on
the wire.

### What the marker frame buys us

The marker is the only thing that distinguishes *our* file from an arbitrary `zstd some.pcap`
produced by the CLI. Both have the extension `.pcap.zst`; both decode to the same pcap. But a
CLI-produced file is typically **one enormous frame** whose boundaries have nothing to do with
pcap records, so none of the streaming properties hold. A reader that assumed frame boundaries
were record-aligned would hand out torn records.

So: **a file without a valid leading marker frame must be rejected**, not "best-efforted".

---

## 2. The marker frame (byte level)

A zstd skippable frame is `magic (4 LE) | payload_size (4 LE) | payload`. Skippable magics occupy
the range `0x184D2A50 .. 0x184D2A5F`; we write the base value.

| Offset | Size | Value | Meaning |
|---|---|---|---|
| 0 | 4 | `0x184D2A50` (LE) | zstd skippable-frame magic |
| 4 | 4 | `6` (LE) | payload size |
| 8 | 4 | `"PSZ1"` (`0x50 0x53 0x5A 0x31`) | pcap-stream zst marker magic |
| 12 | 1 | `1` | format version |
| 13 | 1 | `0` | flags (reserved; must be written as 0) |

Total: **14 bytes**, always at physical offset 0.

Hex, in full:

```
50 2A 4D 18   06 00 00 00   50 53 5A 31   01   00
```

Reader rules:

* The **first frame in the file must be a skippable frame whose payload is a valid marker**
  (magic `PSZ1`, version `1`). Anything else → reject the file (`NotMarked`).
* Validation reads only the first 4 payload bytes (magic) and the version byte. **Ignore the
  flags byte and any trailing payload bytes** — a future version may extend the payload, and the
  `payload_size` field lets you skip whatever you don't understand.
* An unknown *version* is a hard reject, not a warning. Version is bumped only on an
  incompatible container change.
* Skippable frames may in principle appear elsewhere in the file. Treat any skippable frame as
  contributing **zero source bytes** and step over it. (Today only the leading marker exists.)


---

## 3. Frame invariants a reader may rely on

These are guaranteed by every writer of this format. A file that violates them is malformed and
should be rejected rather than reinterpreted.

1. **Frame 0 (physical offset 0) is the marker.** See above.
2. **Frame 1 is the header frame:** it decompresses to exactly the **24-byte pcap global header**,
   and it covers source range `[0, 24)`. If its declared content size is not 24, the file is
   malformed. Parse it as a classic pcap global header (any of the four magics — LE/BE ×
   µs/ns; pcapng is rejected upstream).
3. **Every subsequent frame is a data frame** whose decompressed bytes are a whole number of
   complete pcap records — never a partial record header, never a partial payload. Concatenating
   all data frames' outputs yields the pcap body (everything after byte 24).
4. **Every frame declares its decompressed content size** in its zstd frame header (the writer
   compresses with the input size known, so the field is always present). A frame with an
   undeclared content size is malformed. *This is what lets a reader map physical → source
   offsets without decompressing anything.*
5. **Every frame carries a zstd content checksum.** Decompression verifies it, so single-bit
   corruption surfaces as a decode error rather than as corrupt packets.
6. **Frames are contiguous in source coordinates.** Frame *n*'s source range starts exactly where
   frame *n−1*'s ended. There are no gaps and no overlaps.
7. **Frames are append-only and immutable once written.** Bytes already on disk are never
   rewritten in place. The only exception is truncation of a torn trailing frame at writer
   restart (§7), which happens before any new bytes are appended and always cuts at a frame
   boundary.
8. Data frames target **~256 KiB of uncompressed source bytes** (`frame_target_bytes`, default
   262144) but may be smaller: a flush timer (default 100 ms) cuts a short frame to bound latency,
   and the last frame of a file is whatever is left. **Do not assume any frame size.** Size the
   read window generously (1 MiB is comfortable) and grow it if a single frame ever exceeds it.

Note that packet counts are **not** carried in the container. If you need them, count records
while decompressing.

---

## 4. Two coordinate systems

Keep these strictly separate; conflating them is the single easiest way to get this wrong.

* **Source offset** — a byte offset into the *decompressed* pcap. The global header occupies
  `[0, 24)`; the first data frame starts at 24. This is the coordinate space of the uncompressed pcap.
* **Physical offset** — a byte offset into the `.pcap.zst` on disk. Includes the 14-byte marker
  and all compressed frame bytes.

A frame table entry is therefore four numbers:

```cpp
struct ZstFrame {
    uint64_t src_start;   // source (uncompressed pcap) offset of first byte
    uint64_t src_end;     // exclusive; src_end - src_start == declared content size
    uint64_t phys_start;  // physical offset of the compressed frame
    uint64_t phys_len;    // compressed frame size on disk
};
```

`src_end - src_start` comes from `ZSTD_getFrameContentSize`; `phys_len` comes from
`ZSTD_findFrameCompressedSize`. Neither requires decompressing the frame — which is the whole
trick: you can build the frame table for a multi-GB file by reading only frame headers, and you
only decompress the frames a consumer actually asks for.

The marker frame (and any future skippable frame) advances `phys` and **not** `src`.


---

## 5. Walking the frames

### 5.1 The two zstd calls you need

| Purpose | libzstd (C) |
|---|---|
| Skippable magic test | `(magic & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START` |
| Compressed size of the frame at `p` | `ZSTD_findFrameCompressedSize(p, n)` |
| Declared decompressed size | `ZSTD_getFrameContentSize(p, n)` |
| Decompress one frame | `ZSTD_decompress(dst, dstCap, p, csize)` |

Both size calls are **header-only** reads. Important subtleties:

* `ZSTD_findFrameCompressedSize` returns an error code (test with `ZSTD_isError`) when the buffer
  doesn't contain the whole frame — but it returns the *same* kind of error for a genuinely
  corrupt frame. **You cannot distinguish "need more bytes" from "corrupt" by the error alone.**
  Disambiguate with position: if you have read to end-of-file, treat it as a torn trailing frame
  (stop, retry later); if not, grow your window and retry. This is exactly what the reference
  implementation does.
* `ZSTD_findFrameCompressedSize` may also return a size **larger than the buffer you passed**.
  Guard with `csize <= n` and treat a larger value as "need more bytes".
* `ZSTD_getFrameContentSize` returns `ZSTD_CONTENTSIZE_UNKNOWN` or `ZSTD_CONTENTSIZE_ERROR` as
  sentinels. For this format, either is **malformed** (invariant 4) — do not fall back to
  decompressing to learn the size.
* Skippable frames have no content size. Compute their length yourself: `8 + payload_size`, where
  `payload_size` is the LE `uint32` at offset 4.

### 5.2 The walk

State: `phys_pos` (next physical byte to parse — always a frame boundary), `src_pos` (next source
offset — the source frontier), plus `saw_marker`, `header`, `file_id`.

```
loop until phys_pos == file_size (fstat each poll; the file grows under you):
    read a window at phys_pos  (1 MiB; short reads are normal — retry/accumulate)
    at_eof := (bytes_read == file_size - phys_pos)

    if window starts with a skippable magic:
        if the full skippable frame isn't in the window:
            if !at_eof: grow window, retry
            else:       stop (torn trailing skippable) — retry on a later poll
        if phys_pos == 0: it must be a valid marker, else reject NotMarked
        phys_pos += 8 + payload_size          # src_pos unchanged
        continue

    if !saw_marker: reject NotMarked          # a data frame before any marker

    csize := ZSTD_findFrameCompressedSize(window)
    if error or csize > bytes_read:
        if at_eof: stop (torn trailing frame) — retry on a later poll
        else:      grow window, retry

    content := ZSTD_getFrameContentSize(window[0..csize])
    if unknown or error: reject Malformed

    if header not yet seen:                   # this is the header frame
        if content != 24: reject Malformed
        header := parse_pcap_global_header(decompress(window[0..csize]))
        src_pos  += 24                        # -> 24
        phys_pos += csize
        continue

    emit ZstFrame{ src_start: src_pos, src_end: src_pos + content,
                   phys_start: phys_pos, phys_len: csize }
    src_pos  += content
    phys_pos += csize
```

Key points:

* **A torn trailing frame is normal, not an error.** While the writer is mid-append you will
  routinely see a partial frame at EOF. Stop the walk *before* it, leave `phys_pos` pointing at
  its start, and pick it up on a later poll. Never surface a frame you haven't fully read.
* **`phys_pos` only ever advances across complete frames**, so it is always a valid resume point.
* Cap the work per poll (the reference consumes ~8 MiB of physical bytes per call, rounded up to
  the frame boundary crossed) so that catching up on a large backlog doesn't stall your event
  loop.
* Everything above is `pread`-based on a **held-open fd** — see §7.3 for why that matters.

To read the packets for a frame: `pread` `phys_len` bytes at `phys_start`, `ZSTD_decompress` into
a buffer of `src_end - src_start` bytes, and you have a run of complete pcap records. To relay the
frame to a downstream zstd-speaking consumer: send those `phys_len` bytes **verbatim**. No
decompress, no recompress.

---

## 6. File identity (`FileId`)

Filenames are not unique — a capture restart can recreate the same name with different content —
so files are identified by content, not by path.

```
FileId = { first_packet_ts : u64,   // nanoseconds
           header_crc32    : u32 }
```

* `header_crc32` = **CRC-32 (IEEE, the zlib/`crc32fast` polynomial, init 0xFFFFFFFF, reflected,
  final xor)** over the 40 bytes `global_header[0..24] || first_record_header[0..16]` — i.e. over
  the *decompressed* source bytes `[0, 40)`.
* `first_packet_ts` = the first record's timestamp normalized to nanoseconds: `ts_sec * 1e9 +
  ts_frac * 1000` for a µs-resolution file (magic `0xA1B2C3D4` / `0xD4C3B2A1`), or `ts_sec * 1e9 +
  ts_frac` for a ns-resolution file (magic `0xA1B23C4D` / `0x4D3CB2A1`). Fields are read in the
  file's endianness.
* All-zeros is the **provisional** id: the file exists and has a header but no first packet yet.
  Recompute once the first data frame lands.

For a `.pcap.zst`, this means: decompress the header frame (24 bytes) and just enough of the
**first data frame** to get its first 16 bytes, concatenate, CRC.


---

## 7. Detecting completion: the `.partial` rename

**This is the part that took the most iteration. Read this section carefully — the wrong approach
looks like it works and then loses data or reports the wrong file size in production.**

### 7.1 The scheme

A file being written exists on disk under a name with an in-progress suffix, and is **atomically
renamed** to its final name when — and only when — it is complete:

```
20260714/cap-20260714-14:30:00.pcap.zst.partial     ← still growing
                    │
                    │  rename(2)  — atomic, same directory
                    ▼
20260714/cap-20260714-14:30:00.pcap.zst             ← complete, immutable
```

The suffix is configurable (`partial_suffix`, default `.partial`) and matches SolarCapture's own
convention. The four physical forms a reader must recognize are:

```
<name>.pcap                <name>.pcap.partial
<name>.pcap.zst            <name>.pcap.zst.partial
```

Strip the partial suffix **first**, then a `.zst` suffix, to recover the **logical name**
(`<name>.pcap`). The logical name is what appears in the protocol, in resume tokens, and in your
own bookkeeping — it is *stable across the rename*, which is the whole point. Two flags fall out
of the stripping: `partial` (is it still growing?) and `zst` (is it compressed?).


### 7.2 The rule

> **The rename is the completion signal. Nothing else is.**

Concretely, a reader declares a file complete when **both** of these hold:

1. **The rename has been observed** — the final path exists.
2. **The file is fully drained** — you have walked every byte of it: `phys_pos == file_size`
   (fresh `fstat`), i.e. no torn trailing frame remains.

Only then do you know the final size, which for a `.pcap.zst` is the **source frontier**
(`src_pos`), *not* the physical file size. (For a raw `.pcap` the two coincide, which is a trap if
you generalize from the raw path.)

If (1) holds but (2) doesn't — the rename landed but there's still an unframed tail — **wait**.
Do not declare completion with a size you'd have to revise. Poll again; the tail is already fully
written, so this resolves on the next tick.

### 7.3 What NOT to do (each of these was tried and removed)

* ❌ **Do not use a quiet period / size-stability heuristic** ("no growth for N seconds ⇒ done").
  A capture stall is indistinguishable from a finished file, and it fires early under load. This
  was ripped out in favor of the rename signal alone.
* ❌ **Do not treat the appearance of the *next* file as the completion signal for the current
  one.** Same commit, same reason: it couples two files' lifecycles and it's wrong at a capture
  restart.
* ❌ **Do not re-open the file by path when you notice the rename.** The reader must keep serving
  from the **fd it already holds**. A rename does not invalidate an open fd — it still points at
  the same inode, and reads continue to work through the rename *and even through an unlink*. If
  you close and reopen, you introduce a window where the `.partial` path is gone and the reader
  ENOENTs, and you break continuity for any consumer mid-file.
* ❌ **Do not assume the on-disk file size is the source size for a `.zst`.** It's the compressed
  size. Track the source frontier separately.
* ❌ **Do not re-scan and re-register the renamed file as a new file.** A directory scan after the
  rename sees a *new* directory entry (the final name). Because the logical name is
  suffix-stripped, it maps to the file you already know — dedupe on the **logical name**, not the
  physical path.

### 7.4 The one subtlety that bites: the ENOENT window

There is a gap between "the rename happened" and "we have finished draining and finalized the
file". That gap can be seconds (framing the tail, serving a backlog). During that gap:

* the `.partial` path **no longer exists** (rename is atomic — the old name is gone the instant
  the new one appears), but
* your in-memory state may still be pointing at the `.partial` path for any *new* open.

Anything that opens the file by path in that window — a lazily-rebuilt frame, a new consumer
attaching, a cache miss — gets **ENOENT**, on a file that is perfectly healthy.

The fix (commit `4c5a0e8`) is: **the moment you observe the rename, repoint your stored serve
path at the final path**, immediately — long before you finalize the file. Keep draining through
the fd you already hold (which is unaffected), but make sure any *new* open uses the final name.

```
on scan tick, for each growing file:
    if is_partial && !complete_signal && exists(final_path):
        complete_signal = true
        serve_path = final_path        # ← do this NOW, not at finalization
        # keep tailing via the already-open fd; finalize once drained (§7.2)
```

That ordering — signal and repoint first, drain second, finalize third — is the whole rename
protocol. Get it wrong and you get intermittent ENOENTs under exactly the conditions that are
hardest to reproduce (a consumer reconnecting during the rename).

### 7.5 Detecting the rename

Poll: on each scan, for every file you are tailing that is still `partial`, `stat` its final path.
Existence of the final path = the rename happened. This is cheap (one `stat` per growing file, and
there is normally exactly one) and needs no inotify. The reference implementation drives this on a
1 s timer; a caller-driven scan loop can poll as often as it likes.

If you prefer inotify, watch for `IN_MOVED_TO` in the capture directory — but keep the `stat`
fallback: inotify queues overflow, and missing a rename means never completing a file.

### 7.6 Files discovered already-final

A file discovered with **no** partial suffix (final name, no `.partial` sibling) is **born
complete** — it finished before you ever saw it. Register it as complete immediately. Don't wait
for a rename that already happened.

A reader that builds a frame table (§4) to serve random-access requests should build it lazily,
the first time a consumer actually asks for the file — walking/framing every already-final file at
discovery reads every byte of every file in a directory of a day's captures for nothing. A reader
that just streams packets out sequentially (walking §5.2 once, start to finish, with no frame
table) doesn't have this concern either way.

### 7.7 Writer restart and truncation (reader's perspective)

You never write these files, but the process that does can restart mid-file. On restart it walks
its own `.partial`, and truncates it back to **a frame boundary** — never mid-frame — before
resuming appends.

The guarantee that matters to a reader: **content at a given source offset never changes.** A
restart can only shrink the file back to an earlier frame boundary and re-append from there; the
capture data being re-appended is the same data, so whatever lands at a given source offset is
byte-for-byte identical to what would have been there without the restart. Only the physical
framing (where frame boundaries fall) can differ across a restart — never the decompressed
content.

Two cases follow from that:

* **Ordinary case — the truncated bytes were never surfaced.** A writer only ever truncates a
  *torn* trailing frame (invariant 7 in §3), and a reader following §5.2 never surfaces a torn
  trailing frame in the first place — it stops before it and waits. So `phys_pos` already sits at
  or before the truncation point. There is nothing to undo: the next poll simply finds the file
  has grown again (with corrected content) from the same boundary you were already parked at, and
  the walk continues exactly as if nothing happened.
* **Defensive case — the file shrinks below `phys_pos`.** Check this on every poll with a fresh
  `fstat`, the same as you already do to detect EOF: if `file_size < phys_pos`, trust the frames
  you have already read — they decoded and checksummed cleanly, so they're not in question — and
  simply stop advancing. Wait, polling as usual, until `file_size` grows back past `phys_pos`
  again, then resume the walk from `phys_pos` exactly as before. Because content per source offset
  is guaranteed stable, whatever now occupies the bytes at and beyond `phys_pos` is the correct
  continuation, so no rewind or re-parse of already-consumed frames is needed.

---

## 8. Error handling

| Condition | Action |
|---|---|
| First frame is not a valid marker | **Reject** the file (`NotMarked`). It's a foreign `.zst` whose frames don't align to records. Log the path; do not attempt to stream it. |
| Marker version unknown | **Reject.** |
| Header frame's content size ≠ 24 | **Reject** (`Malformed`). |
| Any frame with undeclared/error content size | **Reject** (`Malformed`). |
| Torn frame at EOF | **Not an error.** Stop before it; retry on the next poll. |
| zstd decode error / checksum failure | **Reject and alarm.** Never resync by scanning forward for a plausible frame magic — you'd hand out garbage records. Stop tailing this file. |
| `incl_len` > 256 KiB (`MAX_INCL_LEN`) or > snaplen | Treat as corruption; stop and alarm, don't resync. (Relevant when validating decompressed records.) |

The consistent principle: **fail loudly, never resync.** A file that violates an invariant is
either corrupt or not ours, and silently guessing at frame boundaries is worse than stopping.

---

## 9. Constants

| Name | Value |
|---|---|
| `SKIPPABLE_MAGIC` | `0x184D2A50` (range `0x184D2A50..=0x184D2A5F`) |
| `MARKER_MAGIC` | `"PSZ1"` |
| `MARKER_VERSION` | `1` |
| Marker frame length | 14 bytes |
| `GLOBAL_HEADER_LEN` | 24 |
| `RECORD_HEADER_LEN` | 16 |
| `MAX_INCL_LEN` | 262144 (256 KiB) |
| `frame_target_bytes` (default) | 262144 (~256 KiB of *source* bytes) |
| `frame_flush_ms` (default) | 100 (may cut a short frame) |
| `zstd_level` (default) | 2 (writer-side; irrelevant to a reader) |
| `partial_suffix` (default) | `.partial` |
| Read window | 1 MiB, grown (×2) if a single frame exceeds it |
| Scan interval (reference default; caller-driven elsewhere) | 1000 ms |
| Poll interval (reference default; caller-driven elsewhere) | 20 ms |

---

## 10. Conformance checklist

A C++ reader is done when it passes these.

- [ ] Walks a marked file (marker + header frame + N data frames) and produces a frame table whose
      first entry starts at source offset 24 and whose entries are contiguous.
- [ ] Rejects an unmarked file — e.g. one produced by plain `zstd file.pcap` — with a distinct
      "not our format" error.
- [ ] Rejects a marker with a bad version byte.
- [ ] Handles a truncated marker (fewer than 14 bytes on disk) as "need more", not as an error.
- [ ] Tails a growing file: a frame appended mid-poll is surfaced on a subsequent poll, and a torn
      trailing frame is *not* surfaced until complete.
- [ ] Whole-file `zstd -d` output equals the source `.pcap` byte for byte.
- [ ] Decompressing frame *n* alone yields exactly `src_end - src_start` bytes, and those bytes are
      a whole number of pcap records.
- [ ] A corrupted byte inside a frame surfaces as a decode error (checksum), not as packets.
- [ ] `FileId` computed from a `.pcap.zst` equals the `FileId` computed from the same capture
      stored as a raw `.pcap`.
- [ ] **Rename:** a `.partial` renamed mid-stream is detected; the file is finalized with the
      **source** size (not the compressed size); a consumer attached across the rename sees no
      interruption and no ENOENT; a new consumer attaching *during* the drain window opens the
      final path successfully.
- [ ] A file discovered already-final (no `.partial`) is treated as complete without waiting for a
      rename.