diff --git a/STREAMING_ZSTD_FORMAT.md b/STREAMING_ZSTD_FORMAT.md new file mode 100644 index 0000000..e02938b --- /dev/null +++ b/STREAMING_ZSTD_FORMAT.md @@ -0,0 +1,475 @@ +# Streaming `.pcap.zst` Format + +**Status:** implemented (Rust reference: `pcapstream-core`, `pcapstream-server`, `pcapstream-client`) +**Audience:** an implementer writing a C++ reader that streams pcap records out of a `.pcap.zst` +file *while it is still being written*. + +--- + +## 1. What this format is and why it exists + +A pcap-stream `.pcap.zst` is a **plain concatenation of independent zstd frames**: + +```text +[skippable marker frame] identity + version (carries no pcap bytes) +[zstd frame] compress(24-byte pcap global header) +[zstd frame] compress(record-aligned chunk of pcap records) \ +[zstd frame] compress(record-aligned chunk of pcap records) | repeats +[zstd frame] ... / +``` + +Two properties fall out of that, and both are load-bearing: + +1. **It is a valid `.zst` file.** `zstd -d file.pcap.zst` concatenates the frame outputs and + reproduces the original `.pcap` byte for byte. The marker is a zstd *skippable* frame, so + compliant decoders ignore it. Nothing about this format requires a special tool to read + offline. +2. **It is streamable and seekable at frame granularity.** Every data frame is cut on a pcap + *record boundary* and declares its decompressed size in its header. A reader can therefore + walk the file frame by frame, know exactly how many uncompressed pcap bytes each frame + contributes, and decompress any single frame independently — without decompressing anything + before it. + +The point of the format is that a file written this way can be tailed as it grows and served (or +relayed) **without recompression**: each stored frame is already exactly the payload you'd put on +the wire. + +### What the marker frame buys us + +The marker is the only thing that distinguishes *our* file from an arbitrary `zstd some.pcap` +produced by the CLI. Both have the extension `.pcap.zst`; both decode to the same pcap. But a +CLI-produced file is typically **one enormous frame** whose boundaries have nothing to do with +pcap records, so none of the streaming properties hold. A reader that assumed frame boundaries +were record-aligned would hand out torn records. + +So: **a file without a valid leading marker frame must be rejected**, not "best-efforted". + +--- + +## 2. The marker frame (byte level) + +A zstd skippable frame is `magic (4 LE) | payload_size (4 LE) | payload`. Skippable magics occupy +the range `0x184D2A50 .. 0x184D2A5F`; we write the base value. + +| Offset | Size | Value | Meaning | +|---|---|---|---| +| 0 | 4 | `0x184D2A50` (LE) | zstd skippable-frame magic | +| 4 | 4 | `6` (LE) | payload size | +| 8 | 4 | `"PSZ1"` (`0x50 0x53 0x5A 0x31`) | pcap-stream zst marker magic | +| 12 | 1 | `1` | format version | +| 13 | 1 | `0` | flags (reserved; must be written as 0) | + +Total: **14 bytes**, always at physical offset 0. + +Hex, in full: + +``` +50 2A 4D 18 06 00 00 00 50 53 5A 31 01 00 +``` + +Reader rules: + +* The **first frame in the file must be a skippable frame whose payload is a valid marker** + (magic `PSZ1`, version `1`). Anything else → reject the file (`NotMarked`). +* Validation reads only the first 4 payload bytes (magic) and the version byte. **Ignore the + flags byte and any trailing payload bytes** — a future version may extend the payload, and the + `payload_size` field lets you skip whatever you don't understand. +* An unknown *version* is a hard reject, not a warning. Version is bumped only on an + incompatible container change. +* Skippable frames may in principle appear elsewhere in the file. Treat any skippable frame as + contributing **zero source bytes** and step over it. (Today only the leading marker exists.) + + +--- + +## 3. Frame invariants a reader may rely on + +These are guaranteed by every writer of this format. A file that violates them is malformed and +should be rejected rather than reinterpreted. + +1. **Frame 0 (physical offset 0) is the marker.** See above. +2. **Frame 1 is the header frame:** it decompresses to exactly the **24-byte pcap global header**, + and it covers source range `[0, 24)`. If its declared content size is not 24, the file is + malformed. Parse it as a classic pcap global header (any of the four magics — LE/BE × + µs/ns; pcapng is rejected upstream). +3. **Every subsequent frame is a data frame** whose decompressed bytes are a whole number of + complete pcap records — never a partial record header, never a partial payload. Concatenating + all data frames' outputs yields the pcap body (everything after byte 24). +4. **Every frame declares its decompressed content size** in its zstd frame header (the writer + compresses with the input size known, so the field is always present). A frame with an + undeclared content size is malformed. *This is what lets a reader map physical → source + offsets without decompressing anything.* +5. **Every frame carries a zstd content checksum.** Decompression verifies it, so single-bit + corruption surfaces as a decode error rather than as corrupt packets. +6. **Frames are contiguous in source coordinates.** Frame *n*'s source range starts exactly where + frame *n−1*'s ended. There are no gaps and no overlaps. +7. **Frames are append-only and immutable once written.** Bytes already on disk are never + rewritten in place. The only exception is truncation of a torn trailing frame at writer + restart (§7), which happens before any new bytes are appended and always cuts at a frame + boundary. +8. Data frames target **~256 KiB of uncompressed source bytes** (`frame_target_bytes`, default + 262144) but may be smaller: a flush timer (default 100 ms) cuts a short frame to bound latency, + and the last frame of a file is whatever is left. **Do not assume any frame size.** Size the + read window generously (1 MiB is comfortable) and grow it if a single frame ever exceeds it. + +Note that packet counts are **not** carried in the container. If you need them, count records +while decompressing. + +--- + +## 4. Two coordinate systems + +Keep these strictly separate; conflating them is the single easiest way to get this wrong. + +* **Source offset** — a byte offset into the *decompressed* pcap. The global header occupies + `[0, 24)`; the first data frame starts at 24. This is the coordinate space of the uncompressed pcap. +* **Physical offset** — a byte offset into the `.pcap.zst` on disk. Includes the 14-byte marker + and all compressed frame bytes. + +A frame table entry is therefore four numbers: + +```cpp +struct ZstFrame { + uint64_t src_start; // source (uncompressed pcap) offset of first byte + uint64_t src_end; // exclusive; src_end - src_start == declared content size + uint64_t phys_start; // physical offset of the compressed frame + uint64_t phys_len; // compressed frame size on disk +}; +``` + +`src_end - src_start` comes from `ZSTD_getFrameContentSize`; `phys_len` comes from +`ZSTD_findFrameCompressedSize`. Neither requires decompressing the frame — which is the whole +trick: you can build the frame table for a multi-GB file by reading only frame headers, and you +only decompress the frames a consumer actually asks for. + +The marker frame (and any future skippable frame) advances `phys` and **not** `src`. + + +--- + +## 5. Walking the frames + +### 5.1 The two zstd calls you need + +| Purpose | libzstd (C) | +|---|---| +| Skippable magic test | `(magic & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START` | +| Compressed size of the frame at `p` | `ZSTD_findFrameCompressedSize(p, n)` | +| Declared decompressed size | `ZSTD_getFrameContentSize(p, n)` | +| Decompress one frame | `ZSTD_decompress(dst, dstCap, p, csize)` | + +Both size calls are **header-only** reads. Important subtleties: + +* `ZSTD_findFrameCompressedSize` returns an error code (test with `ZSTD_isError`) when the buffer + doesn't contain the whole frame — but it returns the *same* kind of error for a genuinely + corrupt frame. **You cannot distinguish "need more bytes" from "corrupt" by the error alone.** + Disambiguate with position: if you have read to end-of-file, treat it as a torn trailing frame + (stop, retry later); if not, grow your window and retry. This is exactly what the reference + implementation does. +* `ZSTD_findFrameCompressedSize` may also return a size **larger than the buffer you passed**. + Guard with `csize <= n` and treat a larger value as "need more bytes". +* `ZSTD_getFrameContentSize` returns `ZSTD_CONTENTSIZE_UNKNOWN` or `ZSTD_CONTENTSIZE_ERROR` as + sentinels. For this format, either is **malformed** (invariant 4) — do not fall back to + decompressing to learn the size. +* Skippable frames have no content size. Compute their length yourself: `8 + payload_size`, where + `payload_size` is the LE `uint32` at offset 4. + +### 5.2 The walk + +State: `phys_pos` (next physical byte to parse — always a frame boundary), `src_pos` (next source +offset — the source frontier), plus `saw_marker`, `header`, `file_id`. + +``` +loop until phys_pos == file_size (fstat each poll; the file grows under you): + read a window at phys_pos (1 MiB; short reads are normal — retry/accumulate) + at_eof := (bytes_read == file_size - phys_pos) + + if window starts with a skippable magic: + if the full skippable frame isn't in the window: + if !at_eof: grow window, retry + else: stop (torn trailing skippable) — retry on a later poll + if phys_pos == 0: it must be a valid marker, else reject NotMarked + phys_pos += 8 + payload_size # src_pos unchanged + continue + + if !saw_marker: reject NotMarked # a data frame before any marker + + csize := ZSTD_findFrameCompressedSize(window) + if error or csize > bytes_read: + if at_eof: stop (torn trailing frame) — retry on a later poll + else: grow window, retry + + content := ZSTD_getFrameContentSize(window[0..csize]) + if unknown or error: reject Malformed + + if header not yet seen: # this is the header frame + if content != 24: reject Malformed + header := parse_pcap_global_header(decompress(window[0..csize])) + src_pos += 24 # -> 24 + phys_pos += csize + continue + + emit ZstFrame{ src_start: src_pos, src_end: src_pos + content, + phys_start: phys_pos, phys_len: csize } + src_pos += content + phys_pos += csize +``` + +Key points: + +* **A torn trailing frame is normal, not an error.** While the writer is mid-append you will + routinely see a partial frame at EOF. Stop the walk *before* it, leave `phys_pos` pointing at + its start, and pick it up on a later poll. Never surface a frame you haven't fully read. +* **`phys_pos` only ever advances across complete frames**, so it is always a valid resume point. +* Cap the work per poll (the reference consumes ~8 MiB of physical bytes per call, rounded up to + the frame boundary crossed) so that catching up on a large backlog doesn't stall your event + loop. +* Everything above is `pread`-based on a **held-open fd** — see §7.3 for why that matters. + +To read the packets for a frame: `pread` `phys_len` bytes at `phys_start`, `ZSTD_decompress` into +a buffer of `src_end - src_start` bytes, and you have a run of complete pcap records. To relay the +frame to a downstream zstd-speaking consumer: send those `phys_len` bytes **verbatim**. No +decompress, no recompress. + +--- + +## 6. File identity (`FileId`) + +Filenames are not unique — a capture restart can recreate the same name with different content — +so files are identified by content, not by path. + +``` +FileId = { first_packet_ts : u64, // nanoseconds + header_crc32 : u32 } +``` + +* `header_crc32` = **CRC-32 (IEEE, the zlib/`crc32fast` polynomial, init 0xFFFFFFFF, reflected, + final xor)** over the 40 bytes `global_header[0..24] || first_record_header[0..16]` — i.e. over + the *decompressed* source bytes `[0, 40)`. +* `first_packet_ts` = the first record's timestamp normalized to nanoseconds: `ts_sec * 1e9 + + ts_frac * 1000` for a µs-resolution file (magic `0xA1B2C3D4` / `0xD4C3B2A1`), or `ts_sec * 1e9 + + ts_frac` for a ns-resolution file (magic `0xA1B23C4D` / `0x4D3CB2A1`). Fields are read in the + file's endianness. +* All-zeros is the **provisional** id: the file exists and has a header but no first packet yet. + Recompute once the first data frame lands. + +For a `.pcap.zst`, this means: decompress the header frame (24 bytes) and just enough of the +**first data frame** to get its first 16 bytes, concatenate, CRC. + + +--- + +## 7. Detecting completion: the `.partial` rename + +**This is the part that took the most iteration. Read this section carefully — the wrong approach +looks like it works and then loses data or reports the wrong file size in production.** + +### 7.1 The scheme + +A file being written exists on disk under a name with an in-progress suffix, and is **atomically +renamed** to its final name when — and only when — it is complete: + +``` +20260714/cap-20260714-14:30:00.pcap.zst.partial ← still growing + │ + │ rename(2) — atomic, same directory + ▼ +20260714/cap-20260714-14:30:00.pcap.zst ← complete, immutable +``` + +The suffix is configurable (`partial_suffix`, default `.partial`) and matches SolarCapture's own +convention. The four physical forms a reader must recognize are: + +``` +.pcap .pcap.partial +.pcap.zst .pcap.zst.partial +``` + +Strip the partial suffix **first**, then a `.zst` suffix, to recover the **logical name** +(`.pcap`). The logical name is what appears in the protocol, in resume tokens, and in your +own bookkeeping — it is *stable across the rename*, which is the whole point. Two flags fall out +of the stripping: `partial` (is it still growing?) and `zst` (is it compressed?). + + +### 7.2 The rule + +> **The rename is the completion signal. Nothing else is.** + +Concretely, a reader declares a file complete when **both** of these hold: + +1. **The rename has been observed** — the final path exists. +2. **The file is fully drained** — you have walked every byte of it: `phys_pos == file_size` + (fresh `fstat`), i.e. no torn trailing frame remains. + +Only then do you know the final size, which for a `.pcap.zst` is the **source frontier** +(`src_pos`), *not* the physical file size. (For a raw `.pcap` the two coincide, which is a trap if +you generalize from the raw path.) + +If (1) holds but (2) doesn't — the rename landed but there's still an unframed tail — **wait**. +Do not declare completion with a size you'd have to revise. Poll again; the tail is already fully +written, so this resolves on the next tick. + +### 7.3 What NOT to do (each of these was tried and removed) + +* ❌ **Do not use a quiet period / size-stability heuristic** ("no growth for N seconds ⇒ done"). + A capture stall is indistinguishable from a finished file, and it fires early under load. This + was ripped out in favor of the rename signal alone. +* ❌ **Do not treat the appearance of the *next* file as the completion signal for the current + one.** Same commit, same reason: it couples two files' lifecycles and it's wrong at a capture + restart. +* ❌ **Do not re-open the file by path when you notice the rename.** The reader must keep serving + from the **fd it already holds**. A rename does not invalidate an open fd — it still points at + the same inode, and reads continue to work through the rename *and even through an unlink*. If + you close and reopen, you introduce a window where the `.partial` path is gone and the reader + ENOENTs, and you break continuity for any consumer mid-file. +* ❌ **Do not assume the on-disk file size is the source size for a `.zst`.** It's the compressed + size. Track the source frontier separately. +* ❌ **Do not re-scan and re-register the renamed file as a new file.** A directory scan after the + rename sees a *new* directory entry (the final name). Because the logical name is + suffix-stripped, it maps to the file you already know — dedupe on the **logical name**, not the + physical path. + +### 7.4 The one subtlety that bites: the ENOENT window + +There is a gap between "the rename happened" and "we have finished draining and finalized the +file". That gap can be seconds (framing the tail, serving a backlog). During that gap: + +* the `.partial` path **no longer exists** (rename is atomic — the old name is gone the instant + the new one appears), but +* your in-memory state may still be pointing at the `.partial` path for any *new* open. + +Anything that opens the file by path in that window — a lazily-rebuilt frame, a new consumer +attaching, a cache miss — gets **ENOENT**, on a file that is perfectly healthy. + +The fix (commit `4c5a0e8`) is: **the moment you observe the rename, repoint your stored serve +path at the final path**, immediately — long before you finalize the file. Keep draining through +the fd you already hold (which is unaffected), but make sure any *new* open uses the final name. + +``` +on scan tick, for each growing file: + if is_partial && !complete_signal && exists(final_path): + complete_signal = true + serve_path = final_path # ← do this NOW, not at finalization + # keep tailing via the already-open fd; finalize once drained (§7.2) +``` + +That ordering — signal and repoint first, drain second, finalize third — is the whole rename +protocol. Get it wrong and you get intermittent ENOENTs under exactly the conditions that are +hardest to reproduce (a consumer reconnecting during the rename). + +### 7.5 Detecting the rename + +Poll: on each scan, for every file you are tailing that is still `partial`, `stat` its final path. +Existence of the final path = the rename happened. This is cheap (one `stat` per growing file, and +there is normally exactly one) and needs no inotify. The reference implementation drives this on a +1 s timer; a caller-driven scan loop can poll as often as it likes. + +If you prefer inotify, watch for `IN_MOVED_TO` in the capture directory — but keep the `stat` +fallback: inotify queues overflow, and missing a rename means never completing a file. + +### 7.6 Files discovered already-final + +A file discovered with **no** partial suffix (final name, no `.partial` sibling) is **born +complete** — it finished before you ever saw it. Register it as complete immediately. Don't wait +for a rename that already happened. + +A reader that builds a frame table (§4) to serve random-access requests should build it lazily, +the first time a consumer actually asks for the file — walking/framing every already-final file at +discovery reads every byte of every file in a directory of a day's captures for nothing. A reader +that just streams packets out sequentially (walking §5.2 once, start to finish, with no frame +table) doesn't have this concern either way. + +### 7.7 Writer restart and truncation (reader's perspective) + +You never write these files, but the process that does can restart mid-file. On restart it walks +its own `.partial`, and truncates it back to **a frame boundary** — never mid-frame — before +resuming appends. + +The guarantee that matters to a reader: **content at a given source offset never changes.** A +restart can only shrink the file back to an earlier frame boundary and re-append from there; the +capture data being re-appended is the same data, so whatever lands at a given source offset is +byte-for-byte identical to what would have been there without the restart. Only the physical +framing (where frame boundaries fall) can differ across a restart — never the decompressed +content. + +Two cases follow from that: + +* **Ordinary case — the truncated bytes were never surfaced.** A writer only ever truncates a + *torn* trailing frame (invariant 7 in §3), and a reader following §5.2 never surfaces a torn + trailing frame in the first place — it stops before it and waits. So `phys_pos` already sits at + or before the truncation point. There is nothing to undo: the next poll simply finds the file + has grown again (with corrected content) from the same boundary you were already parked at, and + the walk continues exactly as if nothing happened. +* **Defensive case — the file shrinks below `phys_pos`.** Check this on every poll with a fresh + `fstat`, the same as you already do to detect EOF: if `file_size < phys_pos`, trust the frames + you have already read — they decoded and checksummed cleanly, so they're not in question — and + simply stop advancing. Wait, polling as usual, until `file_size` grows back past `phys_pos` + again, then resume the walk from `phys_pos` exactly as before. Because content per source offset + is guaranteed stable, whatever now occupies the bytes at and beyond `phys_pos` is the correct + continuation, so no rewind or re-parse of already-consumed frames is needed. + +--- + +## 8. Error handling + +| Condition | Action | +|---|---| +| First frame is not a valid marker | **Reject** the file (`NotMarked`). It's a foreign `.zst` whose frames don't align to records. Log the path; do not attempt to stream it. | +| Marker version unknown | **Reject.** | +| Header frame's content size ≠ 24 | **Reject** (`Malformed`). | +| Any frame with undeclared/error content size | **Reject** (`Malformed`). | +| Torn frame at EOF | **Not an error.** Stop before it; retry on the next poll. | +| zstd decode error / checksum failure | **Reject and alarm.** Never resync by scanning forward for a plausible frame magic — you'd hand out garbage records. Stop tailing this file. | +| `incl_len` > 256 KiB (`MAX_INCL_LEN`) or > snaplen | Treat as corruption; stop and alarm, don't resync. (Relevant when validating decompressed records.) | + +The consistent principle: **fail loudly, never resync.** A file that violates an invariant is +either corrupt or not ours, and silently guessing at frame boundaries is worse than stopping. + +--- + +## 9. Constants + +| Name | Value | +|---|---| +| `SKIPPABLE_MAGIC` | `0x184D2A50` (range `0x184D2A50..=0x184D2A5F`) | +| `MARKER_MAGIC` | `"PSZ1"` | +| `MARKER_VERSION` | `1` | +| Marker frame length | 14 bytes | +| `GLOBAL_HEADER_LEN` | 24 | +| `RECORD_HEADER_LEN` | 16 | +| `MAX_INCL_LEN` | 262144 (256 KiB) | +| `frame_target_bytes` (default) | 262144 (~256 KiB of *source* bytes) | +| `frame_flush_ms` (default) | 100 (may cut a short frame) | +| `zstd_level` (default) | 2 (writer-side; irrelevant to a reader) | +| `partial_suffix` (default) | `.partial` | +| Read window | 1 MiB, grown (×2) if a single frame exceeds it | +| Scan interval (reference default; caller-driven elsewhere) | 1000 ms | +| Poll interval (reference default; caller-driven elsewhere) | 20 ms | + +--- + +## 10. Conformance checklist + +A C++ reader is done when it passes these. + +- [ ] Walks a marked file (marker + header frame + N data frames) and produces a frame table whose + first entry starts at source offset 24 and whose entries are contiguous. +- [ ] Rejects an unmarked file — e.g. one produced by plain `zstd file.pcap` — with a distinct + "not our format" error. +- [ ] Rejects a marker with a bad version byte. +- [ ] Handles a truncated marker (fewer than 14 bytes on disk) as "need more", not as an error. +- [ ] Tails a growing file: a frame appended mid-poll is surfaced on a subsequent poll, and a torn + trailing frame is *not* surfaced until complete. +- [ ] Whole-file `zstd -d` output equals the source `.pcap` byte for byte. +- [ ] Decompressing frame *n* alone yields exactly `src_end - src_start` bytes, and those bytes are + a whole number of pcap records. +- [ ] A corrupted byte inside a frame surfaces as a decode error (checksum), not as packets. +- [ ] `FileId` computed from a `.pcap.zst` equals the `FileId` computed from the same capture + stored as a raw `.pcap`. +- [ ] **Rename:** a `.partial` renamed mid-stream is detected; the file is finalized with the + **source** size (not the compressed size); a consumer attached across the rename sees no + interruption and no ENOENT; a new consumer attaching *during* the drain window opens the + final path successfully. +- [ ] A file discovered already-final (no `.partial`) is treated as complete without waiting for a + rename. +