VoxelForge/OPSTACK-HANDOFF.md

# Handoff — VoxelForge operator stack, 2026-07-29 (T1.d measured in the harness, unproven in the game)

> Paste the block below into a fresh session. Everything it refers to is on disk and in git.
>
> **State:** 8 of 8 archetypes ported and green. **Tile-skipping works and is measured — in the
> automation harness:** 11 of 40 tiles proved `AllSolid` at production defaults, 14641 voxels
> brute-forced, 0 violations. `AUDIT §C2` is fixed. `experimental` is pushed (`origin/experimental`).
>
> **Two things are open, and the second is the more interesting one:**
> 1. commit `e002bd4` (VerticalShafts) is written and **not yet built**;
> 2. **none of this is observable in the running game.** Jahni built a world, looked at it, and said
>    *"I don't know if it dropped any meshing? but it looks alright by the eye."* He is right to be
>    unsure — the plugin has **zero stat counters**, and *skipped correctly* renders identically to
>    *skipped nothing*. `CODEX-TASK-001-tile-skip-stats.md` is the spec that closes this.

---

You're picking up the VoxelForge UE5 voxel plugin on branch `experimental` (already checked out —
do not create another). I'm Jahni. The design and the history are written down so you don't
re-derive them.

## Read first, in this order

1. **`CLAUDE.md`** — project rules. **Rule #1 is absolute: never build, compile, or run the editor.**
   I build everything myself. When code is done, stop, say "ready to build", list the likely
   compile-error spots, and wait.
2. **`OPSTACK-PROGRESS.md` — THE LAST ENTRY FIRST.** Append-only log; the resume point.
3. **`OPSTACK-PLAN.md`** — the plan. **§2.6.1 is the acceptance bar** and supersedes §2.6.
4. **`OPSTACK-DECOMPOSITION.md`** — per-archetype breakdown. **§0.2** (the amplitude bound) is now
   *implemented*, not pending; §2 TunnelNetwork and §8 Underwater are history, not instructions.
5. **`AUDIT-2026-07.md`** — **§C2's SDF-cache half is FIXED (2026-07-28)**, its live-edit half
   (`OC_Chunk` / `BM_Chunk` / `FChunkBiomeCache`) is still open; §C10 is SOLVED, don't reopen;
   §C9's library half is the top open theoretical risk with 0 measured exposure.
6. **`CODEMAP.md`** — navigation. Trust symbol names over line numbers.

## How we work now — Codex writes, you orchestrate

From 2026-07-29 this project runs **in tandem with Codex (Model Luna, xHigh)**. **Codex handles most
of the coding; you orchestrate.** Concretely:

- You read the code and decide *what* to do; you write **precise specs** Codex executes; you **review
  what comes back against the real code, not against its description**; you own the docs
  (`OPSTACK-PROGRESS.md`, `CODEMAP §3`, this file) and the measurements.
- **Hand Codex the INVARIANT, not just the task.** This codebase's traps are invisible in a diff —
  density sign, `Identity` meaning `Sdf ≥ T` (below), cache keys needing params + `LayoutVersion`,
  inserting classes above the anonymous-namespace end marker. A spec that omits these gets code that
  compiles and deletes collision.
- `CODEX-TASK-*.md` at the plugin root are the specs. Each carries a **Why**, the **exact site**, the
  **invariants**, an **acceptance** section, and **notes for the reviewer**. Write the next one the
  same way.
- Unchanged: **never build** (Jahni does), and a plausible patch is not a verified one until a
  measurement says so.

## Where things stand

All 8 archetypes have an operator-stack twin, per-strate opt-in, each equivalence-tested **bit for
bit** against its original density function. The `switch` and the stack are two complete,
interchangeable implementations.

Everything sits behind `UVoxelStrateDefinition::bUseOperatorStack`; the ported list lives **only** in
`UVoxelStrateManager::UsesOperatorStackForChunk` (all 8). **No strate asset has the box ticked** —
that is my call and I still haven't made it. `GetDensityAt` and `ClassifyTile` build the stack
through the **same** factory, `VF_BuildOpStackForChunk` — a second copy would be a hole, not a bug.

### ✅ T1.d — the tile-skipping prize — is real and measured **in the harness** (not yet in the game)

`FRoomGraphSource::EffectOverBox` answers **spatially**. The result, brute-forced voxel by voxel:

```
[production defaults]  11 of 40 tiles proved AllSolid — 14641 voxels checked, 0 violations
[dense fixture]         0 of 40                       — correct, and structurally inevitable
```

One function's verdict is inherited by `FSdfConvertOp`, the twelve detail modifiers (via
`VF_NoCaveOverBox`) **and** `FWormFieldSource` — fourteen operators from one place. That is what the
C1 wiring was built for.

### ⚠️⚠️ THE ONE INVARIANT THAT CAN DELETE COLLISION — read before touching any op

**`FRoomGraphSource::EffectOverBox` returning `Identity` now means `Sdf ≥ T`, NOT `Sdf == FLT_MAX`**,
where `T = max(3·SDFBlendRadius, WormNetworkRange)`. That is sound only because all three consumers
of the SDF channel were read one by one:

| consumer | threshold |
|---|---|
| `FSdfConvertOp::Eval` | `Sdf >= Blend`, and the tunnel stack passes `MakeSdfCarve(P.SDFBlendRadius, …)` ⇒ **K** |
| the twelve modifiers | `VF_NearCaveSurface` ⇒ **3K** |
| `FWormFieldSource::Eval` | `CaveSDF >= WormNetworkRange` ⇒ **WormNetworkRange** |

**Any new consumer of `InOut.Sdf` must have a threshold ≤ `T`, or be added to that `max`.** An op
reading `Sdf < 100` would see false `Identity` verdicts and produce tiles with no geometry **and no
collision**. The warning is written at the site you land on when you add one.

(The `−K` slack covers *any* number of primitives because `SmoothMin`'s penalty is exactly zero once
`|A−B| ≥ K`, so the running minimum saturates at `K` below the smallest term. Without that
observation the slack would scale with the ~88 tunnels in a cache and the criterion would be dead.)

## First actions — one build to read, two tasks to hand Codex

### (a) Hand Codex `CODEX-TASK-001-tile-skip-stats.md`, then `-002-` — 001 first, they chain

Everything in this refactor has been proved in an automation harness on 40 sampled tiles, and
**nothing has ever been observed in the running game.** Task 001 adds a `stat VoxelForge` group with
`TilesClassified / TilesSkippedAllSolid / TilesSkippedAllAir / TilesMeshed`, **plus
`TilesOpStackSolid / TilesOpStackAir` at a second site**. Task 002 adds `ColumnMemoHit / Miss` to the
same group; it needs 001's header to exist, and both should land in one build.

Its deliverable is a **before/after that constitutes the production proof of T1.d**: after ticking
`bUseOperatorStack` on one `TunnelNetwork` strate and flying the same route, **`TilesOpStackSolid`
must go non-zero**. The spec carries the invariants — most importantly that `bTrivialEmpty` decides
whether a tile has **collision**, and that `GenerateTileResult` runs on **worker threads** so a plain
`static int32++` is a data race.

⚠️ **Corrected 2026-08-16 — the earlier version of this bar was unmeasurable.** It said
`TilesSkippedAllSolid` would read 0 underground with no strate opted in. It will not: `ClassifyTile`
also proves `AllSolid` on its **hand-written** path (a bedrock-gap chunk sets `bCanAir = false`,
`VoxelGenerator.cpp` ~2835), which fires with nothing ticked at all. That is the same "~84 %" caveat
below, which the old bar quoted and then contradicted. The op-stack-only counters are zero **by
construction** — the cave branch returns `Mixed` at `UsesOperatorStackForChunk` — so they are the
ones that prove anything.

Interim answer if Jahni wants it before that lands: **Unreal Insights already shows this.** The trace
scopes `VoxelForge_ClassifyTile` and `VoxelForge_GenerateMesh` exist at the site; a skipped tile is a
`ClassifyTile` with no `GenerateMesh` after it. ⚠️ But ~84 % of tiles were *already* being rejected by
the hand-written SurfaceWorld/bedrock paths long before this work, so surface skips will drown the
cave ones — you must be **underground in an opted-in `TunnelNetwork` strate** for the number to mean
anything.

### (b) Build `e002bd4` (VerticalShafts) and read ONE line

Everything before it is built and green.

> Build, run the `VoxelForge` filter, and read
> **`Box verdicts over 60 VerticalShafts tiles`**.
>
> **0 was the number for the whole project's life.** Its `EffectOverBox` used to return `CarveOnly`
> because a shaft merely *existed* within a `Spacing*1.6` halo — true almost everywhere at
> `ShaftSpacing 55 / ShaftDensity 0.6`. It now rebuilds the connectors the way `GetCells` does and
> tests the real capsules, with **Z exact** and XY conservative.
>
> - **Non-zero, and `violations` still 0** ⇒ it worked; record it and move on.
> - **Still 0** ⇒ the warning in that test names what to check **first**: `ExtraReach` inflates both
>   remaining tests, so compare it against `ShaftMaxRadius` before touching either test. **Do not
>   re-derive from scratch** — that is exactly what cost three rounds on TunnelNetwork.

## Then, in order

1. **PERF — the biggest open item, and now AIMED (2026-08-16). Read this before touching it.**
   The op path is measurably slower; one cause was already found and fixed (the column memo
   discarded itself every chunk). Three things were worked out since, all still **unmeasured**:

   - **The A/B needs no new code.** `VoxelForge_ClassifyTile` and `VoxelForge_GenerateMesh` already
     exist, and the world is deterministic, so two Insights traces — flag off, then on, same seed and
     route — are a clean before/after.
   - **But it is unreadable without task 001.** With the stack on, tiles get *skipped*, so
     `GenerateMesh` runs fewer times; a total conflates "cheaper per tile" with "fewer tiles" and
     those pull opposite ways. `TilesMeshed` is the denominator. **⇒ 001 is a PREREQUISITE here, not
     a parallel item.** Order: 001 → traces → attribution → fix.
   - **The suspects don't share an archetype**, so measure one ticked strate at a time.
     *SurfaceWorld* = the hashed column memo. *TunnelNetwork* = **19 virtual calls per voxel**
     (16 from `BuildTunnelNetworkStack` + 3 from `AppendStructuralPost`), plus the known 12× gate
     re-test (stage B5's deliberate trade).

   `CODEX-TASK-002` tests the SurfaceWorld suspect and **fixes nothing on purpose** — the derivation
   says the direct-mapped 4096-slot table evicts ~25 % of columns *every Z plane* (the mesher
   pre-samples **Z-outermost**), for a derived ~9× on column work. **Derived, not measured.**
   **Measure before optimising** — the §C10 lesson, re-learned the hard way last session.
2. **The warp squeeze — PARKED with its ceiling measured, my recommendation is leave it.** The
   `WARP SHARE` line says over half the remaining blocking is the query-box dilation, not geometry
   (production: rooms 0.9 → 0.4, tunnels 2.4 → 1.1 with the dilation zeroed). The only remaining
   route is proving `sup|Perlin3D|` down from the proved **1.5** toward its apparent ~1.0–1.1, worth
   ~27 % of the dilation. Spot-checking a grid is **not** a proof and a wrong sup is a hole.
   **A negative result is already recorded so nobody repeats it:** bounding the warp *locally*
   (evaluate at the box centre, shift, dilate by the variation) is **worse** — a rigorous per-axis
   Lipschitz bound is `4·1.875 + 1 = 8.5` per unit cell, and `8.5 × 0.206` (the half-box in noise
   units) `= 1.75` exceeds the global range bound of 1.5.
3. **`AUDIT §C9` library half** — `sinf`/`cosf` are not IEEE-754 specified, so MSVC's CRT and glibc's
   libm can differ. Currently **0 samples within 1e-6 of the isosurface**, i.e. no measured risk. Run
   `CrossPlatformDigest` on Linux, compare the SHAPE digest, pin it. The real fix if ever needed is a
   deterministic in-house sin/cos.
4. ~~**`AUDIT §C2`'s remaining half**~~ — **✅ CLOSED, verified 2026-08-16. Do not re-open, and do
   not spec a fix for it — I nearly did.** `OC_Chunk`, `BM_Chunk` and `TC_BiomeCache` all carry a
   layout-version guard (`OC_Version` / `BM_Version` / `TC_SeenVersion`), `FChunkBiomeCache` has an
   explicit `Invalidate()` that all four `thread_local` instances call on a version change, and the
   only other two instances in the tree are **function-local**, so they cannot go stale. Recorded in
   `AUDIT-2026-07.md §C2`. The live-edit half was fixed at the same time as the determinism half;
   only this list was stale.
5. **Phase 3 — ops as data assets.** A design conversation, not a transcription. Don't start it
   unprompted. What makes it possible is already in place: ops depend on capabilities
   (`IVoxelBiomeField`), never on `UVoxelGenerator`.

## Debts — status changed, read this before acting on the old text

1. **"Box bounds read STRATE params but a per-room op can raise them" — DORMANT, not urgent.**
   Checked rather than paid, and the check reversed the premise: when the source proves `Identity`
   the twelve modifiers are `Identity` **soundly** (their `bNearCaveSurface` gate never opens, so no
   room op can enable anything), and when it answers `Both` it supplies no `MaxCarveOverBox`, so the
   default `FLT_MAX` kills every hypothesis regardless of what the modifiers claim. **It goes live
   the day `FRoomGraphSource` gains a `MaxCarveOverBox`** — bounding the converter's `2·BaseDensity`
   would make the modifiers' own numbers matter for the first time. Written at the site.
2. **`AUDIT §C2` — FIXED on the `switch` path.** `GetDensityWithParams` now takes **required**
   `ParamsFingerprint` + `LayoutVersion`. Required, not defaulted, so a caller that forgets fails to
   compile. The CRC is taken **once per chunk** where the params memo already lives (`CP_TunnelFP`) —
   a `MemCrc32` per voxel on the hottest path would have been a real regression. Note the audit's own
   suggested alternative ("add chunk Z to the key") is both insufficient (`Interleaved` makes `Alpha`
   depend on chunk **XY** too) and destructive (chunk XY is deliberately absent so `WorldX ± 1`
   gradient probes don't thrash the box — `ARCHITECTURE §8.10`).

## Hard rules that prevent real bugs

- **Density sign:** negative = solid at the mesher. Inside the op stack the convention is INTERNAL
  (**positive = solid**), negated once by the caller. The SDF channel uses standard SDF convention.
- **`Identity` from the room source means `Sdf ≥ T`.** See the boxed invariant above. This is the
  single most dangerous thing in the current code.
- **Never run both density paths in one world.** **Comparing them is legitimate** — §C10 is closed
  since `FPSemantics = Precise`, and all eight equivalence tests compare bit for bit. They are
  **port-correctness oracles**, not fidelity checks: §2.6.1 requires *same seed ⇒ same world on every
  peer*, not resemblance to the pre-refactor world.
- **Every cache key includes `LayoutVersion` AND the params.** See §C2 and the overhang regression of
  2026-07-27, where omitting the params silently deleted the overhang and only 1 sample in 20 000
  crossed the isosurface.
- **A bound in a box verdict must be PROVED, not observed.** `|Perlin3D| ≤ 1.5` is derived from
  `GradDot`'s two-distinct-axes form and the per-axis weighted bound of 0.5 — *not* from the header's
  "~[-1,1]". Over-estimating costs CPU; under-estimating deletes collision.
  **⚠️ USE `VF_PerlinAbsBound` — it is file-scope in `VoxelDensityOpStack.cpp` and it is the ONLY
  copy. Never write a bare `1.0` for a noise amplitude in a reach.** `VoxelNoise::FBM` **normalises**
  (`return Total / MaxValue`), so `sup|FBM| = sup|Perlin3D|` **exactly** — the octave sum neither
  amplifies nor attenuates it, and an `FBM`-driven reach needs the same 1.5. This rule was written
  *before* three `ExtraReach` formulas were found violating it (2026-08-16, fixed in `7dbdf51`):
  VerticalShafts and Maze were unsound at their shipped defaults, FloatingIslands sound only because
  its `SDFBlendRadius` happens to be large. **A rule stated in a doc is not a rule enforced in code** —
  when you add a reach, grep for `VF_PerlinAbsBound` and use it.
- `ProcessQueue` stays `EQueueMode::Mpsc`; `Epoch` carries through every async path; don't "optimize"
  the `ARCHITECTURE §8.10` invariants.
- Commit per coherent unit with a real message. **`experimental` is pushed and tracked
  (`origin/experimental`, since 2026-07-29) — keep it in sync. NEVER push `main`**, which stays the
  known-good fallback at the commit it has always been. ⚠️ A pushed commit here is **not** a
  "verified green" marker: the branch carries unbuilt work by design, and only `OPSTACK-PROGRESS.md`
  says what was actually built.
- Update `CODEMAP §3`, `ARCHITECTURE §8`, tick `OPSTACK-PLAN`, append to `OPSTACK-PROGRESS.md`.
- **When inserting a class into `VoxelDensityOpStack.cpp` / `VoxelHeightOpStack.cpp`, put it ABOVE
  the labelled end of the anonymous namespace.** Anchoring on the FACTORIES banner puts it outside,
  and the brace added with it closes nothing. Made that mistake twice; both files say so.
- **Match the codebase's spelling of engine macros.** `KINDA_SMALL_NUMBER`, not
  `UE_KINDA_SMALL_NUMBER` — the plugin uses the unprefixed form everywhere.

## Method lessons this refactor actually paid for

Ordered by how much they cost.

- **⭐ Instrument what you ASSUMED, not just what you changed.** This is the expensive one, learned
  over four rounds in one session. The warp dilation — `CaveWarpStrength · VOXEL_NOISE_SCALE ·
  PerlinAbsBound`, a constant chosen in the first commit — inflated a 10-voxel tile into a 50-voxel
  query box, **125× the volume**. Four separate tightenings (the worm, the columns, the sampler, the
  tunnel disjunction) were each individually correct and each landed *around* that untouched term.
  The tunnel fix, predicted "an order of magnitude", delivered 25 % — **and the instrument said so,
  and I credited the tunnels.** *When a fix under-delivers against its predicted size, suspect the
  constant you never measured.*
- **Instrument before hypothesising.** §C10 cost six builds and five refuted hypotheses. In this
  session the attribution line (`AllSolid killed by: …`) was written after *two* wrong guesses and
  immediately named a third operator nobody had looked at. **A diagnostic that lists candidate causes
  without measuring them is still a guess wearing rigour** — my "either the tiles straddle cave or
  the source isn't reaching Identity" warning offered two causes and both were wrong.
- **Verify the premise before reasoning from it.** Six times now a confident chain rested on an
  unchecked assumption and the check reversed it. Latest three: `RoomSpacing` was **42** (the fixture
  overrides it) while I did three rounds of arithmetic with the header default of 80 — *the number
  was printing in the report I kept quoting*; "the plugin bets on `|Perlin3D| ≤ 0.8`" was wrong (the
  cache **rebuilds** when the warped query leaves the box, so that expansion is a perf heuristic);
  and the per-room-op debt "must be paid first" was wrong (it is dormant). **Include the premises you
  are confident enough about not to look up — especially a default, when a fixture exists whose whole
  job is overriding defaults.**
- **A sampler must cover at least one period of what it samples.** The tunnel test drew tile XY from
  **±32 voxels** with `RoomSpacing 80` and a guaranteed origin room at (0,0) — it measured the spine
  hub and called it the world. The shaft test had the identical bug (±48 against `ShaftSpacing 55`).
  Both now print their own extent **in units of the pattern's period**.
- **A test fixture tuned for coverage can be antagonistic to the thing you are measuring.**
  `EnableTunnelFeatures` densifies (`RoomSpacing` 80→42, `RoomDensity` 0.35→0.85) so the equivalence
  check isn't comparing solid rock to solid rock — and at that density the room cull radius *equals*
  the lattice spacing, so **no box can ever be proved**. `0 proved` there is the correct answer. The
  box verdict is therefore measured on **both** densities, and the dense run must stay at 0.
- **Diagnostics report THIS run; history goes in the log.** The test output had accumulated hardcoded
  numbers from previous runs beside live ones ("32 of 34 tiles" printed while the live figure was 21
  of 28). Unreadable, and self-inflicted.
- **Read the code, not the comment.** The cliff modifier's comment promises a sampled Z±1 gradient;
  the code samples nothing and uses a Z-stretched Perlin it *calls* `VertGrad`. Ported as written —
  and written down, so nobody "fixes" it from the comment.
- **A perf change can be a correctness change.** The column-memo optimisation silently deleted the
  overhang; the tests caught it the same day. Invisible to inspection, and it produced plausible
  terrain.
- **Coverage is a number, not a boolean.** Four related traps, each producing a green run that proved
  almost nothing:
  - *A test that prints nothing on success is indistinguishable from one that never ran.*
  - *A guard that only trips at zero notices absence, it does not measure coverage.* Use fractions.
  - *A success message that **asserts** coverage instead of reporting it reads as evidence while
    measuring nothing.*
  - *A check can be vacuous as well as a counter.* "Nothing leaked" is worthless unless something
    happened.
- **Enabling a feature is not evidence it fired — ask the structure, not the output.** Setting
  `PitDensity` did nothing (wrong struct). **Prefer the check that can fail for exactly one reason**
  — and when a zero has several possible causes, give each one its own number.
- **An oracle that shares the defect under test proves nothing.** The stale-cache check compares each
  stack against *itself evaluated alone*. (Since §C2 was fixed, the test call sites now pass a real
  params fingerprint, so the original no longer shares the defect either.)
- **One definition, not two kept in sync.** `VF_BuildOpStackForChunk` exists because a tile skipped on
  the verdict of a stack that is not the one producing its density is a hole. The same reasoning is
  why `GetLastRoomBoxDiagnostic` **reads back** what the operator computed instead of letting the
  test re-derive the criterion, and why the two-density tile scan is one lambda called twice.
- **Don't assert a number you want to improve.** Check 4 asserted `0 proved` — honest when written,
  and it would have forbidden the entire T1.d gain. What it asserts now is that **no proved tile is
  wrong** (brute force, every voxel); the proved count is *reported*.