VoxelForge/OPSTACK-PLAN.md

VoxelForge — Density Operator Stack: the plan

What this is: the agreed direction for turning VoxelForge from an archetype dispatcher into a composable density pipeline, so new world ideas become authoring instead of C++. Written 2026-07-26 as a handoff for a future context — read this instead of re-deriving it.

Status: DESIGN AGREED, NOT STARTED. Nothing in the codebase implements this yet.

Read first: CODEMAP.md (navigation) · ARCHITECTURE.md §8.10 (the perf invariants this must not break) · AUDIT-2026-07.md §6 (the 3D hazards this is designed to kill permanently).

Jahni's goal, verbatim (2026-07-26): "a world generator, of any kind, of any possibility, almost — any combination of ideas you could guess, could be happening."

---

0. The one-paragraph version

UVoxelGenerator::GetDensityAt is a switch over 8 hardcoded ECaveGeneratorType values, each owning a bespoke density function and param struct. That means (a) a new world idea costs ~6 edit sites, and (b) ideas cannot combine — one archetype owns the whole voxel. The fix is to make density a stack of small operators, each of which can PrepareChunk / Eval / declare what it can do to a box. That last part is the keystone: it makes ClassifyTile generic and correct forever, instead of a hand-written guard per feature (the thing that already caused one revert and is the top hazard for the current 3D work). Staged so the game never stops working, and so the first useful step is small.

Non-goal, stated deliberately: this must serve the descent-through-strates structure, not dissolve it. "The world is a vertical stack you dig down through, each layer its own place" is the idea of this project — the seals, the (0,0) spine and the passages only mean something because of it. The op stack should make each strate more surprising, not turn the world into undifferentiated composable soup.

---

1. Why this specific shape fits this specific codebase

Not a generic "use composition" argument — three concrete reasons:

(a) It formalises what the code already does by hand. Every archetype already: hoists chunk-constant work into a thread_local cache (PrepareChunk), evaluates cheaply per voxel (Eval), and — in ClassifyTile — has a hand-written statement of what it can do to a tile (Bounds). The operator model isn't a new discipline; it's the existing discipline, named.

(b) The 8 archetypes are ~6 functions wearing costumes. FlatPlain and CrystalChamber are literally the same function with different defaults; Underwater is TunnelNetwork + a water flag. Decomposed, they're roughly 15 orthogonal primitives (floor surface, ceiling surface, hash cylinders, room SDF graph, tunnel capsules, worm noise, lattice corridors, heightfield stack, island blobs, boundary seal, spine carve, passage carve, disturbances, surface ops, diff layer). 15 primitives that combine covers vastly more than 8 that don't.

(c) It kills the recurring hazard permanently. T1.d guards are currently written per feature (AnyPassageNearBox, the spine circle test, chasm/bridge flags, HasAnyModInChunkRange, phase-2's OverhangMargin). Every new 3D feature needs one, forgetting one is a hole, and one was already forgotten badly enough to revert T1.d v1 on 2026-06-26. Under this model a new op cannot ship without answering the question, and a test catches it if the answer is wrong.

---

2. The keystone insight — start with DIRECTION, not intervals

The full version of Bounds returns a numeric interval. Don't start there. Almost every existing operator is one-directional: it only ever carves, or only ever fills.

enum class EVoxelOpEffect : uint8
{
    CarveOnly,   // can only move density toward AIR   → kills the AllSolid hypothesis
    FillOnly,    // can only move density toward SOLID → kills the AllAir  hypothesis
    Both,        // unconstrained
    Identity     // provably no effect on this box (the early-out that makes it fast)
};

virtual EVoxelOpEffect EffectOverBox(const FBox& VoxelBox, const FVoxelOpContext& Ctx) const;

This alone reproduces every hand-written guard in ClassifyTile today, generically. Look at the current code: "passages ⇒ bCanSolid = false" is CarveOnly. "bridges/ridges ⇒ bCanAir = false" is FillOnly. "no passage near this box ⇒ skip" is Identity.

So Phase 1 ships with no numeric bounds at all and already gets the whole safety property. Numeric intervals are a later tightening for gen-cost, not a correctness prerequisite. This is what turns a scary refactor into a small first step — and it's the thing to remember if this plan ever feels too big.

Every existing primitive already has an obvious answer

Filled in here so a future context doesn't have to re-derive it:

Primitive	Effect	Identity test (cheap)	Numeric bound (later)
ApplyBoundarySeal	FillOnly (FMath::Max)	box misses both seal bands	+[0, BaseDensity]
ApplyOriginSpine	CarveOnly	circle-vs-box XY, + Z outside interior	−[0, Base*2 + Seal]
ApplyPassageCarving	CarveOnly	AnyPassageNearBox — already written	−[0, …] via AirTarget
Disturbance chasms	CarveOnly	ChasmDensity == 0, or lattice miss	+[0, Solid] toward air
Disturbance bridges/ridges	FillOnly	density == 0, or lattice miss	−[0, Solid]
F20 overhang	FillOnly, banded	outside (TerrainZ, TerrainZ+Height] — already written	union ⇒ max only
Room / tunnel / column SDF	Both	bounding sphere vs box — already written	Lipschitz-1: SDF ∈ [SDF(c) − r, SDF(c) + r], r = box half-diagonal
fBm / Ridged term k·N(...)	Both	k == 0	±|k| — FBM returns [-1,1] by construction (Total/MaxValue)
Heightfield (TerrainZ − Z)	Both	—	[minT − Zmax, maxT − Zmin]; or sample the lattice exactly (see below)
Diff layer	Both	HasAnyModInChunkRange — already written	±max|Strength| over mods in range

Two things to notice:

• Five of these already exist as code. The work is mostly moving guards, not inventing them.
• Bounds is allowed to be exact-by-sampling, not just analytic. Today's ClassifyTile gets the tightest possible SurfaceWorld verdict by evaluating ComputeSurfaceColumn on the mesher's exact lattice — same functions, same floats, so the verdict is exact rather than estimated. That must survive. The contract is "conservative", not "closed-form": an op may sample to answer.

Prior art (30 minutes, worth it before Phase 1)

• Keeter, Massively Parallel Rendering of Complex Closed-Form Implicit Surfaces + fidget — interval arithmetic per expression node to prune empty regions. The industrial version of this idea.
• Barbier et al., Lipschitz Pruning: Hierarchical Simplification of Primitive-Based SDFs (CGF 2025) — the better fit for us: bounds each primitive's range of influence and treats primitives as black boxes, where full interval arithmetic needs interval semantics defined for every node. Our ops are SDFs (Lipschitz by construction) and bounded-amplitude fBm. Black-box bounds; don't build a node-level IA engine.

---

2.5 ⚠️ The op TAXONOMY — and why this is NOT the old room-ops system

Jahni's objection, 2026-07-27, and it is the correct one: "ops structure — which, let's all be honest, was what I had before, 'room operations' which would modify stuff, so I sure hope your idea is not that. There's a world of difference between a grotto strate and an open world strata."

He is right, and a fresh context must internalise this or it will build something useless.

UVoxelTerrainOpDefinition today can only perturb density near a surface that already exists, inside a fixed archetype (Terrace, LayerLines, Ribbing, Cliff, Scallop, Overhang, Arch, Column, Pit, Chimney, Dome, Pinch — all applied near cave walls where bNearCaveSurface). It cannot turn a grotto into an open world, because it never decides what the field IS. That decision lives in the switch in GetDensityAt, which is exactly the thing we are removing.

So the op stack has four ROLES, and the old system only had role 3:

Role 1 — FIELD SOURCES (the new thing; this is the "world of difference")

Produce a density field from nothing. This is what makes a grotto a grotto and an open world an open world. Each of today's archetypes is fundamentally one of these:

Source	Today's archetype
Heightfield ground + sky-cap ceiling	SurfaceWorld
Room-graph SDF (hash rooms + tunnel capsules)	TunnelNetwork
Floor/ceiling slab void	FlatPlain, CrystalChamber
3D lattice corridors	Maze
Full-height shafts + connectors	VerticalShafts
Suspended island blobs in open void	FloatingIslands

A source is a first-class op with the same three-method contract. A "strate archetype" therefore stops being an enum and becomes source + combiners + modifiers. Two strates differ in their SOURCE first, their modifiers second.

Role 2 — COMBINERS (how sources merge; this is what makes ideas compose)

Replace · Union(min) · Subtract(max) · SmoothUnion/SmoothSubtract (reuse VoxelSDF::SmoothMin/Max) · Mask (scale the next op by a field: biome weight, slope gate, relief, depth).

This is the role that buys the ambition. Floating islands inside a grotto. A maze beneath an open world's ground. A room-graph carved into a mountain. None of those are expressible today at any price.

Role 3 — DETAIL MODIFIERS (the old system, demoted to one role of four)

Roughness, terrace, layer lines, ribbing, scallop, cliff, overhang, domes, pinch. All of today's UVoxelTerrainOpDefinition types land here, essentially unchanged. They keep working; they stop being the whole story.

Role 4 — STRUCTURAL POST (fixed order, non-negotiable, runs last)

ApplyOriginSpine → ApplyBoundarySeal → ApplyPassageCarving → diff layer.

These are world invariants, not creative choices: descent must stay possible, seals must hold, passages must punch through anything, player edits win. They are ops for uniformity but the stack compiler must always append them, in this order, regardless of authoring. An author must not be able to omit them.

Plus: SCOPING

Any op may be gated by a region predicate — Z band, XY region, biome index, slope range, depth. Scoping is what lets one strate hold several sources without them fighting, and it's the mechanism that unifies "strate" and "biome" into one concept in Phase 3.

The test for whether this refactor was worth doing: can you author "an open-world surface strate whose mountains contain a room-graph cave system, with floating islands in the upper void" without writing C++? If no, it collapsed back into the old system and something went wrong.

---

2.6 The acceptance bar — "very close", NOT byte-identical

Jahni, 2026-07-27: "I want not a 1/1 replica of the current strates with the current system, but a possible very close result to what I have right now, else it won't really matter much."

Two different properties get confused here. Keep them apart:

Property	Required?	Meaning
ValidateDeterminism = 0	YES, ALWAYS	the same world point sampled twice, from different cache windows/threads, returns the identical float. This is window invariance (§8.4) — self-consistency. Non-negotiable, it's what prevents seams and MP divergence.
Byte-identical to the OLD system's output	NO	matching the pre-refactor world float-for-float.

This is a deliberate relaxation of what an earlier draft of this plan demanded, and it matters strategically: if bit-identity were required, the cheap path would be to wrap each old density function as one monolithic op — 8 opaque ops that don't compose, i.e. the switch with extra steps and zero gain. Releasing that constraint is what permits real decomposition into the primitives in §2.5.

The bar instead: for each ported archetype, an authored op stack must reproduce the character of the old one — same scale, same navigability, same feel, recognisably the same kind of place. Judged by Jahni on a screenshot at a fixed seed, not by a diff. Expect and accept a one-time re-tune, exactly as the T2.a SIMD noise switch required.

---

3. The contract

// Chunk-constant inputs. Mirrors what the thread_local CP_* block resolves today.
struct FVoxelOpContext
{
    FIntVector ChunkCoord;
    int32      Step;              // LOD step — ops may cheapen themselves (see §7)
    uint32     Seed;
    uint32     LayoutVersion;     // ⚠️ see AUDIT C2 — every cache key MUST include this
    float      StrateTopWorldZ, StrateBottomWorldZ;
    const FBiomeContext* Biome;   // null = strate has no biome field
};

class IVoxelDensityOp
{
public:
    // Hoist all chunk-constant work here (room lists, biome grids, column caches, lattice bakes).
    // Called once per chunk per worker. This is where today's thread_local caches move to.
    virtual void PrepareChunk(const FVoxelOpContext& Ctx) = 0;

    // Per-voxel. InDensity = what the stack produced so far, MC convention (negative = solid).
    virtual float Eval(float X, float Y, float Z, float InDensity) const = 0;

    // CONSERVATIVE. Phase 1: direction only. Phase 3: add a numeric interval overload.
    // Returning Both is always SAFE (costs CPU); returning the wrong one is a HOLE.
    virtual EVoxelOpEffect EffectOverBox(const FBox& VoxelBox, const FVoxelOpContext& Ctx) const = 0;

    // Declares whether Eval depends on Z. XY-pure ops get the T1.a column-cache treatment
    // generically instead of SurfaceWorld having a bespoke one.
    virtual bool IsXYPure() const { return false; }
};

Composition semantics — keep the vocabulary small, and reuse what exists (VoxelSDF::SmoothMin / SmoothMax):

Mode	Meaning
Replace	ignore InDensity (stack roots: heightfield, base density)
Union (min)	add solid — bridges, islands, columns
Subtract (max)	carve air — rooms, tunnels, passages, spine
SmoothUnion/Subtract	the same with SmoothMin/Max(k) — organic junctions
Add	scalar accumulate — noise/roughness terms
Mask	scale the next op by a field (biome weight, slope gate, relief)

Mask is what buys most of the expressiveness: "this op, but only in high-relief regions / only on steep slopes / only in this biome" becomes composition rather than a bespoke gate inside each op.

---

4. Order of work

Phase 0 — THIS WEEK. Ship the 3D caves. Refactor nothing.

The current feature (caves inside mountains, volumetric generation) ships as-is with a hand-written ClassifyTile guard — see AUDIT-2026-07.md §6.1 for why it's mandatory (without it the caves are never meshed: no geometry, no collision, invisible until you fall through them).

One constraint only: write the guard as a standalone function in the shape of the future contract —

// Not a member of anything yet. Just the right shape.
EVoxelOpEffect CaveSystemEffectOverBox(const FBox& VoxelBox, const FVoxelOpContext& Ctx);

Same work, zero architectural commitment, and it establishes the pattern.

Also decide here (see AUDIT §6.2): placed, not fielded. Hash-located cave systems with bounds (like FCachedRoom/FCachedTunnel) let the guard prove most of the mountain still solid and keep T1.d's 44% worker-CPU win. A global 3D noise field makes every deep tile Mixed and hands that back.

✅ Gate: caves render and collide when approached through solid rock, and worker CPU hasn't visibly regressed.

---

Phase 0.5 — The safety net. ~1 day. Do this before Phase 1, not after.

Three automation tests (Source/VoxelForge/Private/Tests/, IMPLEMENT_SIMPLE_AUTOMATION_TEST). There are currently zero tests, and nothing machine-checks the dozens of "bit-identical" claims in the docs.

1. Density purity — sample 10k points, shuffle query order, re-sample, assert bit-equality. Catches every cache-key bug including AUDIT C2. Run it across multiple worker threads, because ValidateDeterminism runs on the game thread and would miss worker-cache divergence.
2. ClassifyTile soundness — for random tiles, if the verdict is AllSolid/AllAir, brute-force the lattice and assert every sample agrees. This is the highest-consequence function in the plugin and it is currently validated only by reasoning.
3. DiffLayer under contention — N readers + a writer; assert no crash, monotonic version.

✅ Gate: all three green on the current code. If #1 or #2 fails, you've found a live bug — fix it before building on top.

---

Phase 1 — The pivot. Port ONE archetype. Timebox it.

Pick Maze. ~100 lines, no cross-chunk connectivity decision, trivial bound (corridor SDF is Lipschitz-1 off a lattice), and it's the least-used archetype so a mistake is cheap.

1. Add IVoxelDensityOp + EVoxelOpEffect + FVoxelOpContext + the four role tags (new file: Public/VoxelDensityOp.h).
2. DECOMPOSE, don't wrap (§2.5, §2.6). Maze becomes a stack, not one op: FLatticeCorridorSource (role 1 — the capsule field off the 3D lattice) → Subtract → FSurfaceRoughnessMod (role 3 — the existing SurfaceRoughness perturbation) → then the four structural-post ops appended automatically. If it comes out as a single FMazeOp, the refactor has failed its own test — that's the switch with extra steps.
3. GetDensityAt gains one branch: strate has an op stack ⇒ run it; else fall through to today's switch. Both systems coexist, indefinitely if needed.
4. ClassifyTile gains a generic path (EffectOverBox folded over the stack) used only by ported strates.

✅ Gate: ValidateDeterminism = 0 delta (§2.6 — always), Phase 0.5 tests green, and the Maze world is recognisably the same place at a fixed seed — same corridor scale, same connectivity, same feel. Judged on a screenshot, not a diff. A one-time re-tune of the params is expected and fine.

✅ The real proof, and the one that answers Jahni's objection: once FLatticeCorridorSource exists, drop it into a SurfaceWorld strate underneath the terrain and confirm you get a maze inside a mountain with no C++ written. If that works, the architecture is doing the thing it was built for.

🛑 Stop-and-reconsider trigger: if Phase 1 exceeds ~2 days, or the source/modifier split doesn't fall out naturally from the existing code, the abstraction is wrong for this domain. Say so plainly, revert, and report — do NOT push through and port a second archetype to prove a point.

---

Phase 2 — Port opportunistically. No big bang.

Port each archetype the next time a feature makes you open it anyway. The switch shrinks on its own. Suggested order when there's a free choice — cheapest and least risky first:

Maze (P1) → FlatPlain/CrystalChamber (one op, two default sets — the first real win: two archetypes collapse into one) → SurfaceWorld (biggest payoff, biggest care: the T1.a column cache and the exact- lattice ClassifyTile bound must both survive) → VerticalShafts → FloatingIslands → TunnelNetwork (last — it owns BuildChunkCache's two-region window-invariance discipline, §8.4, the most delicate code in the plugin).

Along the way, FStrateGenerationParams' 74 fields decompose into per-op structs, which retires the VF_STRATE_PARAM_FIELDS X-macro drift problem for free.

---

Phase 3 — The payoff. Ops become data.

1. UVoxelDensityOpDefinition : UPrimaryDataAsset — one asset per op, mirroring today's UVoxelTerrainOpDefinition (which is already the right authoring shape: an asset + weight + probability in an ordered list).
2. A strate becomes "a Z range + an ordered op list" instead of "an enum + a param bag".
3. A biome becomes "an XY region predicate + an op list" — the same mechanism. Today these are two unrelated systems (ECaveGeneratorType dispatch vs the biome field with its bOverrideTerrain special case that only works for SurfaceWorld). Unifying them is where the expressiveness comes from: any op, scoped by any region, in any combination.
4. Numeric interval bounds where profiling shows Both is costing real gen time.
5. Only if needed: compile the per-chunk stack into a flat opcode tape and interpret with a switch, killing per-voxel virtual dispatch (~43k samples/tile × N ops). Standard technique. Don't do this pre-emptively — measure first; the noise is likely still dominant.

---

5. Invariants this must not break

Non-negotiable. Each has a documented reason and, mostly, a scar.

• Window invariance (§8.4). Every op stays a pure function of world coords + seed. Any op with a connectivity decision over a neighbourhood inherits BuildChunkCache's two-region COLLECT/STORE discipline. Prefer the Maze pattern (pure hash of (lower node, axis) — adjacent chunks cannot disagree, no cache, no COLLECT region) unless connectivity is a gameplay requirement.
• T1.a column cache Z-independence. FSurfaceColumn data is keyed (XY box, StrateKey, Seed) with no ChunkZ and shared down the whole vertical stack. XY-pure data goes in the column cache; Z-dependent evaluation happens per voxel. IsXYPure() exists to make this explicit instead of implicit.
• Cache keys include LayoutVersion (AUDIT C2 — today's CP_Chunk/OC_Chunk/BM_Chunk don't, and serve stale params after a live edit). FVoxelOpContext carries it so a new op can't forget.
• ProcessQueue stays EQueueMode::Mpsc; ops are read-only on workers; Epoch carries through every async path.
• The two-pass MC loop, margin ring, and thread_local grid reuse (§8.10) are untouched by all of this — the op stack lives below GetDensityAt, the mesher never knows.

---

6. Risks, and how each one is detected

Risk	Detection
A wrong EffectOverBox ⇒ a hole	Phase 0.5 test #2 (brute-force vs verdict) — the load-bearing one
A cache key missing an input ⇒ seams	Phase 0.5 test #1 (shuffled order, multi-threaded)
Per-voxel dispatch cost	Insights VoxelForge_GenerateMesh before/after each port; tape compile if it bites
Abstraction is wrong for this domain	The Phase 1 stop-trigger — one archetype, timeboxed, revert cheaply
Scope creep into a node-graph editor	See §7

---

7. Explicitly NOT doing

• No node-graph editor. An ordered TArray of op assets is 90% of the value. A graph UI is a separate project, years later, and chasing it is how generative systems die.
• No GPU density. Same reasons as fable-idea Part I: readback latency, CPU collision, and cross-GPU float determinism is fatal for "replicate the seed, regenerate identically on every peer" (ARCHITECTURE §9.1).
• No node-level interval arithmetic engine. Black-box Lipschitz/amplitude bounds per op (§2).
• No big-bang port. If more than one archetype is mid-port at any time, stop.
• No dissolving the strate structure. See §0.

---

8. Independent fixes — do these regardless, they get worse with time

From AUDIT-2026-07.md §5. None depend on this plan; all become harder inside it.

1. Bound SeedF (AUDIT C1) — const float SeedF = (float)(VoxelHash::Mix((uint32)Seed) & 0x3FFF); at all 6 definition sites. Large seeds currently collapse noise terms to constants. One world re-tune. Do it before tuning 3D caves against a seed you might later randomise.
2. Add GetLayoutVersion() to CP_Chunk / OC_Chunk / BM_Chunk (AUDIT C2).
3. GetPlayerPosition no-player flag (AUDIT C4) — (0,0,0) is the designed spine landing and currently stalls all streaming.
4. Unbounded joins on shutdown (AUDIT C5).
5. Track the .md files (AUDIT P1) — !*.md in .gitignore. This file is currently untracked.
6. Branch the experimental 3D work (AUDIT §6.6) — git switch -c 3d-density.

---

9. Resume here

Next action: Phase 0 — ship the 3D caves with CaveSystemEffectOverBox as a standalone function. Nothing in this plan is started.

When picking this up cold: read §0, §2 (the direction-only insight — that's what makes step 1 small), and §4. The rest is reference. If the plan feels too big, re-read §2: Phase 1 needs no numeric bounds at all, and the first real win is two archetypes collapsing into one.

---

Changelog — 2026-07-26: written by Opus 5 after a full-tree audit, at Jahni's request, as a durable handoff so a future context doesn't re-derive it. Companion to AUDIT-2026-07.md.