research: Astra pass 3, compiled spatial passage evaluator design (read-only)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016dhxM43dnWvUu2B5Ph1iG1

Standalone/Research/WorldAssemblerM3/Astra1/BRIEF-pass3.md

@@ -0,0 +1,74 @@
+Thank you for both earlier passes. Your rec #1 (localize passages) turned out to be right, and here is what we've
+learned since. This is a focused third pass: **passage evaluation design.** Read and reason only; no edits
+anywhere, including temp files. Web allowed. Labels as before.
+
+## What we know now (all measured, ablations verified to remove the work)
+
+- **Owner four-strate world, cold LOD0–2-ready at W12:** FI 2.27 s, TN 2.25 s. Passages off: FI 0.81 s,
+  TN 0.85 s. Boundary bands cost only 0.1–0.2 s. Passage composition costs **~1.4–1.6 s** of near readiness, and
+  it alone separates FI/TN from the ~1 s goal. SW 1.02 s and VS 1.13 s are close already.
+- **Inside the passage phase** (sampled, L83): carve distance ~40%, wall relief ~21–24% (3D value noise, 3 octaves
+  at 43/29/19 voxels), support restoration/columns ~14%, mouth shaping ~4%, other ~11%. Top symbols:
+  `ApplyResolvedStratePassageCarves`, `VF_FieldCachedValueNoise`, `VisitNearestPassageSpatialNode`,
+  `PassagePotentialWithinNode`, the carve lambda.
+- **Exact trims done** (L81–L85):
+  - a density-aware outer reach bound (carve = `max(source, -roughSDF/Q)`, reach = `-d·Q + 0.24 m`);
+  - an exact relief skip when it can't change the max;
+  - per-tile BVH node masks and contiguous ranges;
+  - heavy-tile splitting;
+  - heavy-first order.
+
+  Each gave 5–12%. Readiness is throughput-bound (workers 81–85% busy).
+- **Two representation shortcuts failed:**
+  - L86: a lazy exact per-sample result cache. 24% hit rate, ~700 MB, +47% slower.
+  - L88: a tile-local 4-voxel trilinear cache of passage deltas, exact near the surface. Coarse-node
+    evaluations still paid full route cost, and carve-segment visits rose from 34.5M to 56.2M (FI), making it
+    +46% slower.
+
+  **Lesson:** each route evaluation is expensive in itself, and reorganising *where* we evaluate doesn't help
+  unless the evaluation gets cheaper.
+- **Composer semantics that must keep their meaning:**
+  - the carve is `max(source, -roughSDF/Q)`;
+  - support restoration, applied after later overlapping bores;
+  - segment-local capsule clearance;
+  - mouth shaping and fades;
+  - the walkable tread floor;
+  - Catmull–Rom joins on short segments;
+  - wall/floor relief of ±0.21/0.20 m.
+
+  Reports: `Standalone/Research/WorldAssemblerM3/Linux81`–`Linux88/REPORT.md`; code in `ShapeWorldInternal.h`
+  (`ApplyResolvedStratePassageCarves` and its helpers).
+
+## The owner's rules (unchanged)
+
+- One seed gives the same world on every machine. Byte equality with older versions isn't required; a perf pass
+  may change the world slightly.
+- Passages stay walkable arrival → departure (support, capsule, grade, ripple audits), and look natural.
+- No visible quality regression.
+- Target: **LOD0–2 ready in ~1 s** after cold arrival. The whole view is a bonus; continuous low-priority far
+  generation afterwards.
+
+## Questions
+
+1. **What makes one route evaluation expensive** (read the code path end to end)? Per sample: how many segments are
+   visited, what does each visit compute (closest point on a Catmull–Rom or linear segment, frames, radius
+   interpolation, relief noise, support capsule tests, later-bore checks)? Which parts are redundant per sample, per
+   segment, or per tile?
+2. **Design a passage representation where per-sample cost is near-constant and small,** and the work is not merely
+   moved around. Candidates to evaluate (don't limit yourself):
+   - analytic primitives: polyline/capsule chains with precomputed per-segment frames and an arc-length table;
+   - relief defined in 2D (arc length, angle) with a cheap periodic or table noise;
+   - a precomputed narrow-band distance grid per passage, built once per world at a fixed resolution with
+     deterministic integer construction, plus its build cost;
+   - separating "is this sample within reach" (cheap) from "exact carve value" (expensive);
+   - vectorising across a z-run of samples sharing the same segment candidates.
+
+   For each: expected per-sample cost, build cost, memory, how exact (displacement vs today), determinism risk, and
+   effort.
+3. **Rank by expected gain toward ~1 s,** with arithmetic from the numbers above. For the top choice, describe the
+   smallest prototype and the cheapest falsifier, so the implementer can prove it before building it fully.
+
+## Output
+
+At most ~3,000 words, **ranked recommendation first**, file:line evidence, labels [CODE]/[MEASURED]/[PRIOR ART]/
+[INFERENCE].

Standalone/Research/WorldAssemblerM3/Astra1/REPORT-pass3.md

@@ -0,0 +1,161 @@
+# Astra 1, pass 3 — passage evaluation design (gpt-6-astra, medium, read-only), 2026-10-28
+
+**[INFERENCE] Ranked recommendation: replace recursive route queries with a compiled spatial passage evaluator—not another cache of evaluated densities.**
+
+1. **Compile short lists of prepared carve, tread and clearance primitives into spatial bricks.** Evaluate those lists directly, retaining ordered support restoration and explicit mouth exceptions.
+2. **Add a compact relief representation**, preferably sampled from today’s relief on the passage surface. Treat this as part of recommendation 1’s performance budget, not a sufficient standalone fix.
+3. **Consider a sparse distance grid only if built directly from prepared primitives**, without calling today’s composer at every grid node. Its cold construction cost makes it a less attractive first prototype.
+
+[MEASURED → INFERENCE] The target requires a substantial replacement:
+
+\[
+T_{\text{FI}}\approx0.81+1.46r+B,\qquad
+T_{\text{TN}}\approx0.85+1.40r+B
+\]
+
+Here \(r\) is the fraction of passage cost remaining and \(B\) is added cold critical-path construction time. This is an ablation-based planning model, not a guaranteed linear prediction.
+
+| Passage cost remaining | FI, before added construction | TN |
+|---|---:|---:|
+| 30% | 1.25 s | 1.27 s |
+| 15% | 1.03 s | 1.06 s |
+| 10% | 0.96 s | 0.99 s |
+| 5% | 0.88 s | 0.92 s |
+
+[INFERENCE] **Tenfold passage acceleration barely reaches one second before construction.** With 30 ms additional construction, TN needs approximately **11.7×**. Another 5–12% trim cannot close this gap.
+
+**What one evaluation actually does**
+
+[CODE] The entry point is [`ApplyResolvedStratePassageCarves:1115`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:1115). Its cost is a sequence of different geometric questions:
+
+1. **Reach screening.** Check tile flags, incoming density, passage Z range and floor-profile ranges. Solid samples can invoke `PassageMayAffectDensity` before entering the main passage loop. Inside that loop, another reach test may occur.
+2. **Column/support preparation.** Scan eligible void crossings and possibly evaluate column distance.
+3. **Nearest carve feature.** Search four ranges: descending body, upper endpoint, lower-mouth neighborhood and departure tail.
+4. **Relief and mouth shaping.** Compute wall orientation, route-relative distances, entrance projection, relief and fades.
+5. **Tread selection.** Find the floor-owning segment separately from the nearest carve segment; evaluate up to three neighboring support segments.
+6. **Clearance and composition.** Check the passage’s player capsules, later bores and other passages’ capsules; apply, cancel or defer support. Finally restore mouth support and deferred support.
+
+[CODE] **A carve segment visit is not an iterative closest-point solve on Catmull–Rom.** It computes endpoint differences, squared length, a clamped projection onto the straight chord, then optionally evaluates a Catmull–Rom position at that same parameter. Radius remains interpolated from the endpoints. Distance requires a deterministic square root, followed by stable tie-breaking. [`ShapeWorldInternal.h:1497`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:1497)
+
+[INFERENCE] Consequently, replacing this with a mathematically accurate spline-distance solver would likely increase cost and change the field. Compiling the existing formula—or approximating it deliberately—is the appropriate comparison.
+
+[CODE] **Traversal itself is substantial.** Each nearest-node bound computes distance to an AABB using a square root, subtracts maximum radius, checks range membership and visits children. Potential-within queries perform a separate recursive search. Tile masks and contiguous ranges already remove some overhead; they do not eliminate per-sample traversal. [`:5590`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:5590), [`:5676`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:5676)
+
+[MEASURED → INFERENCE] L88’s approximately 34.5M/33.1M carve visits divided by approximately 14.06M/12.43M passage iterations gives about **2.45/2.66 carve visits per iteration**, assuming those census populations align. This is not visits per density query or per successful route evaluation: iterations include rejected passages, and the visit count excludes internal BVH nodes and separate capsule searches. **The problem is not necessarily hundreds of evaluated segments; it is expensive repeated search around relatively few winners.**
+
+[CODE] **Support is a different geometry query.** It uses horizontal projection, extended tread segments, distinct smoothing rules, native-density-dependent mouth offsets and floor-depth gates. Its floor-profile search deliberately avoids choosing a neighboring helix coil merely because that coil supplies the nearest wall. [`:1833`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:1833), [`:2127`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:2127)
+
+[CODE] Capsule checks project horizontally and test a vertical capsule with 0.34 m radius and 0.54 m axial half-length. Later-bore checks already have precomputed overlap candidates, but can fall back to BVH searches. Deferred support is restored after the passage loop. [`:5708`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:5708), [`:5810`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:5810), [`:2163`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:2163)
+
+[CODE] **Relief still costs work on cache hits.** Each of three octaves computes lattice coordinates, a cache hash, fades and seven interpolations. A miss additionally generates eight corner values. The cache has 64 direct-mapped entries. Wall relief uses wavelengths 43/29/19 and weights 0.50/0.35/0.15. [`VoxelForgeFieldShared.h:260`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/VoxelForgeFieldShared.h:260), [`:394`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/VoxelForgeFieldShared.h:394)
+
+[CODE] Mouth work additionally evaluates noise at projected route positions. Some calculations happen before determining whether the sample belongs to the mouth’s small affected region. [`ShapeWorldInternal.h:1683`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:1683)
+
+[INFERENCE] The reusable information is:
+
+- **Per segment:** chord vectors, inverse squared lengths, lengths, radius slopes, cubic coefficients, support flags, overlap extension and prefix/suffix arc lengths.
+- **Per spatial brick:** possible carve winners, floor owners, clearance capsules, column features and ordered passage dependencies.
+- **Per sample:** projections, distances, source-dependent support blending and final min/max operations.
+- **Across a Z-run:** candidate lists and XY projections; not generally the winning segment or final support decision.
+
+[CODE] `BuildPassageComposerConstants` currently prepares passage-level metadata, not this complete segment representation. [`:5147`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:5147)
+
+**1. Compile a spatial passage program**
+
+[INFERENCE] Build immutable sparse bricks containing contiguous references to prepared features. Use separate lists for **carve, tread, clearance and mouth/column operations**. A single nearest-segment identifier cannot represent their different ownership rules.
+
+For each brick:
+
+- Eliminate segments that cannot win its carve query, using conservative lower/upper distance bounds.
+- Retain every potentially relevant overlapping coil.
+- Store floor candidates independently.
+- Store later-bore and other-capsule dependencies in stable passage order.
+- Subdivide crowded bricks or use an explicit slow fallback; never silently cap candidates.
+
+[INFERENCE] The common query becomes one brick lookup, typically a few direct projections/distances, one relief lookup and a small ordered composition sequence. No reach-BVH traversal followed by nearest-BVH traversal followed by clearance-BVH traversal.
+
+**Important limitation:** “two to four candidates” is a design target to falsify, not something the current route guarantees. Nearby coils and short, densely authored segments may produce much larger lists.
+
+[INFERENCE] Prepare the existing chord/Catmull formulas first. This avoids multiplying segment count through premature tessellation. Subsequently merge or approximate pieces only where a measured error budget permits. Precomputed coefficients and reciprocals may change floating-point rounding; version the evaluator and enforce the same arithmetic across peers.
+
+[PRIOR ART] If route-coordinate relief needs frames, compute stable frames once. Rotation-minimizing frames are established for swept surfaces; ordinary Frenet frames can become discontinuous at inflections. **Keep the walkable floor gravity-referenced**, regardless of the wall-texture frame. [Wang et al., rotation-minimizing frames](https://www.microsoft.com/en-us/research/publication/computation-rotation-minimizing-frames/)
+
+[INFERENCE] **Do not discard the density-dependent reach rule.** A narrow geometric envelope alone is insufficient when `max(source, -roughSDF/Q)` can change deeply negative source values outside the visible tube. Outside a brick’s detailed region, return identity only with a valid source-aware proof. Otherwise use a wider prepared candidate set or an explicit fallback. Count that fallback in the speed budget.
+
+[INFERENCE] Cost model:
+
+- **Query:** target roughly **0.1–0.3 μs** for an ordinary active query with 2–4 prepared candidates, excluding rare complex exceptions. This is an engineering target, not measured performance.
+- **Build:** \(O(S+R)\) segment preparation/reference emission, plus candidate-certification work; \(S\) segments and \(R\) brick references. Target **under 30 ms added cold critical path** initially.
+- **Memory:** approximately 128–256 bytes per prepared segment, plus roughly `16 + 4K` bytes per simple brick list, before multiple semantic lists and directory overhead. Thousands of segments should require megabytes, not L86’s hundreds of megabytes.
+- **Exactness:** prepared original formulas can preserve geometry closely; segment simplification has no automatic displacement guarantee. Keep mouth/support equations source-dependent.
+- **Effort:** medium-to-large; the candidate compiler and semantic separation are the substantial work.
+
+[INFERENCE] Leaving the entire old support phase untouched is unlikely to suffice: **14% of the 1.4-second passage contribution is approximately 0.20 seconds**, already exceeding TN’s available 0.15-second margin. Preserve support’s meaning, but accelerate its feature selection too.
+
+**2. Replace repeated 3D relief evaluation with a small surface field**
+
+[INFERENCE] Use coordinates along the route and around its wall, with periodic angular indexing and continuous frames. A table can combine all three octaves into one lookup. Avoid evaluating general-purpose `atan2` unnecessarily; a specified octant-based angular mapping is an option, but its distortion must be checked.
+
+Prefer initially **sampling today’s 3D relief at the reference wall surface** rather than inventing unrelated 2D noise. This retains much of its spatial appearance. Extend the value radially only over a controlled band; preserve the floor-alignment weights and separate mouth-floor treatment.
+
+[INFERENCE] Costs and risks:
+
+- **Query:** target **20–80 ns** for coordinate mapping plus a warm four-value lookup; cold memory or frame interpolation can exceed this.
+- **Build/memory:** illustrative 1 km of route, 0.25 m longitudinal spacing, 64 angular samples and int16 values: **256k samples, approximately 0.5 MB**. Build requires those relief evaluations once, not full composer evaluations.
+- **Exactness:** independently regenerated noise can differ by **0.42 m in wall offset** between opposite extrema despite retaining ±0.21 m amplitude. Equal amplitude does not establish equal quality. Surface sampling reduces this risk but does not prove a displacement bound.
+- **Determinism:** fixed integer coordinates, specified interpolation/rounding, periodic endpoints and stable frame construction.
+- **Effort:** medium; seam handling, frame continuity and floor/mouth validation dominate.
+
+[MEASURED → INFERENCE] Even removing **all** the measured 21–24% relief share saves only roughly **0.3 seconds**. An 80% relief reduction alone predicts roughly **2.0 seconds**, not one second. It belongs in the combined representation.
+
+[INFERENCE] Cheapest falsifier: evaluate table interpolation error and normal/ripple changes on one curved wall, one mouth and one self-overlapping coil. Reject a table that needs near-original per-sample 3D noise to hide seams or preserve appearance.
+
+**3. Sparse narrow-band distance grids**
+
+[INFERENCE] A grid gives genuinely near-constant queries, but only if its construction avoids L88’s failure: **do not fill it by invoking the existing full route evaluator per node**.
+
+Rasterize prepared local primitives into owned bricks, reduce candidates deterministically, and construct the band from those primitives. Store separate carve and support/clearance information where needed. A baked final delta cannot represent all source-density-dependent mouth restoration.
+
+[PRIOR ART] OpenVDB’s narrow-band representation stores active distances near the surface and constant signed values elsewhere. That is useful representation prior art, but its clamped exterior is not automatically compatible with your composer. [OpenVDB representation](https://github.com/AcademySoftwareFoundation/openvdb/blob/master/doc/doc.txt)
+
+[INFERENCE] For route length \(L\), radius \(r\), band half-width \(w\) and spacing \(h\), band sample count is approximately:
+
+\[
+N \approx 4\pi r w L/h^3.
+\]
+
+Illustratively, \(L=1{,}000\) m, \(r=3\) m, \(w=0.75\) m:
+
+- **25 cm spacing:** approximately 1.8M samples, **3.6 MB per int16 channel**.
+- **12.5 cm spacing:** approximately 14.5M samples, **29 MB per channel**.
+
+Overlapping passages, mouth volumes, brick padding and multiple channels add memory.
+
+[INFERENCE] **Query:** roughly **0.05–0.25 μs per interpolated channel** as a target, with locality decisive. **Build:** millions of local distance operations; even 0.1–0.3 μs per sample means 0.18–0.54 CPU-seconds for the coarser example, before indexing, extra channels or allocation. Parallel wall time must be measured; it is not free.
+
+[INFERENCE] Grid spacing is not a displacement bound. At 25 cm, narrow tread/clearance transitions need analytic treatment or finer data. Integer storage does not ensure deterministic construction: integer distance/projection arithmetic, tie-breaking, reduction order and interpolation must also be specified.
+
+[INFERENCE] Effort: large. This is attractive only if direct construction plus all arrival queries beats the compiled analytic representation. A tiny benchmark of grid lookups alone proves nothing.
+
+**Other candidates**
+
+- **[INFERENCE] Cheap reach versus exact value:** retain the distinction, but answer reach from brick metadata or squared primitive bounds. The current distinction already exists; repeating its recursive BVH search is the expense.
+- **[INFERENCE] SIMD Z-runs:** useful after candidate compilation. XY projection is reusable for tread/capsule queries, while 3D carve projection varies affinely with Z before clamping. Expect amortized lookup and arithmetic benefits, not eightfold speedup. Memory adds only scratch arrays; construction is unchanged. Deterministic lane arithmetic and stable winner selection are required. Medium effort.
+- **[INFERENCE] Pure capsule-chain replacement:** cheap per primitive, but tessellation can increase candidate count and introduce curvature/radius artifacts. Preserve short-join shape with bounded approximation; do not equate a visually smooth centerline with a walkable tread.
+
+**Smallest prototype and decisive falsifier**
+
+[INFERENCE] Implement one prepared passage evaluator for the **FI/TN junction passage**, including an overlapping coil and tread—not a straight isolated tube.
+
+1. Compile segment constants and brick candidate lists; retain existing geometry formulas and relief.
+2. Run the complete carve/support/clearance sequence from those lists.
+3. Measure construction, query time, candidate-count distribution and fallback time separately.
+4. Add the relief table only after direct evaluation beats the old traversal.
+
+[INFERENCE] The cheapest first falsifier is the **candidate compiler census**: if commonly sampled bricks still need many segments or fallback retains more than approximately 10% of original passage time, the proposed common path cannot meet the target even if nearly free.
+
+[INFERENCE] Then replay representative samples with their incoming densities and original spatial order. Include misses, mouths, support and overlapping bores; exclude benchmark-only allocations. A promising kernel must finally survive cold arrival timing including construction, peer comparisons, capsule/support/grade/ripple audits and visual inspection.
+
+[INFERENCE] Budget acceptance around **90–95% total passage-cost removal**, not an attractive isolated kernel speedup. The measured ceiling makes approximately one second plausible, but only if the new representation replaces the search-heavy semantic pipeline broadly enough.
+
+Read-only review; no files changed or generation/tests run.