research: Astra pass 2, whole generation chain cost map (read-only)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016dhxM43dnWvUu2B5Ph1iG1

Standalone/Research/WorldAssemblerM3/Astra1/BRIEF-pass2.md

@@ -0,0 +1,48 @@
+Thank you, that was a strong first pass. Your scope warning about L76 was right and important. I need to correct my
+brief: "about 75% of field time is the passage composer" was **wrong**.
+- L76's term profiler covered only about 6.0M legacy samples for FI (5.6M for TN).
+- L73 counts about 52M density calls per FI or TN whole view: 512 LOD0 tiles × 44,121, 448 LOD1 × 43,322, and so
+  on.
+- The measured passage/support cost is about 6.5 of FI's ~40 field worker-seconds (~15%), not 75%. Roughly
+  30 worker-seconds of field time is not attributed to any term we've measured.
+
+Your gain arithmetic for recommendation 1 inherited that error. Please redo it.
+
+## Deeper pass (you now have more reasoning budget)
+
+Examine **the whole generation chain**, end to end, for one cold whole view on the legacy FI/TN path. For each
+stage, say what it does, how often it runs, what it costs per call (from code structure and the measured numbers),
+and whether its work is necessary:
+- world/plan resolution;
+- tile key selection;
+- `ClassifyBoxDetailed`;
+- the dispatch into `AnalyticDensityAt` (and whatever runs for the ~46M calls the term profiler did not see);
+- the composer;
+- grid fill;
+- MC;
+- normals;
+- skirts;
+- ground/cap refinement;
+- payload;
+- Unreal publication.
+
+The central question: **where do the unattributed ~30 worker-seconds most plausibly go?** For example:
+- a different evaluation path the profiler doesn't instrument;
+- per-call overhead outside the timed terms (dispatch, stratum lookup, virtual calls, allocation, cache misses);
+- border/halo samples evaluated twice;
+- normals sampling the field again;
+- duplicated work between neighbouring tiles or LODs.
+
+Give file:line for each. Then name the cheapest measurement that would confirm your top explanation, so the next
+implementation round can verify it first.
+
+Then **revise your ranked recommendations** in the light of the whole chain. Keep what still holds and drop what
+doesn't. Add anything the chain review reveals.
+
+Same rules as before:
+- read-only;
+- no edits anywhere, including temp files;
+- web allowed;
+- labels [CODE]/[MEASURED]/[PRIOR ART]/[INFERENCE].
+- At most ~3,500 words.
+- **Lead with the stage-by-stage cost map**, then the revised ranking.

Standalone/Research/WorldAssemblerM3/Astra1/REPORT-pass2.md

@@ -0,0 +1,130 @@
+# Astra 1, pass 2 — whole-chain review (gpt-6-astra, effort medium, read-only), 2026-10-27
+
+Claude verified: PackedDensityAt (ShapeWorldInternal.h ~5909) applies ApplyResolvedStrateComposition outside the L76 term scope; FScopedDensityQueryProfile times whole invocations crossing a 256-point boundary (batch-biased).
+
+**Stage-by-stage cost map**
+
+[MEASURED] The L73 stage timings below are historical, before the later skips. They describe where that run spent time; they cannot be subtracted from L76 term timings to produce a current residual. I also inspected the saved L76 results without executing generation.
+
+[CODE] A major architectural detail: the “compiled-analytic” world **already contains precomputed packed signs and quantized surface-distance proxies**. Many lattice queries never enter `AnalyticDensityAt`.
+
+| Stage | Work and frequency | Cost and necessity |
+|---|---|---|
+| **World/plan resolution** | [CODE] Headless loads the owner export, resolves/validates its layout, constructs a snapshot, then constructs `World`. Once per cold run. UE’s serialization entry is [`VoxelForgeCoreAdapter.cpp:2062`](/home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelForgeCoreAdapter.cpp:2062); headless startup begins at [`StreamBenchmarkHeadless.cpp:1659`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Source/StreamBenchmarkHeadless.cpp:1659). | [MEASURED] Saved L76 FI owner-data loading takes about **25 ms**. No complete isolated resolution cost is supplied. [INFERENCE] Necessary, but not a plausible home for worker time measured inside tile-field queries. |
+| **Shape compilation** | [CODE] `World::compile` builds primitives/bins, packed rock intervals, candidate bands, detailed distance proxies, sign-edge repairs, passage validation and far parents. Once for the base shape; additional pages when needed. [`VoxelForgeCore.cpp:1394`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/VoxelForgeCore.cpp:1394), [`ShapeWorldInternal.h:2724`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:2724). | [MEASURED] Saved L76 FI: **313 ms wall**, including approximately 48 ms shapes, 64 ms intervals/signs, 62 ms surface band and 119 ms far parents. [INFERENCE] Important to a one-second cold budget, but separate from the missing tile-field attribution. It already amortizes substantial base-field work. |
+| **Tile-key selection** | [CODE] `desiredTiles` builds clipmap keys, vertical bounds and additional far surface columns. Once initially, then on streaming-set changes; about 2,250 tiles for FI/TN. [`VoxelForgeCore.cpp:1904`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/VoxelForgeCore.cpp:1904). | [INFERENCE] Thousands of key/set operations, not millions of field evaluations. No measured isolated cost. Necessary selection; exact tile count remains a design choice. |
+| **Whole-tile classification** | [CODE] Each assembly tile calls `ClassifyBoxDetailed`, including a one-step halo. It delegates to `ClassifyBoundsDetailed`, with conditional slicing and conservative vetoes. [`VoxelForgeCore.cpp:2377`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/VoxelForgeCore.cpp:2377), [`ShapeWorldInternal.h:6385`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:6385). | [MEASURED] L73: only **4/3 ms total FI/TN**, roughly **2/1 μs per tile**, but ineffective legacy skips then. These are not current classifier costs. [INFERENCE] Necessary only when avoided work pays for it; retain current profitable proofs. |
+| **Density dispatch and base evaluation** | [CODE] Scalar lattice queries probe native/legacy-surface paths, then choose packed lookup or analytic evaluation. Batches have additional fast paths. [`ShapeWorldInternal.h:3475`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:3475), [`:3602`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:3602), [`:5909`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:5909). | [MEASURED] L73’s approximately 52M calls and 39.9/33.2 field worker-seconds imply roughly **0.77/0.64 μs per counted point**, averaged over heterogeneous paths. [INFERENCE] Fetching/evaluating the base value is necessary; repeated dispatch is not inherently necessary. |
+| **Composer** | [CODE] Packed and scalar analytic paths both call `ApplyResolvedStrateComposition`: cavern graph → boundary → passage/support → stamps → compatibility floor caps. [`ShapeWorldInternal.h:1993`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:1993). Frequency depends on the actual route, not just the strate label. | [MEASURED] L76 measures approximately **6.5/5.5 worker-seconds** only within its eligible analytic scopes. [CODE] Its passage/support bucket also includes `ApplyResolvedCavernFloorCaps` at [`:2053`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:2053). [INFERENCE] Composition is necessary where influential; universal invocation is not. |
+| **Grid fill** | [CODE] One cached lattice per generated tile: typically **35³ = 42,875** points at LOD0/1, including halo. Scalar, X8 or Z-run traversal fills it. Storage is thread-local and reused. [`Standalone mesher:430`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Source/VoxelForge/Private/VoxelMarchingCubesMesher.cpp:430), [`:957`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Source/VoxelForge/Private/VoxelMarchingCubesMesher.cpp:957), [`:1296`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Source/VoxelForge/Private/VoxelMarchingCubesMesher.cpp:1296). | [MEASURED] L73: **36.14/29.37 s**, approximately **16.1/13.2 ms per visited tile**, including evaluation. [INFERENCE] Dense fill is avoidable in proven-uniform regions. This is the primary workload, not merely array writes. |
+| **Marching cubes** | [CODE] Classifies cells using cached corners, computes geometry only for surface cells, interpolates edges and deduplicates vertices. LOD0 midpoint correction can issue extra unclamped queries. [`Mesher:1879`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Source/VoxelForge/Private/VoxelMarchingCubesMesher.cpp:1879), [`:1943`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Source/VoxelForge/Private/VoxelMarchingCubesMesher.cpp:1943). | [MEASURED] L73 MC aggregate: **40.58/32.63 s**, including grid work. Subtracting grid gives about **4.44/3.26 s**, not another 40 seconds. [INFERENCE] Necessary extraction, but replacing MC is not the first lever. |
+| **Normals** | [CODE] Central differences normally read the existing grid. At Step 1, proxy-band fallback values trigger `DensityAtUnclamped`; optional gradient memoization avoids repeated corner gradients. [`Mesher:1654`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Source/VoxelForge/Private/VoxelMarchingCubesMesher.cpp:1654). | [MEASURED] L73: **0.306/0.246 s**, about **0.14/0.11 ms per visited tile**. [INFERENCE] Some repeated field queries exist, but six new field evaluations per vertex is not the general path. Normals cannot plausibly explain 30 seconds under these measurements. |
+| **Skirts** | [CODE] Boundary-edge detection, extrusion and density probes prevent unsuitable skirt geometry. Work scales with boundary geometry, not tile volume. [`Mesher:2111`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Source/VoxelForge/Private/VoxelMarchingCubesMesher.cpp:2111). | [MEASURED] L73: **1.079/1.091 s**, approximately **0.48/0.49 ms per visited tile**. [INFERENCE] Necessary for the current seam solution; bounded opportunity, not the missing dominant cost. |
+| **Ground/cap refinement and far shells** | [CODE] Far stratum/connector repairs and native ground/cap refinement run under eligibility gates, with refinement at LOD≥4 and no overlapping edits. [`VoxelForgeCore.cpp:2792`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/VoxelForgeCore.cpp:2792), [`:2875`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/VoxelForgeCore.cpp:2875). | [MEASURED] L73’s combined “Other,” including these and assembly, is **2.86/2.78 s**. [INFERENCE] Required by the current far representation, but partly corrective work that a better far representation could avoid. Not an explanation for missing grid time. |
+| **Payload** | [CODE] Core assembles material/vertex/index outputs; the headless wrapper converts vertices and moves indices/masks. Per output vertex/index, once per tile. [`VoxelForgeCore.cpp:2969`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/VoxelForgeCore.cpp:2969), [`StreamingWorld.h:934`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Source/StreamingWorld.h:934). | [MEASURED] The separately timed headless conversion is **12 ms total**, about **5 μs per tile**. [INFERENCE] This does not include all Core assembly or Unreal upload/collision costs. It is not a useful primary target. |
+| **Unreal publication** | [CODE] Worker generation → prepared streams → result queue → budgeted game-thread component/upload/section work → asynchronous collision completion. [`VoxelWorld.cpp:13303`](/home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:13303), [`:9400`](/home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:9400), [`:14010`](/home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:14010), [`:14279`](/home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:14279). | [MEASURED] SW has **1.33 s additional readiness latency**. [INFERENCE] That is not 1.33 s of CPU publication work. Per-tile apply/cook costs are not established here. Essential to the target, but cannot explain headless field time. |
+
+**Where the “unattributed 30 seconds” most plausibly went**
+
+[CODE] **First: packed-field composition is outside L76’s measurement scope.** The relevant chain is:
+
+`DensityAtLattice` → `PackedDensityAt` → `ApplyResolvedStrateComposition`.
+
+`PackedDensityAt` reads `RockWords`, `CandidateWords` and possibly `SurfaceDistanceQ`, then applies composition. It never enters the `FScopedLegacyFieldTermScope` created later inside `AnalyticDensityAt` at [`ShapeWorldInternal.h:5967`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:5967).
+
+[CODE] Term timers immediately return when that scope is inactive: [`ShapeWorldInternal.h:2275`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:2275). Thus an expensive passage query can be counted by the outer field profiler but entirely absent from L76’s passage timing.
+
+[CODE] `DensityDispatchPathAt` returns “compiled-analytic/FloatingIslands” from the bound generator kind without distinguishing packed from analytic execution: [`ShapeWorldInternal.h:3455`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:3455). That label is insufficient for performance attribution.
+
+[CODE] There are further differences:
+
+- X8 can return packed proxy values directly without invoking the scalar composer; other conditions make it fall back to scalar calls. [`ShapeWorldInternal.h:3675`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:3675)
+- FI LOD0 preferentially uses scalar vertical traversal. [`Mesher:1288`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Source/VoxelForge/Private/VoxelMarchingCubesMesher.cpp:1288)
+- Z-run dispatch can fall back to scalar lattice queries, including packed queries. Its analytic term scope is reached only after those fallbacks. [`ShapeWorldInternal.h:11460`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:11460)
+- Native/surface branches and non-FI/TN samples are excluded from the legacy-term population.
+
+[INFERENCE] Therefore, the correction establishes that **6.5 seconds is measured passage cost within one subset**, not that total passage cost is only 15%. Packed composition is my leading explanation for substantial uninstrumented work. I cannot assign it 30 seconds from source alone.
+
+[MEASURED] **Second: the residual mixes different generations of the workload.** Existing [L76 saved results](/home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l76/term-profile-before/summary.json) contain:
+
+| Same saved L76 run | FI | TN |
+|---|---:|---:|
+| Density point calls | 26.99M | 25.13M |
+| Eligible legacy points | 6.01M | 5.56M |
+| Reported field estimate | 18.19 worker-s | 15.49 worker-s |
+| Tile-worker total | 33.80 worker-s | 28.56 worker-s |
+| Whole-view wall | 4.621 s | 3.977 s |
+
+[INFERENCE] The approximately 46M unseen calls belong to subtracting L76’s eligible count from L73’s pre-skip population. They are not an established current population. Likewise, `40−measured_terms` is not a coherent current residual.
+
+[CODE] **Third: the aggregate field estimator can be biased by batching.** `FScopedDensityQueryProfile` samples when the point sequence crosses a multiple of 256, timing the entire invocation and recording its point count. The reducer pools sampled time/points and scales by all points. See [`ShapeWorldInternal.h:2355`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:2355) and [`StreamBenchmarkPipeline.h:5611`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Source/StreamBenchmarkPipeline.h:5611).
+
+[INFERENCE] A batch of eight has approximately eight times a scalar invocation’s chance of being selected, and contributes eight sampled points when selected. Pooling both without inclusion-probability correction overrepresents batch points. If packed batches are cheap and scalar composition is expensive, the estimate can substantially understate field time. Periodic sampling also risks correlation with spatial traversal.
+
+[MEASURED] This deserves attention: L76 FI LOD0 records **12.70 s grid fill but only 5.96 s estimated field evaluation**. Conversely, LOD3 estimates **3.04 s field time against 2.58 s total tile-worker time**. Those are diagnostic inconsistencies, not proof of where the discrepancy lies. [Saved log](/home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l76/term-profile-before/strate1-w8.log:12880)
+
+[INFERENCE] **Lower-ranked explanations:**
+
+- **Dispatch/lookups:** real repeated work. Stratum lookup uses binary search and coordinate conversion, sometimes repeatedly per sample: [`ShapeWorldInternal.h:437`](/home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Core/src/ShapeWorldInternal.h:437). But attributing 30 seconds to this alone means roughly **0.65 μs per purported 46M calls**. That needs evidence; hidden composer execution is more concrete.
+- **Allocation/cache misses:** possible contributors, but grid allocation is reused and packed access is direct indexing. There is no evidence of one heap allocation per ordinary density point.
+- **Halos/neighbor duplication:** significant but bounded. `35³−33³ = 6,938`, or **16.2%** of a dense tile’s samples, is the gradient halo. Including shared faces, an ideal large same-LOD region could reduce `35³` toward `32³` unique samples per tile: approximately **23.6%** savings before edges and synchronization. This multiplies existing evaluation cost; it does not create a separate unmeasured stage.
+- **Normals/midpoints/skirts:** issue additional calls, but their measured stage budgets exclude them as the dominant explanation. Cross-LOD duplication remains unquantified.
+
+**Cheapest decisive measurement**
+
+[INFERENCE] First take a sampling call-stack profile of the clean production run, focused on grid fill. It requires no new per-sample timers and should immediately distinguish packed composition from dispatch or memory stalls.
+
+[INFERENCE] The next instrumentation change should be a **mutually exclusive dispatch census**, not another individual-term profiler. Count and sample time for packed-scalar, packed-X8, analytic-scalar, analytic-batch, native and legacy-surface routes, split by LOD and caller stage. Measure composition inside packed-scalar explicitly.
+
+Use equal-probability invocation sampling with correct expansion, or stratify by route **and batch length**. Avoid nested double-counting; compare the resulting total against directly timed grid fill. Separate passage carving from compatibility floor caps.
+
+[INFERENCE] This falsifies my leading explanation if packed-scalar composition occupies little time. Only then prioritize dispatch, cache misses or other overhead.
+
+**Revised ranked recommendations**
+
+**1. Establish complete attribution, then specialize the dominant lattice route.**
+
+[INFERENCE] This replaces my unconditional “passages first” ranking. The first implementation round should produce a closed cost account and identify which route dominates LOD0/1.
+
+If packed composition dominates, give each tile/brick a stable local composition program and an identity fast path. Extend `FScopedPassageTileQueries`, `PackedDensityAt` and `ApplyResolvedStrateComposition`. Hoist Z-only dispatch decisions into plane/run contexts. Before copying the X8 shortcut into scalar traversal, establish why their composition behavior differs and which semantics are required.
+
+[INFERENCE] Effort: small for attribution, medium for specialization. Risks: incorrect support envelopes, changed composition order, scalar/batch disagreement. Preserve peer determinism and audit clearance, support and grade; old-version hashes remain an exactness diagnostic.
+
+[MEASURED → INFERENCE] **Corrected passage arithmetic:** using your historical budget strictly as an illustration:
+
+- FI, removing 80–90% of **6.5/46** worker-seconds: `4.6 × (1−0.8…0.9 × 6.5/46)` = **4.08–4.02 s**.
+- TN, removing 80–90% of **5.5/39**: approximately **3.81–3.75 s**.
+
+These replace my erroneous approximately two-second estimates. They are not forecasts because the operands span rounds. Using the contemporaneous saved L76 totals instead gives approximately **3.91–3.83 s FI**, **3.37–3.29 s TN**, under proportional scaling.
+
+[INFERENCE] Total passage savings could be larger if the missing packed-route work is substantial. Measure it before assigning a percentage.
+
+**2. Remove repeated composed sampling; retain MC.**
+
+[INFERENCE] Keep the retained-proof/local-program recommendation, but build on the existing packed representation. Priorities:
+
+- Retain successful slice/brick proofs rather than requiring a uniform whole-tile verdict.
+- Reuse final composed lattice samples across neighboring tiles where inputs and semantics match.
+- Evaluate exact surface endpoints and normal stencils; leave certified interiors implicit.
+
+The relevant changes remain `ClassifyBoundsDetailed`, `World::generateTile` and the standalone mesher’s grid fill. Sample-cache keys must include evaluator version, edits and relevant resolution/context—not coordinates alone.
+
+[INFERENCE] An alternative worth testing is applying composition once to a shared near-region lattice. Count its cold construction cost: moving the same work earlier is not a speedup. It wins through deduplication, localization or cheaper batching.
+
+[INFERENCE] Effort: medium-to-large. Risks: stale edit dependencies, seams and false uniform proofs. Cheap falsifier: one representative region spanning adjacent tiles and a strate boundary; compare total construction plus meshing cost against independent tiles.
+
+[INFERENCE] Withdraw the earlier combined one-second projection. At unchanged throughput, FI’s current 4.6 seconds requires approximately **78% critical-path reduction**. Border sharing alone cannot supply that; broad sampling elimination still has to prove its economics.
+
+**3. Preserve a separate arrival deadline and reduce end-to-end latency.**
+
+[INFERENCE] Keep dependency-aware scheduling through collision publication, rather than merely increasing queue depth. The existing near-priority and hot-swap mechanisms already provide the foundation.
+
+Shape compilation’s approximately 0.31 seconds matters once generation approaches one second. Far repair/refinement becomes material too. Optimize these after the dominant grid route is understood; do not hide their cost outside the cold timer.
+
+[MEASURED → INFERENCE] SW’s scheduling gap remains a possible **0–1.33-second** latency opportunity, not a guaranteed CPU saving. Scheduling cannot remove required whole-view work.
+
+[INFERENCE] Effort: medium. Risks: starving far coverage or publishing visuals before safe collision. Cheap falsifier: an arrival-only admission experiment plus prerequisite/publication tracing. If its uncontended critical path exceeds one second, scheduling alone cannot meet arrival readiness.
+
+[INFERENCE] GPU porting, SIMD, native-generator replacement and a new mesher stay below these three. The newly visible distinction is between **cheap cached base density and potentially expensive repeated composition**, not simply “52M analytic evaluations.”
+
+Read-only inspection completed; no edits, builds, tests, commits or stashes.