diff --git a/Standalone/Research/WorldAssemblerM3/Linux10/REPORT.md b/Standalone/Research/WorldAssemblerM3/Linux10/REPORT.md new file mode 100644 index 0000000..fd74aa0 --- /dev/null +++ b/Standalone/Research/WorldAssemblerM3/Linux10/REPORT.md @@ -0,0 +1,115 @@ +# Linux10 — startup and region crossing + +**Result:** no world-generation optimization was shipped. The dominant spawn gate is full shape compilation; the current Core API does not let tiles consume a partially built shape safely. L10 adds opt-in phase and crossing telemetry to the headless research path. + +Runs used CPUs 0–11, normal priority, at most `-j12`, a 6 GiB address-space cap, and the unchanged `powersave` governor. Peak monitored RSS was 624,996 KiB (610.3 MiB). Final builds: Clang 19 with FP contraction off, SHA-256 `7d5360e2baa637f8b2b4679df9616580bffe689afd101470e311260f0ecd4975`; GCC 14 default, `5041cf15019766f453eb59fe8ff7fc4cd98ca6a904208b645e3d7ca4584e1195`. The L9 Clang timing/hash baseline is `Standalone/bin/linux/VoxelForgeCoreHeadless-linux9-clang`, SHA-256 `0cd3f0ef993ced2c615418c6e2b37b83f57eff6d024d7d1568c30ea51490384a`. + +## 1. Startup phases + +SurfaceWorld seed 0, W12, no hash; five runs on the optimized Clang build. Wall and process CPU are medians. Process CPU sums work across threads. First-tile time is from cold start; its CPU column is CPU since pool start. + +| Phase | Wall ms | Process CPU ms | Threads used | +|---|---:|---:|---| +| Owner data load and resolve | 5.05 | 5.05 | 1 | +| Assembly plan build | 51.04 | 51.03 | 1 | +| Core snapshot | 0.53 | 0.54 | 1 | +| First desired-set build, 2,553 tiles | 0.74 | 0.74 | 1 | +| Shape request setup | 0.03 | 0.03 | 1 | +| Shape compile | 222.59 | 856.79 | caller + up to 8 scheduler workers | +| Streamer setup | 0.01 | 0.01 | 1 | +| Worker-pool start | 0.41 | 1.27 | 12 requested; 8–11 had entered at print | +| First tile starts at 299.62 ms | — | 1.59 since pool start | 12/12 entered | + +Shape-compile wall ranged 215.89–324.88 ms; first tile started at 286.56–409.76 ms. Owner load, plan, local shape setup, and far-parent construction are serial. Before the pool starts, there are no sleeping stream workers, but most of the 12 allowed CPUs are unused. During the interval and detail-band phases the scheduler uses at most 8 workers, leaving up to 4 CPUs unused. + +The phase-trace mode’s whole-view median was 1.436 s, versus 1.372 s with tracing off. The phase runs are for decomposition only; the performance ranking and A/B below use the optimized build with tracing disabled and hashing off. In that flag-off path, shape wall time was 0.222 s median, consistent with the phase trace. + +## 2. Shape contents and first-tile dependencies + +| Shape phase | Wall ms | Process CPU ms | Work / concurrency | +|---|---:|---:|---| +| Local owner shapes and spatial bins | 40.46 | 40.44 | Extracts owner primitives; 1 thread | +| Rock intervals, signs, candidate bands | 40.20 | 187.14 | Packed column signs and candidate masks; up to 8 workers | +| Detail band and sign-edge distances | 81.00 | 568.78 | Applies coherent detail to candidate nodes and builds distance proxies; up to 8 workers | +| Route validation and shared-field setup | 1.12 | 1.13 | 5,642 passage samples; then shared primitive form | +| Far parents and bins | 55.56 | 55.57 | 6,050 far rooms / 18,348 links; 1 thread | + +The compile extracts 18,348 owner primitives (71 rooms, 74 tunnels, 2 passages), covers 138,188,413 sign nodes, allocates 36.71 MiB of packed words and 263.57 MiB of proxy address capacity, and visits 214,538 detail candidates (6,098 flips). + +For this SurfaceWorld fixture, the first LOD0 tile’s density queries take the native `Heightfield2D` path before packed-shape lookup. Far-parent shapes are not needed by that first tile. However, `World::generateTile` rejects requests until the full base shape is marked built, and `compileTileRegion` also requires that base shape. So the current API gates the near tile on the interval, detail, validation, and far-parent work even when that tile does not sample those arrays. Other recipes and fallback/query paths do use the analytic primitives and packed signs/proxies. + +There is no seed-independent owner shape payload to cache at build time. The assembly plan is built from owner data; region pages also re-run owner primitive extraction, spatial-bin construction, and shared-field packing. Those immutable owner-derived structures are a possible per-world reuse target, but the measured short crossings below built no pages, so this run does not price that work. + +## 3. Moving crossing + +Hashing off, W12, same camera and already-streamed initial view. Region compile was called, but all entering fine keys were covered by the initial shape bounds; it built zero regions. + +| Camera step | New tiles | LOD breakdown | Region compile | Tile work wall | Crossing total | +|---|---:|---|---:|---:|---:| +| 8 m | 144 | LOD2: 128, LOD3: 16 | 0.010 ms | 12.61 ms | 12.81 ms | +| 16 m | 288 | LOD0: 128, LOD1: 16, LOD2: 128, LOD3: 16 | 0.010 ms | 60.49 ms | 60.68 ms | + +The 16 m crossing’s worker-time sum was 633 ms; its wall time was 60.5 ms. No plan or full shape compile repeated. A 24 m attempt terminated with `std::bad_alloc` after base shape compilation and before crossing metrics; the failing allocation site is unknown. + +## 4. Ranked levers + +| Rank | Lever | Upper bound from these measurements | Assessment | +|---:|---|---|---| +| 1 | Overlap near-tile work with shape compilation | 222.6 ms, 16.2% of 1.372 s whole-view-ready; first tile could start earlier by up to the compile time | Largest opportunity. Not built: tiles and region compilation currently require a fully built base shape. A safe version needs immutable partial-page publication and a wait before any tile reads an unfinished sign, proxy, primitive bin, or far field. | +| 2 | Defer far-parent construction until far tiles need it | 55.6 ms, at most 4.1% of whole-view-ready and 18.6% of the first-tile delay | Could improve spawn, but far and near paths share mutable spatial-bin state today. Needs separate immutable bins and a synchronized readiness state; no crossing benefit was measured. | +| 3 | Parallelize assembly-plan construction | 51.0 ms, at most 3.7% of whole-view-ready | Serial and seed/owner-specific. No safe deterministic parallel split was established. | +| 4 | Raise shape scheduler cap from 8 to 12 | About 121 ms currently lies in the two worker phases; ideal linear scaling saves about 40 ms (2.9% of whole-view-ready) | Below the 3% bar before overhead, and no crossing benefit. Not built. | + +For any future partial or deferred build, hashes can fail if a tile observes a partially written sign/proxy array, if region publication races with tile lookup, or if shared bins change while a tile is sampling. The page must be immutable before publication; worker-count and timing tests must compare W1 with W12 and repeat W12. Any asynchronous result must also follow the Epoch/MPSC shutdown-drain rules. + +## 5. Final-build timing and hashes + +Five interleaved W12 runs, no hash. Pair values are `whole_view_ready / stream_wall` in seconds. + +| Pair | L9 `7650abb` Clang | L10 Clang | Delta, ms | +|---:|---:|---:|---:| +| 1 | 1.336 / 1.050 | 1.346 / 1.060 | +10 / +10 | +| 2 | 1.351 / 1.062 | 1.365 / 1.073 | +14 / +11 | +| 3 | 1.359 / 1.073 | 1.374 / 1.083 | +15 / +10 | +| 4 | 1.363 / 1.077 | 1.375 / 1.081 | +12 / +4 | +| 5 | 1.387 / 1.089 | 1.372 / 1.092 | −15 / +3 | +| **Median** | **1.359 / 1.073** | **1.372 / 1.081** | **+13 / +8** | + +The measurement-only candidate is +1.0% whole-view-ready and +0.7% stream median; this is no demonstrated speedup. Hashing was off for these timings. + +Final rebuilt binaries passed SurfaceWorld and FloatingIslands seed-0 hash checks at W1, W8, and five W12 runs per compiler: Clang FP contraction off and GCC default. Every full-detail, whole-view, LOD0–8, and exact-query row matches the compiler-matched L9 baseline. Audit files: `Saved/Linux/l10_final_surface_hash_audit.txt` and `Saved/Linux/l10_islands_hash_audit.txt`. + +| Recipe / scope | Density | Mesh | Payload | Topology | +|---|---|---|---|---| +| SurfaceWorld full detail | `0x0ee0579bd15cbbe4` | `0x32e5d4964d610fee` | `0x1932b96ee4b1d910` | `0x69743c6bad3447dc` | +| SurfaceWorld whole view | `0xc10697a8f48380c7` | `0xe75c9119a1498cd5` | `0x49a9a8d1835a6c8c` | `0xedcb23288b5e8724` | +| FloatingIslands full detail | `0x6d842d2878b08ef6` | `0x4b868a3184f08035` | `0x2cedf2236788752d` | `0xeac17d5e0cb5044a` | +| FloatingIslands whole view | `0x5be4d8cde79b2a76` | `0x96f75041f6861b3a` | `0x3700a516cc89d50c` | `0x6b99948d6eab4f21` | + +Exact-query hashes also match: SurfaceWorld density `0xe4d4a8eb95b35933`, detail `0xefcba20c0c70c415`, collision `0xe604823a249029bf`; FloatingIslands density `0x61daded9c303becb`, detail `0xefcba20c0c70c415`, collision `0x84d37463be9b0573`. + +## 6. Failed or retried runs + +- One phase smoke omitted the explicit owner-data path and resolved it under `Standalone/bin/data`; reran with `Standalone/data/owner_world.json`. +- A no-camera phase smoke produced 2,370 tiles instead of the 2,553-tile fixture; discarded it and reran with the fixed camera. +- A malformed `--workers=w8` command exited during argument parsing; reran with numeric `--workers=8`. +- The first FloatingIslands matrix invocation omitted the three required streaming-mode switches. All 14 attempts exited before generation; corrected the command and all 14 final runs passed. +- The 24 m crossing attempt failed with `std::bad_alloc` as described above. +- The first hash-audit parser expected 13 rows; the actual audit contains 12 scopes. Corrected the parser and all final comparisons passed. + +At spawn, the first tile starts about 300 ms after cold start, with roughly 223 ms in full shape compilation and 51 ms in the single-threaded plan; this round did not make that wait shorter. While moving, the 16 m crossing completed 288 entering tiles in about 61 ms and repeated no shape compile, so the measured delay was tile work rather than region setup. + +## Orchestrator review (Claude) + +- **Not committed:** the L10 telemetry code. It is +1.0% whole-view median with tracing off (4 of 5 + pairs slower), and the rule is zero cost when off. Saved as `Saved/Linux/claude/l10-telemetry.patch` + (gitignored) for reuse; the tree is reset to `7650abb`. +- **Key finding, for L11:** the first SurfaceWorld tiles take the native `Heightfield2D` path and don't + read the cave shape, yet `World::generateTile` waits for the full base shape (~223 ms). A per-tile + readiness check (start tiles whose recipe path provably never touches shape arrays) is lever #1 in + safe form. +- **Open bug, for L11:** the 24 m crossing died with `std::bad_alloc` under the 6 GiB **address-space** + cap (`RLIMIT_AS`). That cap counts reservations, not resident memory; L10 also reports 263.57 MiB + of proxy address *capacity*. It is not yet known whether this is a real memory blow-up on larger + crossings or the cap counting reserved address space. L7's 32 m step also hit the 160M-node + interval bound.