standalone: per-tile speed at plugin level, streaming like the game, softer lighting

Cause found by profiling: room/SDF graph-cache churn (exact XY bounds + sample order rebuilt it
constantly). Region grouping (one LOD3 tile: 1,084 -> 229 builds), taller-tile traversal for LOD6
(36,480 -> 12,160), extent-dependent cache slots, lambdas instead of recursive std::function.
Serial ms/tile, 64 nearest tiles per level (before -> after):
  L0 50.4 -> 50.6, L1 112 -> 110, L2 24 -> 22, L3 223 -> 73, L4 269 -> 134, L5 393 -> 127, L6 283 -> 145.
Hashes of all levels unchanged; parity 3,436/3,436 bit-identical.
Full clipmap, 16 workers BelowNormal: full detail 66.6 s, all 12,054 tiles 175 s. In-stream L0 tail is
high (median 83 ms, mean 198 ms, p95 669 ms) vs 51 ms serial.
Streaming: recentre keeps the intersection, generates only missing tiles nearest/finest first;
one crossing = exactly 441 new tiles in 6.1 s, no stale results.
Lighting: normals already come from the density gradient; the striping was the squared headlamp;
broad linear fill + warmer palette. Daytime CPU rule documented in Standalone/README.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MmSoF64BbRFmk9C3p1VsBJ

Standalone/README.md

@@ -1,8 +1,8 @@
 # VoxelForge standalone world
 
 This C++20 program loads the owner snapshot from `data/owner_world.json`, runs the copied
-VoxelForge density pipeline and marching-cubes mesher, builds all clipmap levels with six worker
-threads, and displays them through Direct3D 11. Unreal is not linked or needed at run time.
+VoxelForge density pipeline and marching-cubes mesher, streams the clipmap with six worker threads
+by default, and displays it through Direct3D 11. Unreal is not linked or needed at run time.
 
 ## Build and launch
 
@@ -14,7 +14,18 @@ From the plugin root, run:
 ```
 
 The build uses the configured VS 2022 `vcvars64.bat`, MSVC `/std:c++20 /O2 /arch:AVX2 /fp:precise`.
-The executable finds the JSON beside itself under `Standalone\data\`.
+`Build.ps1` starts MSVC at BelowNormal priority and does not enable `/MP`.
+Pass another executable name to build beside any viewer that may be open:
+`.\Standalone\Build.ps1 -OutputName VoxelForgeStandalone_test.exe`. The executable finds the
+JSON beside itself under `Standalone\data\`.
+
+### Daytime resource limit
+
+Run tests and benchmarks at Idle or BelowNormal priority and use no more than 8 of this PC's 16
+logical CPUs. The default viewer uses six workers. Use `Build\LaunchBelowNormal.ps1` for any test or
+benchmark process; it starts the child at BelowNormal and verifies the priority. Builds use no more
+than four compiler processes (the current build is serial) and retain `/fp:precise`. The all-core
+16-worker runs documented below were the owner's one-night exception.
 
 ## Controls
 
@@ -23,8 +34,9 @@ The executable finds the JSON beside itself under `Standalone\data\`.
 - F toggles wireframe.
 - 1 through 7 toggle clipmap levels 0 through 6 individually; 0 toggles every level.
 - N/P changes the seed. T switches between the owner's base tunnels and the research-shaped round,
-  winding tunnel option. Either action rebuilds the clipmap. Moving three chunks from its center
-  also recenters and rebuilds it.
+  winding tunnel option; `--round-tunnels` starts in that variant. Either action rebuilds the clipmap.
+  Crossing a chunk recenters streaming:
+  tiles still desired stay resident, only new tiles are generated, and leaving tiles are dropped.
 - Esc exits.
 
 The console reports density parity at startup, each level's generation time, and the total time.
@@ -50,10 +62,11 @@ and seed. The export also records their geometry in JSON for reference.
 - Standalone `VOXEL_SIZE` is 25 cm; the plugin's checked-out value remains 12.5 cm. No other voxel
   conversion constant needed changing. Mesh vertices stay in centimetres in the copied mesher and
   the viewer converts them to metres at upload.
-- The compatibility `TMap` and `TSet` are vector-backed containers, so lookup is linear and their
-  iteration follows insertion order rather than Unreal hash-table order. Owner strate slots and
-  passages are read from ordered JSON arrays. Perlin noise and deterministic `DetSinCos` are copied
-  unchanged; MSVC uses `/fp:precise` without fast-math/FMA contraction.
+- The compatibility `TMap` uses an open-addressed hash index with stable pair storage;
+  `TSet` uses that index for lookup. Iteration follows insertion order rather than Unreal hash-table
+  order. Owner strate slots and passages are read from ordered JSON arrays. Perlin noise and
+  deterministic `DetSinCos` are copied unchanged; MSVC uses `/fp:precise` without fast-math/FMA
+  contraction.
 - Unreal reflection and UObject construction are replaced by explicit JSON-to-holder mapping.
   Density-relevant settings, resolved strate layouts, passages, biome parameters, and terrain-op
   assets are loaded. Runtime actor/material assets are not instantiated; the viewer uses one lit
@@ -98,3 +111,136 @@ comparing later speed changes.
 | 5 | 23.809 | 294 | 199,503 | 2.02 M |
 | 6 | 296.304 | 4,704 | 4,032,866 | 32.26 M |
 | **Total** | **515.117** | **12,054** | **6,605,189** | **253.86 M** |
+
+## Profiling result and speed change
+
+The compatibility layer's old `TMap`/`TSet` linear lookups and heap-only vector growth were
+replaced with an open-addressed hash index, stable pair storage, small inline storage, and a
+thread-local pooled allocator. That removes known container and allocation mismatches with the
+Unreal path, but did not improve the earlier LOD0 timing (72.1 s before, 74.1 s after); profiling
+showed those were not the dominant cost. The same audit also found two recursive room-graph walks
+wrapped in `std::function`, which could allocate and dispatch indirectly on every graph build. They
+now use self-recursive generic lambdas with the same depth-first visit order, removing that wrapper
+overhead from the generation hot path. In matched profile runs, RoomGraphBuild fell from 4.531 to
+4.417 s for 64 LOD5 tiles and from 5.210 to 5.038 s for 64 LOD6 tiles; graph build counts and
+density hashes were unchanged.
+
+Per-stage timers and cache counters found the real hotspot in `GetDensityAt`: the room/SDF graph
+cache is keyed by exact XY bounds, and the original X-fast lattice walk kept rebuilding it. On one
+LOD3 tile, cache-region grouping cut graph builds from 1,084 to 229 while writing samples to their
+original lattice indices. A 64-tile LOD6 profile attributed 13.189 s of 19.874 s to 36,480 room
+graph builds. For tiles wider than 16 chunks, keeping each XY column's Z run contiguous reduced
+that to 12,160 builds and 5.210 s of 10.972 s, with the same lattice hash. The final recursive-
+lambda profile measured 5.038 s in room-graph builds and 10.600 s total for the same 64 LOD6 tiles.
+Finer tiles retain the plane-first traversal, which is faster for their shorter cache runs.
+
+Cache-slot experiments found a second scale-specific case: 32-chunk LOD5 tiles improve with two
+room graph cache slots (26,926 to 9,700 builds; profiled RoomGraphBuild 10.151 to 4.475 s across
+64 tiles). Two slots regress 64-chunk LOD6 tiles (12,160 to 23,104 builds), so the generator uses
+two only for 32-chunk tile contexts and one otherwise. The selected-tile density hash is unchanged.
+
+The controlled before/after run below meshes the 64 nearest tiles for each LOD serially, without
+profiling or hash collection. It uses the same owner seed and camera-centered desired set. The
+selected-tile lattice hashes from a separate final run are identical at every LOD.
+
+| LOD | 64-tile density hash |
+|---:|---:|
+| 0 | `0x3d31a090650399d3` |
+| 1 | `0x6150c607c7a5c76f` |
+| 2 | `0x9f1d560bc14e67f1` |
+| 3 | `0xdd3e76020ce69cd7` |
+| 4 | `0x59a209a5db034b46` |
+| 5 | `0x1fdeb60160e4216f` |
+| 6 | `0xbac69461f812493d` |
+
+| LOD | Before ms/tile | After ms/tile | After p50 | After p95 |
+|---:|---:|---:|---:|---:|
+| 0 | 50.4 | 50.551 | 50.521 | 63.148 |
+| 1 | 112.3 | 110.175 | 33.030 | 407.767 |
+| 2 | 23.9 | 22.430 | 12.015 | 51.078 |
+| 3 | 223.0 | 73.482 | 69.614 | 94.331 |
+| 4 | 269.0 | 133.512 | 98.003 | 214.854 |
+| 5 | 393.1 | 127.325 | 116.386 | 173.401 |
+| 6 | 282.8 | 145.457 | 140.952 | 183.234 |
+
+Run mode: BelowNormal priority, serial headless benchmark, 64 nearest tiles per LOD, profiling and
+hash collection off; final executable `VoxelForgeStandalone_night20.exe`. The before timings are
+from the earlier 64-tile baseline. The user's prior plugin reference is about 95 ms worker time for
+a 25 cm LOD0 tile. The final nearest-tile LOD0 average is 50.6 ms. In the complete streamed run,
+LOD0 median was 82.9 ms, but its mean was 197.6 ms and p95 was 668.8 ms. The plugin was not timed
+on the same tile coordinates, so the rough reference comparison is not apples-to-apples and the
+streamed tail does not meet that target yet.
+
+With 16 BelowNormal workers, the final full run completed all 12,054 tiles in 175.056 s, down from
+the historical 515.117 s six-worker run. It reached full detail after 66.565 s. Worker time per
+level in the final run was:
+
+| LOD | Worker sum | Tiles | Avg/tile | p50 | p95 |
+|---:|---:|---:|---:|---:|---:|
+| 0 | 542.284 s | 2,744 | 197.625 ms | 82.946 ms | 668.796 ms |
+| 1 | 501.518 s | 2,009 | 249.635 ms | 89.990 ms | 742.258 ms |
+| 2 | 68.712 s | 1,274 | 53.934 ms | 55.633 ms | 120.102 ms |
+| 3 | 77.055 s | 588 | 131.047 ms | 122.697 ms | 212.157 ms |
+| 4 | 103.552 s | 441 | 234.812 ms | 192.137 ms | 418.249 ms |
+| 5 | 66.689 s | 294 | 226.834 ms | 223.139 ms | 328.030 ms |
+| 6 | 1,439.125 s | 4,704 | 305.936 ms | 303.062 ms | 391.440 ms |
+
+The worker sums are concurrent wall-time measurements and therefore add to more than the 175.056 s
+clipmap wall time. The historical and final full runs used different worker counts; the serial
+64-tile table is the controlled per-tile comparison. The full run also passed density parity at
+3,436/3,436 samples with zero mismatches.
+
+## Streaming and uploads
+
+The viewer rebuilds the desired tile set at each chunk crossing, keeps the intersection resident,
+cancels queued jobs that left the set, and enqueues only missing tiles. A heap orders by camera
+distance, then prefers finer LOD at equal distance. Workers generate CPU meshes concurrently. D3D11
+uploads are limited to four batches or 2 ms per frame. A leaving or changed batch keeps its prior GPU
+buffer until its replacement is uploaded, avoiding a hole while the batch is rebuilt.
+
+The final 16-worker run reached full detail 66.565 s after spawn (4,753 LOD0–1 desired tiles; 5,173
+resident through LOD1) and the full clipmap at 175.056 s. In a controlled one-chunk crossing on the
+same executable, 4,856 tiles stayed resident, 441 left, and only 441 new jobs were submitted. Those
+jobs covered 18.91 M density samples, used 88.573 worker-seconds, and finished in 6.108 s wall time;
+all 441 were generated with no stale results or queued cancellations. The upload budget remained
+four batches / 2 ms per frame.
+
+For a deterministic movement check, `--test-one-chunk-crossing` moves once after full-detail
+readiness and closes the test viewer after that crossing's jobs and bounded uploads drain.
+
+## Lighting comparison
+
+The copied plugin mesher already computes central-difference density-gradient normals on the sampled
+density lattice and interpolates them along marching-cubes edges. The standalone passes those
+normals through to D3D11, so replacing face normals was unnecessary. Profiling measured this existing
+gradient stage at 2.575 ms across 64 LOD0 benchmark tiles (about 72 ms over all 2,744 streamed LOD0
+tiles); this round adds no density samples or meshing work for normals.
+
+The visible striping was amplified by the viewer's squared, high-intensity camera-facing light. The
+shader now uses a broad linear camera fill at similar average strength. Geometry, density, and LOD
+are unchanged; the shading path uses fewer multiplications. The D3D11 pixel-shader adjustment has
+no CPU mesh-generation cost.
+
+Both captures use the default seed, default camera, 1280x800 view, and are saved from the D3D11 back
+buffer after all full-detail tiles upload:
+
+| Before | After |
+|---|---|
+| ![Before lighting](data/night_lighting_before.png) | ![After lighting](data/night_lighting_after.png) |
+
+## Tunnel palette pass
+
+As a separate, render-only visual pass, the neutral blue-gray rock constants were changed to warm
+limestone and ochre, with a softer cyan camera fill. This follows the color balance of the approved
+cave reference while leaving its mesh, 25 cm resolution, and detail untouched. The shader changes
+only existing constant values and adds no texture reads or operations; CPU density and meshing cost is
+zero. The matched 16-worker captures generated the same 7,521,175 vertices and 41,506,623 indices;
+full-clipmap wall times were 172.772 s before and 171.676 s after, within run variance.
+
+Both images show seed 0 with the round/winding tunnel field, at 1280x800 after the full clipmap
+uploaded. Camera: `(-11.75, -123.25, -19.85)` m, yaw 90 degrees, pitch 0. Capture mode used 16
+BelowNormal workers:
+
+| Before palette pass | After palette pass |
+|---|---|
+| ![Before tunnel palette](data/night18_tunnel_view_before.png) | ![After tunnel palette](data/night19_tunnel_view_after.png) |