Fr0zkaandClaude Opus 5.5
81b7d6d0ff
Cause found by profiling: room/SDF graph-cache churn (exact XY bounds + sample order rebuilt it constantly). Region grouping (one LOD3 tile: 1,084 -> 229 builds), taller-tile traversal for LOD6 (36,480 -> 12,160), extent-dependent cache slots, lambdas instead of recursive std::function. Serial ms/tile, 64 nearest tiles per level (before -> after): L0 50.4 -> 50.6, L1 112 -> 110, L2 24 -> 22, L3 223 -> 73, L4 269 -> 134, L5 393 -> 127, L6 283 -> 145. Hashes of all levels unchanged; parity 3,436/3,436 bit-identical. Full clipmap, 16 workers BelowNormal: full detail 66.6 s, all 12,054 tiles 175 s. In-stream L0 tail is high (median 83 ms, mean 198 ms, p95 669 ms) vs 51 ms serial. Streaming: recentre keeps the intersection, generates only missing tiles nearest/finest first; one crossing = exactly 441 new tiles in 6.1 s, no stale results. Lighting: normals already come from the density gradient; the striping was the squared headlamp; broad linear fill + warmer palette. Daytime CPU rule documented in Standalone/README.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmSoF64BbRFmk9C3p1VsBJ
247 lines
14 KiB
Markdown
247 lines
14 KiB
Markdown
# VoxelForge standalone world
|
||
|
||
This C++20 program loads the owner snapshot from `data/owner_world.json`, runs the copied
|
||
VoxelForge density pipeline and marching-cubes mesher, streams the clipmap with six worker threads
|
||
by default, and displays it through Direct3D 11. Unreal is not linked or needed at run time.
|
||
|
||
## Build and launch
|
||
|
||
From the plugin root, run:
|
||
|
||
```powershell
|
||
.\Standalone\Build.ps1
|
||
.\Standalone\bin\VoxelForgeStandalone.exe
|
||
```
|
||
|
||
The build uses the configured VS 2022 `vcvars64.bat`, MSVC `/std:c++20 /O2 /arch:AVX2 /fp:precise`.
|
||
`Build.ps1` starts MSVC at BelowNormal priority and does not enable `/MP`.
|
||
Pass another executable name to build beside any viewer that may be open:
|
||
`.\Standalone\Build.ps1 -OutputName VoxelForgeStandalone_test.exe`. The executable finds the
|
||
JSON beside itself under `Standalone\data\`.
|
||
|
||
### Daytime resource limit
|
||
|
||
Run tests and benchmarks at Idle or BelowNormal priority and use no more than 8 of this PC's 16
|
||
logical CPUs. The default viewer uses six workers. Use `Build\LaunchBelowNormal.ps1` for any test or
|
||
benchmark process; it starts the child at BelowNormal and verifies the priority. Builds use no more
|
||
than four compiler processes (the current build is serial) and retain `/fp:precise`. The all-core
|
||
16-worker runs documented below were the owner's one-night exception.
|
||
|
||
## Controls
|
||
|
||
- Hold right mouse button and move to look.
|
||
- WASD moves; Space/Ctrl or E/Q moves vertically; Shift moves faster; the wheel changes speed.
|
||
- F toggles wireframe.
|
||
- 1 through 7 toggle clipmap levels 0 through 6 individually; 0 toggles every level.
|
||
- N/P changes the seed. T switches between the owner's base tunnels and the research-shaped round,
|
||
winding tunnel option; `--round-tunnels` starts in that variant. Either action rebuilds the clipmap.
|
||
Crossing a chunk recenters streaming:
|
||
tiles still desired stay resident, only new tiles are generated, and leaving tiles are dropped.
|
||
- Esc exits.
|
||
|
||
The console reports density parity at startup, each level's generation time, and the total time.
|
||
|
||
## Copied code and compatibility layer
|
||
|
||
`Source\VoxelForge\` contains the plugin's density evaluator and operator stack, cave morphology,
|
||
passage geometry, strate definitions/layout/manager/composer, noise, voxel math and types,
|
||
marching-cubes tables/mesher, and clipmap tile selection. These are copies. Plugin runtime source
|
||
is untouched apart from the allowed editor commandlet export addition.
|
||
|
||
`Compat\CoreMinimal.h` supplies the UE-style arrays/maps/sets, vectors and boxes, math helpers,
|
||
strings, smart pointers, locks, logging/stat macros, UObject-like data holders, and platform helpers
|
||
needed by the copied CPU code. The `Compat` subfolders cover the UE include paths used by those
|
||
copies. Perlin 2D/3D uses the copied UE 5.7 implementation in `UnrealPerlin.cpp` and
|
||
`UnrealPerlinHelpers.inl`. `OwnerWorldLoader.h` maps the exported settings, resolved strate
|
||
definitions, biomes, and terrain operations into copied holders; soft references resolve through a
|
||
small local asset registry. The copied strate manager reconstructs runtime passages from that data
|
||
and seed. The export also records their geometry in JSON for reference.
|
||
|
||
## Behavior differences
|
||
|
||
- Standalone `VOXEL_SIZE` is 25 cm; the plugin's checked-out value remains 12.5 cm. No other voxel
|
||
conversion constant needed changing. Mesh vertices stay in centimetres in the copied mesher and
|
||
the viewer converts them to metres at upload.
|
||
- The compatibility `TMap` uses an open-addressed hash index with stable pair storage;
|
||
`TSet` uses that index for lookup. Iteration follows insertion order rather than Unreal hash-table
|
||
order. Owner strate slots and passages are read from ordered JSON arrays. Perlin noise and
|
||
deterministic `DetSinCos` are copied unchanged; MSVC uses `/fp:precise` without fast-math/FMA
|
||
contraction.
|
||
- Unreal reflection and UObject construction are replaced by explicit JSON-to-holder mapping.
|
||
Density-relevant settings, resolved strate layouts, passages, biome parameters, and terrain-op
|
||
assets are loaded. Runtime actor/material assets are not instantiated; the viewer uses one lit
|
||
rock material for all generated geometry.
|
||
- The owner export has no active season. Standalone season loading is not implemented; an active
|
||
season would need to be exported and mapped before it could affect this viewer.
|
||
- `WITH_EDITOR` research/debug branches, Unreal logging, startup trace, profiling counters, and
|
||
reflection metadata do not run in the standalone. The base generator path is the default.
|
||
- T selects a standalone-only tunnel-shape port. It disables the owner's first-strate tunnel
|
||
connections and adds the research field's cubic winding centerline and round elliptical tunnel
|
||
profile (144-voxel pitch, 28-voxel radius, 26-voxel half-height, deterministic variable radius
|
||
and vertical wave), anchored around the camera's current height. It reuses the base rooms and
|
||
passages; it does not port the research field's room planner or full parameterized density field.
|
||
|
||
## Owner data and parity
|
||
|
||
`data/owner_world.json` came from one default-off `VoxelForgeExploreCommandlet` export using the
|
||
guarded `Tools\VoxelForgeTest.ps1 -Scenario standalone-export -Build -Assets owner` path. The
|
||
snapshot contains the resolved settings/strates/passages and 3,436 plugin density samples. For that
|
||
commandlet build, `VOXEL_SIZE` was temporarily set to 25 cm in the worktree and restored afterward;
|
||
the exported JSON records 25 cm and the checked-out plugin source still records 12.5 cm.
|
||
|
||
Standalone comparison: **3,436/3,436 bit-identical, zero mismatches**. Samples covered strate
|
||
boundaries (2,420), varied world coordinates including negative and far-from-origin points (768),
|
||
tunnel interiors (222), room interiors (12), inter-strate passages (10), and passage landings (4).
|
||
|
||
## Baseline clipmap timing
|
||
|
||
Owner seed 0, base world, 25 cm voxels, six workers; the final viewer run generated the full
|
||
seven-level clipmap (12,054 tiles, 6,605,189 vertices, 36,307,137 indices) in 515.117 seconds.
|
||
These level timers cover CPU meshing and queue handoff; D3D11 buffer upload is separate. An earlier
|
||
headless pass over the same tiles took 408.719 seconds, so use one run mode consistently when
|
||
comparing later speed changes.
|
||
|
||
| Level | Seconds | Tiles | Vertices | Density samples |
|
||
|---:|---:|---:|---:|---:|
|
||
| 0 | 70.201 | 2,744 | 957,655 | 117.65 M |
|
||
| 1 | 65.376 | 2,009 | 657,423 | 86.14 M |
|
||
| 2 | 9.042 | 1,274 | 292,267 | 8.74 M |
|
||
| 3 | 25.689 | 588 | 235,755 | 4.03 M |
|
||
| 4 | 24.248 | 441 | 229,720 | 3.02 M |
|
||
| 5 | 23.809 | 294 | 199,503 | 2.02 M |
|
||
| 6 | 296.304 | 4,704 | 4,032,866 | 32.26 M |
|
||
| **Total** | **515.117** | **12,054** | **6,605,189** | **253.86 M** |
|
||
|
||
## Profiling result and speed change
|
||
|
||
The compatibility layer's old `TMap`/`TSet` linear lookups and heap-only vector growth were
|
||
replaced with an open-addressed hash index, stable pair storage, small inline storage, and a
|
||
thread-local pooled allocator. That removes known container and allocation mismatches with the
|
||
Unreal path, but did not improve the earlier LOD0 timing (72.1 s before, 74.1 s after); profiling
|
||
showed those were not the dominant cost. The same audit also found two recursive room-graph walks
|
||
wrapped in `std::function`, which could allocate and dispatch indirectly on every graph build. They
|
||
now use self-recursive generic lambdas with the same depth-first visit order, removing that wrapper
|
||
overhead from the generation hot path. In matched profile runs, RoomGraphBuild fell from 4.531 to
|
||
4.417 s for 64 LOD5 tiles and from 5.210 to 5.038 s for 64 LOD6 tiles; graph build counts and
|
||
density hashes were unchanged.
|
||
|
||
Per-stage timers and cache counters found the real hotspot in `GetDensityAt`: the room/SDF graph
|
||
cache is keyed by exact XY bounds, and the original X-fast lattice walk kept rebuilding it. On one
|
||
LOD3 tile, cache-region grouping cut graph builds from 1,084 to 229 while writing samples to their
|
||
original lattice indices. A 64-tile LOD6 profile attributed 13.189 s of 19.874 s to 36,480 room
|
||
graph builds. For tiles wider than 16 chunks, keeping each XY column's Z run contiguous reduced
|
||
that to 12,160 builds and 5.210 s of 10.972 s, with the same lattice hash. The final recursive-
|
||
lambda profile measured 5.038 s in room-graph builds and 10.600 s total for the same 64 LOD6 tiles.
|
||
Finer tiles retain the plane-first traversal, which is faster for their shorter cache runs.
|
||
|
||
Cache-slot experiments found a second scale-specific case: 32-chunk LOD5 tiles improve with two
|
||
room graph cache slots (26,926 to 9,700 builds; profiled RoomGraphBuild 10.151 to 4.475 s across
|
||
64 tiles). Two slots regress 64-chunk LOD6 tiles (12,160 to 23,104 builds), so the generator uses
|
||
two only for 32-chunk tile contexts and one otherwise. The selected-tile density hash is unchanged.
|
||
|
||
The controlled before/after run below meshes the 64 nearest tiles for each LOD serially, without
|
||
profiling or hash collection. It uses the same owner seed and camera-centered desired set. The
|
||
selected-tile lattice hashes from a separate final run are identical at every LOD.
|
||
|
||
| LOD | 64-tile density hash |
|
||
|---:|---:|
|
||
| 0 | `0x3d31a090650399d3` |
|
||
| 1 | `0x6150c607c7a5c76f` |
|
||
| 2 | `0x9f1d560bc14e67f1` |
|
||
| 3 | `0xdd3e76020ce69cd7` |
|
||
| 4 | `0x59a209a5db034b46` |
|
||
| 5 | `0x1fdeb60160e4216f` |
|
||
| 6 | `0xbac69461f812493d` |
|
||
|
||
| LOD | Before ms/tile | After ms/tile | After p50 | After p95 |
|
||
|---:|---:|---:|---:|---:|
|
||
| 0 | 50.4 | 50.551 | 50.521 | 63.148 |
|
||
| 1 | 112.3 | 110.175 | 33.030 | 407.767 |
|
||
| 2 | 23.9 | 22.430 | 12.015 | 51.078 |
|
||
| 3 | 223.0 | 73.482 | 69.614 | 94.331 |
|
||
| 4 | 269.0 | 133.512 | 98.003 | 214.854 |
|
||
| 5 | 393.1 | 127.325 | 116.386 | 173.401 |
|
||
| 6 | 282.8 | 145.457 | 140.952 | 183.234 |
|
||
|
||
Run mode: BelowNormal priority, serial headless benchmark, 64 nearest tiles per LOD, profiling and
|
||
hash collection off; final executable `VoxelForgeStandalone_night20.exe`. The before timings are
|
||
from the earlier 64-tile baseline. The user's prior plugin reference is about 95 ms worker time for
|
||
a 25 cm LOD0 tile. The final nearest-tile LOD0 average is 50.6 ms. In the complete streamed run,
|
||
LOD0 median was 82.9 ms, but its mean was 197.6 ms and p95 was 668.8 ms. The plugin was not timed
|
||
on the same tile coordinates, so the rough reference comparison is not apples-to-apples and the
|
||
streamed tail does not meet that target yet.
|
||
|
||
With 16 BelowNormal workers, the final full run completed all 12,054 tiles in 175.056 s, down from
|
||
the historical 515.117 s six-worker run. It reached full detail after 66.565 s. Worker time per
|
||
level in the final run was:
|
||
|
||
| LOD | Worker sum | Tiles | Avg/tile | p50 | p95 |
|
||
|---:|---:|---:|---:|---:|---:|
|
||
| 0 | 542.284 s | 2,744 | 197.625 ms | 82.946 ms | 668.796 ms |
|
||
| 1 | 501.518 s | 2,009 | 249.635 ms | 89.990 ms | 742.258 ms |
|
||
| 2 | 68.712 s | 1,274 | 53.934 ms | 55.633 ms | 120.102 ms |
|
||
| 3 | 77.055 s | 588 | 131.047 ms | 122.697 ms | 212.157 ms |
|
||
| 4 | 103.552 s | 441 | 234.812 ms | 192.137 ms | 418.249 ms |
|
||
| 5 | 66.689 s | 294 | 226.834 ms | 223.139 ms | 328.030 ms |
|
||
| 6 | 1,439.125 s | 4,704 | 305.936 ms | 303.062 ms | 391.440 ms |
|
||
|
||
The worker sums are concurrent wall-time measurements and therefore add to more than the 175.056 s
|
||
clipmap wall time. The historical and final full runs used different worker counts; the serial
|
||
64-tile table is the controlled per-tile comparison. The full run also passed density parity at
|
||
3,436/3,436 samples with zero mismatches.
|
||
|
||
## Streaming and uploads
|
||
|
||
The viewer rebuilds the desired tile set at each chunk crossing, keeps the intersection resident,
|
||
cancels queued jobs that left the set, and enqueues only missing tiles. A heap orders by camera
|
||
distance, then prefers finer LOD at equal distance. Workers generate CPU meshes concurrently. D3D11
|
||
uploads are limited to four batches or 2 ms per frame. A leaving or changed batch keeps its prior GPU
|
||
buffer until its replacement is uploaded, avoiding a hole while the batch is rebuilt.
|
||
|
||
The final 16-worker run reached full detail 66.565 s after spawn (4,753 LOD0–1 desired tiles; 5,173
|
||
resident through LOD1) and the full clipmap at 175.056 s. In a controlled one-chunk crossing on the
|
||
same executable, 4,856 tiles stayed resident, 441 left, and only 441 new jobs were submitted. Those
|
||
jobs covered 18.91 M density samples, used 88.573 worker-seconds, and finished in 6.108 s wall time;
|
||
all 441 were generated with no stale results or queued cancellations. The upload budget remained
|
||
four batches / 2 ms per frame.
|
||
|
||
For a deterministic movement check, `--test-one-chunk-crossing` moves once after full-detail
|
||
readiness and closes the test viewer after that crossing's jobs and bounded uploads drain.
|
||
|
||
## Lighting comparison
|
||
|
||
The copied plugin mesher already computes central-difference density-gradient normals on the sampled
|
||
density lattice and interpolates them along marching-cubes edges. The standalone passes those
|
||
normals through to D3D11, so replacing face normals was unnecessary. Profiling measured this existing
|
||
gradient stage at 2.575 ms across 64 LOD0 benchmark tiles (about 72 ms over all 2,744 streamed LOD0
|
||
tiles); this round adds no density samples or meshing work for normals.
|
||
|
||
The visible striping was amplified by the viewer's squared, high-intensity camera-facing light. The
|
||
shader now uses a broad linear camera fill at similar average strength. Geometry, density, and LOD
|
||
are unchanged; the shading path uses fewer multiplications. The D3D11 pixel-shader adjustment has
|
||
no CPU mesh-generation cost.
|
||
|
||
Both captures use the default seed, default camera, 1280x800 view, and are saved from the D3D11 back
|
||
buffer after all full-detail tiles upload:
|
||
|
||
| Before | After |
|
||
|---|---|
|
||
|  |  |
|
||
|
||
## Tunnel palette pass
|
||
|
||
As a separate, render-only visual pass, the neutral blue-gray rock constants were changed to warm
|
||
limestone and ochre, with a softer cyan camera fill. This follows the color balance of the approved
|
||
cave reference while leaving its mesh, 25 cm resolution, and detail untouched. The shader changes
|
||
only existing constant values and adds no texture reads or operations; CPU density and meshing cost is
|
||
zero. The matched 16-worker captures generated the same 7,521,175 vertices and 41,506,623 indices;
|
||
full-clipmap wall times were 172.772 s before and 171.676 s after, within run variance.
|
||
|
||
Both images show seed 0 with the round/winding tunnel field, at 1280x800 after the full clipmap
|
||
uploaded. Camera: `(-11.75, -123.25, -19.85)` m, yaw 90 degrees, pitch 0. Capture mode used 16
|
||
BelowNormal workers:
|
||
|
||
| Before palette pass | After palette pass |
|
||
|---|---|
|
||
|  |  |
|