Fr0zka/VoxelForge
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

wip(standalone): speed round -- interrupted by a PC hard reset at 10:16 (Kernel-Power 41)

Browse Source 
Verified before the reset (density + full mesh hashes equal at 1/8/16 workers, routes 5642/5642,
look audits green):
- merged field lookups, parallel mask merge, compact inline spatial bins, line-local relief hash
  cache, lattice-specific mesher entry, fused sign+magnitude evaluation, far connectors as one
  capsule each (181,836 -> 16,491 far primitives).
- crater edit (sphere+box+capsule, one transaction across LOD0/1): 15-27 ms; player 3x3x3
  neighbourhood meshes ready ~18 ms after streaming starts.
NOT yet verified (in the tree, unhashed): standalone-only skips of UV/colour arrays and of the
ceiling/ground material grouping (best cold whole view 0.958 s @16 workers) and a far-bin reserve
tweak. Those skips drop data the plugin's materials may need: not a game-path win as-is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MmSoF64BbRFmk9C3p1VsBJ
This commit is contained in:
Fr0zkaandClaude Opus 5.5 committed 2026-10-04 12:18:48 +02:00
1 parent 97c0455939
commit ee5b05b357
16 files changed
+1011 -156
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Standalone/Research/OwnerShapeWorld/speed_round/REPORT.md
+141
View File
@@ -0,0 +1,141 @@
# Shape world speed round
## Result
The one-second whole-view goal does **not** pass. On the final rebuilt source, cold process-start to all 6,498 tiles through LOD6 took 1.958 s at 8 workers and 1.629 s at 16. The measured shape build plus uncached mesh generation took 1.805 s / 1.454 s. LOD6 remains the largest single streaming stage. The full-detail 960-tile area took 0.896 s / 0.703 s for shape build plus mesh generation; process-start to completion was 1.057 s / 0.825 s. Collision cooking and GPU upload are not measured by this CPU harness.
Implementation: [shape compiler and field](../../../Source/StandaloneShapeWorld.h), [streaming/edit benchmark](../../../Source/StreamingWorld.h), [viewer measurements](../../../Source/StandaloneViewer.cpp), and [machine runner](RunMachine.ps1).
| Cold job | Workers | Shape build | Uncached mesh | Shape + mesh | Process start → completion | Cache hits |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| 64 m full-detail area, 960 tiles | 8 | 0.725 s | 0.171 s | **0.896 s** | **1.057 s** | 0 |
| 64 m full-detail area, 960 tiles | 16 | 0.549 s | 0.154 s | **0.703 s** | **0.825 s** | 0 |
| Whole view, 6,498 tiles through LOD6 | 8 | 0.844 s | 0.961 s | **1.805 s** | **1.958 s** | 0 |
| Whole view, 6,498 tiles through LOD6 | 16 | 0.672 s | 0.782 s | **1.454 s** | **1.629 s** | 0 |
The first four stage columns use separate cold, no-cache processes. Process-start timing begins immediately before `Start-Process`, includes executable launch and world setup, and ends after the final tile is meshed; it still excludes interactive GPU upload and collision cooking. The worker runner is Normal priority on affinity `0xFFFF` and polls at 50 ms. Peak private memory was 0.41 / 0.47 GiB for whole-view 8 / 16-worker runs.
The 3×3×3 LOD0 neighborhood completed in 18 ms / 19 ms after streaming started. The whole 960-tile full-detail run took 1.057 s / 0.825 s from process start, so at 16 workers even the full-detail area was ready inside one second. The standalone harness does not cook collision, so the required collidable-ready time remains unverified.
The edit probe submits a 10 m sphere crater, a rounded-box cut, and a capsule rim as one field transaction, centered 24 m from the player across the LOD0/LOD1 seam. One remesh batch completed all 14 intersecting tiles: 12 at LOD0 and 2 at LOD1. Remesh wall time was 12 ms / 13 ms and edit-request-to-ready was 15 ms / 18 ms at 8 / 16 workers, with zero cache hits. This CPU harness does not update an interactive GPU scene or cook collision.
## Profile and before/after timings
The baseline is the first cold, profiled source at the start of this speed round (`profile_whole_view_w8/w16`). “After” uses the final selected source (`restored_final_whole_w8/w16`). Timings vary by a few milliseconds between cold runs; the table uses one no-hash run at each worker count.
| Whole-view stage | Before, 8 | After, 8 | Before, 16 | After, 16 |
| --- | ---: | ---: | ---: | ---: |
| Local shapes and line bins | 57.6 ms | 54.6 ms | 57.7 ms | 54.2 ms |
| Rock intervals, packed signs, candidate axes and mask merge | 132.6 ms | 123.8 ms | 126.7 ms | 114.0 ms |
| Surface detail band | 539.0 ms | 537.3 ms | 395.1 ms | 375.2 ms |
| Far parent shapes and spatial bins | 194.7 ms | 120.4 ms | 191.6 ms | 121.1 ms |
| **Shape build** | **932.3 ms** | **843.5 ms** | **779.7 ms** | **672.1 ms** |
| **All LOD mesh generation** | **1,064 ms** | **961 ms** | **850 ms** | **782 ms** |
| **Shape build + mesh** | **1.996 s** | **1.805 s** | **1.630 s** | **1.454 s** |
At 16 workers the final interval subprofile was: local primitives and line bins 54 ms, rock interval/sign lines 20 ms, candidate-axis lines 55 ms, candidate-mask merge 25 ms (about 4 ms serial remainder), surface detail 375 ms, far-parent generation 76 ms, and far spatial-bin construction 45 ms. The 138,188,413-node field used 36.71 MiB of packed words, 55.06 MiB of candidate-axis words, a 131.79 MiB surface proxy, and 72.78 MiB of primitive storage. The far spatial index has 328,944 bins and 1,428,924 primitive references. Its compact bin lists hold up to four IDs inline in 181,237 bins; 147,707 larger bins spill to a small heap array. Whole-view processes peaked at 0.41 / 0.48 GiB private memory at 8 / 16 workers.
Per-LOD completion time is measured from stream start and is cumulative, not a sum of LOD durations:
| LOD | Tiles | Density samples | Before complete, 8 / 16 | After complete, 8 / 16 |
| ---: | ---: | ---: | ---: | ---: |
| 0 | 512 | 21.952 M | 0.107 / 0.092 s | 0.130 / 0.102 s |
| 1 | 448 | 19.208 M | 0.190 / 0.153 s | 0.210 / 0.161 s |
| 2 | 320 | 2.195 M | 0.221 / 0.178 s | 0.233 / 0.178 s |
| 3 | 208 | 1.427 M | 0.244 / 0.196 s | 0.251 / 0.194 s |
| 4 | 144 | 0.988 M | 0.255 / 0.205 s | 0.261 / 0.200 s |
| 5 | 96 | 0.658 M | 0.264 / 0.211 s | 0.268 / 0.206 s |
| 6 | 4,770 | 32.717 M | 1.061 / 0.836 s | 0.948 / 0.778 s |
LOD6 emits 1,795,216 vertices, 10,047,528 indices, and 79.42 MiB of CPU mesh data. It starts at about 0.25 / 0.21 s and finishes at 0.948 / 0.778 s at 8 / 16 workers; while active it keeps the worker pool busy. Whole-stream worker-idle estimates were 1.7% / 1.4%. It is the main mesh bottleneck.
A sequential profile of 32 nearby LOD6 tiles measured 33.1 ms total: density-grid evaluation 14.3 ms, cell classification 2.6 ms, gradients 0.3 ms, edge interpolation 0.3 ms, triangle/vertex emission 12.9 ms, and other mesher work 0.5 ms. This sample diagnoses the per-tile stages; it is not a replacement for the full parallel whole-view timing. Density evaluation and output emission remain similarly expensive.
## Changes and research decisions
- `EffectiveRockDistance` now uses one spatial-bin lookup and evaluates cave and solid-overlay distances in the same primitive pass. Lattice semantics and floating-point result ordering were kept. The two full-clipmap hashes stayed bit-identical.
- Candidate masks from the three interval axes are merged in parallel by line, using atomic OR on disjoint output words. The candidate set did not change; this reduced the interval/sign stage by about 13 ms at 8 workers and 10 ms at 16 in these runs.
- The post-interval far build no longer rebuilds axis-line indices that only the already-finished interval solver uses. Reserving the far spatial hash map from the measured primitive count reduced the original far-bin stage from about 112 ms to 97 ms. Replacing per-bin vectors with a 24-byte four-ID inline/spill list then reduced the measured 16-worker far-bin construction to 45 ms. Primitive IDs are still appended in ascending order, which preserves each bin's field-query order.
- The inline-list build had 328,944 far bins: 181,237 fit up to four IDs inline and 147,707 spill to heap storage. Across the full stream, density and CPU mesh hashes remained unchanged. The mesher now also reserves its existing per-worker vertex map from `2 × cells²`; the 32-tile LOD6 attribution sample moved from 33.513 ms to 33.145 ms. The whole-view difference is small and within run variation, so this reserve is retained as a low-cost allocation hint rather than claimed as a major speedup.
- Research: [Abseil's Swiss-table design notes](https://abseil.io/about/design/swisstables) and Meta's [F14 hash-table write-up](https://engineering.fb.com/2019/04/25/developer-tools/f14/) describe how inline storage and cache-aware lookup can reduce pointer chasing and allocation overhead. That motivated two prototypes. A flat open-addressed spatial index cut 16-worker bin construction from about 45 ms to 33 ms, but increased the measured streaming stage from about 761 ms to 804 ms; shape plus mesh was 1.471 s versus 1.440 s with the compact inline-bin hash map. I rejected the flat index because the query cost erased its build-time win. A first inline-list layout with a separate vector enlarged every map node and also streamed slower; the retained 24-byte union layout avoids that metadata growth.
- Added per-stage/per-LOD timing and worker accounting, a 27-tile neighborhood-ready metric, and a filtered one-queue edit-batch benchmark. The edit brush SDF uses explicit SSE2 square root instructions, with `/fp:precise` and `/arch:AVX2`; it does not add CRT transcendental calls or FMA contraction.
- A thread-local last-bin cache was tried and rejected: hashes matched but the LOD6 run showed no stable speed gain. A parallel far-bin builder was also rejected: hashes matched, but spatial-bin time stayed around 107 ms and whole-view time did not improve. Skipping the unused axis-bin rebuild alone saved only a few milliseconds; the capacity reservation was the useful part.
- Research: VTK's [Flying Edges documentation](https://vtk.org/doc/nightly/release/9.7/html/classvtkFlyingEdges3D.html) and the [LDAV 2015 paper](https://doi.org/10.1109/LDAV.2015.7348069) describe edge-oriented passes, count/prefix-sum allocation, preallocated output, and avoiding coincident-point merging. The current mesher still spends comparable sampled time in density-grid evaluation and vertex/triangle emission. A full multi-pass conversion is the strongest untested mesher option, but it is a larger rewrite with separate material-class, skirt, and degeneracy handling to preserve.
- Built and measured a per-worker lattice-edge vertex cache with the existing position hash retained as a fallback. The first version collapsed 2,477 vertices on the 16-worker whole view (1,792,739 vs 1,795,216) with the same triangle count, so it was rejected as a mesh change. The exact-key fallback version restored vertex counts, but totals were 1.884 / 1.484 s versus 1.886 / 1.471 s at 8 / 16 workers: no stable gain. Both prototypes were removed.
- Tested overlapping early far-tile meshing with the shape build. That path queued LOD2+ before the far-parent shapes and spatial index were built, so it could mesh against an incomplete field; its timing is invalid for a speed claim and the path was removed. Even the invalid trial was slower at 16 workers. Correct overlap needs an immutable, separately built far field before any dependent tile starts.
- The final no-hash composite whole view is 1.805 s at 8 workers and 1.454 s at 16; it still needs another 454 ms removed at 16 to meet the one-second target. The single-critical-tile split from the plugin review does not fit this LOD6 workload: there are 4,770 queued coarse tiles, and the worker pool is busy while they run. The 6-worker point, SIMD primitive batches, forward differences, Morton page order, and explicit coarse surfaces were not prototyped in this pass; explicit surfaces and a multi-pass edge mesher remain plausible larger experiments.
- Intel's [Intrinsics Guide](https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html) cautions that an intrinsic may expand to a slower instruction sequence. AVX2 batching of the irregular primitive/bin lookup was not assumed to help without a result-preserving prototype.
The final inline-bin source improves the comparable shape-build-plus-mesh result by about 9.6% at 8 workers and 10.8% at 16 versus this round's first cold baseline. The earlier round-three report's 2.095 / 1.617 s figures compare with 1.805 / 1.454 s here. Process-start timing is 1.958 / 1.629 s, so the one-second whole-view goal remains unmet.
## Determinism, routes, and surface checks
Four cold no-cache hash runs on the final built source (two at 16 workers, one each at 8 and 1) matched exactly. The field hash covers sampled density floats. The CPU-geometry hash covers each streamed tile's float positions, float normals, and indices, with tile keys included in deterministic LOD/Z/Y/X order. UV and color buffers are not stored in `FStandaloneCpuTile` and are not included in this geometry hash.
| Output | Density hash, identical in all runs | CPU geometry hash, identical in all runs |
| --- | --- | --- |
| Full-detail mesh | `0xa071e48a996e2828` | `0x6f270c621d39302f` |
| Whole clipmap | `0x630c43d7c68d5cbb` | `0xe3caf5b44244a992` |
| LOD0 | `0xc31fc55744c45928` | `0x7a70def3eb3e11ad` |
| LOD1 | `0xd74d6563f970d9f9` | `0xf47463988641b397` |
| LOD2 | `0xb0e440b4c9276bd7` | `0x9ed0581189757ee8` |
| LOD3 | `0xda4d3afb3467dd64` | `0xd8101bfb3ad87925` |
| LOD4 | `0x903537240ad6ae68` | `0x64c0b5e6a2e4f8ce` |
| LOD5 | `0x7c9e900fda72f05f` | `0x58c5139c40172196` |
| LOD6 | `0xc67d1f383fbf29ca` | `0xdc35accf7789860d` |
All runs reported zero mesh-cache hits. Each shape build retained all 5,642/5,642 passage route samples as open.
The saved earlier `final_exact_hash_*` set differs in full-detail, whole-clipmap, LOD0, and LOD1 hashes; LOD2–LOD6 match. The source/output change responsible for that older-to-current hash shift is still unexplained. The current hash set is stable across 1 / 8 / 16 workers and two 16-worker runs, but I do not claim bit identity with that earlier saved executable.
This verifies repeatability on this AVX2 host across 1, 8, and 16 workers. A second hardware host was not available, so cross-machine bit identity remains unverified here.
| Check | Final result |
| --- | --- |
| Chamber grid-stair index | 0.003 (threshold 0.400) |
| Tunnel grid-stair index | 0.028 (threshold 0.400) |
| Passage grid-stair index | 0.205 (threshold 0.400) |
| Shared tunnel joins | 280; endpoint radius and slope deltas 0.000000 / 0.00000000 |
| Whole-view level-zero sheet audit | 0 sub-voxel sheets, 0 isolated small components; 4 small boundary-touching components |
| Chamber flat-patch audit | 5.0% at the 16-voxel patch rule, largest 433 samples; audit passes, though it is not a visual sign-off |
The baseline tunnel stair audit without `--round-tunnels` measured 0.434 and failed the 0.400 threshold; the final tunnel view measures 0.028. The baseline chamber (0.041) and passage (0.276) already passed that threshold, so those two are improvements rather than fail-to-pass proofs. The earlier invalid baseline command with `--round-tunnels` remains recorded in [`round2/REPORT.md`](../round2/REPORT.md).
I reran the owner-field scan on the compact-bin source with the round-two settings: 8 workers, 960 full-resolution tiles for close views, 4-sample vertex stride, and 4,770 LOD6 tiles for far. `mesh_sample_mismatch=0` in each scan. The later vertex-map reservation only changes capacity; final no-cache density and CPU geometry hashes match the pre-reservation build exactly. The older saved scalar hashes differ, so these refreshed sign and distance results replace the stale “unchanged” claim.
| View | Lattice samples | Changed signs, before → current | Mean sampled surface distance, before → current | Max, before → current | Unmatched, before → current |
| --- | ---: | ---: | ---: | ---: | ---: |
| Chamber and tunnel region | 35,103,600 | 5.978% → 5.895% | 1.0325 m → 1.0826 m | 6.4677 m → 6.4840 m | 9,393 → 8,832 |
| Passage | 37,240,000 | 3.311% → 3.304% | 0.9814 m → 1.0292 m | 6.3422 m → 6.3750 m | 12,602 → 12,520 |
| Far view, LOD6 | 9,470,835 | 9.041% → 9.041% | 2.1667 m → 2.1588 m | 5.3070 m → 5.3070 m | 783,976 → 783,967 |
Closest-view sign disagreement fell slightly while mean surface distance rose; the far view changed almost not at all. These are sampled symmetric nearest-vertex distances, not continuous Hausdorff distances, and do not indicate that the generated world looks close to the owner.
## Captures and visual review
The final rebuilt captures are in `rebuilt_after_*.png`. Five are byte-identical to the previous exact captures; the passage differs in 4 of 1,024,000 pixels by at most 1 channel value. The current captures look unchanged at display scale. Against the owner images, the chamber still has a broad smooth ceiling and floor where the owner view has more connected forms; the tunnel has smoother, more repetitive relief; the passage retains a horizontal shelf/dark opening and lacks the owner's layered profile; the far image lacks the owner's larger connected forms. The look remains visibly different and is not a match.
| View | Before | Final after | Owner |
| --- | --- | --- | --- |
| Chamber | [before](../round2/final_after_chamber.png) | [after](rebuilt_after_chamber.png) | [owner](../final/chamber_fixed_owner.png) |
| Tunnel | [before](../round2/final_after_tunnel.png) | [after](rebuilt_after_tunnel.png) | [owner](../final/tunnel_fixed_owner.png) |
| Passage | [before](../round2/final_after_passage.png) | [after](rebuilt_after_passage.png) | [owner](../final/passage_fixed_owner.png) |
| Far view | [before](../round2/final_after_far.png) | [after](rebuilt_after_far.png) | [owner](../final/far_owner.png) |
| Passage-floor close-up | [before](../round2/final_after_passage_floor_closeup.png) | [after](rebuilt_after_passage_floor_closeup.png) | no owner close-up |
| Tunnel bend | [before](../round2/final_after_tunnel_bend.png) | [after](rebuilt_after_tunnel_bend.png) | no owner close-up |
## Build, process limits, and failed attempts
The final build is [`vertex_map_reserve_build.log`](vertex_map_reserve_build.log). It succeeded with `/MP16`; the build host ran at Normal priority and affinity `0xFFFF`. It still reports the existing `vswhere.exe` setup diagnostic and six C4838 narrowing warnings in `VoxelStrateManager.cpp`. Benchmark processes ran Normal with `0xFFFF`; final whole-view no-cache runs peaked at 0.41 / 0.48 GiB private memory at 8 / 16 workers. All test processes were launched by this task; no external viewer was closed.
- One build wrapper invocation used a relative `Tee-Object` path. `Build.ps1` changed the working directory, so the log path resolved under `Standalone\Build` and the wrapper returned 1. I checked that no compiler remained active and reran with an absolute log path; the final build succeeded. This was a logging-path failure, not a compiler failure.
- One benchmark invocation referenced `Research\...\RunMachine.ps1` from the plugin root instead of `Standalone\Research\...`; PowerShell rejected the script path before an executable started. The corrected invocation completed.
- A `rg | head` inspection hit a Cygwin `CreateFileMapping` permission diagnostic; the source query was repeated with PowerShell and no build or benchmark was affected.
- The simple edge-cache prototype that produced fewer vertices and the early-far overlap prototype were rejected as described above. Their build/run logs remain in this directory; no such code remains in the final source.
- The parallel far-bin experiment built and ran successfully and kept exact hashes, but did not improve the bin stage or wall time; its code was removed.
- The historical `resume2_before_tunnel_audit.stdout.log` used `--round-tunnels` and is not a valid base-world comparison. The corrected no-`--round-tunnels` baseline is retained at 0.434 and failed the final 0.400 threshold, as described above.
- No compiler build or benchmark executable failed during this speed round. The flat-map and first larger inline-bin prototypes ran successfully but were rejected for slower total results. The corrected baseline stair audit still fails at 0.434 against the 0.400 threshold; this failure is preserved in the earlier report. No commit, push, or stash was made.
## Logs
Baseline profiles: `profile_whole_w8.stdout.log`, `profile_whole_w16.stdout.log`. Final no-hash whole-view stages: `measured_whole_w8/w16.stdout.log`; process walls: `measured_whole_w8/w16.runner.log`. Full detail: `measured_detail_w8/w16.stdout.log` and `measured_detail_w8/w16.runner.log`. Final determinism on the exact rebuilt source: `final_source_hash_w1.stdout.log`, `_w8.stdout.log`, `_w16_a.stdout.log`, `_w16_b.stdout.log`. Cross-LOD edit latency: `reserve_edit_w8/w16.stdout.log`. Refreshed owner scans: `final_owner_bins_chamber_w8.stdout.log`, `final_owner_bins_passage_w8.stdout.log`, `final_owner_bins_far_w8.stdout.log`. Mesher attribution: `lod6_sample32_profile.stdout.log` and `vertex_map_reserve_profile.stdout.log`. Current audits/captures: `rebuilt_audit_*.stdout.log`, `rebuilt_shard_audit_w16.stdout.log`, `rebuilt_capture_*.stdout.log`, and `rebuilt_after_*.png`.