VoxelForge/Standalone/Research/OwnerShapeWorld/speed_round/REPORT.md

Shape world speed round

Result

The one-second whole-view goal does not pass. On the final rebuilt source, cold process-start to all 6,498 tiles through LOD6 took 1.958 s at 8 workers and 1.629 s at 16. The measured shape build plus uncached mesh generation took 1.805 s / 1.454 s. LOD6 remains the largest single streaming stage. The full-detail 960-tile area took 0.896 s / 0.703 s for shape build plus mesh generation; process-start to completion was 1.057 s / 0.825 s. Collision cooking and GPU upload are not measured by this CPU harness.

Implementation: shape compiler and field, streaming/edit benchmark, viewer measurements, and machine runner.

Cold job	Workers	Shape build	Uncached mesh	Shape + mesh	Process start → completion	Cache hits
64 m full-detail area, 960 tiles	8	0.725 s	0.171 s	0.896 s	1.057 s	0
64 m full-detail area, 960 tiles	16	0.549 s	0.154 s	0.703 s	0.825 s	0
Whole view, 6,498 tiles through LOD6	8	0.844 s	0.961 s	1.805 s	1.958 s	0
Whole view, 6,498 tiles through LOD6	16	0.672 s	0.782 s	1.454 s	1.629 s	0

The first four stage columns use separate cold, no-cache processes. Process-start timing begins immediately before Start-Process, includes executable launch and world setup, and ends after the final tile is meshed; it still excludes interactive GPU upload and collision cooking. The worker runner is Normal priority on affinity 0xFFFF and polls at 50 ms. Peak private memory was 0.41 / 0.47 GiB for whole-view 8 / 16-worker runs.

The 3×3×3 LOD0 neighborhood completed in 18 ms / 19 ms after streaming started. The whole 960-tile full-detail run took 1.057 s / 0.825 s from process start, so at 16 workers even the full-detail area was ready inside one second. The standalone harness does not cook collision, so the required collidable-ready time remains unverified.

The edit probe submits a 10 m sphere crater, a rounded-box cut, and a capsule rim as one field transaction, centered 24 m from the player across the LOD0/LOD1 seam. One remesh batch completed all 14 intersecting tiles: 12 at LOD0 and 2 at LOD1. Remesh wall time was 12 ms / 13 ms and edit-request-to-ready was 15 ms / 18 ms at 8 / 16 workers, with zero cache hits. This CPU harness does not update an interactive GPU scene or cook collision.

Profile and before/after timings

The baseline is the first cold, profiled source at the start of this speed round (profile_whole_view_w8/w16). “After” uses the final selected source (restored_final_whole_w8/w16). Timings vary by a few milliseconds between cold runs; the table uses one no-hash run at each worker count.

Whole-view stage	Before, 8	After, 8	Before, 16	After, 16
Local shapes and line bins	57.6 ms	54.6 ms	57.7 ms	54.2 ms
Rock intervals, packed signs, candidate axes and mask merge	132.6 ms	123.8 ms	126.7 ms	114.0 ms
Surface detail band	539.0 ms	537.3 ms	395.1 ms	375.2 ms
Far parent shapes and spatial bins	194.7 ms	120.4 ms	191.6 ms	121.1 ms
Shape build	932.3 ms	843.5 ms	779.7 ms	672.1 ms
All LOD mesh generation	1,064 ms	961 ms	850 ms	782 ms
Shape build + mesh	1.996 s	1.805 s	1.630 s	1.454 s

At 16 workers the final interval subprofile was: local primitives and line bins 54 ms, rock interval/sign lines 20 ms, candidate-axis lines 55 ms, candidate-mask merge 25 ms (about 4 ms serial remainder), surface detail 375 ms, far-parent generation 76 ms, and far spatial-bin construction 45 ms. The 138,188,413-node field used 36.71 MiB of packed words, 55.06 MiB of candidate-axis words, a 131.79 MiB surface proxy, and 72.78 MiB of primitive storage. The far spatial index has 328,944 bins and 1,428,924 primitive references. Its compact bin lists hold up to four IDs inline in 181,237 bins; 147,707 larger bins spill to a small heap array. Whole-view processes peaked at 0.41 / 0.48 GiB private memory at 8 / 16 workers.

Per-LOD completion time is measured from stream start and is cumulative, not a sum of LOD durations:

LOD	Tiles	Density samples	Before complete, 8 / 16	After complete, 8 / 16
0	512	21.952 M	0.107 / 0.092 s	0.130 / 0.102 s
1	448	19.208 M	0.190 / 0.153 s	0.210 / 0.161 s
2	320	2.195 M	0.221 / 0.178 s	0.233 / 0.178 s
3	208	1.427 M	0.244 / 0.196 s	0.251 / 0.194 s
4	144	0.988 M	0.255 / 0.205 s	0.261 / 0.200 s
5	96	0.658 M	0.264 / 0.211 s	0.268 / 0.206 s
6	4,770	32.717 M	1.061 / 0.836 s	0.948 / 0.778 s

LOD6 emits 1,795,216 vertices, 10,047,528 indices, and 79.42 MiB of CPU mesh data. It starts at about 0.25 / 0.21 s and finishes at 0.948 / 0.778 s at 8 / 16 workers; while active it keeps the worker pool busy. Whole-stream worker-idle estimates were 1.7% / 1.4%. It is the main mesh bottleneck.

A sequential profile of 32 nearby LOD6 tiles measured 33.1 ms total: density-grid evaluation 14.3 ms, cell classification 2.6 ms, gradients 0.3 ms, edge interpolation 0.3 ms, triangle/vertex emission 12.9 ms, and other mesher work 0.5 ms. This sample diagnoses the per-tile stages; it is not a replacement for the full parallel whole-view timing. Density evaluation and output emission remain similarly expensive.

Changes and research decisions

• EffectiveRockDistance now uses one spatial-bin lookup and evaluates cave and solid-overlay distances in the same primitive pass. Lattice semantics and floating-point result ordering were kept. The two full-clipmap hashes stayed bit-identical.
• Candidate masks from the three interval axes are merged in parallel by line, using atomic OR on disjoint output words. The candidate set did not change; this reduced the interval/sign stage by about 13 ms at 8 workers and 10 ms at 16 in these runs.
• The post-interval far build no longer rebuilds axis-line indices that only the already-finished interval solver uses. Reserving the far spatial hash map from the measured primitive count reduced the original far-bin stage from about 112 ms to 97 ms. Replacing per-bin vectors with a 24-byte four-ID inline/spill list then reduced the measured 16-worker far-bin construction to 45 ms. Primitive IDs are still appended in ascending order, which preserves each bin's field-query order.
• The inline-list build had 328,944 far bins: 181,237 fit up to four IDs inline and 147,707 spill to heap storage. Across the full stream, density and CPU mesh hashes remained unchanged. The mesher now also reserves its existing per-worker vertex map from 2 × cells²; the 32-tile LOD6 attribution sample moved from 33.513 ms to 33.145 ms. The whole-view difference is small and within run variation, so this reserve is retained as a low-cost allocation hint rather than claimed as a major speedup.
• Research: Abseil's Swiss-table design notes and Meta's F14 hash-table write-up describe how inline storage and cache-aware lookup can reduce pointer chasing and allocation overhead. That motivated two prototypes. A flat open-addressed spatial index cut 16-worker bin construction from about 45 ms to 33 ms, but increased the measured streaming stage from about 761 ms to 804 ms; shape plus mesh was 1.471 s versus 1.440 s with the compact inline-bin hash map. I rejected the flat index because the query cost erased its build-time win. A first inline-list layout with a separate vector enlarged every map node and also streamed slower; the retained 24-byte union layout avoids that metadata growth.
• Added per-stage/per-LOD timing and worker accounting, a 27-tile neighborhood-ready metric, and a filtered one-queue edit-batch benchmark. The edit brush SDF uses explicit SSE2 square root instructions, with /fp:precise and /arch:AVX2; it does not add CRT transcendental calls or FMA contraction.
• A thread-local last-bin cache was tried and rejected: hashes matched but the LOD6 run showed no stable speed gain. A parallel far-bin builder was also rejected: hashes matched, but spatial-bin time stayed around 107 ms and whole-view time did not improve. Skipping the unused axis-bin rebuild alone saved only a few milliseconds; the capacity reservation was the useful part.
• Research: VTK's Flying Edges documentation and the LDAV 2015 paper describe edge-oriented passes, count/prefix-sum allocation, preallocated output, and avoiding coincident-point merging. The current mesher still spends comparable sampled time in density-grid evaluation and vertex/triangle emission. A full multi-pass conversion is the strongest untested mesher option, but it is a larger rewrite with separate material-class, skirt, and degeneracy handling to preserve.
• Built and measured a per-worker lattice-edge vertex cache with the existing position hash retained as a fallback. The first version collapsed 2,477 vertices on the 16-worker whole view (1,792,739 vs 1,795,216) with the same triangle count, so it was rejected as a mesh change. The exact-key fallback version restored vertex counts, but totals were 1.884 / 1.484 s versus 1.886 / 1.471 s at 8 / 16 workers: no stable gain. Both prototypes were removed.
• Tested overlapping early far-tile meshing with the shape build. That path queued LOD2+ before the far-parent shapes and spatial index were built, so it could mesh against an incomplete field; its timing is invalid for a speed claim and the path was removed. Even the invalid trial was slower at 16 workers. Correct overlap needs an immutable, separately built far field before any dependent tile starts.
• The final no-hash composite whole view is 1.805 s at 8 workers and 1.454 s at 16; it still needs another 454 ms removed at 16 to meet the one-second target. The single-critical-tile split from the plugin review does not fit this LOD6 workload: there are 4,770 queued coarse tiles, and the worker pool is busy while they run. The 6-worker point, SIMD primitive batches, forward differences, Morton page order, and explicit coarse surfaces were not prototyped in this pass; explicit surfaces and a multi-pass edge mesher remain plausible larger experiments.
• Intel's Intrinsics Guide cautions that an intrinsic may expand to a slower instruction sequence. AVX2 batching of the irregular primitive/bin lookup was not assumed to help without a result-preserving prototype.

The final inline-bin source improves the comparable shape-build-plus-mesh result by about 9.6% at 8 workers and 10.8% at 16 versus this round's first cold baseline. The earlier round-three report's 2.095 / 1.617 s figures compare with 1.805 / 1.454 s here. Process-start timing is 1.958 / 1.629 s, so the one-second whole-view goal remains unmet.

Determinism, routes, and surface checks

Four cold no-cache hash runs on the final built source (two at 16 workers, one each at 8 and 1) matched exactly. The field hash covers sampled density floats. The CPU-geometry hash covers each streamed tile's float positions, float normals, and indices, with tile keys included in deterministic LOD/Z/Y/X order. UV and color buffers are not stored in FStandaloneCpuTile and are not included in this geometry hash.

Output	Density hash, identical in all runs	CPU geometry hash, identical in all runs
Full-detail mesh	0xa071e48a996e2828	0x6f270c621d39302f
Whole clipmap	0x630c43d7c68d5cbb	0xe3caf5b44244a992
LOD0	0xc31fc55744c45928	0x7a70def3eb3e11ad
LOD1	0xd74d6563f970d9f9	0xf47463988641b397
LOD2	0xb0e440b4c9276bd7	0x9ed0581189757ee8
LOD3	0xda4d3afb3467dd64	0xd8101bfb3ad87925
LOD4	0x903537240ad6ae68	0x64c0b5e6a2e4f8ce
LOD5	0x7c9e900fda72f05f	0x58c5139c40172196
LOD6	0xc67d1f383fbf29ca	0xdc35accf7789860d

All runs reported zero mesh-cache hits. Each shape build retained all 5,642/5,642 passage route samples as open.

The saved earlier final_exact_hash_* set differs in full-detail, whole-clipmap, LOD0, and LOD1 hashes; LOD2–LOD6 match. The source/output change responsible for that older-to-current hash shift is still unexplained. The current hash set is stable across 1 / 8 / 16 workers and two 16-worker runs, but I do not claim bit identity with that earlier saved executable.

This verifies repeatability on this AVX2 host across 1, 8, and 16 workers. A second hardware host was not available, so cross-machine bit identity remains unverified here.

Check	Final result
Chamber grid-stair index	0.003 (threshold 0.400)
Tunnel grid-stair index	0.028 (threshold 0.400)
Passage grid-stair index	0.205 (threshold 0.400)
Shared tunnel joins	280; endpoint radius and slope deltas 0.000000 / 0.00000000
Whole-view level-zero sheet audit	0 sub-voxel sheets, 0 isolated small components; 4 small boundary-touching components
Chamber flat-patch audit	5.0% at the 16-voxel patch rule, largest 433 samples; audit passes, though it is not a visual sign-off

The baseline tunnel stair audit without --round-tunnels measured 0.434 and failed the 0.400 threshold; the final tunnel view measures 0.028. The baseline chamber (0.041) and passage (0.276) already passed that threshold, so those two are improvements rather than fail-to-pass proofs. The earlier invalid baseline command with --round-tunnels remains recorded in round2/REPORT.md.

I reran the owner-field scan on the compact-bin source with the round-two settings: 8 workers, 960 full-resolution tiles for close views, 4-sample vertex stride, and 4,770 LOD6 tiles for far. mesh_sample_mismatch=0 in each scan. The later vertex-map reservation only changes capacity; final no-cache density and CPU geometry hashes match the pre-reservation build exactly. The older saved scalar hashes differ, so these refreshed sign and distance results replace the stale “unchanged” claim.

View	Lattice samples	Changed signs, before → current	Mean sampled surface distance, before → current	Max, before → current	Unmatched, before → current
Chamber and tunnel region	35,103,600	5.978% → 5.895%	1.0325 m → 1.0826 m	6.4677 m → 6.4840 m	9,393 → 8,832
Passage	37,240,000	3.311% → 3.304%	0.9814 m → 1.0292 m	6.3422 m → 6.3750 m	12,602 → 12,520
Far view, LOD6	9,470,835	9.041% → 9.041%	2.1667 m → 2.1588 m	5.3070 m → 5.3070 m	783,976 → 783,967

Closest-view sign disagreement fell slightly while mean surface distance rose; the far view changed almost not at all. These are sampled symmetric nearest-vertex distances, not continuous Hausdorff distances, and do not indicate that the generated world looks close to the owner.

Captures and visual review

The final rebuilt captures are in rebuilt_after_*.png. Five are byte-identical to the previous exact captures; the passage differs in 4 of 1,024,000 pixels by at most 1 channel value. The current captures look unchanged at display scale. Against the owner images, the chamber still has a broad smooth ceiling and floor where the owner view has more connected forms; the tunnel has smoother, more repetitive relief; the passage retains a horizontal shelf/dark opening and lacks the owner's layered profile; the far image lacks the owner's larger connected forms. The look remains visibly different and is not a match.

View	Before	Final after	Owner
Chamber	before	after	owner
Tunnel	before	after	owner
Passage	before	after	owner
Far view	before	after	owner
Passage-floor close-up	before	after	no owner close-up
Tunnel bend	before	after	no owner close-up

Build, process limits, and failed attempts

The final build is vertex_map_reserve_build.log. It succeeded with /MP16; the build host ran at Normal priority and affinity 0xFFFF. It still reports the existing vswhere.exe setup diagnostic and six C4838 narrowing warnings in VoxelStrateManager.cpp. Benchmark processes ran Normal with 0xFFFF; final whole-view no-cache runs peaked at 0.41 / 0.48 GiB private memory at 8 / 16 workers. All test processes were launched by this task; no external viewer was closed.

• One build wrapper invocation used a relative Tee-Object path. Build.ps1 changed the working directory, so the log path resolved under Standalone\Build and the wrapper returned 1. I checked that no compiler remained active and reran with an absolute log path; the final build succeeded. This was a logging-path failure, not a compiler failure.
• One benchmark invocation referenced Research\...\RunMachine.ps1 from the plugin root instead of Standalone\Research\...; PowerShell rejected the script path before an executable started. The corrected invocation completed.
• A rg | head inspection hit a Cygwin CreateFileMapping permission diagnostic; the source query was repeated with PowerShell and no build or benchmark was affected.
• The simple edge-cache prototype that produced fewer vertices and the early-far overlap prototype were rejected as described above. Their build/run logs remain in this directory; no such code remains in the final source.
• The parallel far-bin experiment built and ran successfully and kept exact hashes, but did not improve the bin stage or wall time; its code was removed.
• The historical resume2_before_tunnel_audit.stdout.log used --round-tunnels and is not a valid base-world comparison. The corrected no---round-tunnels baseline is retained at 0.434 and failed the final 0.400 threshold, as described above.
• No compiler build or benchmark executable failed during this speed round. The flat-map and first larger inline-bin prototypes ran successfully but were rejected for slower total results. The corrected baseline stair audit still fails at 0.434 against the 0.400 threshold; this failure is preserved in the earlier report. No commit, push, or stash was made.

Logs

Baseline profiles: profile_whole_w8.stdout.log, profile_whole_w16.stdout.log. Final no-hash whole-view stages: measured_whole_w8/w16.stdout.log; process walls: measured_whole_w8/w16.runner.log. Full detail: measured_detail_w8/w16.stdout.log and measured_detail_w8/w16.runner.log. Final determinism on the exact rebuilt source: final_source_hash_w1.stdout.log, _w8.stdout.log, _w16_a.stdout.log, _w16_b.stdout.log. Cross-LOD edit latency: reserve_edit_w8/w16.stdout.log. Refreshed owner scans: final_owner_bins_chamber_w8.stdout.log, final_owner_bins_passage_w8.stdout.log, final_owner_bins_far_w8.stdout.log. Mesher attribution: lod6_sample32_profile.stdout.log and vertex_map_reserve_profile.stdout.log. Current audits/captures: rebuilt_audit_*.stdout.log, rebuilt_shard_audit_w16.stdout.log, rebuilt_capture_*.stdout.log, and rebuilt_after_*.png.