wip(assembler): round L1 -- Linux baseline on abyss: Linux world identical to Windows
Headless Linux benchmark/hash tool (Clang 19 / GCC 14) sharing the viewer's plan builder and streaming pipeline (moved verbatim into StreamBenchmarkPipeline.h / WorldAssemblyPrototype.h). Owner Round17 view: density/mesh/payload/topology hashes equal the Windows S13 reference at W1/W8/W12 (Clang, 3 runs each) and W8 (GCC), with -ffp-contract=off. No output change. Baseline (i9-12900H, CPUs 0-11, powersave/balance_performance): whole view W8 3.562 s median, W12 3.512 s, W1 16.555 s; peak RSS < 605 MiB. Review: the default-stack crash is 17.3 MiB of static TLS per thread (glibc puts it in the thread stack; HeightfieldColumnCache 6.8 MB), not recursion -- open, after S14. MSVC compile of the viewer refactor not verified (no MSVC here). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Fr0zkaandClaude Opus 5.5 committed
1 parent
ec2afe9c93
commit
1276fd11e3
11 files changed
+1308
-760
No files matched your search
@@ -0,0 +1,153 @@
|
||||
# L1 — Linux baseline (abyss)
|
||||
|
||||
## What changed
|
||||
|
||||
- `Standalone/Source/StreamBenchmarkPipeline.h` now holds the S13 streaming benchmark, hash helpers, and per-LOD reporting. The Windows viewer and the headless executable call the same functions.
|
||||
- `Standalone/Source/WorldAssemblyPrototype.h` holds the existing Round 1 plan builder; both executables call it.
|
||||
- `Standalone/Source/StreamBenchmarkHeadless.cpp` is the Linux CLI entry point. It loads the same owner JSON, resolves it through `StandaloneOwnerLoader`, builds the same Core snapshot, and runs the shared pipeline. It has no window, renderer, D3D11, or Win32 setup.
|
||||
- `Standalone/BuildLinux.sh` builds the Core library and headless executable. `Linux1/MeasureProcess.py` checks affinity and memory/stack limits and records child peak RSS.
|
||||
- `Standalone/Source/StandaloneViewer.cpp` now includes and calls the shared headers. `Standalone/Core/src/CoreNamespacePreamble.h` and `Standalone/Compat/CoreMinimal.h` include `<cfloat>` for existing `FLT_MAX` use. `Standalone/Compat/StandaloneDataAssetGlue.cpp` now supplies the standalone `UVoxelSeasonAsset::GetPrimaryAssetId` implementation matching the engine source.
|
||||
|
||||
The headless entry point retains the parsed JSON root for the existing snapshot adapter. The route planner, Core shape compile, tile generator, hash code, and pipeline counters are shared with the viewer, rather than reimplemented. Viewer-only command switches for process affinity, priority, and frame limit are accepted for command-line parity; the Linux runner pins and limits itself externally.
|
||||
|
||||
**Windows compile risk:** MSVC was unavailable here, so the Windows viewer was not compiled. The likely risk spots are the new includes and namespaced calls in `StandaloneViewer.cpp`, and the moved inline bodies in `StreamBenchmarkPipeline.h` and `WorldAssemblyPrototype.h`; `Build.ps1` compiles that viewer with `/std:c++20`. The Linux-only main, script, and RSS helper are not in the explicit Windows source list. The compatibility `GetPrimaryAssetId` mirrors `VoxelSeasonAsset.cpp`; keep the engine `.cpp` out of the Standalone compatibility manifest to avoid a duplicate if that manifest is expanded later. No density formula changed, so no recipe-version bump is needed.
|
||||
|
||||
## Build and run commands
|
||||
|
||||
Run from the repo root. `BuildLinux.sh` enforces CPUs 0–11 and a 6 GiB address-space limit itself; the explicit wrapper below makes those limits visible at the call site too. The build output is under `Standalone/Build/linux-*` and `Standalone/bin/linux/`.
|
||||
|
||||
```bash
|
||||
taskset -c 0-11 prlimit --as=6442450944 -- \
|
||||
bash Standalone/BuildLinux.sh --compiler=clang++-19 --variant=clang19 --jobs=12 --fp-contract=off \
|
||||
> Saved/Linux/build-clang19-final.log 2>&1
|
||||
|
||||
taskset -c 0-11 prlimit --as=6442450944 -- \
|
||||
bash Standalone/BuildLinux.sh --compiler=g++-14 --variant=gcc14 --jobs=12 --fp-contract=off \
|
||||
> Saved/Linux/build-gcc14.log 2>&1
|
||||
```
|
||||
|
||||
The measured build flags were `-std=c++20 -O2 -mavx2 -mfma -ffp-contract=off -fno-fast-math`; Clang also suppresses the legacy narrowing diagnostic that MSVC accepts. The benchmark wrapper below enforces the 6 GiB address-space cap, a 64 MiB stack, and CPU affinity, then records `wait4` peak RSS. These are the exact run calls; `yes` adds `--profile-column-overlap`.
|
||||
|
||||
```bash
|
||||
run_bench() {
|
||||
local variant="$1" workers="$2" overlap="$3" name="$4"
|
||||
local profile=()
|
||||
if [[ "$overlap" == yes ]]; then profile+=(--profile-column-overlap); fi
|
||||
taskset -c 0-11 prlimit --as=6442450944 --stack=67108864:67108864 -- \
|
||||
python3 Standalone/Research/WorldAssemblerM3/Linux1/MeasureProcess.py \
|
||||
"Standalone/bin/linux/VoxelForgeCoreHeadless-$variant" Standalone/data/owner_world.json \
|
||||
--workers="$workers" --world-assembler-prototype --stream-benchmark --stream-pipeline --stream-hash \
|
||||
--force-generate --clip-radius=3 --max-clip-level=8 \
|
||||
--camera=0,-35,-487.94,0,-22 --far-horizon-lod8-step=0 \
|
||||
--process-priority=Normal --process-affinity=4095 --frame-limit=60 \
|
||||
"${profile[@]}" > "Saved/Linux/$name.log" 2>&1
|
||||
}
|
||||
|
||||
run_bench clang19 1 no bench-clang19-w1-r1
|
||||
run_bench clang19 8 no bench-clang19-w8-r1
|
||||
run_bench clang19 12 no bench-clang19-w12-r1
|
||||
run_bench clang19 8 no bench-clang19-w8-r2
|
||||
run_bench clang19 12 no bench-clang19-w12-r2
|
||||
run_bench clang19 1 no bench-clang19-w1-r2
|
||||
run_bench clang19 12 no bench-clang19-w12-r3
|
||||
run_bench clang19 1 no bench-clang19-w1-r3
|
||||
run_bench clang19 8 no bench-clang19-w8-r3
|
||||
run_bench clang19 8 yes bench-clang19-w8-columns-r1
|
||||
run_bench clang19 8 yes bench-clang19-w8-columns-r2
|
||||
run_bench clang19 8 yes bench-clang19-w8-columns-r3
|
||||
run_bench gcc14 8 no bench-gcc14-w8
|
||||
```
|
||||
|
||||
Every run used a fresh process and a newly constructed streamer; the pipeline reported `cache_hits=0`, so no process or thread-local tile/column cache carried over. `--force-generate` was passed for CLI parity, not relied on for isolation. The kernel file cache was not purged: doing so would require a system-wide setting outside the allowed write boundary. Owner JSON load is included in the cold-start timer and took about 5 ms.
|
||||
|
||||
## Hashes and determinism
|
||||
|
||||
Every Linux hash below covers the same 2,553-tile Round17 view. Clang used `-ffp-contract=off`, matching the intended MSVC `/fp:precise` behavior; GCC used the same contraction setting. Both builds enabled AVX2/FMA instructions but disabled contraction.
|
||||
|
||||
| Build / workers | Runs | Density | Mesh | Payload | Topology | Tiles |
|
||||
|---|---:|---|---|---|---|---:|
|
||||
| Windows S13 reference, W1/W8/W12 | S13 | `c10697a8f48380c7` | `e75c9119a1498cd5` | `49a9a8d1835a6c8c` | `edcb23288b5e8724` | 2,553 |
|
||||
| Linux Clang 19, W1 | 3 | `c10697a8f48380c7` | `e75c9119a1498cd5` | `49a9a8d1835a6c8c` | `edcb23288b5e8724` | 2,553 |
|
||||
| Linux Clang 19, W8 | 3 | `c10697a8f48380c7` | `e75c9119a1498cd5` | `49a9a8d1835a6c8c` | `edcb23288b5e8724` | 2,553 |
|
||||
| Linux Clang 19, W12 | 3 | `c10697a8f48380c7` | `e75c9119a1498cd5` | `49a9a8d1835a6c8c` | `edcb23288b5e8724` | 2,553 |
|
||||
| Linux GCC 14, W8 | 1 | `c10697a8f48380c7` | `e75c9119a1498cd5` | `49a9a8d1835a6c8c` | `edcb23288b5e8724` | 2,553 |
|
||||
|
||||
There was no divergence in this owner seed/view, so there is no first sample or stage to bisect and no density fix to make. The result covers the full hashed view, not every possible seed or platform math-library input. The tested flags show no observable FMA, libm, integer-width, container-order, or SIMD difference here; they do not establish what default Clang contraction would do.
|
||||
|
||||
Clang W1/W8/W12 each produced identical hashes across all three runs. The GCC W8 run agreed too. The overlap-capture counts varied slightly with worker scheduling, while the generated hashes stayed fixed.
|
||||
|
||||
## Linux timing and per-LOD cost
|
||||
|
||||
Machine: Debian 13, Intel i9-12900H, 15 GB RAM, 24 MiB L3. Processes were pinned to logical CPUs 0–11 (the six P-cores and their SMT siblings); CPUs 12–19 remained free. CPUs 0–11 reported `powersave` governor and `balance_performance` EPP. No governor or system setting was changed; priority remained normal. `RLIMIT_AS` was 6 GiB for every build and benchmark process. The benchmark stack was limited to 64 MiB after the default-stack failures below. No run reached the 30-minute limit.
|
||||
|
||||
`whole_view_ready` is the cold-start timer from process start through all 2,553 tiles, including owner load, plan, shape compile, and streamed generation. Values are median `[min–max]` of three interleaved runs. Peak RSS values are the three `wait4` readings in KiB, in run order; the maximum across all successful runs was 604,652 KiB.
|
||||
|
||||
| Workers | `whole_view_ready` median [range] | Peak RSS per run (KiB) |
|
||||
|---:|---:|---|
|
||||
| W1 | 16.555 s [16.069–16.660] | 218,372 / 329,308 / 329,012 |
|
||||
| W8 | 3.562 s [3.558–3.602] | 445,884 / 462,588 / 462,216 |
|
||||
| W12 | 3.512 s [3.482–3.557] | 536,884 / 537,208 / 539,036 |
|
||||
| W8 with overlap capture | 3.718 s [3.638–3.718] | 603,820 / 603,480 / 604,652 |
|
||||
| GCC 14 W8 hash check | 3.789 s (one run) | 463,276 |
|
||||
|
||||
The S13 W8 Windows reference was 3.518 s on the owner’s 5800X3D. The Linux W8 result is 44 ms (about 1.3%) slower for the same source view and timer, but these are different CPUs and this is a baseline comparison, not an optimization result. Round17 documents no corresponding W1/W12 timing for a direct comparison.
|
||||
|
||||
The next table divides S13’s accumulated `worker_sum_s` by tile count. Each cell is median `[min–max]` milliseconds per tile across the three Clang runs. It is average worker time per tile, not a per-tile p95 or whole-pipeline elapsed time.
|
||||
|
||||
| LOD | W1 ms/tile | W8 ms/tile | W12 ms/tile |
|
||||
|---:|---:|---:|---:|
|
||||
| 0 | 2.131 [2.066–2.144] | 4.222 [4.146–4.343] | 6.405 [6.364–6.489] |
|
||||
| 1 | 4.429 [4.281–4.441] | 7.394 [7.335–7.491] | 11.084 [11.000–11.296] |
|
||||
| 2 | 1.319 [1.282–1.319] | 1.871 [1.842–1.943] | 2.771 [2.740–2.813] |
|
||||
| 3 | 5.520 [5.366–5.527] | 7.916 [7.860–8.179] | 11.777 [11.670–11.916] |
|
||||
| 4 | 16.305 [15.787–16.344] | 23.403 [23.382–23.798] | 34.479 [34.229–34.654] |
|
||||
| 5 | 7.389 [7.223–7.443] | 10.714 [10.567–10.926] | 15.520 [15.288–15.605] |
|
||||
| 6 | 15.521 [15.021–15.638] | 22.183 [21.813–22.350] | 32.063 [31.641–32.163] |
|
||||
| 7 | 17.203 [16.495–17.226] | 24.476 [23.925–24.560] | 33.760 [33.085–33.885] |
|
||||
| 8 | 12.087 [11.673–12.228] | 16.591 [16.559–16.617] | 22.738 [22.610–23.673] |
|
||||
|
||||
S13’s exclusive stage counters are also retained per LOD: column evaluation, marching cubes, ground/cap refinement, and mesh assembly. At W8, column evaluation averaged 21.124 ms/tile at LOD4 `[21.083–21.462]`, versus 6.815 ms at LOD3 `[6.761–7.014]`; LOD4 remains the standout cost. Standalone has matching aggregate stage sums, but no BuildHost-style per-tile p95 or per-LOD classify-exclusive p95. It also has no counterpart for full-scan versus candidate feature-list p95, field-point calls p95, column misses p95, or approximate per-LOD hit rate; its field/cache totals are whole-run totals, and its overlap capture measures cross-tile XY reuse instead. BuildHost `UWorld::Tick`, apply, and stationary-ring timings also have no headless equivalent.
|
||||
|
||||
The W8 overlap capture completed for all tiles and all LODs. Each cell is median `[min–max]` across three captures.
|
||||
|
||||
| LOD | Unique XY columns | Repeated tile evaluations | Neighbor repeats | Cross-worker neighbor repeats |
|
||||
|---:|---:|---:|---:|---:|
|
||||
| 0 | 128,240 [128,240–128,240] | 360,764 [359,953–364,225] | 301,491 [300,316–301,913] | 296,573 [294,996–297,862] |
|
||||
| 1 | 103,957 [103,464–104,384] | 300,707 [296,768–312,183] | 243,479 [236,767–246,556] | 238,030 [228,885–241,780] |
|
||||
| 2 | 32,267 [32,263–32,271] | 106,072 [104,091–106,939] | 90,338 [88,410–90,534] | 87,693 [84,976–87,921] |
|
||||
| 3 | 33,286 [33,286–33,286] | 87,531 [87,310–88,408] | 81,853 [79,915–83,171] | 78,814 [76,126–78,903] |
|
||||
| 4 | 91,325 [91,325–91,325] | 231,088 [230,967–231,122] | 228,336 [226,998–228,369] | 207,383 [191,310–207,789] |
|
||||
| 5 | 71,960 [71,960–71,960] | 118,805 [118,676–118,806] | 118,665 [117,782–118,797] | 111,453 [108,197–112,585] |
|
||||
| 6 | 280,336 [280,336–280,336] | 63,463 [63,148–63,624] | 63,463 [63,148–63,624] | 63,204 [62,944–63,579] |
|
||||
| 7 | 71,530 [71,530–71,530] | 7,762 [7,752–7,807] | 7,762 [7,752–7,807] | 7,155 [7,074–7,262] |
|
||||
| 8 | 106,219 [106,219–106,219] | 32,092 [32,013–32,155] | 32,092 [32,013–32,155] | 28,393 [27,959–28,598] |
|
||||
|
||||
## Failed and retried attempts
|
||||
|
||||
- `build-clang19.log` failed because `FLT_MAX` was undeclared; adding `<cfloat>` fixed it.
|
||||
- `build-clang19-retry1.log` failed on existing double-to-float list initializers in `VoxelStrateManager.cpp` and an incorrect dereference in the new runner. The Linux build suppresses that legacy Clang narrowing diagnostic and passes the snapshot value correctly.
|
||||
- `build-clang19-retry2.log` linked with unresolved `typeinfo for UVoxelSeasonAsset`; the compatibility glue now provides the missing `GetPrimaryAssetId`. `build-clang19-retry3.log` and `build-clang19-final.log` succeeded.
|
||||
- A temporary diagnostic relink in `rebuild-headless-crash-handler.log` omitted `main` and failed; `relink-headless-crash-handler.log` corrected it. This was only for diagnosis and is not part of the final source.
|
||||
- `smoke-clang19-w1.log` stopped before launching because `/usr/bin/time` was absent. No package was installed; `MeasureProcess.py` replaced it.
|
||||
- Seven default-stack W1 attempts (`smoke-clang19-w1-r1.log` through `r7.log`) and three instrumented diagnostic runs (`smoke-clang19-direct.log`, `smoke-clang19-entry.log`, `smoke-clang19-stack.log`) exited with SIGSEGV while entering `FStandaloneStreamingGenerator::Generate` under the default 8 MiB stack. Their peak RSS was 211,056–329,008 KiB. A 64 MiB stack completed the diagnostic run (`smoke-clang19-stack64m.log`, 330,288 KiB peak RSS); temporary tracing pointed to worker stack exhaustion. That diagnostic included an extra direct tile and was excluded from the baseline. All clean baseline runs used the 64 MiB stack.
|
||||
|
||||
## Note for the owner
|
||||
|
||||
For this owner seed and view, a Linux server and Windows client generate identical hashed density, mesh, payload, and topology, and the hashes stayed stable across Linux worker counts and compilers. This i9-12900H result is a local Linux speed baseline; the W8 time is close to the S13 Windows number, but the machines differ and it should not be read as a speedup.
|
||||
|
||||
## Orchestrator review (Claude)
|
||||
|
||||
- **The default-stack crash is thread-local storage, not deep recursion.** `-fstack-usage` shows no frame over
|
||||
~4 KB in Core or the headless tool (`Generate` is 8 KB). But the binary's TLS segment is
|
||||
`0x115068d` = **17.3 MiB per thread** (6.1 MiB of it initialised data), and glibc carves static TLS out of
|
||||
each thread's stack, so an 8 MiB default stack overflows before any work runs. Largest pieces:
|
||||
`world_assembly_detail::HeightfieldColumnCache` 6.8 MB, `GFloorReliefCache` 1.3 MB (x2: Core and
|
||||
legacy copies), `IslandColumnCache` 1.1 MB, `GRoomGraphCache` 0.97 MB (x2), `GSurfColCache` 0.83 MB (x2).
|
||||
Windows allocates TLS separately from the 1 MB thread stack, which is why it never showed there.
|
||||
It matters for a Linux dedicated server: every worker thread pays ~17 MB and copies 6 MB on start.
|
||||
Open for a later round (after S14, which reworks the column cache).
|
||||
- **Untested:** the hash match is with `-ffp-contract=off`. Whether the hash would *detect* contraction
|
||||
(`on`/`fast`) has not been shown, so it is not yet proven the flag is load-bearing.
|
||||
- Viewer move checked: of 738 removed viewer lines, all but 12 signature lines (now `inline`/namespaced)
|
||||
reappear verbatim in the two shared headers; every remaining viewer use is qualified or `using`-imported.
|
||||
MSVC compile still unverified.
|
||||
Reference in new issue
Block a user