Fr0zkaandClaude Opus 5.5
7770074503
VoxelForgeCoreThreadLocal::FLazy<T> (new public header ThreadLocalStorage.h) makes every large thread_local cache a pointer that heap-allocates on that thread's first use (still worker-private, lock-free). Static TLS 17.33 MiB -> 43 KiB (initialised image 5.8 MiB -> 21 KiB); worker start 5.2 -> 0.12 ms; idle worker RSS 17.7 MiB -> 52 KiB. Linux workers back to 8 MiB stacks. World unchanged (L1 hashes Clang off/GCC default, W1/W12; FloatingIslands seed 0). Reviewer interleaved W12: 2.256 -> 2.091 s. MSVC/UBT not compiled here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ED33VnqepD4YY6gDAAA832
115 lines
13 KiB
Markdown
115 lines
13 KiB
Markdown
# WorldAssemblerM3 Linux4 — lazy worker caches
|
||
|
||
## Change
|
||
|
||
Added `VoxelForgeCoreThreadLocal::FLazy<T>`: each cache now has a thread-local pointer, allocates its private table on first use, and deletes it when that thread exits. The same helper is used by the namespaced Core and legacy source copies. No cache locks or cross-thread cache sharing were added. The L3 FP-contract preamble and self-test are unchanged.
|
||
|
||
The change covers the heightfield and island caches in Core, plus the large surface, relief, room-graph, tunnel-density, and diff caches in the shared generator sources. Tunnel-density lookup is now restricted to its `TunnelNetwork` and `Underwater` routes, which are the only routes that populate that table. The standalone TLS probe reports spawn time, RSS and per-thread minor faults, and can touch synthetic SurfaceWorld or FloatingIslands samples. Worker stacks are now 8 MiB.
|
||
|
||
## Where the baseline TLS came from
|
||
|
||
Sizes are from the S14 / `046f2af` Clang binary's TLS symbols. The two raw and namespaced source copies are both linked into the standalone executable; both occupy TLS even though a Core call uses the namespaced copy and the legacy/reference route uses the raw copy. The raw legacy cache is not used by the Core API route. `Standalone/Core/core_sources.txt` describes this dual build.
|
||
|
||
| Cache | Section | Bytes per copy | Copies | Total bytes | Route / use |
|
||
|---|---:|---:|---:|---:|---|
|
||
| `HeightfieldColumnCache` | `.tbss` | 6,815,744 | 1 | 6,815,744 | Core SurfaceWorld height samples; S14 hot path, any LOD that queries the heightfield |
|
||
| `GFloorReliefCache` | `.tdata` | 1,310,720 | 2 | 2,621,440 | Cave room/tunnel/landing floor-relief queries |
|
||
| `GRoomGraphCache` | `.tdata` | 970,752 | 2 | 1,941,504 | Room-graph SDF windows; fine/coarse region banks |
|
||
| `GSurfColCache` | `.tbss` | 827,000 | 2 | 1,654,000 | `ClassifyTile` surface-column pass; classifier is opt-in in runtime streaming |
|
||
| `FSurfaceColumnSource` column table | `.tbss` | 826,856 | 2 | 1,653,712 | Surface operator-stack integer XY columns |
|
||
| `PassageReliefCache` | `.tdata` | 655,360 | 2 | 1,310,720 | Passage-relief samples when that feature is queried |
|
||
| `IslandColumnCache` | `.tbss` | 1,114,112 | 1 | 1,114,112 | Core FloatingIslands column density, all LOD sample paths |
|
||
| `CP_TunnelDensityCache` | `.tbss` | 401,408 | 2 | 802,816 | Native TunnelNetwork/Underwater density chunks |
|
||
| `RootLobeCache` | `.tdata` | 155,648 | 1 | 155,648 | Core FloatingIslands root density |
|
||
| `HeightfieldColumnCacheNextWay` | `.tbss` | 16,384 | 1 | 16,384 | Heightfield cache replacement metadata |
|
||
| `DiffSlots` | `.tdata` | 10,752 | 2 | 21,504 | Only when a sampled chunk has player modifications |
|
||
| `RoomGraphCacheSlots` | `.tdata` | 10,240 | 2 | 20,480 | Room-graph generator route |
|
||
|
||
The listed `.tdata` cache symbols total 6,071,296 bytes of the 6,093,024-byte image; the remaining 21,728 bytes are small TLS state. Their nonzero defaults explain the initialized image: for example, room-graph keys start at `INT32_MIN` / `0xFFFFFFFF`, passage and floor-relief keys use `INDEX_NONE`, diff slots use `INT32_MAX` chunk coordinates, and root-lobe radii default to `1.0`. Zero-default large arrays such as the heightfield and island column tables account for the listed `.tbss` bytes. The listed `.tbss` symbols total 12,056,768 bytes; 21,997 bytes remain for small TLS state and alignment.
|
||
|
||
The caches have no universal per-LOD allocation: a thread allocates a table only if it reaches that route. Heightfield and island tables are keyed by recipe/plan or island identity and sampled coordinates; surface columns can be reused by repeated XY queries. Room-graph and classifier paths carry region/tile data in their keys, so coarse and fine requests can occupy distinct entries. Relief, tunnel and diff caches are likewise conditional on those features or modified chunks. All preserve the existing thread-local ownership and eviction behavior.
|
||
|
||
## TLS and per-thread cost
|
||
|
||
`PT_TLS` was read with `readelf -lW`. The baseline is the committed `046f2af` build; the after builds use this tree.
|
||
|
||
| Build | `PT_TLS` FileSiz (initialized) | `PT_TLS` MemSiz (per-thread block) |
|
||
|---|---:|---:|
|
||
| Clang 19 before | 6,093,024 B (5.81 MiB) | 18,171,789 B (17.33 MiB) |
|
||
| Clang 19 after | 21,728 B (21.2 KiB) | 43,977 B (42.95 KiB) |
|
||
| GCC 14 before | 6,060,396 B (5.78 MiB) | 18,172,504 B (17.33 MiB) |
|
||
| GCC 14 after | 9,580 B (9.36 KiB) | 44,696 B (43.65 KiB) |
|
||
|
||
Serial 12-worker no-touch, SurfaceWorld, and FloatingIslands probes compared baseline and L4. Spawn timing and resident delta use 32 MiB stacks on both sides to isolate TLS cost; the separately tested L4 tool uses 8 MiB stacks. Spawn medians/ranges use the per-worker samples; first-touch fault medians/ranges use each worker's `getrusage(RUSAGE_THREAD)` delta.
|
||
|
||
| Measurement | Before | After |
|
||
|---|---:|---:|
|
||
| Median time to start one worker | 5.175 ms | 0.119 ms |
|
||
| Median RSS added per new worker, no cache query | 17,756 KiB | 52 KiB |
|
||
| 12-worker no-touch probe peak RSS | 268,168 KiB | 32,328 KiB |
|
||
| First SurfaceWorld use: RSS increase per worker | 20 KiB | 6,691 KiB |
|
||
| First SurfaceWorld use: minor faults per worker | median 1 (range 1–4) | median 331 (range 8–649) |
|
||
| First FloatingIslands use: RSS increase per worker | 27–30 KiB | 1,283 KiB |
|
||
| First FloatingIslands use: minor faults per worker | median 0 (range 0–7) | median 314 (range 313–319) |
|
||
|
||
The baseline's first-use deltas are small because its large static TLS cost was already incurred at worker creation. After L4, a SurfaceWorld first use adds about 6.54 MiB per worker, while a FloatingIslands first use adds about 1.25 MiB. The no-touch worker increase is about 99.7% lower. These first-use probes call Core directly with four synthetic samples per worker; the full-view benchmark below measures the hot generation path.
|
||
|
||
The baseline binary's first `pthread_create` with an 8 MiB worker stack fails with `pthread_create: Invalid argument`; the 17.33 MiB static TLS block cannot fit that stack configuration. Baseline probes succeeded with 32 MiB stacks. L4 passed the 8 MiB worker-stack probe and the L1 checks, so the 32 MiB wrapper is no longer needed. Keep the normal 8 MiB worker stack; these runs provide no reason to retain 32 MiB as a safety margin.
|
||
|
||
## Generation speed and output
|
||
|
||
Three interleaved W12 whole-view runs per binary used the 2,553-tile L1 view. `Before` is the exact `046f2af` Clang-off executable.
|
||
|
||
| Build | Whole-view times | Median (range) |
|
||
|---|---|---:|
|
||
| `046f2af` baseline | 2.347, 2.399, 2.432 s | 2.399 s (2.347–2.432) |
|
||
| L4 Clang 19 off | 2.139, 2.197, 2.207 s | 2.197 s (2.139–2.207) |
|
||
|
||
L4 is 0.202 s (8.4%) faster at the median; no generation-speed regression was measured. The speed runs also changed the worker stack from 32 to 8 MiB, so the timing difference is not attributed solely to the cache indirection. Peak process RSS was 550,856 KiB before and 410,052 KiB after.
|
||
|
||
The L1 hashes match at W1 and W12 for both requested compiler settings:
|
||
|
||
| Build | Density / mesh / payload / topology |
|
||
|---|---|
|
||
| Clang 19 off, W1 and W12 | `c10697a8f48380c7 / e75c9119a1498cd5 / 49a9a8d1835a6c8c / edcb23288b5e8724` |
|
||
| GCC 14 default, W1 and W12 | same |
|
||
|
||
FloatingIslands seed 0 also matches at W1 and W12: `7fd1e3389642e6f6 / 577e52a5ba8d96be / 7c6ffe9f5bbbb450 / b4d308de116f4657` (density / mesh / payload / topology). It covers 27 tiles, with 9 nonempty mesh tiles and 20,740 indexed triangles; the existing L3 seed-0 hashes match.
|
||
|
||
## Unreal Linux TLS: documented and inferred
|
||
|
||
**Documented by glibc:** Its dynamic-linker guidance describes the `dlopen`/TLS compatibility issue and recommends global-dynamic TLS for general shared-object use. In glibc's current `dl-tls.c`, DTV slots for dynamically loaded TLS modules are marked for deferred allocation; `tls_get_addr_tail` performs the allocation when that thread first resolves the TLS address. The new-thread path calls `_dl_allocate_tls_init`, which also records deferred allocation for dynamically loaded modules. Thus, for the dynamic path, the large module block is allocated on first access, per thread that touches the module; a thread that never accesses it does not get that module's payload block. [glibc dynamic-linker guidance](https://www.sourceware.org/glibc/manual/latest/html_node/Dynamic-Linker-Hardening.html), [glibc TLS implementation](https://codebrowser.dev/glibc/glibc/elf/dl-tls.c.html#L599), [glibc thread creation path](https://codebrowser.dev/glibc/glibc/nptl/allocatestack.c.html#L141).
|
||
|
||
**Inference for Unreal:** This machine did not build or inspect the owner's UE `.so`, so the Core module's TLS model and load timing are unverified. GCC documents global-dynamic as the default model with `-fPIC`, but compiler/linker relaxation and UBT's exact flags can change the emitted access sequence. If Unreal loads Core after engine threads exist and the module uses general-dynamic TLS, any engine thread that first accesses a Core TLS symbol can incur the module's whole static TLS block once; it is not limited to generation workers. The L4 pointer handles reduce that block to about 43 KiB, with each large cache allocated only when its own route runs. Verify the actual `.so` `PT_TLS`, TLS relocations, and UBT compile commands on the owner's Linux build; `LD_DEBUG=tls` can show loader TLS activity. [GCC TLS model options](https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html#Code-Gen-Options).
|
||
|
||
## Windows/MSVC to-do
|
||
|
||
1. Build the updated standalone: `powershell -ExecutionPolicy Bypass -File Standalone\Build.ps1 -Target All -CompileWorkers 12`. Confirm the Core response file still carries `/fp:precise` and the shared header is included.
|
||
2. Run the L1 view at W1 and W12 with `--world-assembler-prototype --stream-benchmark --stream-pipeline --stream-hash --force-generate --clip-radius=3 --max-clip-level=8 --camera=0,-35,-487.94,0,-22 --far-horizon-lod8-step=0`. Require the four L1 hashes above and no self-test error.
|
||
3. Run the seed-0 FloatingIslands check at W1 and W12 with mesh coverage enabled; require matching density/mesh/payload/topology hashes and nonempty mesh coverage.
|
||
4. Build the owner's UE 5.7 Windows target with the plugin, inspect the actual `cl.exe` options, and exercise worker creation/use/shutdown. MSVC-specific risks are `/fp:precise` versus any `/fp:fast` or `/fp:contract` override, `thread_local` destructor handling on engine task threads, and allocator/container construction and destruction when a thread first touches or exits a cache. Linux ELF sizes do not predict Windows TLS costs.
|
||
|
||
## Machine and run record
|
||
|
||
Builds and benchmarks used CPUs 0–11, at most 12 build jobs, a 6 GiB address-space limit, and normal priority. Governor remained `powersave`; EPP remained `balance_performance`. Clang build: 29.06 s, peak RSS 431,732 KiB. GCC build: 46.14 s, peak RSS 896,076 KiB. Largest benchmark RSS was 550,856 KiB. Logs are in `Saved/Linux/linux4-*.log`; artifacts are under `Standalone/Build/linux-*` and `Standalone/bin/linux/`.
|
||
|
||
Failed or retried runs:
|
||
|
||
- The first baseline probes with an 8 MiB worker stack failed for `none`, `surface-world`, and `floating-islands` (`pthread_create: Invalid argument`, exit `-6`, three attempts each). They were rerun serially at 32 MiB and passed. L4 passed all three probes at 8 MiB.
|
||
- An early before/after probe batch ran the six probe types concurrently, so its timings and RSS were discarded. Each type was rerun serially; the final before/after comparison uses those serial logs. The discarded logs are retained with `linux4-discarded-parallel-` names.
|
||
- No build or world/hash run failed. Both Clang and GCC final builds succeeded.
|
||
|
||
## Note for the owner
|
||
|
||
The shared cache change reduces standalone static TLS from 17.33 MiB to about 43 KiB and preserves the L1 and FloatingIslands hashes; the W12 median moved from 2.399 s to 2.197 s. Please run the listed standalone and Unreal checks on Windows/MSVC before merging, especially worker-thread TLS destruction and the actual UBT FP/TLS options.
|
||
|
||
## Orchestrator review (Claude)
|
||
|
||
- Re-run by me: interleaved W12, L1 view: `046f2af` (L3) 2.212/2.256/2.307 s vs L4 2.063/2.091/2.125 s
|
||
(medians 2.256 → 2.091 s, -7%). Density `c10697a8f48380c7` throughout. L4 `PT_TLS` MemSiz 0xabc9 = 43,977 B.
|
||
- Checked the tunnel-cache route gate in `VoxelGenerator.cpp`: entries are only *stored* when `CP_GenType`
|
||
is TunnelNetwork/Underwater, so for any other route the old lookup could only miss. Gating skips
|
||
guaranteed misses, so no output change is possible. The legacy cave route is not exercised by the
|
||
headless tool, so this rests on code reading, not a hash.
|
||
- Windows: `thread_local` destructors on UE task threads are the MSVC risk. Luna's to-do list covers it.
|