VoxelForge/Standalone/Research/WorldAssemblerM3/Linux4/REPORT.md

WorldAssemblerM3 Linux4 — lazy worker caches

Change

Added VoxelForgeCoreThreadLocal::FLazy<T>: each cache now has a thread-local pointer, allocates its private table on first use, and deletes it when that thread exits. The same helper is used by the namespaced Core and legacy source copies. No cache locks or cross-thread cache sharing were added. The L3 FP-contract preamble and self-test are unchanged.

The change covers the heightfield and island caches in Core, plus the large surface, relief, room-graph, tunnel-density, and diff caches in the shared generator sources. Tunnel-density lookup is now restricted to its TunnelNetwork and Underwater routes, which are the only routes that populate that table. The standalone TLS probe reports spawn time, RSS and per-thread minor faults, and can touch synthetic SurfaceWorld or FloatingIslands samples. Worker stacks are now 8 MiB.

Where the baseline TLS came from

Sizes are from the S14 / 046f2af Clang binary's TLS symbols. The two raw and namespaced source copies are both linked into the standalone executable; both occupy TLS even though a Core call uses the namespaced copy and the legacy/reference route uses the raw copy. The raw legacy cache is not used by the Core API route. Standalone/Core/core_sources.txt describes this dual build.

Cache	Section	Bytes per copy	Copies	Total bytes	Route / use
HeightfieldColumnCache	.tbss	6,815,744	1	6,815,744	Core SurfaceWorld height samples; S14 hot path, any LOD that queries the heightfield
GFloorReliefCache	.tdata	1,310,720	2	2,621,440	Cave room/tunnel/landing floor-relief queries
GRoomGraphCache	.tdata	970,752	2	1,941,504	Room-graph SDF windows; fine/coarse region banks
GSurfColCache	.tbss	827,000	2	1,654,000	ClassifyTile surface-column pass; classifier is opt-in in runtime streaming
FSurfaceColumnSource column table	.tbss	826,856	2	1,653,712	Surface operator-stack integer XY columns
PassageReliefCache	.tdata	655,360	2	1,310,720	Passage-relief samples when that feature is queried
IslandColumnCache	.tbss	1,114,112	1	1,114,112	Core FloatingIslands column density, all LOD sample paths
CP_TunnelDensityCache	.tbss	401,408	2	802,816	Native TunnelNetwork/Underwater density chunks
RootLobeCache	.tdata	155,648	1	155,648	Core FloatingIslands root density
HeightfieldColumnCacheNextWay	.tbss	16,384	1	16,384	Heightfield cache replacement metadata
DiffSlots	.tdata	10,752	2	21,504	Only when a sampled chunk has player modifications
RoomGraphCacheSlots	.tdata	10,240	2	20,480	Room-graph generator route

The listed .tdata cache symbols total 6,071,296 bytes of the 6,093,024-byte image; the remaining 21,728 bytes are small TLS state. Their nonzero defaults explain the initialized image: for example, room-graph keys start at INT32_MIN / 0xFFFFFFFF, passage and floor-relief keys use INDEX_NONE, diff slots use INT32_MAX chunk coordinates, and root-lobe radii default to 1.0. Zero-default large arrays such as the heightfield and island column tables account for the listed .tbss bytes. The listed .tbss symbols total 12,056,768 bytes; 21,997 bytes remain for small TLS state and alignment.

The caches have no universal per-LOD allocation: a thread allocates a table only if it reaches that route. Heightfield and island tables are keyed by recipe/plan or island identity and sampled coordinates; surface columns can be reused by repeated XY queries. Room-graph and classifier paths carry region/tile data in their keys, so coarse and fine requests can occupy distinct entries. Relief, tunnel and diff caches are likewise conditional on those features or modified chunks. All preserve the existing thread-local ownership and eviction behavior.

TLS and per-thread cost

PT_TLS was read with readelf -lW. The baseline is the committed 046f2af build; the after builds use this tree.

Build	PT_TLS FileSiz (initialized)	PT_TLS MemSiz (per-thread block)
Clang 19 before	6,093,024 B (5.81 MiB)	18,171,789 B (17.33 MiB)
Clang 19 after	21,728 B (21.2 KiB)	43,977 B (42.95 KiB)
GCC 14 before	6,060,396 B (5.78 MiB)	18,172,504 B (17.33 MiB)
GCC 14 after	9,580 B (9.36 KiB)	44,696 B (43.65 KiB)

Serial 12-worker no-touch, SurfaceWorld, and FloatingIslands probes compared baseline and L4. Spawn timing and resident delta use 32 MiB stacks on both sides to isolate TLS cost; the separately tested L4 tool uses 8 MiB stacks. Spawn medians/ranges use the per-worker samples; first-touch fault medians/ranges use each worker's getrusage(RUSAGE_THREAD) delta.

Measurement	Before	After
Median time to start one worker	5.175 ms	0.119 ms
Median RSS added per new worker, no cache query	17,756 KiB	52 KiB
12-worker no-touch probe peak RSS	268,168 KiB	32,328 KiB
First SurfaceWorld use: RSS increase per worker	20 KiB	6,691 KiB
First SurfaceWorld use: minor faults per worker	median 1 (range 1–4)	median 331 (range 8–649)
First FloatingIslands use: RSS increase per worker	27–30 KiB	1,283 KiB
First FloatingIslands use: minor faults per worker	median 0 (range 0–7)	median 314 (range 313–319)

The baseline's first-use deltas are small because its large static TLS cost was already incurred at worker creation. After L4, a SurfaceWorld first use adds about 6.54 MiB per worker, while a FloatingIslands first use adds about 1.25 MiB. The no-touch worker increase is about 99.7% lower. These first-use probes call Core directly with four synthetic samples per worker; the full-view benchmark below measures the hot generation path.

The baseline binary's first pthread_create with an 8 MiB worker stack fails with pthread_create: Invalid argument; the 17.33 MiB static TLS block cannot fit that stack configuration. Baseline probes succeeded with 32 MiB stacks. L4 passed the 8 MiB worker-stack probe and the L1 checks, so the 32 MiB wrapper is no longer needed. Keep the normal 8 MiB worker stack; these runs provide no reason to retain 32 MiB as a safety margin.

Generation speed and output

Three interleaved W12 whole-view runs per binary used the 2,553-tile L1 view. Before is the exact 046f2af Clang-off executable.

Build	Whole-view times	Median (range)
046f2af baseline	2.347, 2.399, 2.432 s	2.399 s (2.347–2.432)
L4 Clang 19 off	2.139, 2.197, 2.207 s	2.197 s (2.139–2.207)

L4 is 0.202 s (8.4%) faster at the median; no generation-speed regression was measured. The speed runs also changed the worker stack from 32 to 8 MiB, so the timing difference is not attributed solely to the cache indirection. Peak process RSS was 550,856 KiB before and 410,052 KiB after.

The L1 hashes match at W1 and W12 for both requested compiler settings:

Build	Density / mesh / payload / topology
Clang 19 off, W1 and W12	c10697a8f48380c7 / e75c9119a1498cd5 / 49a9a8d1835a6c8c / edcb23288b5e8724
GCC 14 default, W1 and W12	same

FloatingIslands seed 0 also matches at W1 and W12: 7fd1e3389642e6f6 / 577e52a5ba8d96be / 7c6ffe9f5bbbb450 / b4d308de116f4657 (density / mesh / payload / topology). It covers 27 tiles, with 9 nonempty mesh tiles and 20,740 indexed triangles; the existing L3 seed-0 hashes match.

Unreal Linux TLS: documented and inferred

Documented by glibc: Its dynamic-linker guidance describes the dlopen/TLS compatibility issue and recommends global-dynamic TLS for general shared-object use. In glibc's current dl-tls.c, DTV slots for dynamically loaded TLS modules are marked for deferred allocation; tls_get_addr_tail performs the allocation when that thread first resolves the TLS address. The new-thread path calls _dl_allocate_tls_init, which also records deferred allocation for dynamically loaded modules. Thus, for the dynamic path, the large module block is allocated on first access, per thread that touches the module; a thread that never accesses it does not get that module's payload block. glibc dynamic-linker guidance, glibc TLS implementation, glibc thread creation path.

Inference for Unreal: This machine did not build or inspect the owner's UE .so, so the Core module's TLS model and load timing are unverified. GCC documents global-dynamic as the default model with -fPIC, but compiler/linker relaxation and UBT's exact flags can change the emitted access sequence. If Unreal loads Core after engine threads exist and the module uses general-dynamic TLS, any engine thread that first accesses a Core TLS symbol can incur the module's whole static TLS block once; it is not limited to generation workers. The L4 pointer handles reduce that block to about 43 KiB, with each large cache allocated only when its own route runs. Verify the actual .so PT_TLS, TLS relocations, and UBT compile commands on the owner's Linux build; LD_DEBUG=tls can show loader TLS activity. GCC TLS model options.

Windows/MSVC to-do

1. Build the updated standalone: powershell -ExecutionPolicy Bypass -File Standalone\Build.ps1 -Target All -CompileWorkers 12. Confirm the Core response file still carries /fp:precise and the shared header is included.
2. Run the L1 view at W1 and W12 with --world-assembler-prototype --stream-benchmark --stream-pipeline --stream-hash --force-generate --clip-radius=3 --max-clip-level=8 --camera=0,-35,-487.94,0,-22 --far-horizon-lod8-step=0. Require the four L1 hashes above and no self-test error.
3. Run the seed-0 FloatingIslands check at W1 and W12 with mesh coverage enabled; require matching density/mesh/payload/topology hashes and nonempty mesh coverage.
4. Build the owner's UE 5.7 Windows target with the plugin, inspect the actual cl.exe options, and exercise worker creation/use/shutdown. MSVC-specific risks are /fp:precise versus any /fp:fast or /fp:contract override, thread_local destructor handling on engine task threads, and allocator/container construction and destruction when a thread first touches or exits a cache. Linux ELF sizes do not predict Windows TLS costs.

Machine and run record

Builds and benchmarks used CPUs 0–11, at most 12 build jobs, a 6 GiB address-space limit, and normal priority. Governor remained powersave; EPP remained balance_performance. Clang build: 29.06 s, peak RSS 431,732 KiB. GCC build: 46.14 s, peak RSS 896,076 KiB. Largest benchmark RSS was 550,856 KiB. Logs are in Saved/Linux/linux4-*.log; artifacts are under Standalone/Build/linux-* and Standalone/bin/linux/.

Failed or retried runs:

• The first baseline probes with an 8 MiB worker stack failed for none, surface-world, and floating-islands (pthread_create: Invalid argument, exit -6, three attempts each). They were rerun serially at 32 MiB and passed. L4 passed all three probes at 8 MiB.
• An early before/after probe batch ran the six probe types concurrently, so its timings and RSS were discarded. Each type was rerun serially; the final before/after comparison uses those serial logs. The discarded logs are retained with linux4-discarded-parallel- names.
• No build or world/hash run failed. Both Clang and GCC final builds succeeded.

Note for the owner

The shared cache change reduces standalone static TLS from 17.33 MiB to about 43 KiB and preserves the L1 and FloatingIslands hashes; the W12 median moved from 2.399 s to 2.197 s. Please run the listed standalone and Unreal checks on Windows/MSVC before merging, especially worker-thread TLS destruction and the actual UBT FP/TLS options.

Orchestrator review (Claude)

• Re-run by me: interleaved W12, L1 view: 046f2af (L3) 2.212/2.256/2.307 s vs L4 2.063/2.091/2.125 s (medians 2.256 → 2.091 s, -7%). Density c10697a8f48380c7 throughout. L4 PT_TLS MemSiz 0xabc9 = 43,977 B.
• Checked the tunnel-cache route gate in VoxelGenerator.cpp: entries are only stored when CP_GenType is TunnelNetwork/Underwater, so for any other route the old lookup could only miss. Gating skips guaranteed misses, so no output change is possible. The legacy cave route is not exercised by the headless tool, so this rests on code reading, not a hash.
• Windows: thread_local destructors on UE task threads are the MSVC risk. Luna's to-do list covers it.