VoxelForge/Standalone/Research/WorldAssemblerM3/Research7/REPORT.md

Research 7 — terrain disk cache: Windows and multiplayer

Inspection date: 2026-10-23. Read-only review of Standalone/Source/StreamingWorld.h, Standalone/Source/StandaloneViewer.cpp, Core version declarations, and the verification README.

What exists and Windows blockers

FStandaloneMeshCache is a whole-world archive, not an on-demand disk cache. Its filename is derived from seed, clip radius, and tunnel mode; the header also checks source JSON FNV-1a-64, generation settings, cache/algorithm version, tunnel mode, and voxel size. A tile lookup uses (LOD, X, Y, Z). The VFMESH1\0 v1 archive has an 80-byte raw C++ header, 40-byte raw tile records, float32 position/normal pairs, and uint32 indices. It is uncompressed and host-layout/endian dependent; the endian marker detects mismatch but does not convert fields. Save/load also omit generated SurfaceVertexCount, SkirtTriangleCount, and TriangleSkirtFaceMasks, so a cache hit cannot preserve skirt culling.

Load reads and validates the entire archive synchronously in ConfigureProfile before workers start. Streaming workers then copy hits from an immutable in-memory map. Bake save writes synchronously to the final path with truncation. There is no disk eviction or file lock; leaving tiles evicts resident meshes only. The cache has no mmap, fsync, rename, or zlib use today. Existing checks reject mismatched metadata, invalid LOD/counts, truncated data, duplicate keys, non-finite positions/normals, and out-of-range indices, but there is no payload checksum, total allocation budget, or trailing-data check. Stored density hashes are not checked against the mesh.

Windows concerns are the narrow .string() path conversion and a fixed MAX_PATH buffer in DefaultDataPath; the cache is written next to source data, which may be read-only in a packaged install. Windows long-path support needs application opt-in, and the cache path should stay in TCHAR/UE paths. Direct-to-final truncation can leave a partial archive and destroy the previous one; ofstream::flush is not a durable or atomic replacement. Concurrent bakes can collide. The standalone viewer itself has unguarded Win32/D3D11 headers and should not be pulled into the UE plugin; keep any platform adapter gated and use UE file APIs.

UE5 design

• Store a versioned, endian-neutral per-tile payload under FPaths::ProjectSavedDir()/VoxelForge/TileCache/<project-id>/<user-scope>/. Use ProjectPersistentDownloadDir() when the packaged install’s Saved directory is not writable; append a project identity either way. FPaths API
• Use FPlatformFileManager::Get().GetPlatformFile().OpenAsyncRead() and IAsyncReadFileHandle::ReadRequest() for tile/block reads. Parse and decompress on a worker; never wait for file I/O or decompression on the game thread. UE documents async open as non-blocking and read requests as caller-owned; keep buffers/requests alive through completion and cancellation. FPlatformFileManager, IPlatformFile, ReadRequest
• Use UE FCompression with registered format names and IsFormatValid; its compression/decompression routines are thread-safe. This avoids a separately linked Windows zlib. Keep codec ID/version in each tile record and check availability in the packaged client/server. Oodle Data is supplied as a UE plugin and documents Kraken, Mermaid, and Selkie as distinct size/decode-speed choices; do not assume a packaging setting alone enables a runtime cache codec. FCompression, Oodle Data (UE 5.7)
• Treat the generated terrain as persisted feature data, not as a speed claim. Record schema, world key, tile key, codec, compressed/uncompressed sizes, and payload digest. Include all mesh groups, masks, and collision inputs required to rebuild identical tiles. Write a unique temp beside the target, flush and close it, then replace/rename on the same volume. Do not allow a cross-volume copy fallback. Use ReplaceFileW/same-volume MoveFileExW through a Windows adapter; FlushFileBuffers is the file-data durability step. ReplaceFileW, MoveFileExW, FlushFileBuffers, Windows long paths

Cache misses, corrupt entries, and unavailable codecs fall back to cold generation on workers. Keep eviction as a background size-bounded LRU; cache eviction only causes regeneration. Continue judging the no-cache cold path separately.

Key, multiplayer, invalidation, validation

Key a base tile by authoritative season/world-plan identity (include Core recipe and plan content hashes), recipe/schema version, evaluator version, selected mesher plus an explicit mesher implementation revision, LOD and absolute tile coordinate. Add the development source stamp and cache-format version. Core currently declares WorldAssembly schema 2/evaluator 208; FloatingIsland recipe schema 1/default geometry version 4; mesher choices are MarchingCubesV1, HybridV1, HybridV2, with far-horizon version 4 affecting LOD4–8. Read actual resolved recipe/plan versions and hashes from the active snapshot. The verification README defines a SHA-256 source stamp over a sorted source manifest; it is dynamic and can optionally include the Unreal adapter. The current cache uses only the source-data FNV hash and does not carry that stamp.

Persist immutable base terrain only. Keep player edits in the replicated diff layer and overlay them at runtime. Do not write edited composite tiles into the base cache; if a separate composite cache is ever added, key it by the affected tile’s edit revision/content hash. A client cache is visual convenience only: the server uses its own matching cache or regenerates and builds its own collision; never accept client cache bytes as collision or gameplay authority.

On load, bound both per-record and total tile/byte allocations before allocating; require exact file/block lengths, known codec/schema/key, finite and bounded geometry, valid normals, triangle/index/mask counts, and index ranges. Verify a per-payload digest and regenerate on failure. A digest detects accidental corruption, not a malicious rewrite that also changes the digest; only importable/shared caches need a server signature or MAC. Never trust a cache supplied by a client.

Compression probe and next build round

A small serial Python probe used VFMSH001 exports linux27-fi-anchor-pillar.vfmesh and linux27-sw-column-wide.vfmesh, all non-empty tiles LOD0–8, CPython zlib level 1. Payload excludes scene/tile headers. These are Linux in-memory decode numbers, not UE/MSVC or disk-I/O results; exports lack recipe/source stamps and density samples.

Export	Non-empty tiles	Mean raw → zlib bytes/tile	Payload ratio	Decode ms/tile p50 / p95
FI	1,380	75.6 KB → 24.2 KB	32.0%	0.109 / 0.523
SW	1,416	60.0 KB → 21.5 KB	35.8%	0.114 / 0.459

The serial probe peak RSS was 15.4 MiB. Per-LOD details are in Saved/Linux/research/R7_python_zlib_probe.json; script: Saved/Linux/research/R7_python_zlib_probe.py.

Later compare none, zlib, Oodle Kraken/Mermaid/Selkie, and LZ4; quantized vertices (uint16 positions, compact normals) with delta indices; and density/sign-bit packing with RLE. Use the same owner FI and SW tile sets at LOD0–8, keyed to the exact active recipe/plan. Record bytes/tile (median/p95/total), single-core decode ms/tile (median/p95), peak working memory, and cold disk-read-plus-decode latency separately. Validate geometry error, seams, and server collision against uncompressed output. Sign-bit inputs are absent from these mesh exports, so that candidate needs a later density export and must include remesh time in its comparison.

Owner question

Will cache files remain local to each user/install, or be copied/shared between machines?