wip(L16, side branch): disk terrain cache design + headless Linux prototype

Exact lossless zlib tile records (keyed on world/season/recipe/plan/evaluator/options/tile),
pristine cache plus ordered edits, server never trusts client caches, corruption/stale rejection,
atomic writes, 2 GB cap. Loaded tiles match regenerated hashes (L1 view, 24 m crossing).
NOT for assembler/world yet: the shared StreamBenchmarkPipeline.h includes Linux-only headers
(unistd/fcntl/zlib), which would break the Windows viewer build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CSFYgsdW8ejZSyiMf4qRSX

Standalone/Research/WorldAssemblerM3/Linux16/MeasureZstd.py

@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+"""Per-tile zstd CLI comparison over the pristine L1 records from this round."""
+
+from pathlib import Path
+import shutil
+import struct
+import subprocess
+import time
+import zlib
+
+
+REPO = Path.cwd().resolve()
+CACHE = REPO / "Saved/Linux/cache/l16-prototype"
+CAPTURE_LOG = REPO / "Saved/Linux/l16-cache-l1-q24-full-write.log"
+TEMP = CACHE / "codec-zstd"
+RESULT = REPO / "Saved/Linux/l16-zstd-codec.txt"
+
+assert CACHE.is_dir() and not CACHE.is_symlink()
+assert CACHE.resolve().parent == (REPO / "Saved/Linux/cache").resolve()
+assert CAPTURE_LOG.is_file()
+assert not TEMP.exists()
+
+cutoff_ns = CAPTURE_LOG.stat().st_mtime_ns
+records = [
+    path for world in CACHE.glob("world_*")
+    for path in world.glob("*.vftile")
+    if "_s0_d0000000000000000_" in path.name and path.stat().st_mtime_ns <= cutoff_ns
+]
+records.sort()
+assert len(records) == 2553, f"expected 2553 L1 pristine records, found {len(records)}"
+
+codec = shutil.which("zstd")
+assert codec, "zstd CLI is not available"
+TEMP.mkdir()
+per_lod = {}
+try:
+    for source in records:
+        record = source.read_bytes()
+        assert record[:8] == b"VFTILE16"
+        raw_size, compressed_size = struct.unpack_from("<QQ", record, 124)
+        raw = zlib.decompress(record[144:144 + compressed_size])
+        assert len(raw) == raw_size
+        lod = int(source.name.split("_", 1)[0][3:])
+        entry = per_lod.setdefault(lod, {"raw": 0, "files": [], "encode_ns": 0, "decode_ns": 0, "zstd": 0})
+        raw_file = TEMP / (source.stem + ".raw")
+        raw_file.write_bytes(raw)
+        entry["raw"] += len(raw)
+        entry["files"].append(raw_file)
+
+    for lod in sorted(per_lod):
+        entry = per_lod[lod]
+        start = time.perf_counter_ns()
+        subprocess.run([codec, "-q", "-1", "-f", "--", *map(str, entry["files"])], check=True)
+        entry["encode_ns"] = time.perf_counter_ns() - start
+        compressed_files = [Path(str(path) + ".zst") for path in entry["files"]]
+        entry["zstd"] = sum(path.stat().st_size for path in compressed_files)
+        start = time.perf_counter_ns()
+        subprocess.run([codec, "-q", "-d", "-f", "--", *map(str, compressed_files)], check=True)
+        entry["decode_ns"] = time.perf_counter_ns() - start
+
+    # Verify each decoded output byte-for-byte against the zlib payload again.
+    # The filename is the tile stem, so a deterministic source lookup is sufficient.
+    source_by_stem = {path.stem: path for path in records}
+    for entry in per_lod.values():
+        for raw_file in entry["files"]:
+            source = source_by_stem[raw_file.stem]
+            data = source.read_bytes()
+            _, compressed_size = struct.unpack_from("<QQ", data, 124)
+            assert raw_file.read_bytes() == zlib.decompress(data[144:144 + compressed_size])
+
+    lines = ["lod tiles raw_total_B zstd1_total_B zstd1_avg_B encode_us_per_tile decode_us_per_tile status"]
+    for lod in sorted(per_lod):
+        entry = per_lod[lod]
+        count = len(entry["files"])
+        lines.append(
+            f"{lod} {count} {entry['raw']} {entry['zstd']} {entry['zstd'] / count:.1f} "
+            f"{entry['encode_ns'] / count / 1000:.2f} {entry['decode_ns'] / count / 1000:.2f} PASS"
+        )
+    raw_total = sum(entry["raw"] for entry in per_lod.values())
+    zstd_total = sum(entry["zstd"] for entry in per_lod.values())
+    lines.append(f"total_tiles={len(records)} raw_bytes={raw_total} zstd1_bytes={zstd_total} ratio={zstd_total / raw_total:.6f}")
+    lines.append("decode=byte-exact; timing includes zstd CLI process startup once per LOD")
+    RESULT.write_text("\n".join(lines) + "\n")
+    print("\n".join(lines))
+finally:
+    resolved = TEMP.resolve()
+    assert resolved.parent == CACHE.resolve()
+    shutil.rmtree(TEMP)

Standalone/Research/WorldAssemblerM3/Linux16/REPORT.md

@@ -0,0 +1,232 @@
+# Linux16 — terrain cache design and Core prototype
+
+This is a persistence feature. The measurements judge exactness, storage, and worker responsiveness;
+first-time terrain generation remains a separate cost. The prototype is headless and opt-in. Unreal
+integration remains for the owner's PC.
+
+## Design
+
+- **Cache identity.** Production keys should include a stable world ID; season ID and canonical
+  season-content hash; seed and RNG contract; recipe, plan, biome/manifest and owner-content hashes;
+  schema and `WorldAssemblyEvaluatorVersion` (6 here); tile coordinate and LOD; voxel size and tile
+  dimensions; generation options; material/payload-section schema; mesher contract; and a generator
+  build/FP-contract identity. Include a platform or compiler identity unless cross-build output has
+  been proven equivalent. Clang and GCC matched this round's full hash matrix, but that alone does
+  not make every future build interchangeable. Use canonical content hashes so whitespace or JSON
+  key order does not change the world identity.
+- **Prototype identity.** The header and file name include the raw owner-snapshot hash, Core
+  configuration, recipe/plan/manifest hashes, seed, schema, evaluator, voxel size, tile size,
+  mesher/FP contract, generation-options hash, tile key and LOD. The edited-tile probe also includes
+  edit sequence and tile edit digest. The prototype uses 64-bit FNV for identity inputs and CRC32
+  for accidental file corruption. Production should use a 256-bit content hash and a stable world
+  ID; the prototype hashes are evidence machinery, not a security boundary.
+- **Edits.** Keep the reusable cache pristine. On load, apply the authoritative versioned diff layer
+  in sequence, then build the edited render/collision result. A derived edited-tile cache, if useful,
+  must key on the last applied sequence and a digest of edits intersecting that tile. A newer edit
+  invalidates only derived entries for touched tiles; it leaves pristine entries valid. The edit
+  probe wrote and reread sequence 1 with digest `3cd5603ebf707d0c` and matched every array and hash.
+- **Server.** Never accept a client's cached mesh or collision as authoritative. The server may use
+  its own locally generated cache for pristine meshes, then apply its ordered edit layer and build
+  collision from that result. Client cache contents never cross the network; only validated edits
+  do. A server restart can reuse its own cache after identity and payload validation.
+- **Invalidation and verification.** Season/content, seed/RNG, recipe, plan, evaluator, generator,
+  mesher, FP, voxel-grid, LOD, options, or payload-schema changes select a different key. Evaluator
+  bump and edit-sequence bump were rejected by the prototype. A canonical uncompressed-payload hash
+  should be stored and verified at load; corruption or a mismatch is a cache miss, followed by
+  regeneration on a worker. The prototype verifies CRCs and structure on every load, compares the
+  complete mesh arrays and Core D/M/P/T hashes after every generated write/read, and regenerates a
+  deliberately corrupted cached tile. This proves the sampled write/read route; a file's own hash
+  cannot independently prove generator semantics on every later hit, so complete identity and
+  version discipline remain essential.
+- **Stored data.** The prototype stores the final Core mesh: exact double positions/normals, float
+  UVs, RGBA, section/group counts, indices, skirt masks, density sample count, and D/M/P/T hashes.
+  This avoids remeshing on a hit and maps directly to future RMC sections. The indices and section
+  ranges can feed server-side collision construction after edits. Density/sign tiles offer more
+  freedom for future meshers but require meshing, materials, and collision work on load; density
+  sizes were not benchmarked here. Keep collision ownership server-side in either design.
+- **Worker/smoothness path.** Cache reads, zlib decode, integrity checks, serialization, and writes
+  run in the existing headless Core tile worker/result route. The Unreal game thread was not
+  present on this machine; its later integration should submit cache work through the same worker
+  queue and return the ordinary tile result. The prototype uses that route for all recipes and
+  changes no recipe implementation. It exercises SurfaceWorld for cache round trips; the cache-off
+  matrix and route audits cover SurfaceWorld and FloatingIslands.
+- **Read versus regenerate.** Track per-LOD generation and load/decode EWMA. If a demanded tile's
+  estimated read/decode time exceeds generation time, schedule generation directly; if an attempted
+  load fails, discard it and regenerate without blocking the result queue. Keep the same tile
+  priority and permit the previous coarser LOD to remain visible while work finishes. On this L1
+  set, warm read/decode averaged about 0.11 ms per LOD0 tile and 0.91–1.61 ms per LOD8/7 tile,
+  compared with 3.97 ms LOD0 and 12.89–24.24 ms LOD8/6 mesh generation in the instrumented fill.
+- **Layout, crash safety, and eviction.** One file per tile under a world-content directory keeps
+  reads and eviction independent; the prototype has 2,553 L1 records and 2,986 touched records after
+  the 24 m crossing. It writes a same-directory `.tmp`, flushes and `fsync`s it, atomically renames
+  to `.vftile`, then syncs the directory. Readers only open final `.vftile` names and reject bad
+  lengths, versions, identities, CRCs, counts, indices, and padding. A crash before rename leaves a
+  non-loadable temp file. Production startup should remove old temp files. Use a global 2 GB cap
+  with an LRU index by last use, protect entries currently in use, reserve space under one lock,
+  evict oldest unpinned records before writes, and skip the write if space still cannot be made.
+  The prototype enforces the cap and skips writes when full; it does not implement LRU eviction.
+  Its 4 KiB padding enabled `O_DIRECT` measurements and can be removed in production buffered I/O.
+
+## Compression measurements
+
+Available system zlib headers/library (zlib 1.3.1) were linked with `-lz`. The existing zstd 1.5.7
+CLI had no development headers, so it was tested as a per-tile CLI codec in LOD batches; its times
+include one process start per LOD. No installs or system changes were made. All totals below cover
+all 2,553 L1 tiles except Q16, whose first run had five route-audit tiles prepopulated before codec
+sampling. Sizes are payload bytes; the lossless cache also has headers and per-file 4 KiB padding.
+
+| Encoding | Total bytes | Relative to raw | Exact Core arrays? | L1 gate |
+|---|---:|---:|---|---|
+| Raw canonical mesh | 107,342,349 | 100% | Yes | Pass |
+| zlib level 6 | 47,388,401 | 44.15% | Yes | Pass |
+| zstd level 1 (CLI, per tile) | 46,513,479 | 43.33% | Yes | Pass; 1.85% smaller than zlib payload |
+| Q16 positions + oct16 normals + delta-varint indices | 38,672,768 (2,548 tiles sampled) | 36.17% | No | Fail: 11.4482 px at LOD0; 6 degeneracy changes (2 at LOD4, 4 at LOD6) |
+| Q24 positions + oct16 normals + delta-varint indices | 42,862,996 | 39.94% | No | Pass: 0.0458 px max, 0.000060 Lambert delta max, 0.00361° normal error max, no winding/degeneracy changes |
+
+Q24 is about 9.5% smaller than zlib's payload, but it changes floating-point positions/normals and
+cannot satisfy exact loaded-versus-regenerated hashes. The selected cache path is lossless zlib.
+The zstd CLI decoded every tile byte-for-byte, and was modestly smaller/faster in this test. There
+are no zstd development headers here, so it was not linked into the worker prototype. The L1 zlib
+records occupied 55,685,120 bytes on disk, including record framing and page padding.
+
+Per-tile size averages by LOD:
+
+| LOD | Tiles | Raw B | zlib B | zstd B | Q16 B* | Q24 B |
+|---:|---:|---:|---:|---:|---:|---:|
+| 0 | 512 | 19,232.6 | 7,901.7 | 8,009.4 | 7,115.2 | 7,797.3 |
+| 1 | 448 | 15,437.8 | 6,715.2 | 6,715.6 | 5,394.2 | 6,244.9 |
+| 2 | 448 | 11,152.2 | 4,452.4 | 4,516.8 | 4,129.9 | 4,548.7 |
+| 3 | 384 | 11,750.3 | 4,813.1 | 4,885.5 | 4,361.7 | 4,801.7 |
+| 4 | 320 | 49,186.9 | 18,865.3 | 18,352.3 | 17,858.1 | 19,738.2 |
+| 5 | 144 | 83,106.3 | 37,693.0 | 37,004.3 | 30,247.6 | 33,494.7 |
+| 6 | 96 | 128,102.3 | 59,624.7 | 56,521.2 | 46,390.6 | 51,437.7 |
+| 7 | 48 | 308,206.8 | 141,387.7 | 139,348.3 | 110,361.7 | 121,997.0 |
+| 8 | 153 | 171,712.3 | 81,805.0 | 79,662.9 | 60,755.8 | 67,046.4 |
+
+*Q16 averages are over the 2,548 newly generated tiles; five prepopulated tiles did not enter its
+codec sample. Raw, zlib, and Q24 averages are from the clean 2,553-tile fill.
+
+Average codec encode/decode time per tile, microseconds:
+
+| LOD | Raw E/D | zlib E/D | zstd CLI E/D | Q24 mesh E/D |
+|---:|---:|---:|---:|---:|
+| 0 | 38.60 / 12.31 | 736.63 / 79.49 | 72.14 / 47.97 | 16.28 / 18.78 |
+| 1 | 30.13 / 9.47 | 608.30 / 68.07 | 62.71 / 40.53 | 14.26 / 16.04 |
+| 2 | 18.94 / 4.64 | 286.44 / 37.86 | 48.50 / 34.65 | 8.20 / 7.55 |
+| 3 | 20.21 / 5.18 | 313.54 / 41.77 | 50.73 / 35.90 | 8.81 / 8.07 |
+| 4 | 90.46 / 27.53 | 1,508.23 / 189.40 | 134.50 / 78.66 | 41.58 / 41.93 |
+| 5 | 163.48 / 55.53 | 2,935.24 / 370.92 | 220.23 / 140.52 | 76.90 / 79.86 |
+| 6 | 254.98 / 96.01 | 4,900.50 / 611.07 | 361.50 / 184.35 | 127.39 / 136.19 |
+| 7 | 621.57 / 211.84 | 11,944.70 / 1,475.82 | 861.84 / 442.64 | 327.72 / 391.27 |
+| 8 | 325.34 / 109.80 | 6,478.56 / 784.32 | 486.54 / 289.96 | 167.25 / 186.01 |
+
+## Smoothness, prototype, and budgets
+
+- A clean write generated, losslessly stored, reloaded, and exactly compared all 2,553 L1 tiles.
+  The whole-view D/M/P/T hash matched L14. The warm run hit 2,553/2,553 tiles, returned the same
+  hash, and reached whole-view-ready in 0.450 s including 0.193 s of shape setup; tile streaming
+  took 0.243 s.
+- The 24 m crossing covered 432 entering tiles. Its D/M/P/T hash was
+  `337ca62131576728 / 7612f48e2ddbbaab / 69a67ed4add83b87 / f11ac1ebd0f7ddb8`, equal to L14.
+  The cache route audit passed 9 pristine pairs, one edited pair, and one movement-history pair,
+  all with exact arrays and zero D/M/P/T mismatches. The crossing fill generated and round-tripped
+  428 tiles; four had already been created by the route audit.
+- Safety controls passed: evaluator bump rejected, edit-sequence bump rejected, a flipped compressed
+  payload byte rejected by record CRC, and a truncated file rejected by size validation. A separate
+  L1 run with one real tile corrupted regenerated that tile, hit the other 2,552, and restored the
+  exact L14 whole-view hash.
+- `O_DIRECT` bypassed the kernel page cache for the 55,685,120-byte L1 record set: 0.269 s
+  (197.6 MiB/s). Buffered first touch was 0.0191 s and the repeat was 0.0126 s, but those passes
+  followed file writes/verification and were page-cache warm. No page cache was dropped; device
+  firmware may also cache blocks, so `O_DIRECT` is page-cache-cold evidence rather than proof of a
+  physical cold-device read. The ordinary worker hit path includes buffered read, zlib decode,
+  validation, and tile construction.
+- The instrumentation measures raw, zlib, and mesh candidates on a miss, so its clean-fill
+  whole-view-ready time was 2.674 s versus 1.718 s for the hash-on no-cache L1 run. This includes
+  benchmark-only extra codec work. Production should encode only its selected lossless format on a
+  miss. Warm disk work and decode stay on workers; the first tile was ready in 0.226 s on fill and
+  0.208 s warm. These headless timings show worker completion; an Unreal frame-time check is still
+  for the integration round.
+- The cache root peaked at 60,835,840 bytes after L1, the 24 m crossing, edit probe, and corruption
+  controls; the enforced limit is 2,000,000,000 bytes. No eviction was needed. The actual cache
+  implementation is a headless Core-worker prototype, not the Unreal plugin integration.
+
+## Default-path cost and compatibility
+
+Five interleaved W12 L1 pairs, hash off, 2,553 tiles each:
+
+| Pair | L14 seconds | L16 cache-disabled seconds |
+|---:|---:|---:|
+| 1 | 1.450 | 1.446 |
+| 2 | 1.464 | 1.457 |
+| 3 | 1.458 | 1.453 |
+| 4 | 1.525 | 1.472 |
+| 5 | 1.459 | 1.483 |
+
+Median: 1.459 s L14, 1.457 s L16, −0.14%; no measurable cache-off cost in this run.
+
+All 28 L14 hash configurations passed (SurfaceWorld and FloatingIslands; Clang 19 FP off and GCC
+14 default FP; W1/W8/W12 with five W12 repeats). Every one matched all 12 full-detail, whole-view,
+nine LOD, and exact-query rows. The four default 24 m crossing checks (W1/W12 on both compilers)
+matched all 13 rows, including the 432-tile crossing hash. Default route parity passed for both
+compilers across SurfaceWorld seeds 0, 1, and 17 and FloatingIslands seed 0, using the existing
+96 m history probe. The cache route passed its separate L1 and 24 m crossing audits.
+
+Representative SurfaceWorld seed-0 hashes (D/M/P/T):
+
+| View | Density / mesh / payload / topology |
+|---|---|
+| Full detail, 960 tiles | `0ee0579bd15cbbe4 / 32e5d4964d610fee / 1932b96ee4b1d910 / 69743c6bad3447dc` |
+| Whole L1, 2,553 tiles | `c10697a8f48380c7 / e75c9119a1498cd5 / 49a9a8d1835a6c8c / edcb23288b5e8724` |
+| LOD0, 512 tiles | `769f2a706d585a72 / 33eeeb39fa443816 / 645621a1d968983f / c2457ed905036c9d` |
+| LOD1, 448 tiles | `69cc5a5c0a2c60af / b48335f5410c57d5 / d15c5480162ab8ba / feca1bd025823604` |
+| LOD2, 448 tiles | `85deed5683a77202 / 53469895987aeea2 / e55f8ae7e1532033 / d5feb12a0e85c09b` |
+| LOD3, 384 tiles | `6518caefb3be7e7f / ec513c90b853a614 / f62fa1137ff17f82 / ccf1dcf681965816` |
+| LOD4, 320 tiles | `62d4be96021992f4 / f1e4c6efcc2615a9 / 9dc5f9c88be02f33 / fe6f55125a07c3bf` |
+| LOD5, 144 tiles | `5e4a8c38b8cea6fc / 08ee56998aaf21e5 / 541eea8558f26cc8 / 6539374a8b3336d8` |
+| LOD6, 96 tiles | `e036e95b9006945b / 624a33f941a99b02 / 81b73e14f68bdf86 / 4de4c63ce1f80a56` |
+| LOD7, 48 tiles | `8ee358af56a30d8d / 46ab3c6184048b7c / 0a6164b7e68e80c5 / 0c26b85bb03bc831` |
+| LOD8, 153 tiles | `c3458e1cbdb9c471 / 46c03d0994d75192 / 8f89fc5bde12d230 / d16dc0cf62031898` |
+| Exact queries: density / unclamped / detail / collision | `e4d4a8eb95b35933 / e4d4a8eb95b35933 / efcba20c0c70c415 / e604823a249029bf` |
+
+FloatingIslands full-detail was `6d842d2878b08ef6 / 4b868a3184f08035 / 2cedf2236788752d / eac17d5e0cb5044a`; whole-view was `5be4d8cde79b2a76 / 96f75041f6861b3a / 3700a516cc89d50c / 6b99948d6eab4f21`. Its exact-query hashes also matched L14: `61daded9c303becb / 61daded9c303becb / efcba20c0c70c415 / 84d37463be9b0573`.
+
+## Runs and limits
+
+- First Clang build was interrupted during a silent compile phase. The next build exposed a const
+  error in the prototype file index; making that index mutable fixed it. Final Clang and GCC builds
+  passed.
+- The first L1 command used the binary's incorrect default data path (`Standalone/bin/data`) and
+  exited before writing. Supplying `Standalone/data/owner_world.json` passed.
+- The first codec run sampled 2,548 tiles because route parity had prepopulated five; it was kept
+  for Q16 measurements. A clean fill then sampled all 2,553 Q24 tiles.
+- A route wrapper first required a moving hash when hashing was disabled; the route audit itself
+  passed. Seed 17's 24 m history probe had no departed-tile candidate and failed on both L16 and
+  the L14 control. The same L14 96 m history setup passed on both L16 compilers. The required 24 m
+  crossing was separately checked and passed.
+- The first crossing-hash comparator used an incorrect old-log filename; correcting the filename
+  let all four comparisons pass. No run exceeded 30 minutes or stalled for 15 minutes.
+- Final review found that the cache-hit route audit's `exact_arrays` field initially reported hash
+  agreement without comparing the arrays themselves. The audit was changed to compare complete
+  arrays; after rebuilding, both L1 and 24 m crossing route audits passed with zero mismatches.
+
+Runs used CPUs 0–11, at most 12 build jobs, a 6 GiB address-space cap, and normal priority. The
+governor stayed `powersave`; peak sampled single-process RSS was 943,844 KiB (FloatingIslands route
+audit, GCC). No installs, system changes, or page-cache drops were made. The L3 FP-contract preamble
+and self-test, L4 `FLazy`, L7 shared cache, L9/L12 proofs, L11 readiness/crash fix, L13 route binding,
+and async worker/result rules remain enabled. Cache logic is recipe-agnostic.
+
+**Owner note:** Saving generated terrain works exactly and fits well under the round's disk cap.
+Lossless zlib saved about 56% of the serialized payload. The smaller Q24 mesh encoding passed the
+screen gate but changed exact hashes, so the feature should use the lossless record format.
+
+## Orchestrator review (Claude)
+
+- Design accepted: the key, the pristine cache plus ordered edits, a server that never trusts client
+  caches, corruption/stale rejection, lossless zlib, and atomic writes.
+- **Not merged into `assembler/world`:** `Standalone/Source/StreamBenchmarkPipeline.h` (included by the
+  Windows viewer `StandaloneViewer.cpp`) now unconditionally includes `TileCachePrototype.h`, which
+  includes `<unistd.h>`, `<fcntl.h>` and `<zlib.h>`. That would break the MSVC build. Kept on side
+  branch `wip/l16-tile-cache`. Before merging, the prototype must be platform-guarded (or the cache
+  given a portable I/O and codec layer), and zlib needs a Windows source (UE ships zlib).
+- Cache-off cost re-checked by Luna at −0.14% (within noise).