refactor(core): bind generator fields once per resolved strate (L40, migration step 2); research R7
SurfaceWorld, FloatingIslands and legacy evaluators sit behind one immutable densityAt adapter held in a variant (no virtual call per sample); BindResolvedStratumFields builds a sorted strate table at world setup, so DensityAt, native SW/FI sampling and ClassifyBoxDetailed no longer ask the strate manager per sample. No arithmetic change: owner hashes identical to L37/L39 at W1/W12. Cost vs L39: FI -0.66%, SW -1.31% (noise). verify fast/medium/full PASS (full re-run alone: 407 gates). Research R7: disk cache design for UE5/Windows/multiplayer and a zlib probe (about 1/3 size, 0.1 ms/tile). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016dhxM43dnWvUu2B5Ph1iG1
This commit is contained in:
1 parent
753cc9c259
commit
efa9e89468
8 files changed
+859
-118
No files matched your search
@@ -0,0 +1,45 @@
|
||||
# Research 7 — terrain disk cache: Windows and multiplayer
|
||||
|
||||
Inspection date: 2026-10-23. Read-only review of `Standalone/Source/StreamingWorld.h`, `Standalone/Source/StandaloneViewer.cpp`, Core version declarations, and the verification README.
|
||||
|
||||
## What exists and Windows blockers
|
||||
|
||||
`FStandaloneMeshCache` is a whole-world archive, not an on-demand disk cache. Its filename is derived from seed, clip radius, and tunnel mode; the header also checks source JSON FNV-1a-64, generation settings, cache/algorithm version, tunnel mode, and voxel size. A tile lookup uses `(LOD, X, Y, Z)`. The `VFMESH1\0` v1 archive has an 80-byte raw C++ header, 40-byte raw tile records, float32 position/normal pairs, and uint32 indices. It is uncompressed and host-layout/endian dependent; the endian marker detects mismatch but does not convert fields. Save/load also omit generated `SurfaceVertexCount`, `SkirtTriangleCount`, and `TriangleSkirtFaceMasks`, so a cache hit cannot preserve skirt culling.
|
||||
|
||||
Load reads and validates the entire archive synchronously in `ConfigureProfile` before workers start. Streaming workers then copy hits from an immutable in-memory map. Bake save writes synchronously to the final path with truncation. There is no disk eviction or file lock; leaving tiles evicts resident meshes only. The cache has no `mmap`, `fsync`, rename, or zlib use today. Existing checks reject mismatched metadata, invalid LOD/counts, truncated data, duplicate keys, non-finite positions/normals, and out-of-range indices, but there is no payload checksum, total allocation budget, or trailing-data check. Stored density hashes are not checked against the mesh.
|
||||
|
||||
Windows concerns are the narrow `.string()` path conversion and a fixed `MAX_PATH` buffer in `DefaultDataPath`; the cache is written next to source data, which may be read-only in a packaged install. Windows long-path support needs application opt-in, and the cache path should stay in TCHAR/UE paths. Direct-to-final truncation can leave a partial archive and destroy the previous one; `ofstream::flush` is not a durable or atomic replacement. Concurrent bakes can collide. The standalone viewer itself has unguarded Win32/D3D11 headers and should not be pulled into the UE plugin; keep any platform adapter gated and use UE file APIs.
|
||||
|
||||
## UE5 design
|
||||
|
||||
- Store a versioned, endian-neutral per-tile payload under `FPaths::ProjectSavedDir()/VoxelForge/TileCache/<project-id>/<user-scope>/`. Use `ProjectPersistentDownloadDir()` when the packaged install’s Saved directory is not writable; append a project identity either way. [FPaths API](https://dev.epicgames.com/documentation/en-us/unreal-engine/API/Runtime/Core/FPaths)
|
||||
- Use `FPlatformFileManager::Get().GetPlatformFile().OpenAsyncRead()` and `IAsyncReadFileHandle::ReadRequest()` for tile/block reads. Parse and decompress on a worker; never wait for file I/O or decompression on the game thread. UE documents async open as non-blocking and read requests as caller-owned; keep buffers/requests alive through completion and cancellation. [FPlatformFileManager](https://dev.epicgames.com/documentation/unreal-engine/API/Runtime/Core/FPlatformFileManager), [IPlatformFile](https://dev.epicgames.com/documentation/unreal-engine/API/Runtime/Core/IPlatformFile), [ReadRequest](https://dev.epicgames.com/documentation/unreal-engine/API/Runtime/Core/Async/IAsyncReadFileHandle/ReadRequest?application_version=5.5)
|
||||
- Use UE `FCompression` with registered format names and `IsFormatValid`; its compression/decompression routines are thread-safe. This avoids a separately linked Windows zlib. Keep codec ID/version in each tile record and check availability in the packaged client/server. Oodle Data is supplied as a UE plugin and documents Kraken, Mermaid, and Selkie as distinct size/decode-speed choices; do not assume a packaging setting alone enables a runtime cache codec. [FCompression](https://dev.epicgames.com/documentation/en-us/unreal-engine/API/Runtime/Core/FCompression), [Oodle Data (UE 5.7)](https://dev.epicgames.com/documentation/en-us/unreal-engine/oodle-data?application_version=5.7)
|
||||
- Treat the generated terrain as persisted feature data, not as a speed claim. Record schema, world key, tile key, codec, compressed/uncompressed sizes, and payload digest. Include all mesh groups, masks, and collision inputs required to rebuild identical tiles. Write a unique temp beside the target, flush and close it, then replace/rename on the same volume. Do not allow a cross-volume copy fallback. Use `ReplaceFileW`/same-volume `MoveFileExW` through a Windows adapter; `FlushFileBuffers` is the file-data durability step. [ReplaceFileW](https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-replacefilew), [MoveFileExW](https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-movefileexw), [FlushFileBuffers](https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-flushfilebuffers), [Windows long paths](https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation)
|
||||
|
||||
Cache misses, corrupt entries, and unavailable codecs fall back to cold generation on workers. Keep eviction as a background size-bounded LRU; cache eviction only causes regeneration. Continue judging the no-cache cold path separately.
|
||||
|
||||
## Key, multiplayer, invalidation, validation
|
||||
|
||||
Key a base tile by authoritative season/world-plan identity (include Core recipe and plan content hashes), recipe/schema version, evaluator version, selected mesher plus an explicit mesher implementation revision, LOD and absolute tile coordinate. Add the development source stamp and cache-format version. Core currently declares WorldAssembly schema 2/evaluator 208; FloatingIsland recipe schema 1/default geometry version 4; mesher choices are MarchingCubesV1, HybridV1, HybridV2, with far-horizon version 4 affecting LOD4–8. Read actual resolved recipe/plan versions and hashes from the active snapshot. The verification README defines a SHA-256 source stamp over a sorted source manifest; it is dynamic and can optionally include the Unreal adapter. The current cache uses only the source-data FNV hash and does not carry that stamp.
|
||||
|
||||
Persist immutable base terrain only. Keep player edits in the replicated diff layer and overlay them at runtime. Do not write edited composite tiles into the base cache; if a separate composite cache is ever added, key it by the affected tile’s edit revision/content hash. A client cache is visual convenience only: the server uses its own matching cache or regenerates and builds its own collision; never accept client cache bytes as collision or gameplay authority.
|
||||
|
||||
On load, bound both per-record and total tile/byte allocations before allocating; require exact file/block lengths, known codec/schema/key, finite and bounded geometry, valid normals, triangle/index/mask counts, and index ranges. Verify a per-payload digest and regenerate on failure. A digest detects accidental corruption, not a malicious rewrite that also changes the digest; only importable/shared caches need a server signature or MAC. Never trust a cache supplied by a client.
|
||||
|
||||
## Compression probe and next build round
|
||||
|
||||
A small serial Python probe used `VFMSH001` exports `linux27-fi-anchor-pillar.vfmesh` and `linux27-sw-column-wide.vfmesh`, all non-empty tiles LOD0–8, CPython zlib level 1. Payload excludes scene/tile headers. These are Linux in-memory decode numbers, not UE/MSVC or disk-I/O results; exports lack recipe/source stamps and density samples.
|
||||
|
||||
| Export | Non-empty tiles | Mean raw → zlib bytes/tile | Payload ratio | Decode ms/tile p50 / p95 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| FI | 1,380 | 75.6 KB → 24.2 KB | 32.0% | 0.109 / 0.523 |
|
||||
| SW | 1,416 | 60.0 KB → 21.5 KB | 35.8% | 0.114 / 0.459 |
|
||||
|
||||
The serial probe peak RSS was 15.4 MiB. Per-LOD details are in `Saved/Linux/research/R7_python_zlib_probe.json`; script: `Saved/Linux/research/R7_python_zlib_probe.py`.
|
||||
|
||||
Later compare none, zlib, Oodle Kraken/Mermaid/Selkie, and LZ4; quantized vertices (uint16 positions, compact normals) with delta indices; and density/sign-bit packing with RLE. Use the same owner FI and SW tile sets at LOD0–8, keyed to the exact active recipe/plan. Record bytes/tile (median/p95/total), single-core decode ms/tile (median/p95), peak working memory, and cold disk-read-plus-decode latency separately. Validate geometry error, seams, and server collision against uncompressed output. Sign-bit inputs are absent from these mesh exports, so that candidate needs a later density export and must include remesh time in its comparison.
|
||||
|
||||
## Owner question
|
||||
|
||||
Will cache files remain local to each user/install, or be copied/shared between machines?
|
||||
@@ -0,0 +1,255 @@
|
||||
{
|
||||
"codec": "CPython zlib level 1",
|
||||
"decode_repeats_per_tile": 3,
|
||||
"files": {
|
||||
"FI": {
|
||||
"path": "Saved/Linux/linux27-fi-anchor-pillar.vfmesh",
|
||||
"file_bytes": 104373453,
|
||||
"scene_version": 1,
|
||||
"voxel_size_microns": 250000,
|
||||
"tile_records": 2482,
|
||||
"nonempty_tile_records": 1380,
|
||||
"raw_payload_bytes": 104284041,
|
||||
"zlib_payload_bytes": 33400206,
|
||||
"weighted_ratio_pct": 32.02810869210563,
|
||||
"raw_bytes_per_nonempty_tile_median": 46104.0,
|
||||
"decode_ms_per_tile_median": 0.109374,
|
||||
"decode_ms_per_tile_p95": 0.523225,
|
||||
"lod": {
|
||||
"0": {
|
||||
"nonempty_tiles": 136,
|
||||
"raw_bytes_total": 11914151,
|
||||
"packed_bytes_total": 3218452,
|
||||
"weighted_ratio_pct": 27.01369153370643,
|
||||
"raw_bytes_median": 95849.5,
|
||||
"packed_bytes_median": 33744.0,
|
||||
"decode_ms_median": 0.27110500000000004,
|
||||
"decode_ms_p95": 0.316059,
|
||||
"encode_ms_median": 0.610732,
|
||||
"encode_ms_p95": 0.837242
|
||||
},
|
||||
"1": {
|
||||
"nonempty_tiles": 131,
|
||||
"raw_bytes_total": 16499945,
|
||||
"packed_bytes_total": 5941348,
|
||||
"weighted_ratio_pct": 36.008289724602115,
|
||||
"raw_bytes_median": 92598,
|
||||
"packed_bytes_median": 42032,
|
||||
"decode_ms_median": 0.317582,
|
||||
"decode_ms_p95": 0.818547,
|
||||
"encode_ms_median": 0.837591,
|
||||
"encode_ms_p95": 2.246851
|
||||
},
|
||||
"2": {
|
||||
"nonempty_tiles": 172,
|
||||
"raw_bytes_total": 6666423,
|
||||
"packed_bytes_total": 2031078,
|
||||
"weighted_ratio_pct": 30.467283579214822,
|
||||
"raw_bytes_median": 45768.0,
|
||||
"packed_bytes_median": 11943.0,
|
||||
"decode_ms_median": 0.102419,
|
||||
"decode_ms_p95": 0.124572,
|
||||
"encode_ms_median": 0.2224085,
|
||||
"encode_ms_p95": 0.256617
|
||||
},
|
||||
"3": {
|
||||
"nonempty_tiles": 125,
|
||||
"raw_bytes_total": 5555529,
|
||||
"packed_bytes_total": 1649468,
|
||||
"weighted_ratio_pct": 29.6905659209051,
|
||||
"raw_bytes_median": 45768,
|
||||
"packed_bytes_median": 11960,
|
||||
"decode_ms_median": 0.104118,
|
||||
"decode_ms_p95": 0.168216,
|
||||
"encode_ms_median": 0.2313,
|
||||
"encode_ms_p95": 0.349597
|
||||
},
|
||||
"4": {
|
||||
"nonempty_tiles": 174,
|
||||
"raw_bytes_total": 9645483,
|
||||
"packed_bytes_total": 3115498,
|
||||
"weighted_ratio_pct": 32.30007247952228,
|
||||
"raw_bytes_median": 45768.0,
|
||||
"packed_bytes_median": 13419.0,
|
||||
"decode_ms_median": 0.1148265,
|
||||
"decode_ms_p95": 0.259002,
|
||||
"encode_ms_median": 0.255053,
|
||||
"encode_ms_p95": 0.641356
|
||||
},
|
||||
"5": {
|
||||
"nonempty_tiles": 144,
|
||||
"raw_bytes_total": 4232303,
|
||||
"packed_bytes_total": 1383894,
|
||||
"weighted_ratio_pct": 32.69836776809222,
|
||||
"raw_bytes_median": 24938.0,
|
||||
"packed_bytes_median": 9315.0,
|
||||
"decode_ms_median": 0.06914100000000001,
|
||||
"decode_ms_p95": 0.178538,
|
||||
"encode_ms_median": 0.1752125,
|
||||
"encode_ms_p95": 0.460837
|
||||
},
|
||||
"6": {
|
||||
"nonempty_tiles": 96,
|
||||
"raw_bytes_total": 5255938,
|
||||
"packed_bytes_total": 1770019,
|
||||
"weighted_ratio_pct": 33.67655782849798,
|
||||
"raw_bytes_median": 38395.0,
|
||||
"packed_bytes_median": 10093.5,
|
||||
"decode_ms_median": 0.088979,
|
||||
"decode_ms_p95": 0.317363,
|
||||
"encode_ms_median": 0.1975715,
|
||||
"encode_ms_p95": 0.823547
|
||||
},
|
||||
"7": {
|
||||
"nonempty_tiles": 96,
|
||||
"raw_bytes_total": 11085771,
|
||||
"packed_bytes_total": 3715174,
|
||||
"weighted_ratio_pct": 33.51299607397627,
|
||||
"raw_bytes_median": 105285.5,
|
||||
"packed_bytes_median": 30236.0,
|
||||
"decode_ms_median": 0.271746,
|
||||
"decode_ms_p95": 0.588424,
|
||||
"encode_ms_median": 0.63578,
|
||||
"encode_ms_p95": 1.607345
|
||||
},
|
||||
"8": {
|
||||
"nonempty_tiles": 306,
|
||||
"raw_bytes_total": 33428498,
|
||||
"packed_bytes_total": 10575275,
|
||||
"weighted_ratio_pct": 31.635507524149006,
|
||||
"raw_bytes_median": 100540.0,
|
||||
"packed_bytes_median": 29556.0,
|
||||
"decode_ms_median": 0.258287,
|
||||
"decode_ms_p95": 0.529198,
|
||||
"encode_ms_median": 0.644282,
|
||||
"encode_ms_p95": 1.330977
|
||||
}
|
||||
}
|
||||
},
|
||||
"SW": {
|
||||
"path": "Saved/Linux/linux27-sw-column-wide.vfmesh",
|
||||
"file_bytes": 85051310,
|
||||
"scene_version": 1,
|
||||
"voxel_size_microns": 250000,
|
||||
"tile_records": 2130,
|
||||
"nonempty_tile_records": 1416,
|
||||
"raw_payload_bytes": 84974570,
|
||||
"zlib_payload_bytes": 30390177,
|
||||
"weighted_ratio_pct": 35.76384911391726,
|
||||
"raw_bytes_per_nonempty_tile_median": 38620.5,
|
||||
"decode_ms_per_tile_median": 0.11431150000000001,
|
||||
"decode_ms_per_tile_p95": 0.459004,
|
||||
"lod": {
|
||||
"0": {
|
||||
"nonempty_tiles": 285,
|
||||
"raw_bytes_total": 22447321,
|
||||
"packed_bytes_total": 9051898,
|
||||
"weighted_ratio_pct": 40.32507041708897,
|
||||
"raw_bytes_median": 79066,
|
||||
"packed_bytes_median": 32524,
|
||||
"decode_ms_median": 0.244825,
|
||||
"decode_ms_p95": 0.389183,
|
||||
"encode_ms_median": 0.65028,
|
||||
"encode_ms_p95": 1.155402
|
||||
},
|
||||
"1": {
|
||||
"nonempty_tiles": 149,
|
||||
"raw_bytes_total": 14395111,
|
||||
"packed_bytes_total": 5813866,
|
||||
"weighted_ratio_pct": 40.38778165725849,
|
||||
"raw_bytes_median": 108155,
|
||||
"packed_bytes_median": 46017,
|
||||
"decode_ms_median": 0.348557,
|
||||
"decode_ms_p95": 0.551613,
|
||||
"encode_ms_median": 0.930073,
|
||||
"encode_ms_p95": 1.662883
|
||||
},
|
||||
"2": {
|
||||
"nonempty_tiles": 217,
|
||||
"raw_bytes_total": 6043511,
|
||||
"packed_bytes_total": 1736514,
|
||||
"weighted_ratio_pct": 28.733529234909973,
|
||||
"raw_bytes_median": 27184,
|
||||
"packed_bytes_median": 6351,
|
||||
"decode_ms_median": 0.058163,
|
||||
"decode_ms_p95": 0.14959,
|
||||
"encode_ms_median": 0.14067,
|
||||
"encode_ms_p95": 0.37157
|
||||
},
|
||||
"3": {
|
||||
"nonempty_tiles": 190,
|
||||
"raw_bytes_total": 6058525,
|
||||
"packed_bytes_total": 1902018,
|
||||
"weighted_ratio_pct": 31.394076941169676,
|
||||
"raw_bytes_median": 35120.0,
|
||||
"packed_bytes_median": 7724.5,
|
||||
"decode_ms_median": 0.0678105,
|
||||
"decode_ms_p95": 0.15743,
|
||||
"encode_ms_median": 0.17532,
|
||||
"encode_ms_p95": 0.392782
|
||||
},
|
||||
"4": {
|
||||
"nonempty_tiles": 173,
|
||||
"raw_bytes_total": 8837946,
|
||||
"packed_bytes_total": 2824214,
|
||||
"weighted_ratio_pct": 31.95554713730996,
|
||||
"raw_bytes_median": 35120,
|
||||
"packed_bytes_median": 13578,
|
||||
"decode_ms_median": 0.106095,
|
||||
"decode_ms_p95": 0.267514,
|
||||
"encode_ms_median": 0.252394,
|
||||
"encode_ms_p95": 0.663049
|
||||
},
|
||||
"5": {
|
||||
"nonempty_tiles": 144,
|
||||
"raw_bytes_total": 6093522,
|
||||
"packed_bytes_total": 1982234,
|
||||
"weighted_ratio_pct": 32.530185334524106,
|
||||
"raw_bytes_median": 35013.0,
|
||||
"packed_bytes_median": 10745.0,
|
||||
"decode_ms_median": 0.08670549999999999,
|
||||
"decode_ms_p95": 0.186559,
|
||||
"encode_ms_median": 0.230201,
|
||||
"encode_ms_p95": 0.474221
|
||||
},
|
||||
"6": {
|
||||
"nonempty_tiles": 96,
|
||||
"raw_bytes_total": 6363816,
|
||||
"packed_bytes_total": 2057514,
|
||||
"weighted_ratio_pct": 32.33145018649188,
|
||||
"raw_bytes_median": 61690.0,
|
||||
"packed_bytes_median": 18087.0,
|
||||
"decode_ms_median": 0.1538235,
|
||||
"decode_ms_p95": 0.308712,
|
||||
"encode_ms_median": 0.394647,
|
||||
"encode_ms_p95": 0.820557
|
||||
},
|
||||
"7": {
|
||||
"nonempty_tiles": 96,
|
||||
"raw_bytes_total": 9716690,
|
||||
"packed_bytes_total": 3399953,
|
||||
"weighted_ratio_pct": 34.99085593962553,
|
||||
"raw_bytes_median": 98110.0,
|
||||
"packed_bytes_median": 30712.0,
|
||||
"decode_ms_median": 0.2564655,
|
||||
"decode_ms_p95": 0.53338,
|
||||
"encode_ms_median": 0.6491285,
|
||||
"encode_ms_p95": 1.50398
|
||||
},
|
||||
"8": {
|
||||
"nonempty_tiles": 66,
|
||||
"raw_bytes_total": 5018128,
|
||||
"packed_bytes_total": 1621966,
|
||||
"weighted_ratio_pct": 32.322132875048226,
|
||||
"raw_bytes_median": 74425.0,
|
||||
"packed_bytes_median": 23710.5,
|
||||
"decode_ms_median": 0.187504,
|
||||
"decode_ms_p95": 0.337573,
|
||||
"encode_ms_median": 0.47401950000000004,
|
||||
"encode_ms_p95": 0.885023
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"process_peak_rss_kib": 15772
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Cheap per-tile zlib probe over existing VFMSH001 scenes; read-only inputs."""
|
||||
import collections
|
||||
import json
|
||||
import os
|
||||
import resource
|
||||
import statistics
|
||||
import struct
|
||||
import sys
|
||||
import time
|
||||
import zlib
|
||||
from pathlib import Path
|
||||
|
||||
FILES = {
|
||||
"FI": Path("Saved/Linux/linux27-fi-anchor-pillar.vfmesh"),
|
||||
"SW": Path("Saved/Linux/linux27-sw-column-wide.vfmesh"),
|
||||
}
|
||||
REPEATS = 3
|
||||
|
||||
def percentile(values, fraction):
|
||||
ordered = sorted(values)
|
||||
if not ordered:
|
||||
return None
|
||||
return ordered[min(len(ordered) - 1, int((len(ordered) - 1) * fraction))]
|
||||
|
||||
def scan(path):
|
||||
rows = collections.defaultdict(list)
|
||||
with path.open("rb") as stream:
|
||||
prefix = stream.read(60)
|
||||
if len(prefix) != 60 or prefix[:8] != b"VFMSH001":
|
||||
raise RuntimeError(f"{path}: not a VFMSH001 scene")
|
||||
version, voxel_um, tile_count = struct.unpack_from("<3I", prefix, 8)
|
||||
if version != 1 or tile_count > 1_000_000:
|
||||
raise RuntimeError(f"{path}: unsupported header")
|
||||
for _ in range(tile_count):
|
||||
header = stream.read(36)
|
||||
if len(header) != 36:
|
||||
raise EOFError(f"{path}: truncated record")
|
||||
key = struct.unpack_from("<4i", header)
|
||||
_, _, vertex_count, index_count, mask_count = struct.unpack_from("<5I", header, 16)
|
||||
if key[3] < 0 or key[3] > 8 or index_count % 3:
|
||||
raise RuntimeError(f"{path}: malformed tile record")
|
||||
raw_size = vertex_count * 24 + index_count * 4 + mask_count
|
||||
payload = stream.read(raw_size)
|
||||
if len(payload) != raw_size:
|
||||
raise EOFError(f"{path}: truncated tile payload")
|
||||
if raw_size == 0:
|
||||
continue
|
||||
encode_start = time.perf_counter_ns()
|
||||
packed = zlib.compress(payload, level=1)
|
||||
encode_ms = (time.perf_counter_ns() - encode_start) / 1_000_000.0
|
||||
decode_ms = []
|
||||
for _ in range(REPEATS):
|
||||
start = time.perf_counter_ns()
|
||||
decoded = zlib.decompress(packed)
|
||||
decode_ms.append((time.perf_counter_ns() - start) / 1_000_000.0)
|
||||
if decoded != payload:
|
||||
raise RuntimeError(f"{path}: round-trip mismatch")
|
||||
rows[key[3]].append({
|
||||
"raw": raw_size, "packed": len(packed),
|
||||
"encode_ms": encode_ms, "decode_ms": statistics.median(decode_ms),
|
||||
"vertices": vertex_count, "indices": index_count,
|
||||
})
|
||||
if stream.read(1):
|
||||
raise RuntimeError(f"{path}: trailing data")
|
||||
return version, voxel_um, tile_count, rows
|
||||
|
||||
def main():
|
||||
if hasattr(os, "sched_getaffinity"):
|
||||
os.sched_setaffinity(0, {min(os.sched_getaffinity(0))})
|
||||
result = {"codec": "CPython zlib level 1", "decode_repeats_per_tile": REPEATS, "files": {}}
|
||||
for name, path in FILES.items():
|
||||
version, voxel_um, tile_count, by_lod = scan(path)
|
||||
lod_result = {}
|
||||
for lod, rows in sorted(by_lod.items()):
|
||||
raw = [r["raw"] for r in rows]
|
||||
packed = [r["packed"] for r in rows]
|
||||
dec = [r["decode_ms"] for r in rows]
|
||||
enc = [r["encode_ms"] for r in rows]
|
||||
lod_result[str(lod)] = {
|
||||
"nonempty_tiles": len(rows),
|
||||
"raw_bytes_total": sum(raw),
|
||||
"packed_bytes_total": sum(packed),
|
||||
"weighted_ratio_pct": 100.0 * sum(packed) / sum(raw),
|
||||
"raw_bytes_median": statistics.median(raw),
|
||||
"packed_bytes_median": statistics.median(packed),
|
||||
"decode_ms_median": statistics.median(dec),
|
||||
"decode_ms_p95": percentile(dec, 0.95),
|
||||
"encode_ms_median": statistics.median(enc),
|
||||
"encode_ms_p95": percentile(enc, 0.95),
|
||||
}
|
||||
total_raw = sum(x["raw_bytes_total"] for x in lod_result.values())
|
||||
total_packed = sum(x["packed_bytes_total"] for x in lod_result.values())
|
||||
all_decode = [r["decode_ms"] for rows in by_lod.values() for r in rows]
|
||||
all_raw = [r["raw"] for rows in by_lod.values() for r in rows]
|
||||
result["files"][name] = {
|
||||
"path": str(path), "file_bytes": path.stat().st_size,
|
||||
"scene_version": version, "voxel_size_microns": voxel_um,
|
||||
"tile_records": tile_count, "nonempty_tile_records": sum(len(v) for v in by_lod.values()),
|
||||
"raw_payload_bytes": total_raw, "zlib_payload_bytes": total_packed,
|
||||
"weighted_ratio_pct": 100.0 * total_packed / total_raw,
|
||||
"raw_bytes_per_nonempty_tile_median": statistics.median(all_raw),
|
||||
"decode_ms_per_tile_median": statistics.median(all_decode),
|
||||
"decode_ms_per_tile_p95": percentile(all_decode, 0.95),
|
||||
"lod": lod_result,
|
||||
}
|
||||
result["process_peak_rss_kib"] = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
|
||||
out = Path("Saved/Linux/research/R7_python_zlib_probe.json")
|
||||
out.write_text(json.dumps(result, indent=2) + "\n", encoding="utf-8")
|
||||
for name, info in result["files"].items():
|
||||
print(f"{name}: tiles={info['nonempty_tile_records']}/{info['tile_records']} "
|
||||
f"raw={info['raw_payload_bytes']} packed={info['zlib_payload_bytes']} "
|
||||
f"ratio={info['weighted_ratio_pct']:.1f}% "
|
||||
f"decode_p50/p95={info['decode_ms_per_tile_median']:.3f}/"
|
||||
f"{info['decode_ms_per_tile_p95']:.3f}ms")
|
||||
print(" LOD raw B/tile median | zlib B/tile median | decode p50/p95 ms")
|
||||
for lod, row in info["lod"].items():
|
||||
print(f" {lod}: {row['raw_bytes_median']:.0f} | {row['packed_bytes_median']:.0f} | "
|
||||
f"{row['decode_ms_median']:.3f}/{row['decode_ms_p95']:.3f}")
|
||||
print(f"peak_rss={result['process_peak_rss_kib']} KiB; JSON={out}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,2 @@
|
||||
# Shared notes (researcher R7 <-> implementer L40), newest at bottom
|
||||
|
||||
Reference in new issue
Block a user