trace: Unreal timing traces run without hash capture (L71); verify baselines tracked

The L70 "2x SurfaceWorld GenerateTile in Unreal" was the instrument: the trace enabled
bCaptureVoxelForgeCoreHashesForValidation, which disables the native heightfield-band
shortcut and whole-tile skips (W1 same keys: 9.00 s hash-off vs 28.46 s hash-on). With
hashing off, Unreal and headless match at W8 within 1.2-2.5% (Core 14.42 s vs 14.26 s) on
identical tile keys. LOD0 p95 tail = real cost of non-empty passage tiles near the spawn mouth
(p50/p95 18.8/105 ms); most LOD0 tiles are empty.

verify.py reads its baselines from Standalone/Tools/verify/baselines/ instead of untracked
Saved/Linux. The 11 .vfmesh control meshes (~78 MB) are not committed yet (to be stored
compressed). Fast tier green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016dhxM43dnWvUu2B5Ph1iG1

Standalone/Research/WorldAssemblerM3/Linux71/REPORT.md

@@ -0,0 +1,28 @@
+# Linux71 — SurfaceWorld generation timing
+
+## Like-for-like result
+
+All runs used the same host and CPU affinity. Unreal and headless resolved plan hash `bcc0d0d33ea39435` and the exact same 2,850 tile keys at LOD0–8. The live asset trace still prints clip 6/max 6; the captured tile keys are identical to headless radius 3/max 8.
+
+| Run | Tile workers | Hashing | Sum of Core/worker time |
+|---|---:|---:|---:|
+| L70 Unreal median | 8 slots | on | Core 42.69s; worker 42.91s |
+| L70 headless | 12 | off | worker 20.06s |
+| L71 Unreal median | 8 slots | off | Core 14.42s; worker 14.62s |
+| L71 headless | 8 | off | worker 14.26s |
+
+At matched W8 the Core sum differs by 1.2% from headless end-to-end worker time; Unreal worker time differs by 2.5%. The same-worker L71 per-LOD comparison is within about ±11%. Unreal’s initial-set-ready milestone is 4.71–4.83s; the 53s commandlet lifetime includes the trace’s `UntilEndPlay` sampling after all tiles finished. Headless whole-view readiness was 2.16s.
+
+## Cause and tail
+
+The L70 timing trace enabled `bCaptureVoxelForgeCoreHashesForValidation`, which set `TileGenerationOptions.hashDensity`. That diagnostic disables the native heightfield-band shortcut and whole-tile early skips. A direct W1 headless A/B on the same keys went from 9.00s/hash-off to 28.46s/hash-on: field calls rose from 5.46M to 33.82M and 654 whole-tile skips fell to zero. Turning hashing off in the Unreal trace reduced its Core sum 66%, from 42.69s to 14.42s. The no-hash profile shows LOD0 marching-cubes density-grid work as the largest near-field cost; LOD8 also spends time in MC, ground refinement and caps.
+
+The L71 Core p50/p95/max is 1.78/17.56/130.52ms; worker p50/p95 is 1.80/17.97ms. LOD0 p50 is 0.50ms and p95 34.8–39.1ms; LOD1 is 0.60–0.64ms and 10.3–10.7ms. Most LOD0/1 tiles are empty (441/512 and 372/448); the nonempty passage tiles have p50/p95 of 18.8/105.1ms at LOD0 and 8.6/65.0ms at LOD1. The slow keys cluster around the spawn mouth and first passage bend. This is real per-tile generation variance. Queue p95 is separately 129.45ms, so it increases arrival latency but is not counted in `core_generate_s`.
+
+## Changes and verification
+
+The stream-trace commandlet now leaves hash capture off; dedicated hash/parity tests still enable it. This changes no world data or hashes. No shared Core or build-setting change was warranted by the matched result. No Windows build-flag change is indicated by this Linux run. The only `Source/` change is `Source/VoxelForgeEditor/Private/VoxelForgeExploreCommandlet.cpp`; Linux VFHostEditor Development UBT succeeded with `-MaxParallelActions=4` (log: `Saved/Linux/l71/ubt-source-change.log`). No Unreal build ran during timing.
+
+Historical fixtures and controls used directly or indirectly by verify tiers now live under `Standalone/Tools/verify/baselines/`; `verify.py` and its helper scripts point there. Gate thresholds and outcomes are unchanged. The Clang fast tier passed in 267.2s (log: `Saved/Linux/l71/verify-fast-tier.log`).
+
+`perf` and stack-sampling tools were unavailable, so stage counters and per-tile traces supplied the profile. Initial UE launches used an output path outside the host plugin’s Saved tree and stopped before measurement; corrected launches all passed. The first UBT attempt could not open its default user log; retrying with an explicit `-Log` succeeded. The first fast invocation lacked its output binary; after the Clang build, the rerun passed. No commit, push or stash was made.

Standalone/Research/WorldAssemblerM3/Linux71/RunEngineSurfaceTracesL71.py

@@ -0,0 +1,107 @@
+#!/usr/bin/env python3
+"""Measure production-path SurfaceWorld tile work without validation hashing."""
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+import time
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[4]
+OUT = ROOT / "Saved/Linux/l71"
+ENGINE_OUT = OUT / "engine"
+HOST_PLUGIN_SAVED = Path("/home/jahni/VFHost/Plugins/VoxelForge/Saved/Linux/l71")
+PROJECT = Path("/home/jahni/VFHost/VFHost.uproject")
+EDITOR = Path("/home/jahni/UnrealEngine/Engine/Binaries/Linux/UnrealEditor-Cmd")
+SPAWN_M = (8.0, 16.0, -372.2626953125)
+
+
+def run(trial: int) -> dict:
+    label = f"strate0-trial{trial}"
+    traces = ENGINE_OUT / "traces"
+    trace = traces / f"{label}.trace.jsonl"
+    abslog = traces / f"{label}.log"
+    stdout = traces / f"{label}.stdout.log"
+    # VoxelForgeExplore restricts exported content to the host plugin's Saved tree.
+    output = HOST_PLUGIN_SAVED / "output" / label
+    userdir = ENGINE_OUT / "user" / label
+    zendata = ENGINE_OUT / "zen" / label
+    ddc = ENGINE_OUT / "DDC"
+    for directory in (traces, output, userdir, zendata, ddc):
+        directory.mkdir(parents=True, exist_ok=True)
+
+    spawn_cm = ",".join(f"{value * 100.0:.3f}" for value in SPAWN_M)
+    command = [
+        str(EDITOR), str(PROJECT), "-nullrhi", "-unattended", "-nop4", "-nosplash",
+        "-NoSound", "-corelimit=12", "-NoZenAutoLaunch", "-notraceserver",
+        "-nocrashreports", "-DisablePlugins=AndroidFileServer", "-ddc=NoZenLocalFallback",
+        f"-userdir={userdir}", f"-ZenDataPath={zendata}",
+        f"-LocalDataCachePath={ddc}", "-run=VoxelForgeExplore", "-seed=0",
+        "-archetype=TunnelNetwork", "-slot=0", "-modes=export", "-gameconfig=1",
+        "-coreworldtest=1", "-ownerfourstrateplan=1", "-corevoxelsizecm=25",
+        "-coreislandtest=0", "-corefallbacktest=0", "-coreofftest=0",
+        "-coreplantest=0", "-settingsref=/Game/VoxelForge/DA_Settings.DA_Settings",
+        "-corehybridmesher=0", f"-out={output}", "-voxel.StreamingCoarseFirst=0",
+        "-voxel.StreamingDynamicNearReserve=1", "-voxel.StreamingLegacyPendingResultCap=0",
+        "-voxel.CoreRegionConcurrentAdmission=1", "-voxel.StreamingResultBacklogCap=256",
+        "-voxel.StreamingReadyQueueDepth=8", "-voxel.StreamingBackgroundWorkerCap=1",
+        "-voxel.CoreColumnCacheDirectMapped=0", "-voxel.CoreMCGradientMemoization=1",
+        "-voxel.StreamingCollisionLookahead=1", "-coreworldstreamtrace=1", "-voxel.TestMove=0",
+        "-voxel.TestMoveSpeedCmPerSecond=0", f"-voxel.StartupTraceSpawnCm={spawn_cm}",
+        f"-voxel.StartupTraceFile={trace}", "-voxel.StartupTraceUntilEndPlay=1",
+        "-voxel.StartupTraceMaxTiles=16384", "-voxel.CollisionGate=0", f"-abslog={abslog}",
+    ]
+    started = time.monotonic()
+    with stdout.open("w", encoding="utf-8") as stream:
+        # UE itself is CPU-pinned and time-bounded. Linux does not enforce --rss.
+        result = subprocess.run(
+            ["taskset", "-c", "0-11", "timeout", "1740s", *command],
+            cwd=ROOT, stdout=stream, stderr=subprocess.STDOUT, check=False, timeout=1740,
+        )
+    elapsed = time.monotonic() - started
+    events = set()
+    if trace.is_file():
+        for line in trace.read_text(encoding="utf-8", errors="replace").splitlines():
+            try:
+                events.add(json.loads(line).get("type", ""))
+            except json.JSONDecodeError:
+                continue
+    row = {
+        "label": label,
+        "trial": trial,
+        "spawn_m": SPAWN_M,
+        "elapsed_s": round(elapsed, 3),
+        "exit_code": result.returncode,
+        "trace": str(trace),
+        "stdout": str(stdout),
+        "events": sorted(events),
+        "has_walkable_gate": "walkable_gate" in events,
+        "has_lod0_ring": "lod0_ring_ready" in events,
+        "has_initial_set": "stream_trace_milestone" in events,
+        "has_summary": "summary" in events,
+    }
+    print(f"[L71EngineTrace] {json.dumps(row, sort_keys=True)}", flush=True)
+    return row
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--trials", type=int, default=3, choices=range(1, 4))
+    args = parser.parse_args()
+    if not EDITOR.is_file() or not PROJECT.is_file():
+        raise SystemExit(f"missing UnrealEditor-Cmd or host project: {EDITOR} {PROJECT}")
+    rows = [run(trial) for trial in range(1, args.trials + 1)]
+    summary = ENGINE_OUT / "surface-trace-summary.json"
+    summary.write_text(json.dumps(rows, indent=2) + "\n", encoding="utf-8")
+    complete = all(row["exit_code"] == 0 and row["has_walkable_gate"]
+                   and row["has_lod0_ring"] and row["has_initial_set"] and row["has_summary"]
+                   for row in rows)
+    print(f"[L71EngineTraceSet] trials={len(rows)} status={'PASS' if complete else 'INCOMPLETE'} "
+          f"summary={summary}", flush=True)
+    return 0 if complete else 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())