diff --git a/Standalone/Research/WorldAssemblerM3/Linux167/REPORT.md b/Standalone/Research/WorldAssemblerM3/Linux167/REPORT.md new file mode 100644 index 0000000..aefdaaf --- /dev/null +++ b/Standalone/Research/WorldAssemblerM3/Linux167/REPORT.md @@ -0,0 +1,39 @@ +# L167 — Re-test L157 and L160 on the faster base + +**Result:** No candidate meets the full default bar. Leave L157 and L160 disabled. L160 is tick-safe in this W8 run but lowers all-around near readiness and delays arrival. L157 improves readiness but still spikes; combining them improves readiness most, but worsens worst ticks and delays arrival. + +## Measurement + +Ran base, L157, L160, and both in seven interleaved repetitions each at 30 m/s. Each measured run covered a 12-second move and 5-second dwell, with eight effective tile slots and the four-strata owner plan. One base warm-up was excluded. Timing began with 1/5/15-minute loads at 0.77/0.96/1.46 and no other build or editor process running. Values below are medians across seven runs; p99 range and maximum worst tick show the full run spread. + +The common base had `StreamingIncrementalDesiredSet=1` with audits off, matching the faster L165/L166 path. The checked-in CVar default remains off; this was a controlled runtime setting for the comparison. Null-RHI provided no camera-forward samples, so the 55-degree forward cone uses travel direction. + +| Condition | Tick p99 median (run range), ms | Worst tick median / run max, ms | Near readiness, % | Forward cone, % | Far holes, % | LOD0–2 arrival, s | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| Base | 7.289 (6.786–7.513) | 12.061 / 12.465 | 43.39 | 15.77 | 3.00 | 2.230 | +| L157 | 8.582 (8.046–12.462) | 12.462 / 16.689 | 55.44 | 18.61 | 2.64 | 2.469 | +| L160 | 7.187 (6.962–8.009) | 11.747 / 12.503 | 38.03 | 20.32 | 2.59 | 2.447 | +| Both | 8.932 (8.064–10.973) | 12.594 / 14.278 | 45.55 | 25.24 | 2.85 | 2.461 | + +Near readiness, cone readiness, and far holes improved for L157 and the combination; L160 lowered all-around near readiness. All three candidates delayed median arrival by 0.216–0.239 seconds. L157 had one run above the 11 ms p99 limit and a 16.689 ms worst tick. The combination stayed within the p99 limit in all seven runs, but its median worst tick was 0.533 ms worse than base and one run reached 14.278 ms. L160 passed the tick limits but failed readiness and arrival gates. Therefore none qualifies as default. + +## Spike count and cause + +Trace data attributes the L157 spikes to main-thread desired-set and scheduler work, not the 1 ms apply budget: + +- At the 16.689 ms worst tick, `UpdateChunks` took 15.500 ms. `SubmitTiles` took 8.807 ms, including 8.336 ms in level sorting; desired rebuild took 5.353 ms. Apply took only 0.374 ms. The scheduler trace counted 912 reserve-declined tiles and three near tasks submitted. +- A second 16.198 ms tick had 13.852 ms in `UpdateChunks` and an 11.777 ms desired rebuild, including 6.679 ms of overlay materialization. `SubmitTiles` took 0.813 ms and apply took 1.087 ms. +- The combined run's 14.278 ms maximum had 10.279 ms in `UpdateChunks`, 2.859 ms draining pending results, and one applied result taking 2.799 ms against the 1 ms budget (`time_limit_reached=1`). Its level sort was only 0.095 ms. The budget is checked after a result completes, so it cannot bound one expensive apply. + +These traces show more than one source: level sorting under L157 admission, expensive desired-set rebuild/materialization, and individual result-apply cost in the combined mode. No tuning change was attempted because the measured work does not support a single cap adjustment that would address all three; all candidates also fail another promotion gate. + +## Checks and limits + +- VFHost Linux Development build succeeded with `-NoUBA -MaxParallelActions=4`; the temporary VFHost plugin source link was restored afterward. +- `verify.py fast` passed in 298.0 seconds: 268 pass, 0 fail, 5 diagnostics. Its 84 binary runs had 82 pass and 2 expected failures. Geometry checks and Clang/GCC W1/W8/W12 determinism passed. The L167 source stamp matched the L166 verifier binary (`ba79279c16018d2996e724df0f7c0e8a210929baebc1aa14bef6a76fbda96faa`). +- Movement timings used W8, the effective slot count read back by this VFHost command. W12 movement was not run; the command's 12-core affinity exposed eight TaskGraph background workers and eight effective streaming slots. Fast-tier W12 determinism did run. The live movement `TileHashDump` hook was not enabled; no geometry-generation source changed. +- Runner `py_compile` and `git diff --check` passed on the final files. No Unreal `Source/` files changed. + +Changed Unreal `Source/` files: none. + +Artifacts are under `Saved/Linux/l167/`; the interleaved movement runner is `RunL167Movement.py` alongside this report. diff --git a/Standalone/Research/WorldAssemblerM3/Linux167/RunL167Movement.py b/Standalone/Research/WorldAssemblerM3/Linux167/RunL167Movement.py new file mode 100644 index 0000000..ef5e705 --- /dev/null +++ b/Standalone/Research/WorldAssemblerM3/Linux167/RunL167Movement.py @@ -0,0 +1,639 @@ +#!/usr/bin/env python3 +"""Interleaved L167 UE movement runs for the L157/L160 levers on the L165/L166 base.""" +from __future__ import annotations + +import argparse +import json +import math +import statistics +import subprocess +import time +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[4] +HOST = Path("/home/jahni/VFHost") +EDITOR = Path("/home/jahni/UnrealEngine/Engine/Binaries/Linux/UnrealEditor-Cmd") +OUT = ROOT / "Saved/Linux/l167/ue/movement" +HOST_SAVED = HOST / "Saved/Linux/l167" +HOST_PLUGIN_SAVED = HOST / "Plugins/VoxelForge/Saved/Linux/l167" +SPAWN_M = (19.166015625, -14.2001953125, -753.3740234375) +MOVE_START_S = 6.0 + +_COMMON = { + # Keep the faster L165/L166 path active for every condition; omit its audit + # during timing so the audit itself cannot create a frame spike. + "incremental_desired": 1, "audit_every": 0, + "materialization_audit_every": 0, "inject_mismatch": 0, +} +CONDITIONS = { + "base": { + **_COMMON, "wait_for_critical": 1, "apply_budget_ms": 0.0, + "forward_first": 0, "rebuild_on_region_completion": 1, + "core_desired_cache": 0, + }, + "l157": { + **_COMMON, "wait_for_critical": 0, "apply_budget_ms": 1.0, + "forward_first": 0, "rebuild_on_region_completion": 1, + "core_desired_cache": 0, + }, + "l160": { + **_COMMON, "wait_for_critical": 1, "apply_budget_ms": 0.0, + "forward_first": 1, "rebuild_on_region_completion": 0, + "core_desired_cache": 1, + }, + "both": { + **_COMMON, "wait_for_critical": 0, "apply_budget_ms": 1.0, + "forward_first": 1, "rebuild_on_region_completion": 0, + "core_desired_cache": 1, + }, +} +ORDER = { + 1: ("base", "l157", "l160", "both"), + 2: ("l157", "l160", "both", "base"), + 3: ("l160", "both", "base", "l157"), + 4: ("both", "base", "l157", "l160"), + 5: ("l160", "l157", "base", "both"), + 6: ("both", "l160", "l157", "base"), + 7: ("base", "both", "l160", "l157"), +} + + +def percentile(values: list[float], fraction: float = 0.99) -> float | None: + if not values: + return None + ordered = sorted(values) + return ordered[max(0, min(len(ordered) - 1, math.ceil(len(ordered) * fraction) - 1))] + + +def read_events(path: Path) -> list[dict]: + rows = [] + if path.is_file(): + for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + if not line.startswith("{"): + continue + try: + rows.append(json.loads(line)) + except json.JSONDecodeError: + continue + return rows + + +def mean(values: list[float]) -> float | None: + return statistics.mean(values) if values else None + + +def desired_breakdown(events: list[dict], move_start_s: float, stop_s: float) -> dict: + rows = [event for event in events + if event.get("type") == "desired_rebuild_breakdown" + and move_start_s <= float(event.get("t_s", 0.0)) < stop_s] + keys = ("total_ms", "core_prepare_ms", "core_delta_ms", "fallback_ms", + "overlay_materialize_ms", "priority_sort_ms", "critical_order_ms") + result = {"rebuild_count": len(rows)} + result["incremental_materialization_update_count"] = sum( + int(event.get("materialization_update", 0)) != 0 for event in rows) + result["priority_order_reuse_count"] = sum( + int(event.get("priority_order_reused", 0)) != 0 for event in rows) + result["priority_order_update_count"] = sum( + int(event.get("priority_order_updated", 0)) != 0 for event in rows) + result["incremental_materialization_update_share_pct"] = percent( + result["incremental_materialization_update_count"], len(rows)) + result["priority_order_reuse_share_pct"] = percent( + result["priority_order_reuse_count"], len(rows)) + result["priority_order_update_share_pct"] = percent( + result["priority_order_update_count"], len(rows)) + for key in keys: + values = [float(event.get(key, 0.0)) for event in rows] + result[f"{key}_p50"] = statistics.median(values) if values else None + result[f"{key}_p95"] = percentile(values, 0.95) + result[f"{key}_max"] = max(values) if values else None + fallback_rows = [event for event in events + if event.get("type") == "core_desired_cache" + and move_start_s <= float(event.get("t_s", 0.0)) < stop_s + and int(event.get("full_fallback", 0)) != 0] + result["fallback_count"] = len(fallback_rows) + result["lod4plus_fallback_count"] = sum( + int(event.get("fallback_reason", 0)) == 1 for event in fallback_rows) + result["other_fallback_count"] = sum( + int(event.get("fallback_reason", 0)) == 2 for event in fallback_rows) + result["fallback_cost_sum_ms"] = sum( + float(event.get("fallback_ms", 0.0)) for event in fallback_rows) + result["fallback_cost_max_ms"] = max( + (float(event.get("fallback_ms", 0.0)) for event in fallback_rows), default=None) + return result + + +def percent(numerator: int | float, denominator: int | float) -> float | None: + return 100.0 * numerator / denominator if denominator > 0 else None + + +def run(condition: str, repetition: int, tile_hash_dump: Path | None = None, + expected_mismatch: bool = False) -> dict: + config = CONDITIONS[condition] + run_name = f"fast-r{repetition}" + out = OUT / condition + host_out = HOST_PLUGIN_SAVED / condition / "output" + userdir = HOST_SAVED / "user" / condition / run_name + zendata = HOST_SAVED / "zen" / condition / run_name + ddc = HOST_SAVED / "DDC" + trace = out / f"{run_name}.trace.jsonl" + abslog = out / f"{run_name}.log" + stdout = out / f"{run_name}.stdout.log" + export = host_out / run_name + for directory in (out, export, userdir, zendata, ddc): + directory.mkdir(parents=True, exist_ok=True) + trace.unlink(missing_ok=True) + + spawn_cm = ",".join(f"{value * 100.0:.3f}" for value in SPAWN_M) + command = [ + str(EDITOR), str(HOST / "VFHost.uproject"), "-nullrhi", "-unattended", + "-nop4", "-nosplash", "-NoSound", "-corelimit=12", "-NoZenAutoLaunch", + "-notraceserver", "-nocrashreports", "-DisablePlugins=AndroidFileServer", + "-ddc=NoZenLocalFallback", f"-userdir={userdir}", f"-ZenDataPath={zendata}", + f"-LocalDataCachePath={ddc}", "-run=VoxelForgeExplore", "-seed=0", + "-archetype=TunnelNetwork", "-slot=0", "-modes=export", "-gameconfig=1", + "-coreworldtest=1", "-ownerfourstrateplan=1", "-corevoxelsizecm=25", + "-coreislandtest=0", "-corefallbacktest=0", "-coreofftest=0", + "-coreplantest=0", "-settingsref=/Game/VoxelForge/DA_Settings.DA_Settings", + "-corehybridmesher=0", f"-out={export}", "-voxel.StreamingCoarseFirst=1", + "-voxel.StreamingDynamicNearReserve=1", + "-voxel.StreamingLegacyPendingResultCap=0", + "-voxel.CoreRegionConcurrentAdmission=1", + "-voxel.StreamingResultBacklogCap=256", "-voxel.StreamingReadyQueueDepth=8", + "-voxel.StreamingBackgroundWorkerCap=1", "-voxel.CoreColumnCacheDirectMapped=0", + "-voxel.StreamingFarApplySupplement=1", + f"-voxel.StreamingFastTravelWaitForAllCriticalTiles={config['wait_for_critical']}", + f"-voxel.StreamingFastTravelApplyBudgetMilliseconds={config['apply_budget_ms']:.1f}", + f"-voxel.StreamingFastTravelForwardFirst={config['forward_first']}", + f"-voxel.StreamingRebuildOnCoreRegionCompletion={config['rebuild_on_region_completion']}", + f"-voxel.StreamingCoreDesiredCacheByCenter={config['core_desired_cache']}", + f"-voxel.StreamingIncrementalDesiredSet={config['incremental_desired']}", + f"-voxel.StreamingIncrementalDesiredAuditEvery={config['audit_every']}", + f"-voxel.StreamingIncrementalDesiredMaterializationAuditEvery={config['materialization_audit_every']}", + f"-voxel.StreamingIncrementalDesiredAuditInjectMismatch={config['inject_mismatch']}", + "-voxel.CoreMCGradientMemoization=1", "-voxel.StreamingCollisionLookahead=1", + "-voxel.TestMove=1", "-voxel.TestMoveSpeedCmPerSecond=3000.0", + f"-voxel.TestMoveStartSeconds={MOVE_START_S:.1f}", + "-voxel.TestMoveDurationSeconds=12.0", "-voxel.TestExitSeconds=0.0", + "-voxel.TestMoveDirection=1,0,0", "-voxel.StreamingSchedulerPolicy=3", + "-voxel.TestExitOnSteady=0", "-coreworldstreamtrace=1", + f"-voxel.StartupTraceSpawnCm={spawn_cm}", + "-voxel.StartupTraceUntilEndPlay=1", "-voxel.StartupTraceThroughCrossing=1", + "-voxel.StartupTraceMaxTiles=16384", f"-voxel.StartupTraceFile={trace}", + f"-abslog={abslog}", + ] + if tile_hash_dump is not None: + tile_hash_dump.parent.mkdir(parents=True, exist_ok=True) + tile_hash_dump.unlink(missing_ok=True) + command.append(f"-voxel.TileHashDump={tile_hash_dump.resolve()}") + wrapped = ["taskset", "-c", "0-11", "timeout", "1740s", *command] + started = time.monotonic() + with stdout.open("w", encoding="utf-8") as stream: + process = subprocess.run( + wrapped, cwd=ROOT, stdout=stream, stderr=subprocess.STDOUT, + check=False, timeout=1740) + + events = read_events(trace) + if expected_mismatch: + diagnostic = "VoxelForgeIncrementalDesiredAudit" + logs = "\n".join(path.read_text(encoding="utf-8", errors="replace") + for path in (abslog, stdout) if path.is_file()) + audit_events = [event for event in events + if event.get("type") in {"incremental_desired_audit", + "incremental_desired_order_audit"}] + failed_audit = any(int(event.get("pass", 1)) == 0 for event in audit_events) + failed_audit = failed_audit or ( + diagnostic in logs and "mismatch" in logs.lower()) + if process.returncode == 0 or diagnostic not in logs or "mismatch" not in logs.lower(): + raise RuntimeError( + f"injected mismatch did not fail as expected: exit={process.returncode}; " + f"audit_events={len(audit_events)}; log={abslog}") + print("[L166MismatchControl] " + json.dumps({ + "exit_code": process.returncode, + "diagnostic_seen": True, + "failed_audit_event_seen": failed_audit, + "trace": str(trace.relative_to(ROOT)), + "log": str(abslog.relative_to(ROOT)), + }, sort_keys=True), flush=True) + return {"condition": condition, "exit_code": process.returncode, + "failed_audit_event_seen": failed_audit} + samples = [event for event in events if event.get("type") == "streaming_sample"] + moving = [event for event in samples if float(event.get("speed_cm_s", 0.0)) >= 50.0] + stop_event = next((event for event in events + if event.get("type") == "streaming_test_move_stopped"), None) + backend = next((event for event in events if event.get("type") == "worker_backend_config"), {}) + scheduler_config = next((event for event in events + if event.get("type") == "scheduler_admission"), {}) + worker_cap = next((event for event in events + if event.get("type") == "streaming_worker_cap_readback"), {}) + if stop_event is None: + raise RuntimeError(f"missing move-stop event: {trace}") + stop_s = float(stop_event.get("t_s", 0.0)) + move_start_s = min((float(event.get("t_s", 0.0)) for event in moving), default=float("inf")) + dwell = [event for event in samples if float(event.get("t_s", 0.0)) >= stop_s + and float(event.get("speed_cm_s", 0.0)) < 50.0] + arrival = [event for event in samples if float(event.get("t_s", 0.0)) < move_start_s + and float(event.get("speed_cm_s", 0.0)) < 50.0] + desired_stage_breakdown = desired_breakdown(events, move_start_s, stop_s) + if process.returncode or len(moving) < 40 or len(dwell) < 16: + raise RuntimeError(f"incomplete {condition} run: {trace}; exit={process.returncode}; " + f"move={len(moving)} dwell={len(dwell)}") + if (int(backend.get("streaming_fast_travel_wait_for_all_critical_tiles", -1)) + != config["wait_for_critical"] + or abs(float(backend.get("streaming_fast_travel_apply_budget_ms", -1.0)) + - config["apply_budget_ms"]) > 0.01 + or int(worker_cap.get("streaming_background_worker_cap", -1)) != 1 + or int(backend.get("effective_tile_slots", -1)) != 8 + or int(backend.get("streaming_far_apply_supplement", -1)) != 1 + or int(scheduler_config.get("forward_first", -1)) != config["forward_first"] + or int(scheduler_config.get("rebuild_on_core_region_completion", -1)) + != config["rebuild_on_region_completion"] + or int(scheduler_config.get("core_desired_cache_by_center", -1)) + != config["core_desired_cache"] + or int(scheduler_config.get("incremental_desired_set", -1)) + != config["incremental_desired"] + or int(scheduler_config.get("incremental_desired_audit_every", -1)) + != config["audit_every"] + or int(scheduler_config.get("incremental_desired_materialization_audit_every", -1)) + != config["materialization_audit_every"] + or int(scheduler_config.get("incremental_desired_audit_inject_mismatch", -1)) + != config["inject_mismatch"]): + raise RuntimeError(f"CVar/worker readback mismatch: {backend}; {worker_cap}; {scheduler_config}") + + moving_metrics = [] + for event in moving: + desired = event.get("desired_by_lod", [0] * 9) + resident = event.get("resident_by_lod", [0] * 9) + near_desired = sum(desired[:3]) + cone_desired = int(event.get("near_forward_cone_desired", 0)) + view_cone_desired = int(event.get("near_view_forward_cone_desired", 0)) + view_valid = bool(event.get("near_view_forward_cone_valid", 0)) + weighted_desired = float(event.get("near_forward_weighted_desired", 0.0)) + travel_cone_ready = percent( + int(event.get("near_forward_cone_resident", 0)), cone_desired) + view_cone_ready = percent( + int(event.get("near_view_forward_cone_resident", 0)), view_cone_desired) + moving_metrics.append({ + "near_ready_pct": percent(sum(resident[:3]), near_desired) or 0.0, + "lod0_ready_pct": percent(resident[0], desired[0]) or 0.0, + "lod1_ready_pct": percent(resident[1], desired[1]) or 0.0, + "lod2_ready_pct": percent(resident[2], desired[2]) or 0.0, + "far_hole_pct": percent( + sum(desired[3:]) - sum(event.get("covered_by_lod", resident)[3:]), + sum(desired[3:])) or 0.0, + "ahead_share_pct": percent(int(event.get("near_ahead_desired", 0)), near_desired), + "behind_share_pct": percent(int(event.get("near_behind_desired", 0)), near_desired), + "side_share_pct": 100.0 - ( + percent(int(event.get("near_ahead_desired", 0)), near_desired) or 0.0 + ) - ( + percent(int(event.get("near_behind_desired", 0)), near_desired) or 0.0 + ), + "ahead_ready_pct": percent( + int(event.get("near_ahead_resident", 0)), + int(event.get("near_ahead_desired", 0))), + "behind_ready_pct": percent( + int(event.get("near_behind_resident", 0)), + int(event.get("near_behind_desired", 0))), + "travel_forward_cone_ready_pct": travel_cone_ready, + "view_forward_cone_ready_pct": view_cone_ready, + "forward_cone_ready_pct": view_cone_ready if view_valid else travel_cone_ready, + "forward_cone_source": "camera" if view_valid else "travel_fallback", + "forward_cone_share_pct": percent(cone_desired, near_desired), + "forward_weighted_ready_pct": percent( + float(event.get("near_forward_weighted_resident", 0.0)), weighted_desired), + }) + + arrival_near_settled = next((event for event in arrival + if sum(event.get("desired_by_lod", [0] * 9)[:3]) > 0 + and sum(event.get("settled_by_lod", [0] * 9)[:3]) + >= sum(event.get("desired_by_lod", [0] * 9)[:3])), None) + dwell_complete = next((event for event in dwell + if sum(event.get("resident_by_lod", [0] * 9)[:3]) + >= sum(event.get("desired_by_lod", [0] * 9)[:3]) + and sum(event.get("desired_by_lod", [0] * 9)[3:]) + <= sum(event.get("covered_by_lod", [0] * 9)[3:])), None) + + work = [event for event in events if event.get("type") == "game_thread_work_frame" + and move_start_s <= float(event.get("t_s", 0.0)) < stop_s] + applies = {event.get("frame"): event for event in events + if event.get("type") == "apply_frame" + and move_start_s <= float(event.get("t_s", 0.0)) < stop_s} + scheduler = {event.get("frame"): event for event in events + if event.get("type") == "scheduler_admission" + and move_start_s <= float(event.get("t_s", 0.0)) < stop_s} + frames = [] + for event in work: + apply = applies.get(event.get("frame"), {}) + admit = scheduler.get(event.get("frame"), {}) + frames.append({ + "frame": event.get("frame"), + "tick_ms": float(event.get("tick_ms", 0.0)), + "update_chunks_ms": float(event.get("update_chunks_ms", 0.0)), + "pending_drain_ms": float(event.get("pending_drain_ms", 0.0)), + "apply_ms": float(apply.get("apply_ms", 0.0)), + "apply_budget_ms": float(apply.get("apply_budget_ms", 0.0)), + "meshes_applied": int(apply.get("meshes_applied", 0)), + "time_budget_hit": int(apply.get("time_limit_reached", 0)), + "desired_rebuild_ms": float(admit.get("desired_rebuild_ms", 0.0)), + "desired_rebuilt": int(admit.get("desired_rebuilt", 0)), + "submit_tiles_ms": float(admit.get("submit_tiles_ms", 0.0)), + "level_admit_ms": float(admit.get("level_admit_ms", 0.0)), + }) + p99_field = lambda key: percentile([float(row[key]) for row in frames]) + max_field = lambda key: max((float(row[key]) for row in frames), default=None) + rebuild_frames = [row for row in frames if row["desired_rebuilt"]] + rebuild_costs = [float(row["desired_rebuild_ms"]) for row in rebuild_frames] + moving_scheduler = [event for event in scheduler.values()] + center_crossings = 0 + previous_center = None + for event in sorted(moving_scheduler, key=lambda item: int(item.get("frame", 0))): + center = tuple(event.get("center", ())) + if previous_center is not None and center != previous_center: + center_crossings += 1 + previous_center = center + core_requests = [event for event in events + if event.get("type") == "core_region_compile_request" + and move_start_s <= float(event.get("t_s", 0.0)) < stop_s] + same_center_core_requests = sum( + event.get("reason") == "desired_set_changed_same_center" for event in core_requests) + core_desired_queries = [event for event in events + if event.get("type") == "core_desired_cache" + and move_start_s <= float(event.get("t_s", 0.0)) < stop_s] + core_desired_query_ms = [float(event.get("query_ms", 0.0)) + for event in core_desired_queries if float(event.get("query_ms", 0.0)) > 0.0] + core_desired_cache_hits = sum(int(event.get("hit", 0)) for event in core_desired_queries) + incremental_audits = [event for event in events + if event.get("type") == "incremental_desired_audit" + and move_start_s <= float(event.get("t_s", 0.0)) < stop_s] + incremental_audit_failures = sum(int(event.get("pass", 1)) == 0 + for event in incremental_audits) + order_audits = [event for event in events + if event.get("type") == "incremental_desired_order_audit" + and move_start_s <= float(event.get("t_s", 0.0)) < stop_s] + order_audit_failures = sum(int(event.get("pass", 1)) == 0 for event in order_audits) + materialization_audits = [event for event in incremental_audits + if int(event.get("materialization_audit", 0)) != 0] + if config["audit_every"] > 0 and ( + not incremental_audits or incremental_audit_failures + or not order_audits or order_audit_failures + or (config["materialization_audit_every"] > 0 + and not materialization_audits)): + raise RuntimeError( + f"incremental desired equality audit missing/failed: {trace}; " + f"set={len(incremental_audits)}/{incremental_audit_failures} " + f"order={len(order_audits)}/{order_audit_failures} " + f"materialization={len(materialization_audits)}") + direction_submissions = [event.get("near_submitted_by_direction", [0, 0, 0, 0]) + for event in moving_scheduler] + direction_submissions = [counts for counts in direction_submissions + if isinstance(counts, list) and len(counts) == 4] + direction_submission_totals = [ + sum(int(counts[index]) for counts in direction_submissions) for index in range(4) + ] + first_admission_order = next((event.get("near_admission_order_first", []) + for event in moving_scheduler + if event.get("near_admission_order_first")), []) + camera_samples = sum(bool(event.get("near_view_forward_cone_valid", 0)) for event in moving) + dwell_complete_s = (float(dwell_complete.get("t_s", 0.0)) - stop_s + if dwell_complete else None) + + row = { + "condition": condition, + "repetition": repetition, + "speed_mps": 30.0, + "wait_for_all_critical_tiles": config["wait_for_critical"], + "fast_travel_apply_budget_ms": config["apply_budget_ms"], + "forward_first": config["forward_first"], + "rebuild_on_core_region_completion": config["rebuild_on_region_completion"], + "core_desired_cache_by_center": config["core_desired_cache"], + "incremental_desired_set": config["incremental_desired"], + "incremental_desired_audit_every": config["audit_every"], + "incremental_desired_materialization_audit_every": config["materialization_audit_every"], + "incremental_desired_audit_count": len(incremental_audits), + "incremental_desired_audit_failures": incremental_audit_failures, + "incremental_desired_order_audit_count": len(order_audits), + "incremental_desired_order_audit_failures": order_audit_failures, + "incremental_desired_materialization_audit_count": len(materialization_audits), + "effective_tile_slots": backend.get("effective_tile_slots"), + "exit_code": process.returncode, + "wall_s": round(time.monotonic() - started, 3), + "moving_samples": len(moving), + "dwell_samples": len(dwell), + "arrival_samples": len(arrival), + "near_ready_moving_mean_pct": mean([m["near_ready_pct"] for m in moving_metrics]), + "lod0_ready_moving_mean_pct": mean([m["lod0_ready_pct"] for m in moving_metrics]), + "lod1_ready_moving_mean_pct": mean([m["lod1_ready_pct"] for m in moving_metrics]), + "lod2_ready_moving_mean_pct": mean([m["lod2_ready_pct"] for m in moving_metrics]), + "far_holes_moving_mean_pct": mean([m["far_hole_pct"] for m in moving_metrics]), + "near_ahead_desired_share_pct": mean([m["ahead_share_pct"] for m in moving_metrics + if m["ahead_share_pct"] is not None]), + "near_behind_desired_share_pct": mean([m["behind_share_pct"] for m in moving_metrics + if m["behind_share_pct"] is not None]), + "near_side_desired_share_pct": mean([m["side_share_pct"] for m in moving_metrics]), + "near_ahead_ready_pct": mean([m["ahead_ready_pct"] for m in moving_metrics + if m["ahead_ready_pct"] is not None]), + "near_behind_ready_pct": mean([m["behind_ready_pct"] for m in moving_metrics + if m["behind_ready_pct"] is not None]), + "near_forward_cone_half_angle_deg": 55, + "near_forward_cone_desired_share_pct": mean([m["forward_cone_share_pct"] + for m in moving_metrics + if m["forward_cone_share_pct"] is not None]), + "near_forward_cone_ready_pct": mean([m["forward_cone_ready_pct"] + for m in moving_metrics + if m["forward_cone_ready_pct"] is not None]), + "near_travel_forward_cone_ready_pct": mean( + [m["travel_forward_cone_ready_pct"] for m in moving_metrics + if m["travel_forward_cone_ready_pct"] is not None]), + "near_view_forward_cone_ready_pct": mean( + [m["view_forward_cone_ready_pct"] for m in moving_metrics + if m["view_forward_cone_ready_pct"] is not None]), + "camera_forward_samples": camera_samples, + "camera_forward_sample_share_pct": percent(camera_samples, len(moving)), + "near_forward_cosine_weighted_ready_pct": mean( + [m["forward_weighted_ready_pct"] for m in moving_metrics + if m["forward_weighted_ready_pct"] is not None]), + "arrival_lod02_settled_t_s": ( + float(arrival_near_settled.get("t_s", 0.0)) if arrival_near_settled else None), + "near_ready_dwell_end_pct": (100.0 * sum(dwell[-1].get("resident_by_lod", [0] * 9)[:3]) + / max(1, sum(dwell[-1].get("desired_by_lod", [0] * 9)[:3]))), + "far_holes_dwell_end_pct": (100.0 * ( + sum(dwell[-1].get("desired_by_lod", [0] * 9)[3:]) + - sum(dwell[-1].get("covered_by_lod", [0] * 9)[3:])) + / max(1, sum(dwell[-1].get("desired_by_lod", [0] * 9)[3:]))), + "dwell_complete_after_stop_s": dwell_complete_s, + "game_thread_tick_p99_ms": p99_field("tick_ms"), + "game_thread_tick_max_ms": max_field("tick_ms"), + "stream_update_p99_ms": p99_field("update_chunks_ms"), + "pending_drain_p99_ms": p99_field("pending_drain_ms"), + "apply_p99_ms": p99_field("apply_ms"), + "apply_max_ms": max_field("apply_ms"), + "apply_budget_min_ms": min((f["apply_budget_ms"] for f in frames), default=None), + "apply_budget_max_ms": max((f["apply_budget_ms"] for f in frames), default=None), + "apply_time_budget_hit_frames": sum(f["time_budget_hit"] for f in frames), + "apply_frames": sum(f["meshes_applied"] > 0 for f in frames), + "apply_meshes_total": sum(f["meshes_applied"] for f in frames), + "desired_rebuild_p99_ms": p99_field("desired_rebuild_ms"), + "desired_rebuild_count": len(rebuild_frames), + "desired_rebuilds_per_second": len(rebuild_frames) / 12.0, + "center_crossings": center_crossings, + "same_center_core_region_requests": same_center_core_requests, + "core_desired_cache_hit_count": core_desired_cache_hits, + "core_desired_cache_hit_share_pct": percent( + core_desired_cache_hits, len(core_desired_queries)), + "core_desired_query_nonzero_p50_ms": statistics.median(core_desired_query_ms) + if core_desired_query_ms else None, + "core_desired_query_nonzero_p95_ms": percentile(core_desired_query_ms, 0.95), + "core_desired_query_nonzero_max_ms": max(core_desired_query_ms) + if core_desired_query_ms else None, + "desired_rebuild_nonzero_p50_ms": statistics.median(rebuild_costs) + if rebuild_costs else None, + "desired_rebuild_nonzero_p95_ms": percentile(rebuild_costs, 0.95), + "desired_rebuild_nonzero_max_ms": max(rebuild_costs) if rebuild_costs else None, + "desired_breakdown": desired_stage_breakdown, + "near_submitted_by_direction": direction_submission_totals, + "near_admission_first12": first_admission_order, + "submit_tiles_p99_ms": p99_field("submit_tiles_ms"), + "level_admit_p99_ms": p99_field("level_admit_ms"), + "top_tick_frames": sorted(frames, key=lambda f: f["tick_ms"], reverse=True)[:5], + "trace": str(trace.relative_to(ROOT)), + "log": str(abslog.relative_to(ROOT)), + } + print("[L167Movement] " + json.dumps(row, sort_keys=True), flush=True) + return row + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--smoke", action="store_true", + help="run one unmeasured base warm-up and validate CVar readback") + parser.add_argument("--resume", action="store_true", + help="resume an incomplete matrix from runs-partial.json") + parser.add_argument("--summarize-existing", action="store_true", + help="summarize 28 completed matrix rows without rerunning UE") + args = parser.parse_args() + missing = [path for path in (EDITOR, HOST / "VFHost.uproject") if not path.exists()] + if missing: + raise SystemExit(f"missing UE runner input: {missing}") + OUT.mkdir(parents=True, exist_ok=True) + partial = OUT / "runs-partial.json" + if args.smoke: + run("base", 0) + return 0 + if args.summarize_existing: + if not partial.is_file(): + raise RuntimeError(f"no saved movement rows: {partial}") + rows = json.loads(partial.read_text(encoding="utf-8")) + else: + if partial.is_file() and not args.resume: + raise RuntimeError(f"partial matrix exists; use --resume or move it first: {partial}") + rows = json.loads(partial.read_text(encoding="utf-8")) if partial.is_file() else [] + if not rows: + # One editor binary is used by every condition, so one warm-up is enough. + run("base", 0) + completed = {(row["repetition"], row["condition"]) for row in rows} + for repetition in range(1, 8): + for condition in ORDER[repetition]: + if (repetition, condition) in completed: + continue + row = run(condition, repetition) + rows.append(row) + partial.write_text(json.dumps(rows, indent=2, sort_keys=True) + "\n", + encoding="utf-8") + + metrics = ( + "near_ready_moving_mean_pct", "far_holes_moving_mean_pct", + "game_thread_tick_p99_ms", "game_thread_tick_max_ms", + "arrival_lod02_settled_t_s", "dwell_complete_after_stop_s", + "near_ahead_ready_pct", "near_behind_ready_pct", + "near_forward_cone_ready_pct", "near_forward_cosine_weighted_ready_pct", + "apply_p99_ms", "apply_max_ms", "desired_rebuild_nonzero_p95_ms", + "stream_update_p99_ms", "pending_drain_p99_ms", "submit_tiles_p99_ms", + "level_admit_p99_ms", "desired_rebuild_count", "center_crossings", + ) + + def median_for(condition: str, metric: str) -> float | None: + values = [row[metric] for row in rows + if row["condition"] == condition and row.get(metric) is not None] + return statistics.median(values) if values else None + + medians = { + condition: {metric: median_for(condition, metric) for metric in metrics} + for condition in CONDITIONS + } + gates = {} + paired_deltas = {} + for candidate in ("l157", "l160", "both"): + candidate_rows = [row for row in rows if row["condition"] == candidate] + candidate_median_worst = median_for(candidate, "game_thread_tick_max_ms") + base_median_worst = median_for("base", "game_thread_tick_max_ms") + candidate_near = median_for(candidate, "near_ready_moving_mean_pct") + base_near = median_for("base", "near_ready_moving_mean_pct") + candidate_cone = median_for(candidate, "near_forward_cone_ready_pct") + base_cone = median_for("base", "near_forward_cone_ready_pct") + candidate_far = median_for(candidate, "far_holes_moving_mean_pct") + base_far = median_for("base", "far_holes_moving_mean_pct") + candidate_arrival = median_for(candidate, "arrival_lod02_settled_t_s") + base_arrival = median_for("base", "arrival_lod02_settled_t_s") + candidate_gates = { + "seven_runs_complete": len(candidate_rows) == 7, + "every_run_p99_le_11_ms": len(candidate_rows) == 7 and all( + row["game_thread_tick_p99_ms"] <= 11.0 for row in candidate_rows), + "median_worst_tick_no_worse": ( + candidate_median_worst is not None and base_median_worst is not None + and candidate_median_worst <= base_median_worst), + "no_run_worst_tick_above_14_ms": len(candidate_rows) == 7 and all( + row["game_thread_tick_max_ms"] <= 14.0 for row in candidate_rows), + "median_near_readiness_up": ( + candidate_near is not None and base_near is not None + and candidate_near > base_near), + "median_forward_cone_readiness_up": ( + candidate_cone is not None and base_cone is not None + and candidate_cone > base_cone), + "median_far_holes_no_worse": ( + candidate_far is not None and base_far is not None + and candidate_far <= base_far), + "median_arrival_no_worse": ( + candidate_arrival is not None and base_arrival is not None + and candidate_arrival <= base_arrival), + } + gates[candidate] = { + **candidate_gates, + "passes_default_bar": all(candidate_gates.values()), + } + paired_deltas[candidate] = [] + for repetition in range(1, 8): + by_condition = {row["condition"]: row for row in rows + if row["repetition"] == repetition} + if candidate not in by_condition or "base" not in by_condition: + continue + paired_deltas[candidate].append({ + "repetition": repetition, + **{f"{metric}_delta": by_condition[candidate][metric] + - by_condition["base"][metric] + for metric in metrics if by_condition[candidate].get(metric) is not None + and by_condition["base"].get(metric) is not None}, + }) + + summary = { + "source_commit": subprocess.run( + ["git", "-c", f"safe.directory={ROOT}", "rev-parse", "HEAD"], + cwd=ROOT, check=True, capture_output=True, text=True).stdout.strip(), + "protocol": "Seven interleaved 30 m/s W8 UE runs per condition; 12 s move plus 5 s dwell; 8 effective tile slots; one base warm-up.", + "conditions": CONDITIONS, + "movement_medians": medians, + "candidate_gates": gates, + "paired_deltas_vs_base": paired_deltas, + "runs": rows, + } + (OUT / "summary.json").write_text( + json.dumps(summary, indent=2, sort_keys=True) + "\n", encoding="utf-8") + print("[L167Summary] " + json.dumps({ + "movement_medians": medians, + "candidate_gates": gates, + "runs": len(rows), + }, sort_keys=True), flush=True) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/Standalone/Research/WorldAssemblerM3/Linux167/__pycache__/RunL167Movement.cpython-313.pyc b/Standalone/Research/WorldAssemblerM3/Linux167/__pycache__/RunL167Movement.cpython-313.pyc new file mode 100644 index 0000000..6706b3a Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux167/__pycache__/RunL167Movement.cpython-313.pyc differ