passages: memoized exact MC midpoints + wider conservative sweep bounds (L101, exact vs L100)
Off-grid census by caller (MC midpoints, skirts, far shells); all 63.0M sweep misses accounted (1.9M exact-route fallbacks, 61.1M counted bound skips). Exact MC midpoint queries memoized by float bits (Unreal enables it too); feature bounds expanded. L100/L101 whole-view hashes equal in all four strata (v216 unchanged); 0 added/removed crossings; 15 capture pairs pixel-identical. Near-ready @W12 L100 -> L101: FI 1.77 -> 1.65, TN 1.66 -> 1.54, SW 0.85 -> 0.84, VS 1.03 -> 1.03 s (L98 2.28/2.23/1.02/1.11). Fast tier PASS; Linux UE host builds. Unreal arrival traces not obtained (the runner wrongly applied the 6 GiB address-space cap to Unreal). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016dhxM43dnWvUu2B5Ph1iG1
This commit is contained in:
Fr0zkaandClaude Opus 5.5 committed
1 parent
2d5822cdf7
commit
55e20f4a7e
48 files changed
+2087
-38
No files matched your search
@@ -0,0 +1,93 @@
|
||||
# Linux101
|
||||
|
||||
Branch: `wip/l99-swept-tube`, based on L100 commit `2d5822c`.
|
||||
|
||||
## Off-grid census
|
||||
|
||||
Whole-view counts by caller and stratum. Midpoint counts are unique exact off-grid points after memo hits; the other rows count off-grid float samples.
|
||||
|
||||
| Caller | SW | FI | TN | VS |
|
||||
|---|---:|---:|---:|---:|
|
||||
| MC midpoint correction | 32,177 | 29,101 | 29,864 | 17,534 |
|
||||
| Skirt surface vertex | 25,479 | 21,380 | 21,082 | 13,651 |
|
||||
| Skirt extruded vertex | 231,844 | 192,868 | 191,186 | 169,574 |
|
||||
| Skirt clipped vertex | 14,171 | 26,365 | 22,643 | 11,117 |
|
||||
| Skirt refinement face | 213,293 | 188,382 | 184,353 | 113,766 |
|
||||
| Skirt coarse centroid | 253,581 | 210,056 | 209,077 | 210,502 |
|
||||
| Far-shell surface query | 539,321 | 539,321 | 539,321 | 539,321 |
|
||||
| Far-shell projection normal | 192 | 192 | 192 | 192 |
|
||||
| Far-shell stratum refinement | 755,908 | 1,131,895 | 1,145,781 | 1,057,949 |
|
||||
|
||||
`density_grid`, `gradient_normal_fallback`, and `unknown` had zero generic off-grid float samples. Midpoint attempts / memo hits / unique points were SW 122,286 / 90,109 / 32,177; FI 109,114 / 80,013 / 29,101; TN 112,098 / 82,234 / 29,864; VS 65,236 / 47,702 / 17,534.
|
||||
|
||||
Near LOD0–2 column and support probes:
|
||||
|
||||
| Stratum | Column distance | Support visits / capsule checks | Mouth support / route checks | Relief calls |
|
||||
|---|---:|---:|---:|---:|
|
||||
| SW | 0 | 1,403,489 / 536,084 | 195,083 / 1,630,474 | 3,285,696 |
|
||||
| FI | 11,804 | 10,599,809 / 4,088,836 | 0 / 12,221,614 | 15,809,366 |
|
||||
| TN | 0 | 8,369,673 / 3,378,670 | 0 / 10,074,675 | 13,766,961 |
|
||||
| VS | 0 | 1,997,148 / 670,793 | 0 / 1,999,034 | 2,367,426 |
|
||||
|
||||
Sweep miss accounting, aggregated across the four strata. `Skipped` is the sum of the listed conservative skip classes.
|
||||
|
||||
| Caller | Misses | Exact-route fallbacks | Skipped |
|
||||
|---|---:|---:|---:|
|
||||
| Density grid | 57,887,114 | 1,157,693 | 56,729,421 |
|
||||
| Gradient normal fallback | 186,042 | 80 | 185,962 |
|
||||
| MC midpoint correction | 250,008 | 94,840 | 155,168 |
|
||||
| Skirt surface vertex | 148,792 | 51,147 | 97,645 |
|
||||
| Skirt extruded vertex | 1,625,366 | 113,314 | 1,512,052 |
|
||||
| Skirt clipped vertex | 116,856 | 23,567 | 93,289 |
|
||||
| Skirt refinement face | 1,298,204 | 463,302 | 834,902 |
|
||||
| Skirt coarse centroid | 1,523,877 | 4,619 | 1,519,258 |
|
||||
| **Total** | **63,036,259** | **1,908,562** | **61,127,697** |
|
||||
|
||||
Skip totals: feature bound 59,646,691; tile-free 147,272; vertical bound 160,272; negative reach 1,158,700; positive reach 14,762. Unaccounted misses: 0. All caller totals matched, and every caller’s miss count equaled its exact fallbacks plus skip counts. Near LOD0–2 sweep misses were SW 3,413,227; FI 13,903,288; TN 9,086,664; VS 2,721,371.
|
||||
|
||||
## Fix and ownership checks
|
||||
|
||||
L101 memoizes exact MC midpoint queries by exact float bits and expands the sweep’s conservative feature bounds. In-bound misses still take the exact route; support-free cases use counted bound skips. No trilinear approximation was added.
|
||||
|
||||
The fast tier passed: 274 checks in 277.8 s, including the walkability gates. Determinism passed for 24/24 comparisons across Clang/GCC, W1/W8/W12, all four strata, and L100 W1.
|
||||
|
||||
The special exact-versus-sweep reach audit covered every near tile and found zero added or removed crossings in all strata. It reported 56 added surface triangles in FI and none in the other strata. Its older displacement/normal composite check still returned exit 14: maximum displacement was 19.80–25.88 cm and maximum normal delta 29.91–71.90 degrees.
|
||||
|
||||
| Stratum | Tiles | Added / removed crossings | Added / removed triangles |
|
||||
|---|---:|---:|---:|
|
||||
| SW | 1,408 / 1,408 | 0 / 0 | 0 / 0 |
|
||||
| FI | 1,216 / 1,216 | 0 / 0 | 56 / 0 |
|
||||
| TN | 1,344 / 1,344 | 0 / 0 | 0 / 0 |
|
||||
| VS | 1,344 / 1,344 | 0 / 0 | 0 / 0 |
|
||||
|
||||
Capture check: 15 matched views (30 PNGs), all with 0 changed pixels. I also inspected the mouth and mid-route pairs for each of the three passage links; the openings and silhouettes match by eye. L100/L101 whole-view density, mesh, payload, and topology hashes matched in all four strata, so no version bump was needed.
|
||||
|
||||
## Readiness
|
||||
|
||||
L80 protocol, interleaved versions, four measured runs per cell, with a separate warmup. Each cell is `near LOD0–2 / whole view` seconds (median).
|
||||
|
||||
| Workers | Stratum | L98 | L99 | L100 | L101 |
|
||||
|---:|---|---:|---:|---:|---:|
|
||||
| 8 | SW | 1.031 / 2.244 | 0.795 / 1.965 | 0.845 / 2.021 | 0.829 / 2.013 |
|
||||
| 8 | FI | 2.400 / 3.662 | 1.509 / 2.546 | 1.874 / 2.930 | 1.703 / 2.723 |
|
||||
| 8 | TN | 2.395 / 3.494 | 1.520 / 2.489 | 1.834 / 2.789 | 1.653 / 2.627 |
|
||||
| 8 | VS | 1.279 / 2.302 | 1.099 / 2.087 | 1.184 / 2.210 | 1.165 / 2.173 |
|
||||
| 12 | SW | 1.017 / 2.163 | 0.806 / 1.873 | 0.852 / 1.932 | 0.840 / 1.926 |
|
||||
| 12 | FI | 2.279 / 3.477 | 1.448 / 2.420 | 1.769 / 2.757 | 1.652 / 2.641 |
|
||||
| 12 | TN | 2.225 / 3.258 | 1.392 / 2.295 | 1.663 / 2.570 | 1.535 / 2.449 |
|
||||
| 12 | VS | 1.112 / 2.062 | 0.989 / 1.918 | 1.034 / 1.967 | 1.029 / 1.970 |
|
||||
|
||||
L101 improved near-ready medians over L100 in all eight cells and whole-view medians in seven of eight; W12 VS was 0.003 s slower. L101 remains slower than L99.
|
||||
|
||||
## Unreal arrival traces
|
||||
|
||||
The final Linux Development host build succeeded in 4.0 s with `-NoUBA -MaxParallelActions=4` and no `--as`.
|
||||
|
||||
| Stratum | Complete runs | First admission | LOD0–2 settled | Max frame tick |
|
||||
|---|---:|---:|---:|---:|
|
||||
| FI | 0 of 3 | 0.372 s in the last partial attempt | — | — |
|
||||
| SW | 0 of 3 | — | — | — |
|
||||
|
||||
The runner recorded 17 FI launch attempts; none completed. The last FI probe reached first admission, then the 6 GiB address-space limit rejected a 29,425,664-byte allocation in `FStandaloneShapeWorld::AddTaperedCapsule` (peak virtual use 6,436,151,296 bytes). A Binned allocator and smaller temporary editor pools still did not complete the trace. Earlier attempts also failed at Unreal startup with `pthread_create`/`EAGAIN`; one `-NoEnginePlugins` attempt could not find the project commandlet. No arrival or frame-tick timing is inferred from these failures, and the required three FI plus three SW traces were not obtained.
|
||||
|
||||
All attempted runs stayed under 30 minutes. Temporary `VFHost.uproject` and `DefaultEngine.ini` changes were restored. Logs and crash contexts are under `Saved/Linux/l101/unreal-arrival/` and `/home/jahni/VFHost/Saved/Linux/l101/user/`.
|
||||
@@ -0,0 +1,602 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Capture matched L100/L101 passage mouth and mid-route views."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import struct
|
||||
import subprocess
|
||||
import time
|
||||
import zlib
|
||||
from array import array
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[4]
|
||||
OUT = ROOT / "Saved/Linux/l101/captures"
|
||||
IMAGE_DIR = Path(__file__).resolve().parent / "captures"
|
||||
DATA = ROOT / "Standalone/data/owner_world.json"
|
||||
LAYOUT_PATH = ROOT / "Saved/Linux/l99/resolved-sw.json"
|
||||
BINARIES = {
|
||||
"before-l100": ROOT / "Saved/Linux/l100/bin/VoxelForgeCoreHeadless-l100-clang",
|
||||
"after-l101": ROOT / "Saved/Linux/l101/bin/VoxelForgeCoreHeadless-l101-clang",
|
||||
}
|
||||
RENDERER = ROOT / "Saved/Linux/l101/bin/VoxelForgeOffscreenCapture-l101-clang"
|
||||
VOXEL_Q = 256.0
|
||||
CORE_UM_PER_VOXEL = 250_000.0
|
||||
ROUTE_POINTS = {}
|
||||
ROUTE_LENGTHS = {}
|
||||
|
||||
|
||||
def add(a, b):
|
||||
return tuple(x + y for x, y in zip(a, b))
|
||||
|
||||
|
||||
def sub(a, b):
|
||||
return tuple(x - y for x, y in zip(a, b))
|
||||
|
||||
|
||||
def mul(a, s):
|
||||
return tuple(x * s for x in a)
|
||||
|
||||
|
||||
def dot(a, b):
|
||||
return sum(x * y for x, y in zip(a, b))
|
||||
|
||||
|
||||
def norm(a):
|
||||
length = math.sqrt(dot(a, a))
|
||||
if length < 1.0e-8:
|
||||
raise ValueError("degenerate capture direction")
|
||||
return tuple(x / length for x in a)
|
||||
|
||||
|
||||
def fmt(values):
|
||||
return ",".join(f"{value:.9f}" for value in values)
|
||||
|
||||
|
||||
def angles(camera, target):
|
||||
delta = sub(target, camera)
|
||||
return (math.degrees(math.atan2(delta[0], delta[1])),
|
||||
math.degrees(math.atan2(delta[2], math.hypot(delta[0], delta[1]))))
|
||||
|
||||
|
||||
def interpolate(points, distance):
|
||||
remaining = max(0.0, distance)
|
||||
for a, b in zip(points, points[1:]):
|
||||
delta = sub(b[0], a[0])
|
||||
length = math.sqrt(dot(delta, delta))
|
||||
if length < 1.0e-8:
|
||||
continue
|
||||
if remaining <= length:
|
||||
t = remaining / length
|
||||
return (tuple(a[0][i] + delta[i] * t for i in range(3)),
|
||||
a[1] + (b[1] - a[1]) * t)
|
||||
remaining -= length
|
||||
return points[-1]
|
||||
|
||||
|
||||
def route_frame(points, distance):
|
||||
center, radius = interpolate(points, distance)
|
||||
before, _ = interpolate(points, max(0.0, distance - 0.25))
|
||||
after, _ = interpolate(points, min(sum(
|
||||
math.sqrt(dot(sub(b[0], a[0]), sub(b[0], a[0])))
|
||||
for a, b in zip(points, points[1:])), distance + 0.25))
|
||||
tangent = norm(sub(after, before))
|
||||
horizontal = norm((tangent[0], tangent[1], 0.0))
|
||||
side = (-horizontal[1], horizontal[0], 0.0)
|
||||
return center, radius, tangent, side
|
||||
|
||||
|
||||
def route_length(points):
|
||||
return sum(math.sqrt(dot(sub(b[0], a[0]), sub(b[0], a[0])))
|
||||
for a, b in zip(points, points[1:]))
|
||||
|
||||
|
||||
def nearest_route(points, point):
|
||||
best = None
|
||||
distance_along = 0.0
|
||||
for a, b in zip(points, points[1:]):
|
||||
delta = sub(b[0], a[0])
|
||||
length_sq = dot(delta, delta)
|
||||
if length_sq < 1.0e-10:
|
||||
continue
|
||||
segment_length = math.sqrt(length_sq)
|
||||
t = max(0.0, min(1.0, dot(sub(point, a[0]), delta) / length_sq))
|
||||
center = add(a[0], mul(delta, t))
|
||||
tangent = norm(delta)
|
||||
radial = sub(point, center)
|
||||
radial = sub(radial, mul(tangent, dot(radial, tangent)))
|
||||
distance = math.sqrt(dot(sub(point, center), sub(point, center)))
|
||||
if best is None or distance < best[0]:
|
||||
radius = a[1] + (b[1] - a[1]) * t
|
||||
best = (distance, center, tangent, radial, radius,
|
||||
distance_along + t * segment_length)
|
||||
distance_along += segment_length
|
||||
if best is None:
|
||||
raise ValueError("passage route has no non-degenerate segment")
|
||||
return best
|
||||
|
||||
|
||||
def run(name, command, headless):
|
||||
log = OUT / f"{name}.log"
|
||||
wrapped = ["taskset", "-c", "0-11"]
|
||||
if headless:
|
||||
wrapped += ["prlimit", "--as=6442450944", "--"]
|
||||
wrapped += ["timeout", "1740s", *map(str, command)]
|
||||
started = time.monotonic()
|
||||
with log.open("w", encoding="utf-8") as stream:
|
||||
result = subprocess.run(wrapped, cwd=ROOT, stdout=stream,
|
||||
stderr=subprocess.STDOUT, check=False, timeout=1745)
|
||||
row = {"name": name, "exit_code": result.returncode,
|
||||
"wall_s": round(time.monotonic() - started, 3),
|
||||
"log": str(log.relative_to(ROOT))}
|
||||
print("[L101CaptureRun] " + json.dumps(row, sort_keys=True), flush=True)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"capture step failed: {name}; see {log}")
|
||||
return row
|
||||
|
||||
|
||||
def nearest_surface_anchor(mesh, point, tile_search_margin_m=20.0):
|
||||
"""Return the nearest surface vertex and outward mesh normal from a VFMSH scene."""
|
||||
best_distance_sq = float("inf")
|
||||
best_position = best_normal = None
|
||||
with mesh.open("rb") as stream:
|
||||
magic = stream.read(8)
|
||||
has_materials = magic == b"VFMSH002"
|
||||
if magic not in (b"VFMSH001", b"VFMSH002"):
|
||||
raise RuntimeError(f"unsupported mesh scene header: {mesh}")
|
||||
version, voxel_um, tile_count = struct.unpack("<3I", stream.read(12))
|
||||
if version != (2 if has_materials else 1):
|
||||
raise RuntimeError(f"mesh scene version mismatch: {mesh}")
|
||||
stream.seek(40, 1) # stored camera position, yaw, and pitch
|
||||
for _ in range(tile_count):
|
||||
key = struct.unpack("<4i", stream.read(16))
|
||||
surface_count, _, vertex_count, index_count, mask_count = struct.unpack(
|
||||
"<5I", stream.read(20))
|
||||
extent = (32 << key[3]) * voxel_um / 1_000_000.0
|
||||
origin = (key[0] * extent, key[1] * extent, key[2] * extent)
|
||||
if any(point[axis] < origin[axis] - tile_search_margin_m or
|
||||
point[axis] > origin[axis] + extent + tile_search_margin_m
|
||||
for axis in range(3)):
|
||||
stream.seek(vertex_count * 24 + (10 * vertex_count if has_materials else 0)
|
||||
+ index_count * 4 + mask_count, 1)
|
||||
continue
|
||||
positions = array("f")
|
||||
positions.frombytes(stream.read(vertex_count * 12))
|
||||
normals = array("f")
|
||||
normals.frombytes(stream.read(vertex_count * 12))
|
||||
if has_materials:
|
||||
stream.seek(vertex_count * 10, 1)
|
||||
stream.seek(index_count * 4 + mask_count, 1)
|
||||
if struct.pack("=I", 1) != struct.pack("<I", 1):
|
||||
positions.byteswap()
|
||||
normals.byteswap()
|
||||
for index in range(min(surface_count, vertex_count)):
|
||||
offset = index * 3
|
||||
# VFMSH stores vertex coordinates in world space; tile origin
|
||||
# is metadata for edge rendering and must not be added again.
|
||||
position = tuple(float(positions[offset + axis]) for axis in range(3))
|
||||
delta = sub(position, point)
|
||||
distance_sq = dot(delta, delta)
|
||||
if distance_sq >= best_distance_sq:
|
||||
continue
|
||||
raw_normal = tuple(float(normals[offset + axis]) for axis in range(3))
|
||||
if dot(raw_normal, raw_normal) < 1.0e-10:
|
||||
continue
|
||||
best_distance_sq = distance_sq
|
||||
best_position = position
|
||||
best_normal = norm(raw_normal)
|
||||
if best_position is None:
|
||||
raise RuntimeError(f"no nearby surface vertices in mesh scene: {mesh}")
|
||||
return {"position_m": best_position, "normal": best_normal,
|
||||
"distance_m": math.sqrt(best_distance_sq)}
|
||||
|
||||
|
||||
def make_passages(layout):
|
||||
q_to_m = layout["core_voxel_size_um"] / (VOXEL_Q * 1_000_000.0)
|
||||
passages = []
|
||||
all_routes = []
|
||||
for passage in layout["passages"]:
|
||||
controls = [(tuple(float(q) * q_to_m for q in p["position_q"]),
|
||||
float(p["radius_q"]) * q_to_m)
|
||||
for p in passage["control_points"]]
|
||||
upper = tuple(float(q) * q_to_m for q in passage["upper_mouth_q"])
|
||||
lower = tuple(float(q) * q_to_m for q in passage["lower_mouth_q"])
|
||||
ui = min(range(len(controls)), key=lambda i: dot(sub(controls[i][0], upper),
|
||||
sub(controls[i][0], upper)))
|
||||
li = min(range(len(controls)), key=lambda i: dot(sub(controls[i][0], lower),
|
||||
sub(controls[i][0], lower)))
|
||||
points = controls[ui:li + 1]
|
||||
if len(points) < 2:
|
||||
raise RuntimeError(f"invalid mouth path: {passage['stable_id']}")
|
||||
upper_offset = math.sqrt(dot(sub(points[0][0], upper), sub(points[0][0], upper)))
|
||||
lower_offset = math.sqrt(dot(sub(points[-1][0], lower), sub(points[-1][0], lower)))
|
||||
length = route_length(points)
|
||||
passage_views = []
|
||||
for endpoint, distance, camera_distance, outward_sign in (
|
||||
("upper-mouth", 0.0, min(length, 7.4), -1.0),
|
||||
("lower-mouth", length, max(0.0, length - 5.5), 1.0)):
|
||||
center, _, tangent, side = route_frame(points, distance)
|
||||
camera_center, _, _, camera_side = route_frame(points, camera_distance)
|
||||
camera = (camera_center[0] + camera_side[0] * 2.5,
|
||||
camera_center[1] + camera_side[1] * 2.5, camera_center[2])
|
||||
outward = mul(tangent, outward_sign)
|
||||
target = (center[0] + outward[0] * 8.0 + side[0] * 2.5,
|
||||
center[1] + outward[1] * 8.0 + side[1] * 2.5, center[2])
|
||||
yaw, pitch = angles(camera, target)
|
||||
passage_views.append({"name": f"{passage['stable_id']}-{endpoint}",
|
||||
"kind": "mouth", "focus": center, "camera": camera, "target": target,
|
||||
"yaw": yaw, "pitch": pitch, "clip": None, "fov": 80.0,
|
||||
"focus_radius": 20.0, "passage": passage["stable_id"],
|
||||
"camera_location": "passage-air"})
|
||||
for fraction in (0.25, 0.50, 0.75):
|
||||
distance = length * fraction
|
||||
center, radius, _, side = route_frame(points, distance)
|
||||
camera = (center[0] + side[0] * 0.75,
|
||||
center[1] + side[1] * 0.75,
|
||||
center[2] - radius + 1.6)
|
||||
target_pos, target_radius = interpolate(
|
||||
points, min(length, distance + 14.0))
|
||||
target = (target_pos[0] + side[0] * 2.35,
|
||||
target_pos[1] + side[1] * 2.35,
|
||||
target_pos[2] - target_radius + 1.8)
|
||||
yaw, pitch = angles(camera, target)
|
||||
pct = int(round(fraction * 100))
|
||||
passage_views.append({"name": f"{passage['stable_id']}-mid-{pct:02d}",
|
||||
"kind": "mid-tunnel", "route_fraction": fraction,
|
||||
"focus": center, "camera": camera, "target": target,
|
||||
"yaw": yaw, "pitch": pitch, "clip": None,
|
||||
"fov": 78.0, "focus_radius": 24.0,
|
||||
"passage": passage["stable_id"]})
|
||||
passages.append({"stable_id": passage["stable_id"], "route_length_m": length,
|
||||
"control_points": len(points), "upper_mouth_anchor_offset_m": upper_offset,
|
||||
"lower_mouth_anchor_offset_m": lower_offset, "views": passage_views})
|
||||
all_routes.append((passage["stable_id"], points))
|
||||
return passages, all_routes
|
||||
|
||||
|
||||
def make_hotspot_views(routes, reach_summary):
|
||||
views = []
|
||||
for series, rows, value_key, location_key, region_key in (
|
||||
("displacement", reach_summary["top_displacement_tiles"],
|
||||
"max_displacement_cm", "displacement_location_metres", "displacement_region"),
|
||||
("normal", reach_summary["top_normal_tiles"],
|
||||
"max_normal_degrees", "normal_location_metres", "normal_region"),
|
||||
):
|
||||
for index, row in enumerate(rows, 1):
|
||||
point = tuple(row[location_key])
|
||||
candidates = [(nearest_route(route, point), name, route)
|
||||
for name, route in routes]
|
||||
(distance, center, tangent, radial, radius, along), nearest_id, route = min(
|
||||
candidates, key=lambda item: item[0][0])
|
||||
if math.sqrt(dot(radial, radial)) < 0.25:
|
||||
horizontal = norm((tangent[0], tangent[1], 0.0))
|
||||
radial = (-horizontal[1], horizontal[0], 0.0)
|
||||
outward = norm(radial)
|
||||
camera = add(point, mul(outward, 20.0))
|
||||
target = point
|
||||
plane = None
|
||||
yaw, pitch = angles(camera, target)
|
||||
views.append({"name": f"top-{series}-{index:02d}", "kind": f"top-{series}",
|
||||
"focus": point, "camera": camera, "target": target,
|
||||
"yaw": yaw, "pitch": pitch, "clip": plane, "fov": 68.0,
|
||||
"focus_radius": 18.0, "strate": row["strate"],
|
||||
"value": row[value_key], "region": row[region_key],
|
||||
"camera_anchor": "nearest-surface-l100-v1", "nearest_passage": nearest_id,
|
||||
"route_distance_m": distance})
|
||||
return views
|
||||
|
||||
|
||||
def decode_png(path):
|
||||
"""Decode 8-bit RGB/RGBA PNG pixels using only the Python standard library."""
|
||||
data = path.read_bytes()
|
||||
if data[:8] != b"\x89PNG\r\n\x1a\n":
|
||||
raise RuntimeError(f"not a PNG: {path}")
|
||||
width = height = bit_depth = color_type = interlace = None
|
||||
compressed = bytearray()
|
||||
offset = 8
|
||||
while offset < len(data):
|
||||
length = struct.unpack_from(">I", data, offset)[0]
|
||||
kind = data[offset + 4:offset + 8]
|
||||
chunk = data[offset + 8:offset + 8 + length]
|
||||
if kind == b"IHDR":
|
||||
width, height, bit_depth, color_type, compression, filtering, interlace = \
|
||||
struct.unpack(">IIBBBBB", chunk)
|
||||
if compression or filtering or interlace:
|
||||
raise RuntimeError(f"unsupported PNG coding in {path}")
|
||||
elif kind == b"IDAT":
|
||||
compressed.extend(chunk)
|
||||
elif kind == b"IEND":
|
||||
break
|
||||
offset += 12 + length
|
||||
if bit_depth != 8 or color_type not in (2, 6):
|
||||
raise RuntimeError(f"expected 8-bit RGB/RGBA PNG: {path}")
|
||||
channels = 3 if color_type == 2 else 4
|
||||
stride = width * channels
|
||||
encoded = zlib.decompress(compressed)
|
||||
pixels = bytearray(height * stride)
|
||||
source_offset = 0
|
||||
for y in range(height):
|
||||
filter_type = encoded[source_offset]
|
||||
source_offset += 1
|
||||
row = bytearray(encoded[source_offset:source_offset + stride])
|
||||
source_offset += stride
|
||||
prior_offset = (y - 1) * stride
|
||||
for index in range(stride):
|
||||
left = row[index - channels] if index >= channels else 0
|
||||
above = pixels[prior_offset + index] if y else 0
|
||||
upper_left = pixels[prior_offset + index - channels] \
|
||||
if y and index >= channels else 0
|
||||
if filter_type == 1:
|
||||
row[index] = (row[index] + left) & 255
|
||||
elif filter_type == 2:
|
||||
row[index] = (row[index] + above) & 255
|
||||
elif filter_type == 3:
|
||||
row[index] = (row[index] + ((left + above) >> 1)) & 255
|
||||
elif filter_type == 4:
|
||||
estimate = left + above - upper_left
|
||||
left_distance = abs(estimate - left)
|
||||
above_distance = abs(estimate - above)
|
||||
upper_left_distance = abs(estimate - upper_left)
|
||||
if left_distance <= above_distance and left_distance <= upper_left_distance:
|
||||
predictor = left
|
||||
elif above_distance <= upper_left_distance:
|
||||
predictor = above
|
||||
else:
|
||||
predictor = upper_left
|
||||
row[index] = (row[index] + predictor) & 255
|
||||
elif filter_type != 0:
|
||||
raise RuntimeError(f"unsupported PNG filter {filter_type}: {path}")
|
||||
start = y * stride
|
||||
pixels[start:start + stride] = row
|
||||
if channels == 3:
|
||||
return width, height, pixels
|
||||
rgb = bytearray(width * height * 3)
|
||||
rgb[0::3] = pixels[0::4]
|
||||
rgb[1::3] = pixels[1::4]
|
||||
rgb[2::3] = pixels[2::4]
|
||||
return width, height, rgb
|
||||
|
||||
|
||||
def image_geometry_metrics(path):
|
||||
width, height, rgb = decode_png(path)
|
||||
color_counts = {}
|
||||
for offset in range(0, len(rgb), 3):
|
||||
color = (rgb[offset], rgb[offset + 1], rgb[offset + 2])
|
||||
color_counts[color] = color_counts.get(color, 0) + 1
|
||||
total = width * height
|
||||
visible = total - max(color_counts.values())
|
||||
return {"width": width, "height": height, "unique_rgb": len(color_counts),
|
||||
"visible_geometry_pixels": visible,
|
||||
"visible_geometry_fraction": visible / total}
|
||||
|
||||
|
||||
def changed_pixel_count(before, after):
|
||||
before_width, before_height, before_rgb = decode_png(before)
|
||||
after_width, after_height, after_rgb = decode_png(after)
|
||||
if (before_width, before_height) != (after_width, after_height):
|
||||
raise RuntimeError(f"capture dimensions differ: {before} / {after}")
|
||||
return sum(before_rgb[offset] != after_rgb[offset]
|
||||
or before_rgb[offset + 1] != after_rgb[offset + 1]
|
||||
or before_rgb[offset + 2] != after_rgb[offset + 2]
|
||||
for offset in range(0, len(before_rgb), 3))
|
||||
|
||||
|
||||
def mesh_triangle_count(mesh):
|
||||
with mesh.open("rb") as stream:
|
||||
magic = stream.read(8)
|
||||
has_materials = magic == b"VFMSH002"
|
||||
if magic not in (b"VFMSH001", b"VFMSH002"):
|
||||
raise RuntimeError(f"unsupported mesh scene header: {mesh}")
|
||||
version, _, tile_count = struct.unpack("<3I", stream.read(12))
|
||||
if version != (2 if has_materials else 1):
|
||||
raise RuntimeError(f"mesh scene version mismatch: {mesh}")
|
||||
stream.seek(40, 1)
|
||||
triangles = 0
|
||||
for _ in range(tile_count):
|
||||
stream.seek(16, 1)
|
||||
_, _, vertex_count, index_count, mask_count = struct.unpack(
|
||||
"<5I", stream.read(20))
|
||||
triangles += index_count // 3
|
||||
stream.seek(vertex_count * 24 + (10 * vertex_count if has_materials else 0)
|
||||
+ index_count * 4 + mask_count, 1)
|
||||
return triangles
|
||||
|
||||
|
||||
def set_route_station(view, distance):
|
||||
points = ROUTE_POINTS[view["passage"]]
|
||||
length = ROUTE_LENGTHS[view["passage"]]
|
||||
center, radius, _, side = route_frame(points, distance)
|
||||
camera = (center[0] + side[0] * 0.75,
|
||||
center[1] + side[1] * 0.75,
|
||||
center[2] - radius + 1.6)
|
||||
target_pos, target_radius = interpolate(points, min(length, distance + 14.0))
|
||||
target = (target_pos[0] + side[0] * 2.35,
|
||||
target_pos[1] + side[1] * 2.35,
|
||||
target_pos[2] - target_radius + 1.8)
|
||||
yaw, pitch = angles(camera, target)
|
||||
if "route_fraction_requested" not in view:
|
||||
view["route_fraction_requested"] = view["route_fraction"]
|
||||
view["route_requested_focus"] = view["focus"]
|
||||
view["route_fraction"] = distance / length
|
||||
view["route_station_adjustment_m"] = distance - length * view["route_fraction_requested"]
|
||||
view["focus"] = center
|
||||
view["camera"] = camera
|
||||
view["target"] = target
|
||||
view["yaw"], view["pitch"] = yaw, pitch
|
||||
view["clip"] = None
|
||||
|
||||
|
||||
def render_command(view, mesh, image, camera, yaw, pitch):
|
||||
args = [RENDERER, mesh, image,
|
||||
"--size=1600x900", f"--fov-deg={view['fov']:.1f}",
|
||||
f"--camera-m={fmt(camera)}", f"--yaw-deg={yaw:.6f}",
|
||||
f"--pitch-deg={pitch:.6f}", "--shade=readable", "--headlamp=on",
|
||||
"--height-tint=on", "--tile-edges=off", "--tile-skirts=off",
|
||||
f"--height-center-m={view['focus'][2]:.6f}", "--height-range-m=10",
|
||||
f"--focus-region={fmt((*view['focus'], view['focus_radius']))}"]
|
||||
if view["clip"] is not None:
|
||||
args.append(f"--clip-plane={fmt(view['clip'])}")
|
||||
return args
|
||||
|
||||
|
||||
def export_scene(view, binary, mesh, stem):
|
||||
camera = view["camera"]
|
||||
yaw, pitch = angles(camera, view["target"])
|
||||
export_args = [binary, DATA, "--world-assembler-prototype", "--stream-benchmark",
|
||||
"--stream-pipeline", "--workers=8", "--force-generate", "--clip-radius=3",
|
||||
"--max-clip-level=0", "--max-level=0", "--render-distance-chunks=0",
|
||||
f"--camera={fmt((*camera, yaw, pitch))}", f"--mesh-export={mesh}"]
|
||||
row = run(f"{stem}-export", export_args, headless=True)
|
||||
row["scene_bytes"] = mesh.stat().st_size if mesh.exists() else 0
|
||||
row["scene_triangles"] = mesh_triangle_count(mesh) if row["scene_bytes"] else 0
|
||||
return row
|
||||
|
||||
|
||||
def export_and_render(view, label, binary):
|
||||
safe = view["name"].replace("/", "-")
|
||||
stem = f"{safe}-{label}"
|
||||
mesh = OUT / f"{stem}.vfmesh"
|
||||
image = IMAGE_DIR / f"{safe}-{label}.png"
|
||||
camera = view["camera"]
|
||||
yaw, pitch = angles(camera, view["target"])
|
||||
export_row = export_scene(view, binary, mesh, stem)
|
||||
if export_row["scene_bytes"] <= 0:
|
||||
raise RuntimeError(f"empty mesh scene: {mesh}")
|
||||
if export_row["scene_triangles"] == 0:
|
||||
if label != "before-l100" or view.get("kind") != "mid-tunnel":
|
||||
raise RuntimeError(f"route station exports no surface triangles: {mesh}")
|
||||
points = ROUTE_POINTS[view["passage"]]
|
||||
length = ROUTE_LENGTHS[view["passage"]]
|
||||
requested_fraction = view["route_fraction"]
|
||||
requested_distance = length * requested_fraction
|
||||
found_surface = False
|
||||
for offset in (5.0, -5.0, 10.0, -10.0, 20.0, -20.0, 35.0, -35.0,
|
||||
50.0, -50.0, 75.0, -75.0, 100.0, -100.0, 150.0,
|
||||
-150.0, 200.0, -200.0):
|
||||
distance = max(0.0, min(length, requested_distance + offset))
|
||||
if distance == requested_distance:
|
||||
continue
|
||||
set_route_station(view, distance)
|
||||
adjusted_stem = f"{stem}-station-{view['route_fraction']:.5f}"
|
||||
export_row = export_scene(view, binary, mesh, adjusted_stem)
|
||||
if export_row["scene_triangles"] > 0:
|
||||
view["camera_anchor"] = "route-station-shifted-to-nearby-geometry"
|
||||
found_surface = True
|
||||
break
|
||||
if not found_surface:
|
||||
raise RuntimeError(f"no geometry along nearby route stations: {mesh}")
|
||||
camera = view["camera"]
|
||||
yaw, pitch = angles(camera, view["target"])
|
||||
if label == "before-l100" and view.get("camera_anchor") == "nearest-surface-l100-v1":
|
||||
anchor = nearest_surface_anchor(mesh, view["focus"], tile_search_margin_m=256.0)
|
||||
camera = add(anchor["position_m"], mul(anchor["normal"], 8.0))
|
||||
view["camera"] = camera
|
||||
view["target"] = anchor["position_m"]
|
||||
view["yaw"], view["pitch"] = angles(camera, view["target"])
|
||||
view["clip"] = None
|
||||
view["surface_anchor"] = anchor
|
||||
yaw, pitch = view["yaw"], view["pitch"]
|
||||
render_args = render_command(view, mesh, image, camera, yaw, pitch)
|
||||
try:
|
||||
render_row = run(f"{stem}-render", render_args, headless=True)
|
||||
except RuntimeError:
|
||||
if view["clip"] is None:
|
||||
raise
|
||||
retry_args = [arg for arg in render_args
|
||||
if not (isinstance(arg, str) and arg.startswith("--clip-plane="))]
|
||||
render_row = run(f"{stem}-render-no-clip-retry", retry_args, headless=True)
|
||||
render_row["clip_fallback"] = True
|
||||
render_row["image"] = str(image.relative_to(ROOT))
|
||||
render_row["image_bytes"] = image.stat().st_size if image.exists() else 0
|
||||
if render_row["image_bytes"] <= 0:
|
||||
raise RuntimeError(f"empty image: {image}")
|
||||
geometry = image_geometry_metrics(image)
|
||||
if label == "before-l100" and (
|
||||
geometry["visible_geometry_pixels"] < 3000 or geometry["unique_rgb"] < 2):
|
||||
# At a few deep route stations the default forward view looks into empty
|
||||
# sky. Keep the camera on the route station and aim at the nearest L98
|
||||
# surface in the exported scene; retain that matched target for L101.
|
||||
anchor = nearest_surface_anchor(mesh, view["focus"], tile_search_margin_m=256.0)
|
||||
if anchor["distance_m"] > 60.0:
|
||||
raise RuntimeError(f"no nearby surface for fallback camera: {image} {anchor}")
|
||||
camera = view["camera"]
|
||||
target = anchor["position_m"]
|
||||
yaw, pitch = angles(camera, target)
|
||||
view["target"] = target
|
||||
view["yaw"], view["pitch"] = yaw, pitch
|
||||
view["clip"] = None
|
||||
view["camera_anchor"] = "route-station-nearest-surface-target"
|
||||
view["surface_anchor"] = anchor
|
||||
retry_args = render_command(view, mesh, image, camera, yaw, pitch)
|
||||
render_row = run(f"{stem}-render-surface-fallback", retry_args, headless=True)
|
||||
render_row["image"] = str(image.relative_to(ROOT))
|
||||
render_row["image_bytes"] = image.stat().st_size if image.exists() else 0
|
||||
geometry = image_geometry_metrics(image)
|
||||
render_row["geometry"] = geometry
|
||||
if geometry["visible_geometry_pixels"] < 3000 or geometry["unique_rgb"] < 2:
|
||||
raise RuntimeError(f"camera sees sky only or too little geometry: {image} {geometry}")
|
||||
mesh.unlink()
|
||||
return export_row, render_row
|
||||
|
||||
|
||||
def main():
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
IMAGE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
for path in (*BINARIES.values(), RENDERER, DATA, LAYOUT_PATH):
|
||||
if not path.is_file():
|
||||
raise SystemExit(f"missing input: {path}")
|
||||
layout = json.loads(LAYOUT_PATH.read_text(encoding="utf-8"))
|
||||
passages, routes = make_passages(layout)
|
||||
ROUTE_POINTS.clear()
|
||||
ROUTE_LENGTHS.clear()
|
||||
for stable_id, points in routes:
|
||||
ROUTE_POINTS[stable_id] = points
|
||||
ROUTE_LENGTHS[stable_id] = route_length(points)
|
||||
views = [view for passage in passages for view in passage["views"]]
|
||||
partial_path = OUT / "runs-partial.json"
|
||||
existing_records = {}
|
||||
if partial_path.is_file():
|
||||
for item in json.loads(partial_path.read_text(encoding="utf-8")):
|
||||
existing_records[item["view"]["name"]] = item
|
||||
records = []
|
||||
for view in views:
|
||||
prior = existing_records.get(view["name"])
|
||||
camera_matches = (prior is not None and
|
||||
prior["view"].get("camera_anchor") == view.get("camera_anchor"))
|
||||
if camera_matches:
|
||||
for key in ("camera", "target", "yaw", "pitch", "clip", "surface_anchor"):
|
||||
if key in prior["view"]:
|
||||
view[key] = prior["view"][key]
|
||||
if camera_matches and all(version in prior and
|
||||
(ROOT / prior[version]["render"]["image"]).is_file()
|
||||
for version in BINARIES):
|
||||
records.append(prior)
|
||||
print(f"[L101CaptureSkip] complete_pair={view['name']}", flush=True)
|
||||
continue
|
||||
pair = {"view": view}
|
||||
for label, binary in BINARIES.items():
|
||||
export_row, render_row = export_and_render(view, label, binary)
|
||||
pair[label] = {"export": export_row, "render": render_row}
|
||||
(OUT / "runs-partial.json").write_text(json.dumps(records + [pair], indent=2) + "\n",
|
||||
encoding="utf-8")
|
||||
pair["comparison"] = {"changed_pixels": changed_pixel_count(
|
||||
ROOT / pair["before-l100"]["render"]["image"],
|
||||
ROOT / pair["after-l101"]["render"]["image"])}
|
||||
(OUT / "runs-partial.json").write_text(json.dumps(records + [pair], indent=2) + "\n",
|
||||
encoding="utf-8")
|
||||
records.append(pair)
|
||||
summary = {"source_layout_identity": layout["stable_identity"],
|
||||
"view_count": len(views), "paired_png_count": len(views) * 2,
|
||||
"passages": passages, "views": records}
|
||||
(OUT / "summary.json").write_text(json.dumps(summary, indent=2) + "\n",
|
||||
encoding="utf-8")
|
||||
print("[L101CaptureSummary] " + json.dumps({
|
||||
"views": len(views), "paired_png_count": len(views) * 2,
|
||||
"images": [row[version]["render"]["image"] for row in records
|
||||
for version in BINARIES]}, sort_keys=True), flush=True)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,204 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Count measured off-grid and passage-composer work by tile and caller."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[4]
|
||||
OUT = ROOT / "Saved/Linux/l101/census"
|
||||
DATA = ROOT / "Standalone/data/owner_world.json"
|
||||
HEADLESS = ROOT / "Saved/Linux/l101/bin/VoxelForgeCoreHeadless-l101-clang"
|
||||
CAMERAS = {
|
||||
"SW": (8.0, 16.0, -372.2626953125),
|
||||
"FI": (25.8369140625, 3.19921875, -737.0),
|
||||
"TN": (19.166015625, -14.2001953125, -753.3740234375),
|
||||
"VS": (-20.0625, -5.068359375, -900.353515625),
|
||||
}
|
||||
CALLERS = (
|
||||
"unknown", "density_grid", "gradient_normal_fallback", "mc_midpoint_correction",
|
||||
"skirt_surface_vertex", "skirt_extruded_vertex", "skirt_clipped_vertex",
|
||||
"skirt_refinement_face", "skirt_coarse_centroid", "far_shell_surface_query",
|
||||
"far_shell_projection_normal", "far_shell_stratum_refinement",
|
||||
)
|
||||
SKIRT_CALLERS = (
|
||||
"skirt_surface_vertex", "skirt_extruded_vertex", "skirt_clipped_vertex",
|
||||
"skirt_refinement_face", "skirt_coarse_centroid",
|
||||
)
|
||||
CPU_LINE = re.compile(r"^\[CpuTileStages\] key=\([^,]+,[^,]+,[^,]+,lod(\d+)\) (.*)$", re.M)
|
||||
COMPOSER_LINE = re.compile(
|
||||
r"^\[PassageComposerTile\] key=\([^,]+,[^,]+,[^,]+,lod(\d+)\) (.*)$", re.M)
|
||||
FIELDS = re.compile(r"([a-zA-Z0-9_]+)=([^ ]+)")
|
||||
ARRAY_FIELDS = (
|
||||
"sweep_missing_by_caller", "sweep_exact_fallback_by_caller",
|
||||
"sweep_bound_skip_by_caller", "sweep_tile_free_skip_by_caller",
|
||||
"sweep_vertical_bound_skip_by_caller", "sweep_negative_reach_skip_by_caller",
|
||||
"sweep_positive_reach_skip_by_caller", "density_offgrid_float_by_caller",
|
||||
"density_offgrid_unclamped_by_caller", "density_offgrid_detail_by_caller",
|
||||
)
|
||||
SCALAR_FIELDS = (
|
||||
"density_float_points", "density_lattice_points", "density_unclamped_points",
|
||||
"grad_calls", "grad_memo_hits", "grad_fallback_calls", "midpoint_attempts",
|
||||
"midpoint_memo_hits", "skirt_density_probes", "skirt_surface_vertex",
|
||||
"skirt_surface_vertex_offgrid", "skirt_extruded_vertex",
|
||||
"skirt_extruded_vertex_offgrid", "skirt_clipped_vertex",
|
||||
"skirt_clipped_vertex_offgrid", "skirt_refinement_face",
|
||||
"skirt_refinement_face_offgrid", "skirt_coarse_centroid",
|
||||
"skirt_coarse_centroid_offgrid",
|
||||
)
|
||||
COMPOSER_FIELDS = (
|
||||
"queries", "boundary_band_queries", "column_distance_calls",
|
||||
"carve_segment_visits", "support_segment_visits", "support_capsule_checks",
|
||||
"later_bore_checks", "other_capsule_checks", "mouth_support_calls",
|
||||
"mouth_route_segment_checks", "relief_field_calls", "sweep_segments",
|
||||
"sweep_pairs", "sweep_lookups", "sweep_cache_hits", "sweep_missing",
|
||||
"tile_bound_checks", "tile_bound_rejects", "tile_bound_skipped_iterations",
|
||||
)
|
||||
|
||||
|
||||
def counts(value: str | None) -> list[int]:
|
||||
result = [int(part) for part in value.split(",")] if value else []
|
||||
if len(result) != len(CALLERS):
|
||||
raise ValueError(f"expected {len(CALLERS)} caller counts, got {len(result)}")
|
||||
return result
|
||||
|
||||
|
||||
def run(strate: str) -> dict:
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
log = OUT / f"census-{strate.lower()}.log"
|
||||
camera = ",".join(f"{value:.12f}" for value in CAMERAS[strate]) + ",0,0"
|
||||
command = [
|
||||
"taskset", "-c", "0-11", "prlimit", "--as=6442450944", "--",
|
||||
"timeout", "1740s", str(HEADLESS), str(DATA), "--world-assembler-prototype",
|
||||
"--stream-benchmark", "--stream-pipeline", "--workers=8", "--clip-radius=3",
|
||||
"--max-clip-level=8", "--max-level=8", f"--camera={camera}",
|
||||
"--profile-passage-composer", "--profile-lod-work",
|
||||
]
|
||||
start = time.monotonic()
|
||||
with log.open("w", encoding="utf-8") as stream:
|
||||
result = subprocess.run(command, cwd=ROOT, stdout=stream,
|
||||
stderr=subprocess.STDOUT, check=False, timeout=1740)
|
||||
output = log.read_text(encoding="utf-8", errors="replace")
|
||||
if result.returncode:
|
||||
raise RuntimeError(f"off-grid census failed: exit={result.returncode} {log}")
|
||||
cpu_rows, composer_rows = [], []
|
||||
for pattern, destination, scalar_fields in (
|
||||
(CPU_LINE, cpu_rows, SCALAR_FIELDS),
|
||||
(COMPOSER_LINE, composer_rows, COMPOSER_FIELDS),
|
||||
):
|
||||
for level_text, tail in pattern.findall(output):
|
||||
values = dict(FIELDS.findall(tail))
|
||||
row = {"lod": int(level_text)}
|
||||
row.update({key: int(values.get(key, 0)) for key in scalar_fields})
|
||||
if pattern is CPU_LINE:
|
||||
row.update({key: counts(values.get(key)) for key in ARRAY_FIELDS})
|
||||
destination.append(row)
|
||||
if not cpu_rows or not composer_rows:
|
||||
raise RuntimeError(f"missing tile census output: cpu={len(cpu_rows)} composer={len(composer_rows)} {log}")
|
||||
|
||||
def aggregate(rows: list[dict], fields: tuple[str, ...], maximum_lod: int | None) -> dict:
|
||||
selected = [row for row in rows if maximum_lod is None or row["lod"] <= maximum_lod]
|
||||
result = {field: sum(row[field] for row in selected) for field in fields}
|
||||
if selected and "midpoint_attempts" in fields:
|
||||
for key in ARRAY_FIELDS:
|
||||
result[key] = [sum(row[key][index] for row in selected)
|
||||
for index in range(len(CALLERS))]
|
||||
result["tiles"] = len(selected)
|
||||
result["tiles_by_lod"] = {
|
||||
str(lod): sum(row["lod"] == lod for row in selected)
|
||||
for lod in sorted({row["lod"] for row in selected})
|
||||
}
|
||||
return result
|
||||
|
||||
all_cpu = aggregate(cpu_rows, SCALAR_FIELDS, None)
|
||||
near_cpu = aggregate(cpu_rows, SCALAR_FIELDS, 2)
|
||||
all_composer = aggregate(composer_rows, COMPOSER_FIELDS, None)
|
||||
near_composer = aggregate(composer_rows, COMPOSER_FIELDS, 2)
|
||||
|
||||
def caller_map(values: list[int]) -> dict[str, int]:
|
||||
return dict(zip(CALLERS, values))
|
||||
|
||||
validation = {
|
||||
"near_midpoint_exact_queries_match_unclamped_offgrid":
|
||||
near_cpu["midpoint_attempts"] - near_cpu["midpoint_memo_hits"]
|
||||
== near_cpu["density_offgrid_unclamped_by_caller"][
|
||||
CALLERS.index("mc_midpoint_correction")],
|
||||
"all_sweep_misses_match_caller_totals":
|
||||
sum(all_cpu["sweep_missing_by_caller"])
|
||||
== all_composer["sweep_missing"],
|
||||
"all_sweep_misses_have_exact_disposition":
|
||||
all(sum(all_cpu[key][index] for key in (
|
||||
"sweep_exact_fallback_by_caller", "sweep_bound_skip_by_caller",
|
||||
"sweep_tile_free_skip_by_caller", "sweep_vertical_bound_skip_by_caller",
|
||||
"sweep_negative_reach_skip_by_caller", "sweep_positive_reach_skip_by_caller"))
|
||||
== all_cpu["sweep_missing_by_caller"][index]
|
||||
for index in range(len(CALLERS))),
|
||||
}
|
||||
# Validate each skirt caller independently against the generic density entry counter.
|
||||
skirt_pairs = {
|
||||
"skirt_surface_vertex": "skirt_surface_vertex_offgrid",
|
||||
"skirt_extruded_vertex": "skirt_extruded_vertex_offgrid",
|
||||
"skirt_clipped_vertex": "skirt_clipped_vertex_offgrid",
|
||||
"skirt_refinement_face": "skirt_refinement_face_offgrid",
|
||||
"skirt_coarse_centroid": "skirt_coarse_centroid_offgrid",
|
||||
}
|
||||
skirt_validation = {
|
||||
caller: near_cpu["density_offgrid_float_by_caller"][CALLERS.index(caller)]
|
||||
== near_cpu[probe_count]
|
||||
for caller, probe_count in skirt_pairs.items()
|
||||
}
|
||||
validation["near_skirt_callers_match"] = all(skirt_validation.values())
|
||||
validation["near_skirt_caller_checks"] = skirt_validation
|
||||
unresolved = [all_cpu["sweep_missing_by_caller"][index]
|
||||
- all_cpu["sweep_exact_fallback_by_caller"][index]
|
||||
- all_cpu["sweep_bound_skip_by_caller"][index]
|
||||
- all_cpu["sweep_tile_free_skip_by_caller"][index]
|
||||
- all_cpu["sweep_vertical_bound_skip_by_caller"][index]
|
||||
- all_cpu["sweep_negative_reach_skip_by_caller"][index]
|
||||
- all_cpu["sweep_positive_reach_skip_by_caller"][index]
|
||||
for index in range(len(CALLERS))]
|
||||
row = {
|
||||
"strate": strate, "workers": 8, "exit_code": result.returncode,
|
||||
"wall_s": round(time.monotonic() - start, 3),
|
||||
"caller_order": CALLERS, "near_lod0_2": near_cpu,
|
||||
"all_lods": all_cpu, "near_composer_lod0_2": near_composer,
|
||||
"all_composer_lods": all_composer,
|
||||
"offgrid_float_by_caller": caller_map(all_cpu["density_offgrid_float_by_caller"]),
|
||||
"offgrid_unclamped_by_caller": caller_map(
|
||||
all_cpu["density_offgrid_unclamped_by_caller"]),
|
||||
"offgrid_detail_by_caller": caller_map(all_cpu["density_offgrid_detail_by_caller"]),
|
||||
"sweep_misses_by_caller": caller_map(all_cpu["sweep_missing_by_caller"]),
|
||||
"exact_route_fallbacks_by_caller": caller_map(
|
||||
all_cpu["sweep_exact_fallback_by_caller"]),
|
||||
"feature_bound_skips_by_caller": caller_map(
|
||||
all_cpu["sweep_bound_skip_by_caller"]),
|
||||
"tile_free_skips_by_caller": caller_map(
|
||||
all_cpu["sweep_tile_free_skip_by_caller"]),
|
||||
"vertical_bound_skips_by_caller": caller_map(
|
||||
all_cpu["sweep_vertical_bound_skip_by_caller"]),
|
||||
"negative_reach_skips_by_caller": caller_map(
|
||||
all_cpu["sweep_negative_reach_skip_by_caller"]),
|
||||
"positive_reach_skips_by_caller": caller_map(
|
||||
all_cpu["sweep_positive_reach_skip_by_caller"]),
|
||||
"misses_without_recorded_disposition_by_caller":
|
||||
caller_map(unresolved),
|
||||
"validation": validation, "log": str(log.relative_to(ROOT)),
|
||||
}
|
||||
print("[L101Census] " + json.dumps(row, sort_keys=True), flush=True)
|
||||
return row
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if not HEADLESS.is_file() or not DATA.is_file():
|
||||
raise SystemExit("missing L101 headless binary or owner world")
|
||||
rows = [run(strate) for strate in CAMERAS]
|
||||
(OUT / "summary.json").write_text(json.dumps({"runs": rows}, indent=2) + "\n",
|
||||
encoding="utf-8")
|
||||
return 0 if all(all(row["validation"].values()) for row in rows) else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,144 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compare the direct passage evaluator with the bounded L101 swept tube."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[4]
|
||||
OUT = ROOT / "Saved/Linux/l101/density-compare"
|
||||
DATA = ROOT / "Standalone/data/owner_world.json"
|
||||
HEADLESS = ROOT / "Saved/Linux/l101/bin/VoxelForgeCoreHeadless-l101-clang"
|
||||
CAMERAS = {
|
||||
"SW": (8.0, 16.0, -372.2626953125),
|
||||
"FI": (25.8369140625, 3.19921875, -737.0),
|
||||
"TN": (19.166015625, -14.2001953125, -753.3740234375),
|
||||
"VS": (-20.0625, -5.068359375, -900.353515625),
|
||||
}
|
||||
SUMMARY = re.compile(r"^\[L93PassageReachSummary\] reach_m=4\.00 (.*)$", re.M)
|
||||
COVERAGE = re.compile(r"^\[L93PassageReachCoverage\] (.*)$", re.M)
|
||||
TILE = re.compile(
|
||||
r"^\[L93PassageReachTile\] reach_m=4\.00 key=\(([-\d]+),([-\d]+),([-\d]+),lod(\d+)\) "
|
||||
r"samples=(\d+) changed=(\d+) sign_flips=(\d+) moved_edges=(\d+) "
|
||||
r"added_crossings=(\d+) removed_crossings=(\d+) "
|
||||
r"added_surface_triangles=(\d+) removed_surface_triangles=(\d+) "
|
||||
r"max_displacement_cm=([\deE+.-]+) max_displacement_xyz_vox=\(([^)]*)\) "
|
||||
r"displacement_region=(\d+) normal_vertices=(\d+) max_normal_degrees=([\deE+.-]+) "
|
||||
r"max_normal_xyz_vox=\(([^)]*)\) normal_region=(\d+) changed_mesh=(\d+)$", re.M)
|
||||
VOXEL_SIZE_M = 0.25
|
||||
|
||||
|
||||
def xyz(text: str) -> list[float]:
|
||||
return [float(part) for part in text.split(",")]
|
||||
|
||||
|
||||
def parse_locations(output: str, strate: str) -> tuple[list[dict], list[dict]]:
|
||||
rows = []
|
||||
for match in TILE.finditer(output):
|
||||
(x, y, z, lod, samples, changed, flips, moved, added, removed,
|
||||
added_triangles, removed_triangles, displacement, displacement_xyz,
|
||||
displacement_region, normal_vertices, angle, normal_xyz, normal_region,
|
||||
changed_mesh) = match.groups()
|
||||
disp_vox = xyz(displacement_xyz)
|
||||
normal_vox = xyz(normal_xyz)
|
||||
rows.append({
|
||||
"strate": strate, "tile": [int(x), int(y), int(z), int(lod)],
|
||||
"samples": int(samples), "changed_samples": int(changed),
|
||||
"sign_flips": int(flips), "moved_edges": int(moved),
|
||||
"surface_added_crossings": int(added),
|
||||
"surface_removed_crossings": int(removed),
|
||||
"surface_added_triangles": int(added_triangles),
|
||||
"surface_removed_triangles": int(removed_triangles),
|
||||
"max_displacement_cm": float(displacement),
|
||||
"displacement_location_voxels": disp_vox,
|
||||
"displacement_location_metres": [value * VOXEL_SIZE_M for value in disp_vox],
|
||||
"displacement_region": int(displacement_region),
|
||||
"normal_vertices": int(normal_vertices), "max_normal_degrees": float(angle),
|
||||
"normal_location_voxels": normal_vox,
|
||||
"normal_location_metres": [value * VOXEL_SIZE_M for value in normal_vox],
|
||||
"normal_region": int(normal_region), "changed_mesh": bool(int(changed_mesh)),
|
||||
})
|
||||
top_displacement = sorted(rows, key=lambda row: row["max_displacement_cm"], reverse=True)[:5]
|
||||
top_normals = sorted(rows, key=lambda row: row["max_normal_degrees"], reverse=True)[:5]
|
||||
return top_displacement, top_normals
|
||||
|
||||
|
||||
def top_unique(rows: list[dict], value_key: str, location_key: str) -> list[dict]:
|
||||
chosen, seen = [], set()
|
||||
for row in sorted(rows, key=lambda item: item[value_key], reverse=True):
|
||||
if row[value_key] <= 0.0:
|
||||
continue
|
||||
location = tuple(round(value, 4) for value in row[location_key])
|
||||
if location in seen:
|
||||
continue
|
||||
seen.add(location)
|
||||
chosen.append(row)
|
||||
if len(chosen) == 5:
|
||||
break
|
||||
return chosen
|
||||
|
||||
|
||||
def run(strate: str) -> dict:
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
log = OUT / f"reach-{strate.lower()}.log"
|
||||
camera = ",".join(f"{value:.12f}" for value in CAMERAS[strate]) + ",0,0"
|
||||
command = [
|
||||
"taskset", "-c", "0-11", "prlimit", "--as=6442450944", "--",
|
||||
"timeout", "1740s", str(HEADLESS), str(DATA), "--world-assembler-prototype",
|
||||
"--stream-benchmark", "--stream-pipeline", "--workers=8", "--clip-radius=3",
|
||||
"--max-clip-level=8", "--max-level=8", f"--camera={camera}",
|
||||
"--audit-passage-carve-reach",
|
||||
]
|
||||
start = time.monotonic()
|
||||
with log.open("w", encoding="utf-8") as stream:
|
||||
result = subprocess.run(command, cwd=ROOT, stdout=stream,
|
||||
stderr=subprocess.STDOUT, check=False, timeout=1740)
|
||||
output = log.read_text(encoding="utf-8", errors="replace")
|
||||
summary_match, coverage_match = SUMMARY.search(output), COVERAGE.search(output)
|
||||
if summary_match is None or coverage_match is None or result.returncode in (-9, 124):
|
||||
raise RuntimeError(f"incomplete carve-reach audit: exit={result.returncode} {log}")
|
||||
reach = dict(re.findall(r"([a-zA-Z0-9_]+)=([^ ]+)", summary_match.group(1)))
|
||||
coverage = dict(re.findall(r"([a-zA-Z0-9_]+)=([^ ]+)", coverage_match.group(1)))
|
||||
top_displacement, top_normals = parse_locations(output, strate)
|
||||
row = {
|
||||
"strate": strate, "exit_code": result.returncode,
|
||||
"wall_s": round(time.monotonic() - start, 3), "coverage": coverage,
|
||||
"bounded_sweep_summary": reach,
|
||||
"bounded_sweep_tile_rows": len(TILE.findall(output)),
|
||||
"top_displacement_tiles": top_displacement,
|
||||
"top_normal_tiles": top_normals,
|
||||
"log": str(log.relative_to(ROOT)),
|
||||
}
|
||||
print("[L101DensityCompareRun] " + json.dumps(row, sort_keys=True), flush=True)
|
||||
return row
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if not HEADLESS.is_file() or not DATA.is_file():
|
||||
raise SystemExit(f"missing input: {HEADLESS if not HEADLESS.is_file() else DATA}")
|
||||
if "--summarize-only" in sys.argv:
|
||||
rows = json.loads((OUT / "runs-partial.json").read_text(encoding="utf-8"))
|
||||
else:
|
||||
rows = []
|
||||
for strate in CAMERAS:
|
||||
rows.append(run(strate))
|
||||
(OUT / "runs-partial.json").write_text(json.dumps(rows, indent=2) + "\n",
|
||||
encoding="utf-8")
|
||||
all_displacements = [row for result in rows for row in result["top_displacement_tiles"]]
|
||||
all_normals = [row for result in rows for row in result["top_normal_tiles"]]
|
||||
top_displacement = top_unique(all_displacements, "max_displacement_cm",
|
||||
"displacement_location_metres")
|
||||
top_normals = top_unique(all_normals, "max_normal_degrees", "normal_location_metres")
|
||||
summary = {"runs": rows, "top_displacement_tiles": top_displacement,
|
||||
"top_normal_tiles": top_normals}
|
||||
(OUT / "summary.json").write_text(json.dumps(summary, indent=2) + "\n",
|
||||
encoding="utf-8")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,130 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check L101 whole-view hashes across W1/W8/W12 and Clang/GCC."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[4]
|
||||
OUT = ROOT / "Saved/Linux/l101/hashes"
|
||||
DATA = ROOT / "Standalone/data/owner_world.json"
|
||||
BINARIES = {
|
||||
"Clang": ROOT / "Saved/Linux/l101/bin/VoxelForgeCoreHeadless-l101-clang",
|
||||
"GCC": ROOT / "Saved/Linux/l101/bin/VoxelForgeCoreHeadless-l101-gcc",
|
||||
}
|
||||
L100_BINARY = ROOT / "Saved/Linux/l100/bin/VoxelForgeCoreHeadless-l100-clang"
|
||||
CAMERAS = {
|
||||
"SW": (8.0, 16.0, -372.2626953125),
|
||||
"FI": (25.8369140625, 3.19921875, -737.0),
|
||||
"TN": (19.166015625, -14.2001953125, -753.3740234375),
|
||||
"VS": (-20.0625, -5.068359375, -900.353515625),
|
||||
}
|
||||
TILE_HASH = re.compile(
|
||||
r"^\[PipelineTileHash\] lod=(-?\d+) x=(-?\d+) y=(-?\d+) z=(-?\d+) "
|
||||
r"density=(0x[0-9a-f]+) mesh=(0x[0-9a-f]+) payload=(0x[0-9a-f]+) "
|
||||
r"topology=(0x[0-9a-f]+)$", re.M)
|
||||
WHOLE_HASH = re.compile(
|
||||
r"^\[PipelineWholeViewHash\] tiles=(\d+) density=(0x[0-9a-f]+) "
|
||||
r"mesh=(0x[0-9a-f]+) payload=(0x[0-9a-f]+) topology=(0x[0-9a-f]+)$", re.M)
|
||||
FIELDS = ("density", "mesh", "payload", "topology")
|
||||
|
||||
|
||||
def run(strate: str, compiler: str, workers: int, binary: Path | None = None) -> dict:
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
log = OUT / f"{strate.lower()}-{compiler.lower()}-w{workers}.log"
|
||||
camera = ",".join(f"{value:.12f}" for value in CAMERAS[strate]) + ",0,0"
|
||||
selected_binary = binary or BINARIES[compiler]
|
||||
command = [
|
||||
"taskset", "-c", "0-11", "prlimit", "--as=6442450944", "--",
|
||||
"timeout", "1740s", str(selected_binary), str(DATA),
|
||||
"--world-assembler-prototype", "--stream-benchmark", "--stream-pipeline",
|
||||
f"--workers={workers}", "--clip-radius=3", "--max-clip-level=8",
|
||||
"--max-level=8", f"--camera={camera}", "--stream-hash-tiles",
|
||||
]
|
||||
start = time.monotonic()
|
||||
with log.open("w", encoding="utf-8") as stream:
|
||||
result = subprocess.run(command, cwd=ROOT, stdout=stream,
|
||||
stderr=subprocess.STDOUT, check=False, timeout=1740)
|
||||
output = log.read_text(encoding="utf-8", errors="replace")
|
||||
hashes = {
|
||||
(int(m.group(1)), int(m.group(2)), int(m.group(3)), int(m.group(4))):
|
||||
m.group(5, 6, 7, 8) for m in TILE_HASH.finditer(output)
|
||||
}
|
||||
whole = WHOLE_HASH.search(output)
|
||||
stamp = re.search(r"^\[VoxelForgeSourceStamp\] sha256=([0-9a-f]{64})$", output, re.M)
|
||||
if (result.returncode or whole is None or stamp is None
|
||||
or len(hashes) != int(whole.group(1))):
|
||||
raise RuntimeError(f"hash run failed or incomplete: exit={result.returncode} {log}")
|
||||
row = {
|
||||
"strate": strate, "compiler": compiler, "workers": workers,
|
||||
"source_stamp": stamp.group(1), "exit_code": result.returncode,
|
||||
"wall_s": round(time.monotonic() - start, 3), "tile_count": len(hashes),
|
||||
"tiles_by_lod": {str(lod): sum(1 for key in hashes if key[0] == lod)
|
||||
for lod in sorted({key[0] for key in hashes})},
|
||||
"whole_hash": list(whole.groups()), "tile_hashes": hashes,
|
||||
"log": str(log.relative_to(ROOT)),
|
||||
}
|
||||
print("[L101HashRun] " + json.dumps(
|
||||
{key: value for key, value in row.items() if key != "tile_hashes"},
|
||||
sort_keys=True), flush=True)
|
||||
return row
|
||||
|
||||
|
||||
def compare(strate: str, reference: dict, candidate: dict) -> dict:
|
||||
keys = set(reference["tile_hashes"]) | set(candidate["tile_hashes"])
|
||||
mismatches = {
|
||||
field: sum(reference["tile_hashes"].get(key, (None,) * 4)[index]
|
||||
!= candidate["tile_hashes"].get(key, (None,) * 4)[index]
|
||||
for key in keys)
|
||||
for index, field in enumerate(FIELDS)
|
||||
}
|
||||
row = {
|
||||
"strate": strate, "reference": f"{reference['compiler']}-w{reference['workers']}",
|
||||
"candidate": f"{candidate['compiler']}-w{candidate['workers']}",
|
||||
"tile_counts_match": reference["tile_count"] == candidate["tile_count"],
|
||||
"whole_hashes_match": reference["whole_hash"] == candidate["whole_hash"],
|
||||
"tile_hashes_match": all(value == 0 for value in mismatches.values())
|
||||
and reference["tile_hashes"].keys() == candidate["tile_hashes"].keys(),
|
||||
"component_mismatch_tiles": mismatches,
|
||||
"tiles_reference": reference["tile_count"], "tiles_candidate": candidate["tile_count"],
|
||||
"reference_whole_hash": reference["whole_hash"],
|
||||
"candidate_whole_hash": candidate["whole_hash"],
|
||||
}
|
||||
print("[L101HashComparison] " + json.dumps(row, sort_keys=True), flush=True)
|
||||
return row
|
||||
|
||||
|
||||
def main() -> int:
|
||||
for binary in (*BINARIES.values(), L100_BINARY, DATA):
|
||||
if not binary.is_file():
|
||||
raise SystemExit(f"missing input: {binary}")
|
||||
detail, comparisons = [], []
|
||||
for strate in CAMERAS:
|
||||
group = {(compiler, workers): run(strate, compiler, workers)
|
||||
for compiler in BINARIES for workers in (1, 8, 12)}
|
||||
reference = group[("Clang", 1)]
|
||||
for key, candidate in group.items():
|
||||
if key == ("Clang", 1):
|
||||
continue
|
||||
comparisons.append(compare(strate, reference, candidate))
|
||||
l100 = run(strate, "L100-Clang", 1, L100_BINARY)
|
||||
l100_comparison = compare(strate, l100, reference)
|
||||
l100_comparison["comparison_scope"] = "L100 versus L101 output identity"
|
||||
comparisons.append(l100_comparison)
|
||||
detail.extend({key: value for key, value in row.items() if key != "tile_hashes"}
|
||||
for row in group.values())
|
||||
(OUT / "runs-partial.json").write_text(json.dumps(
|
||||
{"runs": detail, "comparisons": comparisons}, indent=2) + "\n",
|
||||
encoding="utf-8")
|
||||
summary = {"runs": detail, "comparisons": comparisons}
|
||||
(OUT / "summary.json").write_text(json.dumps(summary, indent=2) + "\n",
|
||||
encoding="utf-8")
|
||||
return 0 if all(row["tile_counts_match"] and row["whole_hashes_match"]
|
||||
and row["tile_hashes_match"] for row in comparisons) else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env python3
|
||||
"""L80 protocol with balanced, rotating L98/L99/L100/L101 order."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import statistics
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[4]
|
||||
OUT = ROOT / "Saved/Linux/l101/readiness/l98-l99-l100-l101-exact-bounds"
|
||||
DATA = ROOT / "Standalone/data/owner_world.json"
|
||||
BINARIES = {
|
||||
"L98": ROOT / "Saved/Linux/l98/bin/VoxelForgeCoreHeadless-l98-clang19",
|
||||
"L99": ROOT / "Saved/Linux/l99/bin/VoxelForgeCoreHeadless-l99-clang",
|
||||
"L100": ROOT / "Saved/Linux/l100/bin/VoxelForgeCoreHeadless-l100-clang",
|
||||
"L101": ROOT / "Saved/Linux/l101/bin/VoxelForgeCoreHeadless-l101-clang",
|
||||
}
|
||||
CAMERAS = {
|
||||
"SW": (8.0, 16.0, -372.2626953125),
|
||||
"FI": (25.8369140625, 3.19921875, -737.0),
|
||||
"TN": (19.166015625, -14.2001953125, -753.3740234375),
|
||||
"VS": (-20.0625, -5.068359375, -900.353515625),
|
||||
}
|
||||
PIPELINE = re.compile(r"^\[StreamPipeline\] (.*)$", re.M)
|
||||
FIELDS = re.compile(r"([a-zA-Z0-9_]+)=([^ ]+)")
|
||||
NEAR = re.compile(r"^\[WorldAssemblerNearLODReady\] lods=0-2 tiles=(\d+)/(\d+) "
|
||||
r"cold_ready_s=([0-9.]+)$", re.M)
|
||||
STAMP = re.compile(r"^\[VoxelForgeSourceStamp\] sha256=([0-9a-f]{64})$", re.M)
|
||||
|
||||
|
||||
def run(strate: str, workers: int, label: str, phase: str, rep: int) -> dict:
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
log = OUT / f"w{workers}-{strate.lower()}-{label.lower()}-{phase}{rep}.log"
|
||||
camera = ",".join(f"{value:.12f}" for value in CAMERAS[strate]) + ",0,0"
|
||||
command = [
|
||||
"taskset", "-c", "0-11", "prlimit", "--as=6442450944", "--",
|
||||
"env", "timeout", "1740s", str(BINARIES[label]), str(DATA),
|
||||
"--world-assembler-prototype", "--stream-benchmark", "--stream-pipeline",
|
||||
f"--workers={workers}", "--clip-radius=3", "--max-clip-level=8",
|
||||
"--max-level=8", f"--camera={camera}", "--profile-lod-work",
|
||||
"--stream-scheduler-policy=3",
|
||||
]
|
||||
start = time.monotonic()
|
||||
with log.open("w", encoding="utf-8") as stream:
|
||||
result = subprocess.run(command, cwd=ROOT, stdout=stream,
|
||||
stderr=subprocess.STDOUT, check=False, timeout=1740)
|
||||
output = log.read_text(encoding="utf-8", errors="replace")
|
||||
pipeline_match, near_match, stamp_match = (
|
||||
PIPELINE.search(output), NEAR.search(output), STAMP.search(output))
|
||||
if result.returncode or pipeline_match is None or near_match is None or stamp_match is None:
|
||||
raise RuntimeError(f"readiness run failed: exit={result.returncode} {log}")
|
||||
pipeline = dict(FIELDS.findall(pipeline_match.group(1)))
|
||||
row = {
|
||||
"strate": strate, "workers": workers, "binary": label, "phase": phase,
|
||||
"replicate": rep, "exit_code": result.returncode,
|
||||
"source_stamp": stamp_match.group(1), "wall_s": round(time.monotonic() - start, 3),
|
||||
"lod02_tiles": f"{near_match.group(1)}/{near_match.group(2)}",
|
||||
"lod02_ready_s": float(near_match.group(3)),
|
||||
"whole_view_ready_s": float(pipeline["whole_view_ready"].removesuffix("s")),
|
||||
"whole_tiles": pipeline["whole_tiles"].split("/")[0],
|
||||
"log": str(log.relative_to(ROOT)),
|
||||
}
|
||||
print("[L101ReadinessRun] " + json.dumps(row, sort_keys=True), flush=True)
|
||||
return row
|
||||
|
||||
|
||||
def main() -> int:
|
||||
for binary in (*BINARIES.values(), DATA):
|
||||
if not binary.is_file():
|
||||
raise SystemExit(f"missing input: {binary}")
|
||||
rows = []
|
||||
labels = tuple(BINARIES)
|
||||
orders = tuple(labels[index:] + labels[:index] for index in range(len(labels)))
|
||||
for workers in (8, 12):
|
||||
for strate in CAMERAS:
|
||||
for label in labels:
|
||||
run(strate, workers, label, "warmup", 0)
|
||||
for strate in CAMERAS:
|
||||
for rep, order in enumerate(orders, 1):
|
||||
for label in order:
|
||||
rows.append(run(strate, workers, label, "measured", rep))
|
||||
(OUT / "runs-partial.json").write_text(json.dumps(rows, indent=2) + "\n",
|
||||
encoding="utf-8")
|
||||
summary = []
|
||||
for workers in (8, 12):
|
||||
for strate in CAMERAS:
|
||||
group = [row for row in rows if row["workers"] == workers
|
||||
and row["strate"] == strate]
|
||||
item = {"workers": workers, "strate": strate, "measured_runs": 3}
|
||||
for label in labels:
|
||||
subset = [row for row in group if row["binary"] == label]
|
||||
for metric in ("lod02_ready_s", "whole_view_ready_s"):
|
||||
values = [row[metric] for row in subset]
|
||||
item[f"{label}_{metric}_runs"] = values
|
||||
item[f"{label}_{metric}_median"] = statistics.median(values)
|
||||
item[f"{label}_process_wall_s_runs"] = [row["wall_s"] for row in subset]
|
||||
summary.append(item)
|
||||
(OUT / "summary.json").write_text(json.dumps(
|
||||
{"comparison_triple": labels, "runs": rows, "summary": summary}, indent=2) + "\n",
|
||||
encoding="utf-8")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,426 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run one pinned null-RHI SW/FI arrival for Linux 101."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from collections import Counter
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[4]
|
||||
HOST = Path("/home/jahni/VFHost")
|
||||
HOST_PLUGIN_SAVED = HOST / "Plugins/VoxelForge/Saved/Linux/l101"
|
||||
HOST_SAVED = HOST / "Saved/Linux/l101"
|
||||
EDITOR = Path("/home/jahni/UnrealEngine/Engine/Binaries/Linux/UnrealEditor-Cmd")
|
||||
OUT = ROOT / "Saved/Linux/l101/unreal-arrival"
|
||||
SPAWNS_M = {
|
||||
"FI": (25.8369140625, 3.19921875, -737.0),
|
||||
"SW": (25.8369140625, 3.19921875, -200.0),
|
||||
}
|
||||
MOVE_START_S = 20.0
|
||||
MOVE_DURATION_S = 12.0
|
||||
EXIT_S = MOVE_START_S + MOVE_DURATION_S + 5.0
|
||||
RUN_TIMEOUT_S = 600
|
||||
|
||||
|
||||
def read_events(path: Path) -> list[dict]:
|
||||
events = []
|
||||
if path.is_file():
|
||||
for line in path.read_text(encoding="utf-8", errors="replace").splitlines():
|
||||
if line.startswith("{"):
|
||||
try:
|
||||
events.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
return events
|
||||
|
||||
|
||||
def percentile(values: list[float], fraction: float) -> float | None:
|
||||
if not values:
|
||||
return None
|
||||
ordered = sorted(values)
|
||||
return ordered[min(len(ordered) - 1, int(round((len(ordered) - 1) * fraction)))]
|
||||
|
||||
|
||||
def run(stratum: str, repeat: int, phase: str, ready_depth: int,
|
||||
background_cap: int, worker_reserve: int, helper_background: int,
|
||||
tile_background_low: int, tile_background_normal: int, startup_boost: int,
|
||||
tick_logs: bool, insights: bool) -> dict:
|
||||
label = (f"{phase}-{stratum.lower()}-rd{ready_depth}-bg{background_cap}"
|
||||
f"-reserve{worker_reserve}-helperbg{helper_background}"
|
||||
f"-tilelow{tile_background_low}-tilenormal{tile_background_normal}"
|
||||
f"-ticklog{int(tick_logs)}"
|
||||
f"-boost{startup_boost}-{repeat}")
|
||||
trace = OUT / f"{label}.trace.jsonl"
|
||||
utrace = OUT / f"{label}.utrace"
|
||||
abslog = OUT / f"{label}.log"
|
||||
stdout = OUT / f"{label}.stdout.log"
|
||||
output = HOST_PLUGIN_SAVED / "output" / label
|
||||
userdir = HOST_SAVED / "user" / label
|
||||
zendata = HOST_SAVED / "zen" / label
|
||||
ddc = HOST_SAVED / "DDC"
|
||||
for directory in (OUT, output, userdir, zendata, ddc):
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
trace.unlink(missing_ok=True)
|
||||
utrace.unlink(missing_ok=True)
|
||||
spawn_cm = ",".join(f"{value * 100.0:.3f}" for value in SPAWNS_M[stratum])
|
||||
command = [
|
||||
str(EDITOR), str(HOST / "VFHost.uproject"), "-nullrhi", "-unattended",
|
||||
"-nop4", "-nosplash", "-NoSound", "-corelimit=12", "-NoZenAutoLaunch",
|
||||
"-notraceserver", "-nocrashreports",
|
||||
"-DisablePlugins=AndroidFileServer",
|
||||
"-ddc=NoZenLocalFallback", f"-userdir={userdir}", f"-ZenDataPath={zendata}",
|
||||
f"-LocalDataCachePath={ddc}", "-run=VoxelForgeExplore", "-seed=0",
|
||||
"-archetype=TunnelNetwork", f"-slot={1 if stratum == 'FI' else 0}",
|
||||
"-modes=export", "-gameconfig=1", "-coreworldtest=1",
|
||||
"-ownerfourstrateplan=1", "-corevoxelsizecm=25", "-coreislandtest=0",
|
||||
"-corefallbacktest=0", "-coreofftest=0", "-coreplantest=0",
|
||||
"-settingsref=/Game/VoxelForge/DA_Settings.DA_Settings", "-corehybridmesher=0",
|
||||
f"-out={output}", "-voxel.StreamingCoarseFirst=1",
|
||||
"-voxel.StreamingDynamicNearReserve=1",
|
||||
"-voxel.StreamingLegacyPendingResultCap=0",
|
||||
"-voxel.CoreRegionConcurrentAdmission=1",
|
||||
"-voxel.StreamingResultBacklogCap=256",
|
||||
f"-voxel.StreamingReadyQueueDepth={ready_depth}",
|
||||
f"-voxel.StreamingBackgroundWorkerCap={background_cap}",
|
||||
f"-voxel.StreamingBackgroundWorkerReserve={worker_reserve}",
|
||||
f"-voxel.CoreParallelForBackgroundPriority={helper_background}",
|
||||
f"-voxel.StreamingTileBackgroundLow={tile_background_low}",
|
||||
f"-voxel.StreamingTileBackgroundNormal={tile_background_normal}",
|
||||
f"-voxel.TestLogTickFunctions={int(tick_logs)}",
|
||||
"-voxel.CoreColumnCacheDirectMapped=0", "-voxel.CoreMCGradientMemoization=1",
|
||||
"-voxel.StreamingCollisionLookahead=1",
|
||||
f"-voxel.StreamingStartupApplyBoost={startup_boost}",
|
||||
"-voxel.TestMove=1", "-voxel.TestMoveSpeedCmPerSecond=140.0",
|
||||
f"-voxel.TestMoveStartSeconds={MOVE_START_S:.1f}",
|
||||
f"-voxel.TestMoveDurationSeconds={MOVE_DURATION_S:.1f}",
|
||||
f"-voxel.TestExitSeconds={EXIT_S:.1f}", "-voxel.TestMoveDirection=1,0,0",
|
||||
"-voxel.StreamingSchedulerPolicy=3", "-voxel.TestExitOnSteady=0",
|
||||
"-coreworldstreamtrace=1", f"-voxel.StartupTraceSpawnCm={spawn_cm}",
|
||||
"-voxel.StartupTraceUntilEndPlay=1", "-voxel.StartupTraceThroughCrossing=1",
|
||||
"-voxel.StartupTraceMaxTiles=16384", f"-voxel.StartupTraceFile={trace}",
|
||||
f"-abslog={abslog}",
|
||||
]
|
||||
if insights:
|
||||
command += ["-trace=cpu,task,frame", "-statnamedevents",
|
||||
f"-tracefile={utrace}", "-tracefiletrunc"]
|
||||
wrapped = ["taskset", "-c", "0-11", "prlimit", "--as=6442450944", "--",
|
||||
"timeout", f"{RUN_TIMEOUT_S}s", *command]
|
||||
started = time.monotonic()
|
||||
with stdout.open("w", encoding="utf-8") as stream:
|
||||
result = subprocess.run(wrapped, cwd=ROOT, stdout=stream,
|
||||
stderr=subprocess.STDOUT, check=False, timeout=RUN_TIMEOUT_S)
|
||||
|
||||
events = read_events(trace)
|
||||
worker_backend_config = next(
|
||||
(e for e in events if e.get("type") == "worker_backend_config"), {})
|
||||
worker_cap_readback = next(
|
||||
(e for e in events if e.get("type") == "streaming_worker_cap_readback"), {})
|
||||
samples = [e for e in events if e.get("type") == "streaming_sample"]
|
||||
arrival = [e for e in samples if float(e.get("t_s", 0.0)) < MOVE_START_S]
|
||||
full = next((e for e in arrival
|
||||
if sum(e.get("desired_by_lod", [0] * 9)[:3]) > 0
|
||||
and sum(e.get("settled_by_lod", [0] * 9)[:3])
|
||||
>= sum(e.get("desired_by_lod", [0] * 9)[:3])), None)
|
||||
first_admission = next((e for e in events if e.get("type") == "scheduler_admission"
|
||||
and e.get("desired_rebuilt") == 1), None)
|
||||
request_events = [e for e in events if e.get("type") == "request"]
|
||||
initial_request_events = [e for e in request_events if e.get("epoch") == 0]
|
||||
tile_by_ordinal = {e.get("request_ordinal"): e for e in events
|
||||
if e.get("type") == "tile"}
|
||||
collision_count = sum(1 for e in events if e.get("type") == "collision_ready"
|
||||
and e.get("level", 9) <= 2
|
||||
and full and e.get("t_s", 1e30) <= full["t_s"])
|
||||
initial_tiles = [tile_by_ordinal[r.get("request_ordinal")] for r in initial_request_events
|
||||
if full and r.get("t_s", 1e30) <= full["t_s"]
|
||||
and r.get("request_ordinal") in tile_by_ordinal
|
||||
and tile_by_ordinal[r["request_ordinal"]].get("level", 9) <= 2]
|
||||
burst_start_s = float(first_admission["t_s"]) if first_admission else -1.0
|
||||
burst_end_s = float(full["t_s"]) if full else -1.0
|
||||
work_by_frame = {int(e.get("frame", -1)): e for e in events
|
||||
if e.get("type") == "game_thread_work_frame"}
|
||||
work_values = [e for e in work_by_frame.values()
|
||||
if burst_start_s <= float(e.get("t_s", -1.0)) <= burst_end_s]
|
||||
arrival_frame_ids = {int(e.get("frame", -1)) for e in work_values}
|
||||
scheduler_by_frame = {int(e.get("frame", -1)): e for e in events
|
||||
if e.get("type") == "scheduler_admission"}
|
||||
scheduler_values = [e for e in events if e.get("type") == "scheduler_admission"
|
||||
and burst_start_s <= float(e.get("t_s", -1.0)) <= burst_end_s]
|
||||
apply_by_frame = {int(e.get("frame", -1)): e for e in events
|
||||
if e.get("type") == "apply_frame"}
|
||||
commandlet_tick_by_frame = {int(e.get("frame", -1)): e for e in events
|
||||
if e.get("type") == "commandlet_world_tick_frame"}
|
||||
garbage_collect_by_frame = {int(e.get("frame", -1)): e for e in events
|
||||
if e.get("type") == "garbage_collect"}
|
||||
# apply_frame N+1 records the interval containing AVoxelWorld::Tick N. Its apply_ms
|
||||
# belongs to Tick N+1, so use apply_by_frame[N] for the same-tick apply breakdown.
|
||||
near_intervals = [apply_by_frame[frame + 1] for frame in arrival_frame_ids
|
||||
if frame + 1 in apply_by_frame]
|
||||
rmc_by_frame: dict[int, dict[str, float]] = {}
|
||||
for event in events:
|
||||
frame = int(event.get("frame", -1))
|
||||
if event.get("type") != "rmc_submit" or frame not in arrival_frame_ids:
|
||||
continue
|
||||
values = rmc_by_frame.setdefault(frame, {
|
||||
"component_acquire_ms": 0.0,
|
||||
"create_section_group_ms": 0.0,
|
||||
"create_sections_ms": 0.0,
|
||||
"collision_handoff_ms": 0.0,
|
||||
})
|
||||
for key in values:
|
||||
values[key] += float(event.get(key, 0.0))
|
||||
collision_completion_by_frame: dict[int, float] = {}
|
||||
for event in events:
|
||||
frame = int(event.get("frame", -1))
|
||||
if event.get("type") == "collision_completion" and frame in arrival_frame_ids:
|
||||
collision_completion_by_frame[frame] = collision_completion_by_frame.get(frame, 0.0) \
|
||||
+ float(event.get("handler_ms", 0.0))
|
||||
near_requests = [r for r in request_events if full and r.get("t_s", 1e30) <= full["t_s"]
|
||||
and r.get("level", 9) <= 2]
|
||||
initial_lod0_counts = Counter(
|
||||
tuple(r.get("tile", ())) for r in near_requests if r.get("level") == 0)
|
||||
repeated_lod0_keys = [
|
||||
{"tile": list(tile), "requests": count}
|
||||
for tile, count in sorted(initial_lod0_counts.items()) if count > 1]
|
||||
initial_scheduler_frames = [e for e in events
|
||||
if e.get("type") == "scheduler_admission" and full
|
||||
and e.get("t_s", 1e30) <= full["t_s"]]
|
||||
worst_interval = max(near_intervals, key=lambda e: float(e.get("frame_ms", 0.0)), default=None)
|
||||
worst_interval_frame_id = int(worst_interval.get("frame", -1)) if worst_interval else -1
|
||||
worst_work_frame_id = worst_interval_frame_id - 1
|
||||
worst_work = work_by_frame.get(worst_work_frame_id, {})
|
||||
worst_apply = apply_by_frame.get(worst_work_frame_id, {})
|
||||
worst_scheduler = scheduler_by_frame.get(worst_work_frame_id, {})
|
||||
worst_rmc = rmc_by_frame.get(worst_work_frame_id, {})
|
||||
worst_collision_completion_ms = collision_completion_by_frame.get(worst_work_frame_id, 0.0)
|
||||
rmc_values = [rmc_by_frame.get(frame, {}) for frame in arrival_frame_ids]
|
||||
apply_values = [apply_by_frame[frame] for frame in arrival_frame_ids
|
||||
if frame in apply_by_frame]
|
||||
commandlet_tick_values = [commandlet_tick_by_frame[frame] for frame in arrival_frame_ids
|
||||
if frame in commandlet_tick_by_frame]
|
||||
garbage_collect_values = [garbage_collect_by_frame[frame] for frame in arrival_frame_ids
|
||||
if frame in garbage_collect_by_frame]
|
||||
|
||||
def work_stat(name: str, fraction: float | None = None) -> float | None:
|
||||
values = [float(e.get(name, 0.0)) for e in work_values]
|
||||
return percentile(values, fraction) if fraction is not None else max(values, default=0.0)
|
||||
|
||||
def scheduler_stat(name: str, fraction: float | None = None) -> float | None:
|
||||
values = [float(e.get(name, 0.0)) for e in scheduler_values]
|
||||
return percentile(values, fraction) if fraction is not None else max(values, default=0.0)
|
||||
|
||||
def rmc_stat(name: str, fraction: float | None = None) -> float | None:
|
||||
values = [float(e.get(name, 0.0)) for e in rmc_values]
|
||||
return percentile(values, fraction) if fraction is not None else max(values, default=0.0)
|
||||
|
||||
def commandlet_stat(name: str, fraction: float | None = None) -> float | None:
|
||||
values = [float(e.get(name, 0.0)) for e in commandlet_tick_values]
|
||||
return percentile(values, fraction) if fraction is not None else max(values, default=0.0)
|
||||
|
||||
row = {
|
||||
"label": label, "stratum": stratum, "repeat": repeat, "phase": phase,
|
||||
"ready_queue_depth": ready_depth,
|
||||
"requested_background_worker_cap": background_cap,
|
||||
"background_worker_cap": worker_cap_readback.get(
|
||||
"streaming_background_worker_cap", background_cap),
|
||||
"requested_background_worker_reserve": worker_reserve,
|
||||
"background_worker_reserve": worker_backend_config.get(
|
||||
"streaming_background_worker_reserve", worker_cap_readback.get(
|
||||
"background_worker_reserve", worker_reserve)),
|
||||
"requested_core_parallel_for_background_priority": helper_background,
|
||||
"core_parallel_for_background_priority": worker_backend_config.get(
|
||||
"core_parallel_for_background_priority", helper_background),
|
||||
"requested_streaming_tile_background_low": tile_background_low,
|
||||
"streaming_tile_background_low": worker_backend_config.get(
|
||||
"streaming_tile_background_low", tile_background_low),
|
||||
"requested_streaming_tile_background_normal": tile_background_normal,
|
||||
"streaming_tile_background_normal": worker_backend_config.get(
|
||||
"streaming_tile_background_normal", tile_background_normal),
|
||||
"effective_tile_slots": worker_backend_config.get("effective_tile_slots"),
|
||||
"background_worker_threads": worker_backend_config.get("taskgraph_background_threads"),
|
||||
"startup_apply_boost": startup_boost, "insights": insights,
|
||||
"engine_tick_logs": tick_logs,
|
||||
"exit_code": result.returncode,
|
||||
"wall_s": round(time.monotonic() - started, 3),
|
||||
"desired_t_s": next((e.get("t_s") for e in events if e.get("type") == "desired_set"), None),
|
||||
"first_admission_t_s": first_admission.get("t_s") if first_admission else None,
|
||||
"lod02_full_settled_t_s": full.get("t_s") if full else None,
|
||||
"near_settle_from_first_admission_s": round(
|
||||
float(full["t_s"]) - float(first_admission["t_s"]), 4)
|
||||
if full and first_admission else None,
|
||||
"desired_by_lod": full.get("desired_by_lod", [])[:3] if full else None,
|
||||
"settled_by_lod": full.get("settled_by_lod", [])[:3] if full else None,
|
||||
"near_admitted_tiles": len(near_requests),
|
||||
"initial_lod0_admissions": sum(initial_lod0_counts.values()),
|
||||
"repeated_lod0_request_keys_before_settle": repeated_lod0_keys,
|
||||
"peak_result_backlog_before_settle": max(
|
||||
(int(e.get("result_backlog", 0)) for e in initial_scheduler_frames), default=0),
|
||||
"last_admission_t_s": max((e.get("t_s", 0.0) for e in near_requests), default=None),
|
||||
"near_tile_events": len(initial_tiles),
|
||||
"near_worker_seconds": round(sum(e.get("worker_task_s", 0.0) for e in initial_tiles), 4),
|
||||
"near_core_seconds": round(sum(e.get("core_generate_s", 0.0) for e in initial_tiles), 4),
|
||||
"collision_ready_by_settle": collision_count,
|
||||
"burst_frame_count": len(near_intervals),
|
||||
"burst_frame_ms_p95": round(sorted(e.get("frame_ms", 0.0) for e in near_intervals)[
|
||||
min(len(near_intervals) - 1, int(.95 * (len(near_intervals) - 1)))], 3)
|
||||
if near_intervals else None,
|
||||
"burst_frame_ms_max": round(max((e.get("frame_ms", 0.0) for e in near_intervals), default=0.0), 3),
|
||||
"burst_full_world_tick_ms_p99": round(commandlet_stat("world_tick_ms", .99) or 0.0, 3),
|
||||
"burst_full_world_tick_ms_max": round(commandlet_stat("world_tick_ms") or 0.0, 3),
|
||||
"burst_world_pre_actor_ms_p99": round(commandlet_stat("world_pre_actor_ms", .99) or 0.0, 3),
|
||||
"burst_world_pre_actor_ms_max": round(commandlet_stat("world_pre_actor_ms") or 0.0, 3),
|
||||
"burst_world_actor_phase_ms_p99": round(commandlet_stat("world_actor_phase_ms", .99) or 0.0, 3),
|
||||
"burst_world_actor_phase_ms_max": round(commandlet_stat("world_actor_phase_ms") or 0.0, 3),
|
||||
"burst_world_post_actor_ms_p99": round(commandlet_stat("world_post_actor_ms", .99) or 0.0, 3),
|
||||
"burst_world_post_actor_ms_max": round(commandlet_stat("world_post_actor_ms") or 0.0, 3),
|
||||
"burst_commandlet_readiness_check_ms_max": round(
|
||||
commandlet_stat("readiness_check_ms") or 0.0, 3),
|
||||
"burst_commandlet_sleep_ms_p99": round(commandlet_stat("sleep_ms", .99) or 0.0, 3),
|
||||
"burst_commandlet_sleep_ms_max": round(commandlet_stat("sleep_ms") or 0.0, 3),
|
||||
"burst_commandlet_loop_ms_max": round(commandlet_stat("loop_ms") or 0.0, 3),
|
||||
"burst_gc_count": len(garbage_collect_values),
|
||||
"burst_gc_duration_ms_max": round(max(
|
||||
(float(e.get("duration_ms", 0.0)) for e in garbage_collect_values), default=0.0), 3),
|
||||
"burst_max_meshes_per_frame": max((e.get("meshes_applied", 0) for e in apply_values), default=0),
|
||||
"burst_max_apply_ms": round(max((e.get("apply_ms", 0.0) for e in apply_values), default=0.0), 3),
|
||||
"burst_max_admit_per_frame": max((sum(e.get("submitted_by_lod", [0] * 9)[:3])
|
||||
for e in scheduler_values), default=0),
|
||||
"burst_frame_timing_count": len(work_values),
|
||||
"burst_active_generation_workers_p95": round(work_stat("active_generation_workers", .95) or 0.0, 3),
|
||||
"burst_active_generation_workers_max": round(work_stat("active_generation_workers") or 0.0, 3),
|
||||
"burst_active_tile_tasks_max": round(work_stat("active_tile_tasks") or 0.0, 3),
|
||||
"burst_world_tick_ms_p99": round(work_stat("tick_ms", .99) or 0.0, 3),
|
||||
"burst_world_tick_ms_max": round(work_stat("tick_ms") or 0.0, 3),
|
||||
"burst_update_chunks_ms_p99": round(work_stat("update_chunks_ms", .99) or 0.0, 3),
|
||||
"burst_update_chunks_ms_max": round(work_stat("update_chunks_ms") or 0.0, 3),
|
||||
"burst_pending_drain_ms_p99": round(work_stat("pending_drain_ms", .99) or 0.0, 3),
|
||||
"burst_pending_drain_ms_max": round(work_stat("pending_drain_ms") or 0.0, 3),
|
||||
"burst_post_apply_callbacks_ms_p99": round(work_stat("post_apply_callbacks_ms", .99) or 0.0, 3),
|
||||
"burst_post_apply_callbacks_ms_max": round(work_stat("post_apply_callbacks_ms") or 0.0, 3),
|
||||
"burst_visibility_reconcile_ms_p99": round(work_stat("visibility_reconcile_ms", .99) or 0.0, 3),
|
||||
"burst_visibility_reconcile_ms_max": round(work_stat("visibility_reconcile_ms") or 0.0, 3),
|
||||
"burst_collision_probe_ms_max": round(work_stat("collision_probe_ms") or 0.0, 3),
|
||||
"burst_visible_apply_probe_ms_max": round(work_stat("visible_apply_probe_ms") or 0.0, 3),
|
||||
"burst_lod0_ready_ms_max": round(work_stat("lod0_ready_ms") or 0.0, 3),
|
||||
"burst_steady_trace_ms_max": round(work_stat("steady_trace_ms") or 0.0, 3),
|
||||
"burst_modification_ms_max": round(work_stat("modification_ms") or 0.0, 3),
|
||||
"burst_lifecycle_audit_ms_p99": round(work_stat("lifecycle_audit_ms", .99) or 0.0, 3),
|
||||
"burst_lifecycle_audit_ms_max": round(work_stat("lifecycle_audit_ms") or 0.0, 3),
|
||||
"burst_collision_gate_ms_p99": round(work_stat("collision_gate_ms", .99) or 0.0, 3),
|
||||
"burst_collision_gate_ms_max": round(work_stat("collision_gate_ms") or 0.0, 3),
|
||||
"burst_unload_ms_p99": round(work_stat("unload_ms", .99) or 0.0, 3),
|
||||
"burst_unload_ms_max": round(work_stat("unload_ms") or 0.0, 3),
|
||||
"burst_trace_json_ms_p99": round(work_stat("trace_json_ms", .99) or 0.0, 3),
|
||||
"burst_trace_json_ms_max": round(work_stat("trace_json_ms") or 0.0, 3),
|
||||
"burst_desired_rebuild_ms_max": round(scheduler_stat("desired_rebuild_ms") or 0.0, 3),
|
||||
"burst_rmc_create_group_ms_p99": round(rmc_stat("create_section_group_ms", .99) or 0.0, 3),
|
||||
"burst_rmc_create_group_ms_max": round(rmc_stat("create_section_group_ms") or 0.0, 3),
|
||||
"burst_rmc_collision_handoff_ms_max": round(rmc_stat("collision_handoff_ms") or 0.0, 3),
|
||||
"burst_collision_completion_ms_max": round(max(
|
||||
collision_completion_by_frame.values(), default=0.0), 3),
|
||||
"worst_frame_breakdown": {
|
||||
"frame": worst_work_frame_id,
|
||||
"world_tick_frame": worst_work_frame_id,
|
||||
"frame_interval_end_frame": worst_interval_frame_id,
|
||||
"frame_interval_ms": round(float(worst_interval.get("frame_ms", 0.0)), 3)
|
||||
if worst_interval else None,
|
||||
"apply_ms": round(float(worst_apply.get("apply_ms", 0.0)), 3)
|
||||
if worst_apply else None,
|
||||
"result_drain_ms": round(float(worst_apply.get("drain_ms", 0.0)), 3)
|
||||
if worst_apply else None,
|
||||
"outside_world_tick_ms": round(max(0.0, float(worst_interval.get("frame_ms", 0.0))
|
||||
- float(worst_work.get("tick_ms", 0.0))), 3)
|
||||
if worst_interval else None,
|
||||
"outside_full_world_tick_ms": round(max(0.0, float(worst_interval.get("frame_ms", 0.0))
|
||||
- float(commandlet_tick_by_frame.get(worst_work_frame_id, {}).get("world_tick_ms", 0.0))), 3)
|
||||
if worst_interval else None,
|
||||
"full_world_tick_ms": round(float(commandlet_tick_by_frame.get(
|
||||
worst_work_frame_id, {}).get("world_tick_ms", 0.0)), 3),
|
||||
"world_pre_actor_ms": round(float(commandlet_tick_by_frame.get(
|
||||
worst_work_frame_id, {}).get("world_pre_actor_ms", 0.0)), 3),
|
||||
"world_actor_phase_ms": round(float(commandlet_tick_by_frame.get(
|
||||
worst_work_frame_id, {}).get("world_actor_phase_ms", 0.0)), 3),
|
||||
"world_post_actor_ms": round(float(commandlet_tick_by_frame.get(
|
||||
worst_work_frame_id, {}).get("world_post_actor_ms", 0.0)), 3),
|
||||
"commandlet_readiness_check_ms": round(float(commandlet_tick_by_frame.get(
|
||||
worst_work_frame_id, {}).get("readiness_check_ms", 0.0)), 3),
|
||||
"commandlet_sleep_ms": round(float(commandlet_tick_by_frame.get(
|
||||
worst_work_frame_id, {}).get("sleep_ms", 0.0)), 3),
|
||||
"commandlet_loop_ms": round(float(commandlet_tick_by_frame.get(
|
||||
worst_work_frame_id, {}).get("loop_ms", 0.0)), 3),
|
||||
"gc_duration_ms": round(float(garbage_collect_by_frame.get(
|
||||
worst_work_frame_id, {}).get("duration_ms", 0.0)), 3),
|
||||
"world_tick_ms": round(float(worst_work.get("tick_ms", 0.0)), 3),
|
||||
"active_generation_workers": int(worst_work.get("active_generation_workers", 0)),
|
||||
"active_tile_tasks": int(worst_work.get("active_tile_tasks", 0)),
|
||||
"collision_gate_ms": round(float(worst_work.get("collision_gate_ms", 0.0)), 3),
|
||||
"update_chunks_ms": round(float(worst_work.get("update_chunks_ms", 0.0)), 3),
|
||||
"desired_rebuild_ms": round(float(worst_scheduler.get("desired_rebuild_ms", 0.0)), 3),
|
||||
"pending_drain_ms": round(float(worst_work.get("pending_drain_ms", 0.0)), 3),
|
||||
"post_apply_callbacks_ms": round(float(worst_work.get("post_apply_callbacks_ms", 0.0)), 3),
|
||||
"collision_probe_ms": round(float(worst_work.get("collision_probe_ms", 0.0)), 3),
|
||||
"visibility_reconcile_ms": round(float(worst_work.get("visibility_reconcile_ms", 0.0)), 3),
|
||||
"visible_apply_probe_ms": round(float(worst_work.get("visible_apply_probe_ms", 0.0)), 3),
|
||||
"lod0_ready_ms": round(float(worst_work.get("lod0_ready_ms", 0.0)), 3),
|
||||
"unload_ms": round(float(worst_work.get("unload_ms", 0.0)), 3),
|
||||
"steady_trace_ms": round(float(worst_work.get("steady_trace_ms", 0.0)), 3),
|
||||
"modification_ms": round(float(worst_work.get("modification_ms", 0.0)), 3),
|
||||
"lifecycle_audit_ms": round(float(worst_work.get("lifecycle_audit_ms", 0.0)), 3),
|
||||
"trace_json_ms": round(float(worst_work.get("trace_json_ms", 0.0)), 3),
|
||||
"trace_archive_serialize_ms": round(float(
|
||||
worst_work.get("trace_archive_serialize_ms", 0.0)), 3),
|
||||
"collision_completion_ms": round(worst_collision_completion_ms, 3),
|
||||
**{key: round(float(worst_rmc.get(key, 0.0)), 3)
|
||||
for key in ("component_acquire_ms", "create_section_group_ms",
|
||||
"create_sections_ms", "collision_handoff_ms")},
|
||||
},
|
||||
"trace": str(trace.relative_to(ROOT)), "utrace": str(utrace.relative_to(ROOT)),
|
||||
"abslog": str(abslog.relative_to(ROOT)), "stdout": str(stdout.relative_to(ROOT)),
|
||||
}
|
||||
stdout_text = stdout.read_text(encoding="utf-8", errors="replace")
|
||||
failure = re.search(r"\[VoxelForgeCoreWorldTest\] failed: ([^\r\n]+)", stdout_text)
|
||||
if failure:
|
||||
row["world_test_failure"] = failure.group(1)
|
||||
with (OUT / "runs.jsonl").open("a", encoding="utf-8") as stream:
|
||||
stream.write(json.dumps(row, sort_keys=True) + "\n")
|
||||
(OUT / f"{label}.summary.json").write_text(json.dumps(row, indent=2) + "\n",
|
||||
encoding="utf-8")
|
||||
print("[L101UnrealArrival] " + json.dumps(row, sort_keys=True), flush=True)
|
||||
if result.returncode:
|
||||
raise SystemExit(f"Unreal commandlet failed: {stdout}; exit={result.returncode}")
|
||||
if not full or len(arrival) < 8:
|
||||
raise SystemExit(f"Unreal arrival trace incomplete: {stdout}; exit={result.returncode}")
|
||||
if insights and (not utrace.is_file() or utrace.stat().st_size < 1024):
|
||||
raise SystemExit(f"Unreal Insights trace missing or empty: {utrace}")
|
||||
return row
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--stratum", choices=("FI", "SW"), required=True)
|
||||
parser.add_argument("--repeat", type=int, default=1)
|
||||
parser.add_argument("--phase", choices=("probe", "before", "after", "snapshot"), default="probe")
|
||||
parser.add_argument("--ready-depth", type=int, default=64)
|
||||
parser.add_argument("--background-cap", type=int, choices=(0, 1), default=1)
|
||||
parser.add_argument("--worker-reserve", type=int, choices=(0, 1, 2, 3), default=0)
|
||||
parser.add_argument("--helper-background", type=int, choices=(0, 1), default=1)
|
||||
parser.add_argument("--tile-background-low", type=int, choices=(0, 1), default=1)
|
||||
parser.add_argument("--tile-background-normal", type=int, choices=(0, 1), default=0)
|
||||
parser.add_argument("--startup-boost", type=int, choices=(0, 1), default=1)
|
||||
parser.add_argument("--tick-logs", action="store_true")
|
||||
parser.add_argument("--insights", action="store_true")
|
||||
args = parser.parse_args()
|
||||
if not EDITOR.is_file():
|
||||
raise SystemExit(f"missing Unreal commandlet executable: {EDITOR}")
|
||||
run(args.stratum, args.repeat, args.phase, args.ready_depth,
|
||||
args.background_cap, args.worker_reserve, args.helper_background,
|
||||
args.tile_background_low, args.tile_background_normal, args.startup_boost,
|
||||
args.tick_logs, args.insights)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
|
After Width: | Height: | Size: 193 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 193 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 74 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 74 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 135 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 135 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 125 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 125 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 142 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 142 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 173 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 173 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 93 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 93 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 266 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 266 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 22 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 22 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 8.3 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 8.3 KiB |
Reference in new issue
Block a user