diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/REPORT.md b/Standalone/Research/WorldAssemblerM3/Linux106/REPORT.md new file mode 100644 index 0000000..14546b1 --- /dev/null +++ b/Standalone/Research/WorldAssemblerM3/Linux106/REPORT.md @@ -0,0 +1,32 @@ +# Linux 106 + +## Decision + +**Floor/support ribbon: stopped at the falsifier.** A route-only floor profile cannot replace the current support decisions. Support is gated by pointwise native density (`density < 0`), player-capsule clearance, later-bore deferral, and neighboring-passage capsule checks. Mouth support also blends back the native field and its relief under another capsule check. The overlapping-coil tile exercises the later-bore and neighboring-capsule paths. + +At W1, the ordinary FI tile `(2,-2,-93,lod0)` had 81,816 support-segment visits and 33,422 capsule checks, with no later-bore or other-capsule checks. The overlapping-coil tile `(1,-2,-92,lod0)` had 181,956 visits, 93,034 capsule checks, 3,219 later-bore checks, and 2,442 other-capsule checks. These counts matched L105. Both tiles also had identical L105/L106 density, mesh, payload, and topology hashes. + +No local floor candidate was emitted, so candidate-specific crossing and capsule-clearance deltas are **not measured**. For a minimal hybrid, precompute route-floor geometry in the tile field while retaining the native-density gate, both capsule checks, later-bore deferral, and mouth restoration. This can remove route projection work without changing those support rules. + +**Boundary bands:** added a per-tile XY-column cache for active transition pairs. It serves exact lattice XY only; off-grid queries still call the original exact calculation. The FI profile measured 681,651 column/pair evaluations, 4,530,549 lattice cache hits, and 75,758 exact off-grid fallbacks. A seam-centered profile measured 623,891 evaluations, 2,578,260 hits, and 21,328 fallbacks. + +## Verification + +- Exact hashes: **24/24** Clang/GCC × W1/W8/W12 × SW/FI/TN/VS runs matched L105 tile and whole-view hashes; zero mismatches. Source stamp: `a033e6b53537f3550549a7a77f93f5a5d6594263c208cbbffc5d403f6d7ad7b5`. +- Fast tier: **PASS**, 274 checks, 269 PASS, 5 historical diagnostics, 0 failures. Walkability, capsule clearance, support, floor/mouth profiles, route binding, and headroom gates passed. +- L80 readiness: the required 3-pair medians are below. Near and whole sets completed in every cell. + +| Stratum | Near W8 | Near W12 | Whole W8 | Whole W12 | +|---|---:|---:|---:|---:| +| SW | +4.8% | −0.6% | +8.7% | −1.1% | +| FI | −4.2% | +0.1% | −1.5% | +0.6% | +| TN | −3.4% | −2.4% | −2.0% | −1.5% | +| VS | −2.3% | −2.4% | −1.2% | −1.1% | + +Readiness was noisy. A second full 3-pair set produced FI W12 deltas of +11.9% near and +7.6% whole; a focused 3-pair FI W12 retest measured −0.9% and −0.5%. All three result sets are under `Saved/Linux/l106/readiness-paired*`; I do not claim a consistent readiness speedup. + +- Captures: visually checked all nine L105/L106 pairs; every pair had 0 changed pixels. Listed views: `passage/0-1/0`, `passage/1-2/0`, and `passage/2-3/0`, each at upper mouth, lower mouth, and mid-50. The [contact sheet](captures/contact-sheet.png) shows every pair; exact PNG paths are in `Saved/Linux/l106/captures/summary.json`. +- Unreal Linux Development build: **PASS**, 27/27 actions, `-NoUBA -MaxParallelActions=4`, after timing and without `prlimit`. +- FI Unreal arrivals: **3/3 PASS**, each settled 1,216 near tiles in 1.543–1.550 s from first admission. They ran with `taskset`, without `prlimit`. + +No world behavior changed, so the evaluator version remains 216. No commit, push, or stash was made. diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/RunL106Captures.py b/Standalone/Research/WorldAssemblerM3/Linux106/RunL106Captures.py new file mode 100644 index 0000000..21fb328 --- /dev/null +++ b/Standalone/Research/WorldAssemblerM3/Linux106/RunL106Captures.py @@ -0,0 +1,602 @@ +#!/usr/bin/env python3 +"""Capture matched L105/L106 passage mouth and mid-route views.""" +from __future__ import annotations + +import json +import math +import struct +import subprocess +import time +import zlib +from array import array +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[4] +OUT = ROOT / "Saved/Linux/l106/captures" +IMAGE_DIR = Path(__file__).resolve().parent / "captures" +DATA = ROOT / "Standalone/data/owner_world.json" +LAYOUT_PATH = ROOT / "Saved/Linux/l99/resolved-sw.json" +BINARIES = { + "before-l105": ROOT / "Saved/Linux/l105/bin/VoxelForgeCoreHeadless-l105-clang", + "after-l106": ROOT / "Saved/Linux/l106/bin/VoxelForgeCoreHeadless-l106-clang", +} +RENDERER = ROOT / "Saved/Linux/l106/bin/VoxelForgeOffscreenCapture-l106-clang" +VOXEL_Q = 256.0 +CORE_UM_PER_VOXEL = 250_000.0 +ROUTE_POINTS = {} +ROUTE_LENGTHS = {} + + +def add(a, b): + return tuple(x + y for x, y in zip(a, b)) + + +def sub(a, b): + return tuple(x - y for x, y in zip(a, b)) + + +def mul(a, s): + return tuple(x * s for x in a) + + +def dot(a, b): + return sum(x * y for x, y in zip(a, b)) + + +def norm(a): + length = math.sqrt(dot(a, a)) + if length < 1.0e-8: + raise ValueError("degenerate capture direction") + return tuple(x / length for x in a) + + +def fmt(values): + return ",".join(f"{value:.9f}" for value in values) + + +def angles(camera, target): + delta = sub(target, camera) + return (math.degrees(math.atan2(delta[0], delta[1])), + math.degrees(math.atan2(delta[2], math.hypot(delta[0], delta[1])))) + + +def interpolate(points, distance): + remaining = max(0.0, distance) + for a, b in zip(points, points[1:]): + delta = sub(b[0], a[0]) + length = math.sqrt(dot(delta, delta)) + if length < 1.0e-8: + continue + if remaining <= length: + t = remaining / length + return (tuple(a[0][i] + delta[i] * t for i in range(3)), + a[1] + (b[1] - a[1]) * t) + remaining -= length + return points[-1] + + +def route_frame(points, distance): + center, radius = interpolate(points, distance) + before, _ = interpolate(points, max(0.0, distance - 0.25)) + after, _ = interpolate(points, min(sum( + math.sqrt(dot(sub(b[0], a[0]), sub(b[0], a[0]))) + for a, b in zip(points, points[1:])), distance + 0.25)) + tangent = norm(sub(after, before)) + horizontal = norm((tangent[0], tangent[1], 0.0)) + side = (-horizontal[1], horizontal[0], 0.0) + return center, radius, tangent, side + + +def route_length(points): + return sum(math.sqrt(dot(sub(b[0], a[0]), sub(b[0], a[0]))) + for a, b in zip(points, points[1:])) + + +def nearest_route(points, point): + best = None + distance_along = 0.0 + for a, b in zip(points, points[1:]): + delta = sub(b[0], a[0]) + length_sq = dot(delta, delta) + if length_sq < 1.0e-10: + continue + segment_length = math.sqrt(length_sq) + t = max(0.0, min(1.0, dot(sub(point, a[0]), delta) / length_sq)) + center = add(a[0], mul(delta, t)) + tangent = norm(delta) + radial = sub(point, center) + radial = sub(radial, mul(tangent, dot(radial, tangent))) + distance = math.sqrt(dot(sub(point, center), sub(point, center))) + if best is None or distance < best[0]: + radius = a[1] + (b[1] - a[1]) * t + best = (distance, center, tangent, radial, radius, + distance_along + t * segment_length) + distance_along += segment_length + if best is None: + raise ValueError("passage route has no non-degenerate segment") + return best + + +def run(name, command, headless): + log = OUT / f"{name}.log" + wrapped = ["taskset", "-c", "0-11"] + if headless: + wrapped += ["prlimit", "--as=6442450944", "--"] + wrapped += ["timeout", "1740s", *map(str, command)] + started = time.monotonic() + with log.open("w", encoding="utf-8") as stream: + result = subprocess.run(wrapped, cwd=ROOT, stdout=stream, + stderr=subprocess.STDOUT, check=False, timeout=1745) + row = {"name": name, "exit_code": result.returncode, + "wall_s": round(time.monotonic() - started, 3), + "log": str(log.relative_to(ROOT))} + print("[L106CaptureRun] " + json.dumps(row, sort_keys=True), flush=True) + if result.returncode != 0: + raise RuntimeError(f"capture step failed: {name}; see {log}") + return row + + +def nearest_surface_anchor(mesh, point, tile_search_margin_m=20.0): + """Return the nearest surface vertex and outward mesh normal from a VFMSH scene.""" + best_distance_sq = float("inf") + best_position = best_normal = None + with mesh.open("rb") as stream: + magic = stream.read(8) + has_materials = magic == b"VFMSH002" + if magic not in (b"VFMSH001", b"VFMSH002"): + raise RuntimeError(f"unsupported mesh scene header: {mesh}") + version, voxel_um, tile_count = struct.unpack("<3I", stream.read(12)) + if version != (2 if has_materials else 1): + raise RuntimeError(f"mesh scene version mismatch: {mesh}") + stream.seek(40, 1) # stored camera position, yaw, and pitch + for _ in range(tile_count): + key = struct.unpack("<4i", stream.read(16)) + surface_count, _, vertex_count, index_count, mask_count = struct.unpack( + "<5I", stream.read(20)) + extent = (32 << key[3]) * voxel_um / 1_000_000.0 + origin = (key[0] * extent, key[1] * extent, key[2] * extent) + if any(point[axis] < origin[axis] - tile_search_margin_m or + point[axis] > origin[axis] + extent + tile_search_margin_m + for axis in range(3)): + stream.seek(vertex_count * 24 + (10 * vertex_count if has_materials else 0) + + index_count * 4 + mask_count, 1) + continue + positions = array("f") + positions.frombytes(stream.read(vertex_count * 12)) + normals = array("f") + normals.frombytes(stream.read(vertex_count * 12)) + if has_materials: + stream.seek(vertex_count * 10, 1) + stream.seek(index_count * 4 + mask_count, 1) + if struct.pack("=I", 1) != struct.pack("= best_distance_sq: + continue + raw_normal = tuple(float(normals[offset + axis]) for axis in range(3)) + if dot(raw_normal, raw_normal) < 1.0e-10: + continue + best_distance_sq = distance_sq + best_position = position + best_normal = norm(raw_normal) + if best_position is None: + raise RuntimeError(f"no nearby surface vertices in mesh scene: {mesh}") + return {"position_m": best_position, "normal": best_normal, + "distance_m": math.sqrt(best_distance_sq)} + + +def make_passages(layout): + q_to_m = layout["core_voxel_size_um"] / (VOXEL_Q * 1_000_000.0) + passages = [] + all_routes = [] + for passage in layout["passages"]: + controls = [(tuple(float(q) * q_to_m for q in p["position_q"]), + float(p["radius_q"]) * q_to_m) + for p in passage["control_points"]] + upper = tuple(float(q) * q_to_m for q in passage["upper_mouth_q"]) + lower = tuple(float(q) * q_to_m for q in passage["lower_mouth_q"]) + ui = min(range(len(controls)), key=lambda i: dot(sub(controls[i][0], upper), + sub(controls[i][0], upper))) + li = min(range(len(controls)), key=lambda i: dot(sub(controls[i][0], lower), + sub(controls[i][0], lower))) + points = controls[ui:li + 1] + if len(points) < 2: + raise RuntimeError(f"invalid mouth path: {passage['stable_id']}") + upper_offset = math.sqrt(dot(sub(points[0][0], upper), sub(points[0][0], upper))) + lower_offset = math.sqrt(dot(sub(points[-1][0], lower), sub(points[-1][0], lower))) + length = route_length(points) + passage_views = [] + for endpoint, distance, camera_distance, outward_sign in ( + ("upper-mouth", 0.0, min(length, 7.4), -1.0), + ("lower-mouth", length, max(0.0, length - 5.5), 1.0)): + center, _, tangent, side = route_frame(points, distance) + camera_center, _, _, camera_side = route_frame(points, camera_distance) + camera = (camera_center[0] + camera_side[0] * 2.5, + camera_center[1] + camera_side[1] * 2.5, camera_center[2]) + outward = mul(tangent, outward_sign) + target = (center[0] + outward[0] * 8.0 + side[0] * 2.5, + center[1] + outward[1] * 8.0 + side[1] * 2.5, center[2]) + yaw, pitch = angles(camera, target) + passage_views.append({"name": f"{passage['stable_id']}-{endpoint}", + "kind": "mouth", "focus": center, "camera": camera, "target": target, + "yaw": yaw, "pitch": pitch, "clip": None, "fov": 80.0, + "focus_radius": 20.0, "passage": passage["stable_id"], + "camera_location": "passage-air"}) + for fraction in (0.50,): + distance = length * fraction + center, radius, _, side = route_frame(points, distance) + camera = (center[0] + side[0] * 0.75, + center[1] + side[1] * 0.75, + center[2] - radius + 1.6) + target_pos, target_radius = interpolate( + points, min(length, distance + 14.0)) + target = (target_pos[0] + side[0] * 2.35, + target_pos[1] + side[1] * 2.35, + target_pos[2] - target_radius + 1.8) + yaw, pitch = angles(camera, target) + pct = int(round(fraction * 100)) + passage_views.append({"name": f"{passage['stable_id']}-mid-{pct:02d}", + "kind": "mid-tunnel", "route_fraction": fraction, + "focus": center, "camera": camera, "target": target, + "yaw": yaw, "pitch": pitch, "clip": None, + "fov": 78.0, "focus_radius": 24.0, + "passage": passage["stable_id"]}) + passages.append({"stable_id": passage["stable_id"], "route_length_m": length, + "control_points": len(points), "upper_mouth_anchor_offset_m": upper_offset, + "lower_mouth_anchor_offset_m": lower_offset, "views": passage_views}) + all_routes.append((passage["stable_id"], points)) + return passages, all_routes + + +def make_hotspot_views(routes, reach_summary): + views = [] + for series, rows, value_key, location_key, region_key in ( + ("displacement", reach_summary["top_displacement_tiles"], + "max_displacement_cm", "displacement_location_metres", "displacement_region"), + ("normal", reach_summary["top_normal_tiles"], + "max_normal_degrees", "normal_location_metres", "normal_region"), + ): + for index, row in enumerate(rows, 1): + point = tuple(row[location_key]) + candidates = [(nearest_route(route, point), name, route) + for name, route in routes] + (distance, center, tangent, radial, radius, along), nearest_id, route = min( + candidates, key=lambda item: item[0][0]) + if math.sqrt(dot(radial, radial)) < 0.25: + horizontal = norm((tangent[0], tangent[1], 0.0)) + radial = (-horizontal[1], horizontal[0], 0.0) + outward = norm(radial) + camera = add(point, mul(outward, 20.0)) + target = point + plane = None + yaw, pitch = angles(camera, target) + views.append({"name": f"top-{series}-{index:02d}", "kind": f"top-{series}", + "focus": point, "camera": camera, "target": target, + "yaw": yaw, "pitch": pitch, "clip": plane, "fov": 68.0, + "focus_radius": 18.0, "strate": row["strate"], + "value": row[value_key], "region": row[region_key], + "camera_anchor": "nearest-surface-l100-v1", "nearest_passage": nearest_id, + "route_distance_m": distance}) + return views + + +def decode_png(path): + """Decode 8-bit RGB/RGBA PNG pixels using only the Python standard library.""" + data = path.read_bytes() + if data[:8] != b"\x89PNG\r\n\x1a\n": + raise RuntimeError(f"not a PNG: {path}") + width = height = bit_depth = color_type = interlace = None + compressed = bytearray() + offset = 8 + while offset < len(data): + length = struct.unpack_from(">I", data, offset)[0] + kind = data[offset + 4:offset + 8] + chunk = data[offset + 8:offset + 8 + length] + if kind == b"IHDR": + width, height, bit_depth, color_type, compression, filtering, interlace = \ + struct.unpack(">IIBBBBB", chunk) + if compression or filtering or interlace: + raise RuntimeError(f"unsupported PNG coding in {path}") + elif kind == b"IDAT": + compressed.extend(chunk) + elif kind == b"IEND": + break + offset += 12 + length + if bit_depth != 8 or color_type not in (2, 6): + raise RuntimeError(f"expected 8-bit RGB/RGBA PNG: {path}") + channels = 3 if color_type == 2 else 4 + stride = width * channels + encoded = zlib.decompress(compressed) + pixels = bytearray(height * stride) + source_offset = 0 + for y in range(height): + filter_type = encoded[source_offset] + source_offset += 1 + row = bytearray(encoded[source_offset:source_offset + stride]) + source_offset += stride + prior_offset = (y - 1) * stride + for index in range(stride): + left = row[index - channels] if index >= channels else 0 + above = pixels[prior_offset + index] if y else 0 + upper_left = pixels[prior_offset + index - channels] \ + if y and index >= channels else 0 + if filter_type == 1: + row[index] = (row[index] + left) & 255 + elif filter_type == 2: + row[index] = (row[index] + above) & 255 + elif filter_type == 3: + row[index] = (row[index] + ((left + above) >> 1)) & 255 + elif filter_type == 4: + estimate = left + above - upper_left + left_distance = abs(estimate - left) + above_distance = abs(estimate - above) + upper_left_distance = abs(estimate - upper_left) + if left_distance <= above_distance and left_distance <= upper_left_distance: + predictor = left + elif above_distance <= upper_left_distance: + predictor = above + else: + predictor = upper_left + row[index] = (row[index] + predictor) & 255 + elif filter_type != 0: + raise RuntimeError(f"unsupported PNG filter {filter_type}: {path}") + start = y * stride + pixels[start:start + stride] = row + if channels == 3: + return width, height, pixels + rgb = bytearray(width * height * 3) + rgb[0::3] = pixels[0::4] + rgb[1::3] = pixels[1::4] + rgb[2::3] = pixels[2::4] + return width, height, rgb + + +def image_geometry_metrics(path): + width, height, rgb = decode_png(path) + color_counts = {} + for offset in range(0, len(rgb), 3): + color = (rgb[offset], rgb[offset + 1], rgb[offset + 2]) + color_counts[color] = color_counts.get(color, 0) + 1 + total = width * height + visible = total - max(color_counts.values()) + return {"width": width, "height": height, "unique_rgb": len(color_counts), + "visible_geometry_pixels": visible, + "visible_geometry_fraction": visible / total} + + +def changed_pixel_count(before, after): + before_width, before_height, before_rgb = decode_png(before) + after_width, after_height, after_rgb = decode_png(after) + if (before_width, before_height) != (after_width, after_height): + raise RuntimeError(f"capture dimensions differ: {before} / {after}") + return sum(before_rgb[offset] != after_rgb[offset] + or before_rgb[offset + 1] != after_rgb[offset + 1] + or before_rgb[offset + 2] != after_rgb[offset + 2] + for offset in range(0, len(before_rgb), 3)) + + +def mesh_triangle_count(mesh): + with mesh.open("rb") as stream: + magic = stream.read(8) + has_materials = magic == b"VFMSH002" + if magic not in (b"VFMSH001", b"VFMSH002"): + raise RuntimeError(f"unsupported mesh scene header: {mesh}") + version, _, tile_count = struct.unpack("<3I", stream.read(12)) + if version != (2 if has_materials else 1): + raise RuntimeError(f"mesh scene version mismatch: {mesh}") + stream.seek(40, 1) + triangles = 0 + for _ in range(tile_count): + stream.seek(16, 1) + _, _, vertex_count, index_count, mask_count = struct.unpack( + "<5I", stream.read(20)) + triangles += index_count // 3 + stream.seek(vertex_count * 24 + (10 * vertex_count if has_materials else 0) + + index_count * 4 + mask_count, 1) + return triangles + + +def set_route_station(view, distance): + points = ROUTE_POINTS[view["passage"]] + length = ROUTE_LENGTHS[view["passage"]] + center, radius, _, side = route_frame(points, distance) + camera = (center[0] + side[0] * 0.75, + center[1] + side[1] * 0.75, + center[2] - radius + 1.6) + target_pos, target_radius = interpolate(points, min(length, distance + 14.0)) + target = (target_pos[0] + side[0] * 2.35, + target_pos[1] + side[1] * 2.35, + target_pos[2] - target_radius + 1.8) + yaw, pitch = angles(camera, target) + if "route_fraction_requested" not in view: + view["route_fraction_requested"] = view["route_fraction"] + view["route_requested_focus"] = view["focus"] + view["route_fraction"] = distance / length + view["route_station_adjustment_m"] = distance - length * view["route_fraction_requested"] + view["focus"] = center + view["camera"] = camera + view["target"] = target + view["yaw"], view["pitch"] = yaw, pitch + view["clip"] = None + + +def render_command(view, mesh, image, camera, yaw, pitch): + args = [RENDERER, mesh, image, + "--size=1600x900", f"--fov-deg={view['fov']:.1f}", + f"--camera-m={fmt(camera)}", f"--yaw-deg={yaw:.6f}", + f"--pitch-deg={pitch:.6f}", "--shade=readable", "--headlamp=on", + "--height-tint=on", "--tile-edges=off", "--tile-skirts=off", + f"--height-center-m={view['focus'][2]:.6f}", "--height-range-m=10", + f"--focus-region={fmt((*view['focus'], view['focus_radius']))}"] + if view["clip"] is not None: + args.append(f"--clip-plane={fmt(view['clip'])}") + return args + + +def export_scene(view, binary, mesh, stem): + camera = view["camera"] + yaw, pitch = angles(camera, view["target"]) + export_args = [binary, DATA, "--world-assembler-prototype", "--stream-benchmark", + "--stream-pipeline", "--workers=8", "--force-generate", "--clip-radius=3", + "--max-clip-level=0", "--max-level=0", "--render-distance-chunks=0", + f"--camera={fmt((*camera, yaw, pitch))}", f"--mesh-export={mesh}"] + row = run(f"{stem}-export", export_args, headless=True) + row["scene_bytes"] = mesh.stat().st_size if mesh.exists() else 0 + row["scene_triangles"] = mesh_triangle_count(mesh) if row["scene_bytes"] else 0 + return row + + +def export_and_render(view, label, binary): + safe = view["name"].replace("/", "-") + stem = f"{safe}-{label}" + mesh = OUT / f"{stem}.vfmesh" + image = IMAGE_DIR / f"{safe}-{label}.png" + camera = view["camera"] + yaw, pitch = angles(camera, view["target"]) + export_row = export_scene(view, binary, mesh, stem) + if export_row["scene_bytes"] <= 0: + raise RuntimeError(f"empty mesh scene: {mesh}") + if export_row["scene_triangles"] == 0: + if label != "before-l105" or view.get("kind") != "mid-tunnel": + raise RuntimeError(f"route station exports no surface triangles: {mesh}") + points = ROUTE_POINTS[view["passage"]] + length = ROUTE_LENGTHS[view["passage"]] + requested_fraction = view["route_fraction"] + requested_distance = length * requested_fraction + found_surface = False + for offset in (5.0, -5.0, 10.0, -10.0, 20.0, -20.0, 35.0, -35.0, + 50.0, -50.0, 75.0, -75.0, 100.0, -100.0, 150.0, + -150.0, 200.0, -200.0): + distance = max(0.0, min(length, requested_distance + offset)) + if distance == requested_distance: + continue + set_route_station(view, distance) + adjusted_stem = f"{stem}-station-{view['route_fraction']:.5f}" + export_row = export_scene(view, binary, mesh, adjusted_stem) + if export_row["scene_triangles"] > 0: + view["camera_anchor"] = "route-station-shifted-to-nearby-geometry" + found_surface = True + break + if not found_surface: + raise RuntimeError(f"no geometry along nearby route stations: {mesh}") + camera = view["camera"] + yaw, pitch = angles(camera, view["target"]) + if label == "before-l105" and view.get("camera_anchor") == "nearest-surface-l100-v1": + anchor = nearest_surface_anchor(mesh, view["focus"], tile_search_margin_m=256.0) + camera = add(anchor["position_m"], mul(anchor["normal"], 8.0)) + view["camera"] = camera + view["target"] = anchor["position_m"] + view["yaw"], view["pitch"] = angles(camera, view["target"]) + view["clip"] = None + view["surface_anchor"] = anchor + yaw, pitch = view["yaw"], view["pitch"] + render_args = render_command(view, mesh, image, camera, yaw, pitch) + try: + render_row = run(f"{stem}-render", render_args, headless=True) + except RuntimeError: + if view["clip"] is None: + raise + retry_args = [arg for arg in render_args + if not (isinstance(arg, str) and arg.startswith("--clip-plane="))] + render_row = run(f"{stem}-render-no-clip-retry", retry_args, headless=True) + render_row["clip_fallback"] = True + render_row["image"] = str(image.relative_to(ROOT)) + render_row["image_bytes"] = image.stat().st_size if image.exists() else 0 + if render_row["image_bytes"] <= 0: + raise RuntimeError(f"empty image: {image}") + geometry = image_geometry_metrics(image) + if label == "before-l105" and ( + geometry["visible_geometry_pixels"] < 3000 or geometry["unique_rgb"] < 2): + # At a few deep route stations the default forward view looks into empty + # sky. Keep the camera on the route station and aim at the nearest L98 + # surface in the exported scene; retain that matched target for L101. + anchor = nearest_surface_anchor(mesh, view["focus"], tile_search_margin_m=256.0) + if anchor["distance_m"] > 60.0: + raise RuntimeError(f"no nearby surface for fallback camera: {image} {anchor}") + camera = view["camera"] + target = anchor["position_m"] + yaw, pitch = angles(camera, target) + view["target"] = target + view["yaw"], view["pitch"] = yaw, pitch + view["clip"] = None + view["camera_anchor"] = "route-station-nearest-surface-target" + view["surface_anchor"] = anchor + retry_args = render_command(view, mesh, image, camera, yaw, pitch) + render_row = run(f"{stem}-render-surface-fallback", retry_args, headless=True) + render_row["image"] = str(image.relative_to(ROOT)) + render_row["image_bytes"] = image.stat().st_size if image.exists() else 0 + geometry = image_geometry_metrics(image) + render_row["geometry"] = geometry + if geometry["visible_geometry_pixels"] < 3000 or geometry["unique_rgb"] < 2: + raise RuntimeError(f"camera sees sky only or too little geometry: {image} {geometry}") + mesh.unlink() + return export_row, render_row + + +def main(): + OUT.mkdir(parents=True, exist_ok=True) + IMAGE_DIR.mkdir(parents=True, exist_ok=True) + for path in (*BINARIES.values(), RENDERER, DATA, LAYOUT_PATH): + if not path.is_file(): + raise SystemExit(f"missing input: {path}") + layout = json.loads(LAYOUT_PATH.read_text(encoding="utf-8")) + passages, routes = make_passages(layout) + ROUTE_POINTS.clear() + ROUTE_LENGTHS.clear() + for stable_id, points in routes: + ROUTE_POINTS[stable_id] = points + ROUTE_LENGTHS[stable_id] = route_length(points) + views = [view for passage in passages for view in passage["views"]] + partial_path = OUT / "runs-partial.json" + existing_records = {} + if partial_path.is_file(): + for item in json.loads(partial_path.read_text(encoding="utf-8")): + existing_records[item["view"]["name"]] = item + records = [] + for view in views: + prior = existing_records.get(view["name"]) + camera_matches = (prior is not None and + prior["view"].get("camera_anchor") == view.get("camera_anchor")) + if camera_matches: + for key in ("camera", "target", "yaw", "pitch", "clip", "surface_anchor"): + if key in prior["view"]: + view[key] = prior["view"][key] + if camera_matches and all(version in prior and + (ROOT / prior[version]["render"]["image"]).is_file() + for version in BINARIES): + records.append(prior) + print(f"[L106CaptureSkip] complete_pair={view['name']}", flush=True) + continue + pair = {"view": view} + for label, binary in BINARIES.items(): + export_row, render_row = export_and_render(view, label, binary) + pair[label] = {"export": export_row, "render": render_row} + (OUT / "runs-partial.json").write_text(json.dumps(records + [pair], indent=2) + "\n", + encoding="utf-8") + pair["comparison"] = {"changed_pixels": changed_pixel_count( + ROOT / pair["before-l105"]["render"]["image"], + ROOT / pair["after-l106"]["render"]["image"])} + (OUT / "runs-partial.json").write_text(json.dumps(records + [pair], indent=2) + "\n", + encoding="utf-8") + records.append(pair) + summary = {"source_layout_identity": layout["stable_identity"], + "view_count": len(views), "paired_png_count": len(views) * 2, + "passages": passages, "views": records} + (OUT / "summary.json").write_text(json.dumps(summary, indent=2) + "\n", + encoding="utf-8") + print("[L106CaptureSummary] " + json.dumps({ + "views": len(views), "paired_png_count": len(views) * 2, + "images": [row[version]["render"]["image"] for row in records + for version in BINARIES]}, sort_keys=True), flush=True) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/RunL106Hashes.py b/Standalone/Research/WorldAssemblerM3/Linux106/RunL106Hashes.py new file mode 100644 index 0000000..5d514e2 --- /dev/null +++ b/Standalone/Research/WorldAssemblerM3/Linux106/RunL106Hashes.py @@ -0,0 +1,117 @@ +#!/usr/bin/env python3 +"""Compare L106 exact tile hashes with L105 across all strata, compilers and workers.""" +from __future__ import annotations + +import json +import re +import subprocess +import time +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[4] +OUT = ROOT / "Saved/Linux/l106/hashes" +DATA = ROOT / "Standalone/data/owner_world.json" +OLD = ROOT / "Saved/Linux/l105/hashes" +BINARIES = { + "clang": ROOT / "Saved/Linux/l106/bin/VoxelForgeCoreHeadless-l106-clang", + "gcc": ROOT / "Saved/Linux/l106/bin/VoxelForgeCoreHeadless-l106-gcc", +} +CAMERAS = { + "sw": ((8.0, 16.0, -372.2626953125), 2850), + "fi": ((25.8369140625, 3.19921875, -737.0), 2249), + "tn": ((19.166015625, -14.2001953125, -753.3740234375), 2233), + "vs": ((-20.0625, -5.068359375, -900.353515625), 2329), +} +TILE = re.compile( + r"^\[PipelineTileHash\] lod=(-?\d+) x=(-?\d+) y=(-?\d+) z=(-?\d+) " + r"density=(0x[0-9a-f]+) mesh=(0x[0-9a-f]+) payload=(0x[0-9a-f]+) " + r"topology=(0x[0-9a-f]+)$", re.M) +WHOLE = re.compile( + r"^\[PipelineWholeViewHash\] tiles=(\d+) density=(0x[0-9a-f]+) " + r"mesh=(0x[0-9a-f]+) payload=(0x[0-9a-f]+) topology=(0x[0-9a-f]+)$", re.M) +STAMP = re.compile(r"^\[VoxelForgeSourceStamp\] sha256=([0-9a-f]{64})$", re.M) + + +def parse(path: Path) -> tuple[dict, tuple[str, ...], str]: + content = path.read_text(encoding="utf-8", errors="replace") + tiles = {(int(m.group(1)), int(m.group(2)), int(m.group(3)), int(m.group(4))): + m.group(5, 6, 7, 8) for m in TILE.finditer(content)} + whole, stamp = WHOLE.search(content), STAMP.search(content) + if whole is None or stamp is None or len(tiles) != int(whole.group(1)): + raise RuntimeError(f"incomplete hash output: {path}") + return tiles, whole.groups(), stamp.group(1) + + +def run(stratum: str, compiler: str, workers: int) -> dict: + OUT.mkdir(parents=True, exist_ok=True) + log = OUT / f"{stratum}-{compiler}-w{workers}.log" + camera, expected_tiles = CAMERAS[stratum] + camera_arg = ",".join(f"{v:.12f}" for v in camera) + ",0,0" + command = [ + "taskset", "-c", "0-11", "prlimit", "--as=6442450944", "--", "timeout", "1740s", + str(BINARIES[compiler]), str(DATA), "--world-assembler-prototype", + "--stream-benchmark", "--stream-pipeline", f"--workers={workers}", + "--clip-radius=3", "--max-clip-level=8", "--max-level=8", + f"--camera={camera_arg}", "--stream-hash-tiles", + ] + started = time.monotonic() + with log.open("w", encoding="utf-8") as stream: + process = subprocess.run(command, cwd=ROOT, stdout=stream, + stderr=subprocess.STDOUT, check=False, timeout=1740) + if process.returncode: + raise RuntimeError(f"hash process exited {process.returncode}: {log}") + tiles, whole, stamp = parse(log) + if len(tiles) != expected_tiles: + raise RuntimeError(f"expected {expected_tiles} tiles, got {len(tiles)}: {log}") + old_log = OLD / f"{stratum}-{compiler}-w{workers}.log" + old_tiles, old_whole, _old_stamp = parse(old_log) + clang_reference, clang_whole, _ = parse(OLD / f"{stratum}-clang-w1.log") + + def difference(reference: dict, candidate: dict) -> tuple[int, int]: + keys = set(reference) | set(candidate) + mismatches = sum(reference.get(key) != candidate.get(key) for key in keys) + return mismatches, len(keys) + + old_mismatches, old_keys = difference(old_tiles, tiles) + cross_mismatches, cross_keys = difference(clang_reference, tiles) + row = { + "stratum": stratum, "compiler": compiler, "workers": workers, + "source_stamp": stamp, "tiles": len(tiles), + "l105_same_cell_tile_mismatches": old_mismatches, + "l105_same_cell_key_count": old_keys, + "l105_same_cell_whole_hash_match": whole == old_whole, + "clang_w1_tile_mismatches": cross_mismatches, + "clang_w1_key_count": cross_keys, + "clang_w1_whole_hash_match": whole == clang_whole, + "wall_s": round(time.monotonic() - started, 3), + "log": str(log.relative_to(ROOT)), + } + print("[L106HashRun] " + json.dumps(row, sort_keys=True), flush=True) + row["tile_hashes"] = {"|".join(map(str, key)): value for key, value in tiles.items()} + return row + + +def main() -> int: + for path in (*BINARIES.values(), DATA): + if not path.is_file(): + raise SystemExit(f"missing input: {path}") + rows = [] + for stratum in CAMERAS: + for compiler in BINARIES: + for workers in (1, 8, 12): + rows.append(run(stratum, compiler, workers)) + (OUT / "runs-partial.json").write_text( + json.dumps(rows, indent=2) + "\n", encoding="utf-8") + failures = [row for row in rows if row["l105_same_cell_tile_mismatches"] + or not row["l105_same_cell_whole_hash_match"] + or row["clang_w1_tile_mismatches"] + or not row["clang_w1_whole_hash_match"]] + summary = {"runs": [{key: value for key, value in row.items() if key != "tile_hashes"} + for row in rows], "failed_runs": len(failures)} + (OUT / "summary.json").write_text(json.dumps(summary, indent=2) + "\n", + encoding="utf-8") + return 1 if failures else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/RunL106ReadinessExtra.py b/Standalone/Research/WorldAssemblerM3/Linux106/RunL106ReadinessExtra.py new file mode 100755 index 0000000..7fabcdf --- /dev/null +++ b/Standalone/Research/WorldAssemblerM3/Linux106/RunL106ReadinessExtra.py @@ -0,0 +1,118 @@ +#!/usr/bin/env python3 +"""L80-protocol L105/L106 near and whole-view readiness pairs.""" +from __future__ import annotations + +import json +import re +import statistics +import subprocess +import time +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[4] +OUT = ROOT / "Saved/Linux/l106/readiness-paired-extra" +DATA = ROOT / "Standalone/data/owner_world.json" +BINARIES = { + "before": ROOT / "Saved/Linux/l105/bin/VoxelForgeCoreHeadless-l105-clang", + "after": ROOT / "Saved/Linux/l106/bin/VoxelForgeCoreHeadless-l106-clang", +} +STRATA = { + "sw": {"camera": (8.0, 16.0, -372.2626953125), "near": 1408, "whole": 2850}, + "fi": {"camera": (25.8369140625, 3.19921875, -737.0), "near": 1216, "whole": 2249}, + "tn": {"camera": (19.166015625, -14.2001953125, -753.3740234375), "near": 1344, "whole": 2233}, + "vs": {"camera": (-20.0625, -5.068359375, -900.353515625), "near": 1344, "whole": 2329}, +} +NEAR = re.compile(r"^\[WorldAssemblerNearLODReady\] lods=0-2 tiles=(\d+)/(\d+) cold_ready_s=([0-9.]+)$", re.M) +PIPELINE = re.compile(r"^\[StreamPipeline\] (.*)$", re.M) +CONFIG = re.compile(r"^\[HeadlessConfig\] (.*)$", re.M) +STAMP = re.compile(r"^\[VoxelForgeSourceStamp\] sha256=([0-9a-f]{64})$", re.M) +FIELDS = re.compile(r"([a-zA-Z0-9_]+)=([^ ]+)") + + +def run(stratum: str, workers: int, build: str, phase: str, replicate: int) -> dict: + OUT.mkdir(parents=True, exist_ok=True) + suffix = f"-r{replicate}" if phase == "measured" else "" + log = OUT / f"{stratum}-w{workers}-{build}-{phase}{suffix}.log" + camera = ",".join(f"{value:.12f}" for value in STRATA[stratum]["camera"]) + ",0,0" + command = [ + "taskset", "-c", "0-11", "prlimit", "--as=6442450944", "--", "timeout", "1740s", + str(BINARIES[build]), str(DATA), "--world-assembler-prototype", + "--stream-benchmark", "--stream-pipeline", f"--workers={workers}", + "--clip-radius=3", "--max-clip-level=8", "--max-level=8", + f"--camera={camera}", "--profile-lod-work", "--stream-scheduler-policy=3", + ] + started = time.monotonic() + with log.open("w", encoding="utf-8") as stream: + process = subprocess.run(command, cwd=ROOT, stdout=stream, + stderr=subprocess.STDOUT, check=False, timeout=1740) + content = log.read_text(encoding="utf-8", errors="replace") + near, pipeline, config, stamp = (NEAR.search(content), PIPELINE.search(content), + CONFIG.search(content), STAMP.search(content)) + if process.returncode or not all((near, pipeline, config, stamp)): + raise RuntimeError(f"readiness run failed or missing marker: {log}; exit={process.returncode}") + expected = STRATA[stratum] + config_fields = dict(FIELDS.findall(config.group(1))) + pipeline_fields = dict(FIELDS.findall(pipeline.group(1))) + if int(config_fields["workers"]) != workers: + raise RuntimeError(f"worker mismatch: {log}") + if (int(near.group(1)), int(near.group(2))) != (expected["near"], expected["near"]): + raise RuntimeError(f"incomplete LOD0-2 set: {log}: {near.group(1)}/{near.group(2)}") + whole_done, whole_total = map(int, pipeline_fields["whole_tiles"].split("/")) + if (whole_done, whole_total) != (expected["whole"], expected["whole"]): + raise RuntimeError(f"incomplete whole view: {log}: {whole_done}/{whole_total}") + row = { + "stratum": stratum, "workers": workers, "build": build, + "phase": phase, "replicate": replicate, "source_stamp": stamp.group(1), + "near_tiles": expected["near"], "whole_tiles": whole_done, + "lod02_ready_s": float(near.group(3)), + "whole_view_ready_s": float(pipeline_fields["whole_view_ready"].removesuffix("s")), + "shape_s": float(pipeline_fields["shape"].removesuffix("s")), + "far_shape_s": float(pipeline_fields["far_shape"].removesuffix("s")), + "wall_s": round(time.monotonic() - started, 3), + "log": str(log.relative_to(ROOT)), + } + print("[L106ReadinessRun] " + json.dumps(row, sort_keys=True), flush=True) + return row + + +def main() -> int: + for path in (*BINARIES.values(), DATA): + if not path.is_file(): + raise SystemExit(f"missing input: {path}") + cells = [(stratum, workers) for stratum in ("vs", "tn", "fi", "sw") + for workers in (12, 8)] + for stratum, workers in cells: + for build in ("before", "after"): + run(stratum, workers, build, "warmup", 0) + orders = (("before", "after"), ("after", "before"), ("before", "after")) + rows = [] + for stratum, workers in cells: + for replicate, order in enumerate(orders, 1): + for build in order: + rows.append(run(stratum, workers, build, "measured", replicate)) + (OUT / "runs-partial.json").write_text( + json.dumps(rows, indent=2) + "\n", encoding="utf-8") + summary = [] + for stratum in STRATA: + for workers in (8, 12): + group = [row for row in rows if row["stratum"] == stratum + and row["workers"] == workers] + item = {"stratum": stratum, "workers": workers, "measured_runs_per_build": 3} + for build in ("before", "after"): + subset = [row for row in group if row["build"] == build] + for metric in ("lod02_ready_s", "whole_view_ready_s"): + values = [row[metric] for row in subset] + item[f"{build}_{metric}"] = values + item[f"{build}_{metric}_median"] = statistics.median(values) + for metric in ("lod02_ready_s", "whole_view_ready_s"): + item[f"{metric}_delta_pct"] = 100.0 * ( + item[f"after_{metric}_median"] / item[f"before_{metric}_median"] - 1.0) + summary.append(item) + (OUT / "summary.json").write_text( + json.dumps({"runs": rows, "summary": summary}, indent=2) + "\n", + encoding="utf-8") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/RunL106ReadinessFIW12.py b/Standalone/Research/WorldAssemblerM3/Linux106/RunL106ReadinessFIW12.py new file mode 100755 index 0000000..a458385 --- /dev/null +++ b/Standalone/Research/WorldAssemblerM3/Linux106/RunL106ReadinessFIW12.py @@ -0,0 +1,117 @@ +#!/usr/bin/env python3 +"""L80-protocol L105/L106 near and whole-view readiness pairs.""" +from __future__ import annotations + +import json +import re +import statistics +import subprocess +import time +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[4] +OUT = ROOT / "Saved/Linux/l106/readiness-paired-fi-w12-extra" +DATA = ROOT / "Standalone/data/owner_world.json" +BINARIES = { + "before": ROOT / "Saved/Linux/l105/bin/VoxelForgeCoreHeadless-l105-clang", + "after": ROOT / "Saved/Linux/l106/bin/VoxelForgeCoreHeadless-l106-clang", +} +STRATA = { + "sw": {"camera": (8.0, 16.0, -372.2626953125), "near": 1408, "whole": 2850}, + "fi": {"camera": (25.8369140625, 3.19921875, -737.0), "near": 1216, "whole": 2249}, + "tn": {"camera": (19.166015625, -14.2001953125, -753.3740234375), "near": 1344, "whole": 2233}, + "vs": {"camera": (-20.0625, -5.068359375, -900.353515625), "near": 1344, "whole": 2329}, +} +NEAR = re.compile(r"^\[WorldAssemblerNearLODReady\] lods=0-2 tiles=(\d+)/(\d+) cold_ready_s=([0-9.]+)$", re.M) +PIPELINE = re.compile(r"^\[StreamPipeline\] (.*)$", re.M) +CONFIG = re.compile(r"^\[HeadlessConfig\] (.*)$", re.M) +STAMP = re.compile(r"^\[VoxelForgeSourceStamp\] sha256=([0-9a-f]{64})$", re.M) +FIELDS = re.compile(r"([a-zA-Z0-9_]+)=([^ ]+)") + + +def run(stratum: str, workers: int, build: str, phase: str, replicate: int) -> dict: + OUT.mkdir(parents=True, exist_ok=True) + suffix = f"-r{replicate}" if phase == "measured" else "" + log = OUT / f"{stratum}-w{workers}-{build}-{phase}{suffix}.log" + camera = ",".join(f"{value:.12f}" for value in STRATA[stratum]["camera"]) + ",0,0" + command = [ + "taskset", "-c", "0-11", "prlimit", "--as=6442450944", "--", "timeout", "1740s", + str(BINARIES[build]), str(DATA), "--world-assembler-prototype", + "--stream-benchmark", "--stream-pipeline", f"--workers={workers}", + "--clip-radius=3", "--max-clip-level=8", "--max-level=8", + f"--camera={camera}", "--profile-lod-work", "--stream-scheduler-policy=3", + ] + started = time.monotonic() + with log.open("w", encoding="utf-8") as stream: + process = subprocess.run(command, cwd=ROOT, stdout=stream, + stderr=subprocess.STDOUT, check=False, timeout=1740) + content = log.read_text(encoding="utf-8", errors="replace") + near, pipeline, config, stamp = (NEAR.search(content), PIPELINE.search(content), + CONFIG.search(content), STAMP.search(content)) + if process.returncode or not all((near, pipeline, config, stamp)): + raise RuntimeError(f"readiness run failed or missing marker: {log}; exit={process.returncode}") + expected = STRATA[stratum] + config_fields = dict(FIELDS.findall(config.group(1))) + pipeline_fields = dict(FIELDS.findall(pipeline.group(1))) + if int(config_fields["workers"]) != workers: + raise RuntimeError(f"worker mismatch: {log}") + if (int(near.group(1)), int(near.group(2))) != (expected["near"], expected["near"]): + raise RuntimeError(f"incomplete LOD0-2 set: {log}: {near.group(1)}/{near.group(2)}") + whole_done, whole_total = map(int, pipeline_fields["whole_tiles"].split("/")) + if (whole_done, whole_total) != (expected["whole"], expected["whole"]): + raise RuntimeError(f"incomplete whole view: {log}: {whole_done}/{whole_total}") + row = { + "stratum": stratum, "workers": workers, "build": build, + "phase": phase, "replicate": replicate, "source_stamp": stamp.group(1), + "near_tiles": expected["near"], "whole_tiles": whole_done, + "lod02_ready_s": float(near.group(3)), + "whole_view_ready_s": float(pipeline_fields["whole_view_ready"].removesuffix("s")), + "shape_s": float(pipeline_fields["shape"].removesuffix("s")), + "far_shape_s": float(pipeline_fields["far_shape"].removesuffix("s")), + "wall_s": round(time.monotonic() - started, 3), + "log": str(log.relative_to(ROOT)), + } + print("[L106ReadinessRun] " + json.dumps(row, sort_keys=True), flush=True) + return row + + +def main() -> int: + for path in (*BINARIES.values(), DATA): + if not path.is_file(): + raise SystemExit(f"missing input: {path}") + cells = [("fi", 12)] + for stratum, workers in cells: + for build in ("before", "after"): + run(stratum, workers, build, "warmup", 0) + orders = (("before", "after"), ("after", "before"), ("before", "after")) + rows = [] + for stratum, workers in cells: + for replicate, order in enumerate(orders, 1): + for build in order: + rows.append(run(stratum, workers, build, "measured", replicate)) + (OUT / "runs-partial.json").write_text( + json.dumps(rows, indent=2) + "\n", encoding="utf-8") + summary = [] + for stratum in ("fi",): + for workers in (12,): + group = [row for row in rows if row["stratum"] == stratum + and row["workers"] == workers] + item = {"stratum": stratum, "workers": workers, "measured_runs_per_build": 3} + for build in ("before", "after"): + subset = [row for row in group if row["build"] == build] + for metric in ("lod02_ready_s", "whole_view_ready_s"): + values = [row[metric] for row in subset] + item[f"{build}_{metric}"] = values + item[f"{build}_{metric}_median"] = statistics.median(values) + for metric in ("lod02_ready_s", "whole_view_ready_s"): + item[f"{metric}_delta_pct"] = 100.0 * ( + item[f"after_{metric}_median"] / item[f"before_{metric}_median"] - 1.0) + summary.append(item) + (OUT / "summary.json").write_text( + json.dumps({"runs": rows, "summary": summary}, indent=2) + "\n", + encoding="utf-8") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/RunL106ReadinessPairs.py b/Standalone/Research/WorldAssemblerM3/Linux106/RunL106ReadinessPairs.py new file mode 100644 index 0000000..770a68e --- /dev/null +++ b/Standalone/Research/WorldAssemblerM3/Linux106/RunL106ReadinessPairs.py @@ -0,0 +1,118 @@ +#!/usr/bin/env python3 +"""L80-protocol L105/L106 near and whole-view readiness pairs.""" +from __future__ import annotations + +import json +import re +import statistics +import subprocess +import time +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[4] +OUT = ROOT / "Saved/Linux/l106/readiness-paired" +DATA = ROOT / "Standalone/data/owner_world.json" +BINARIES = { + "before": ROOT / "Saved/Linux/l105/bin/VoxelForgeCoreHeadless-l105-clang", + "after": ROOT / "Saved/Linux/l106/bin/VoxelForgeCoreHeadless-l106-clang", +} +STRATA = { + "sw": {"camera": (8.0, 16.0, -372.2626953125), "near": 1408, "whole": 2850}, + "fi": {"camera": (25.8369140625, 3.19921875, -737.0), "near": 1216, "whole": 2249}, + "tn": {"camera": (19.166015625, -14.2001953125, -753.3740234375), "near": 1344, "whole": 2233}, + "vs": {"camera": (-20.0625, -5.068359375, -900.353515625), "near": 1344, "whole": 2329}, +} +NEAR = re.compile(r"^\[WorldAssemblerNearLODReady\] lods=0-2 tiles=(\d+)/(\d+) cold_ready_s=([0-9.]+)$", re.M) +PIPELINE = re.compile(r"^\[StreamPipeline\] (.*)$", re.M) +CONFIG = re.compile(r"^\[HeadlessConfig\] (.*)$", re.M) +STAMP = re.compile(r"^\[VoxelForgeSourceStamp\] sha256=([0-9a-f]{64})$", re.M) +FIELDS = re.compile(r"([a-zA-Z0-9_]+)=([^ ]+)") + + +def run(stratum: str, workers: int, build: str, phase: str, replicate: int) -> dict: + OUT.mkdir(parents=True, exist_ok=True) + suffix = f"-r{replicate}" if phase == "measured" else "" + log = OUT / f"{stratum}-w{workers}-{build}-{phase}{suffix}.log" + camera = ",".join(f"{value:.12f}" for value in STRATA[stratum]["camera"]) + ",0,0" + command = [ + "taskset", "-c", "0-11", "prlimit", "--as=6442450944", "--", "timeout", "1740s", + str(BINARIES[build]), str(DATA), "--world-assembler-prototype", + "--stream-benchmark", "--stream-pipeline", f"--workers={workers}", + "--clip-radius=3", "--max-clip-level=8", "--max-level=8", + f"--camera={camera}", "--profile-lod-work", "--stream-scheduler-policy=3", + ] + started = time.monotonic() + with log.open("w", encoding="utf-8") as stream: + process = subprocess.run(command, cwd=ROOT, stdout=stream, + stderr=subprocess.STDOUT, check=False, timeout=1740) + content = log.read_text(encoding="utf-8", errors="replace") + near, pipeline, config, stamp = (NEAR.search(content), PIPELINE.search(content), + CONFIG.search(content), STAMP.search(content)) + if process.returncode or not all((near, pipeline, config, stamp)): + raise RuntimeError(f"readiness run failed or missing marker: {log}; exit={process.returncode}") + expected = STRATA[stratum] + config_fields = dict(FIELDS.findall(config.group(1))) + pipeline_fields = dict(FIELDS.findall(pipeline.group(1))) + if int(config_fields["workers"]) != workers: + raise RuntimeError(f"worker mismatch: {log}") + if (int(near.group(1)), int(near.group(2))) != (expected["near"], expected["near"]): + raise RuntimeError(f"incomplete LOD0-2 set: {log}: {near.group(1)}/{near.group(2)}") + whole_done, whole_total = map(int, pipeline_fields["whole_tiles"].split("/")) + if (whole_done, whole_total) != (expected["whole"], expected["whole"]): + raise RuntimeError(f"incomplete whole view: {log}: {whole_done}/{whole_total}") + row = { + "stratum": stratum, "workers": workers, "build": build, + "phase": phase, "replicate": replicate, "source_stamp": stamp.group(1), + "near_tiles": expected["near"], "whole_tiles": whole_done, + "lod02_ready_s": float(near.group(3)), + "whole_view_ready_s": float(pipeline_fields["whole_view_ready"].removesuffix("s")), + "shape_s": float(pipeline_fields["shape"].removesuffix("s")), + "far_shape_s": float(pipeline_fields["far_shape"].removesuffix("s")), + "wall_s": round(time.monotonic() - started, 3), + "log": str(log.relative_to(ROOT)), + } + print("[L106ReadinessRun] " + json.dumps(row, sort_keys=True), flush=True) + return row + + +def main() -> int: + for path in (*BINARIES.values(), DATA): + if not path.is_file(): + raise SystemExit(f"missing input: {path}") + cells = [(stratum, workers) for stratum in ("vs", "tn", "fi", "sw") + for workers in (12, 8)] + for stratum, workers in cells: + for build in ("before", "after"): + run(stratum, workers, build, "warmup", 0) + orders = (("before", "after"), ("after", "before"), ("before", "after")) + rows = [] + for stratum, workers in cells: + for replicate, order in enumerate(orders, 1): + for build in order: + rows.append(run(stratum, workers, build, "measured", replicate)) + (OUT / "runs-partial.json").write_text( + json.dumps(rows, indent=2) + "\n", encoding="utf-8") + summary = [] + for stratum in STRATA: + for workers in (8, 12): + group = [row for row in rows if row["stratum"] == stratum + and row["workers"] == workers] + item = {"stratum": stratum, "workers": workers, "measured_runs_per_build": 3} + for build in ("before", "after"): + subset = [row for row in group if row["build"] == build] + for metric in ("lod02_ready_s", "whole_view_ready_s"): + values = [row[metric] for row in subset] + item[f"{build}_{metric}"] = values + item[f"{build}_{metric}_median"] = statistics.median(values) + for metric in ("lod02_ready_s", "whole_view_ready_s"): + item[f"{metric}_delta_pct"] = 100.0 * ( + item[f"after_{metric}_median"] / item[f"before_{metric}_median"] - 1.0) + summary.append(item) + (OUT / "summary.json").write_text( + json.dumps({"runs": rows, "summary": summary}, indent=2) + "\n", + encoding="utf-8") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/RunL106UnrealArrival.py b/Standalone/Research/WorldAssemblerM3/Linux106/RunL106UnrealArrival.py new file mode 100755 index 0000000..28cc6f1 --- /dev/null +++ b/Standalone/Research/WorldAssemblerM3/Linux106/RunL106UnrealArrival.py @@ -0,0 +1,484 @@ +#!/usr/bin/env python3 +"""Run one pinned null-RHI SW/FI arrival for Linux 106.""" +from __future__ import annotations + +import argparse +from collections import Counter +from datetime import datetime +import json +import re +import subprocess +import time +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[4] +HOST = Path("/home/jahni/VFHost") +HOST_PLUGIN_SAVED = HOST / "Plugins/VoxelForge/Saved/Linux/l106" +HOST_SAVED = HOST / "Saved/Linux/l106" +EDITOR = Path("/home/jahni/UnrealEngine/Engine/Binaries/Linux/UnrealEditor-Cmd") +OUT = ROOT / "Saved/Linux/l106/unreal-arrival" +SPAWNS_M = { + "FI": (25.8369140625, 3.19921875, -737.0), + "SW": (25.8369140625, 3.19921875, -200.0), +} +MOVE_START_S = 20.0 +MOVE_DURATION_S = 12.0 +EXIT_S = MOVE_START_S + MOVE_DURATION_S + 5.0 +RUN_TIMEOUT_S = 1740 + + +def read_events(path: Path) -> list[dict]: + events = [] + if path.is_file(): + for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + if line.startswith("{"): + try: + events.append(json.loads(line)) + except json.JSONDecodeError: + pass + return events + + +def percentile(values: list[float], fraction: float) -> float | None: + if not values: + return None + ordered = sorted(values) + return ordered[min(len(ordered) - 1, int(round((len(ordered) - 1) * fraction)))] + + +def sample_unreal_peak_rss() -> tuple[int, int] | None: + executable = str(EDITOR).encode() + project = str(HOST / "VFHost.uproject").encode() + peak: tuple[int, int] | None = None + for process in Path("/proc").glob("[0-9]*"): + try: + command_line = (process / "cmdline").read_bytes() + if executable not in command_line or project not in command_line: + continue + status = (process / "status").read_text(encoding="ascii", errors="ignore") + match = re.search(r"^VmHWM:\s+(\d+)\s+kB$", status, re.MULTILINE) + if match: + rss_kib = int(match.group(1)) + if peak is None or rss_kib > peak[0]: + peak = (rss_kib, int(process.name)) + except (FileNotFoundError, PermissionError, ProcessLookupError, OSError): + continue + return peak + + +def endphysics_gap(stdout: Path) -> tuple[float | None, int | None]: + release_times: dict[int, dict[int, datetime]] = {} + pattern = re.compile( + r"^\[(\d{4}\.\d{2}\.\d{2}-\d{2}\.\d{2}\.\d{2}:\d{3})\].*" + r"LogTick: tick\s+(\d+)\s+-+\s+Release tick group ([34])") + with stdout.open(encoding="utf-8", errors="replace") as stream: + for line in stream: + match = pattern.match(line) + if not match: + continue + timestamp = datetime.strptime(match.group(1), "%Y.%m.%d-%H.%M.%S:%f") + frame = int(match.group(2)) + group = int(match.group(3)) + release_times.setdefault(frame, {})[group] = timestamp + gaps = [(round((times[4] - times[3]).total_seconds() * 1000.0, 3), frame) + for frame, times in release_times.items() if 3 in times and 4 in times] + return max(gaps, default=(None, None)) + + +def run(stratum: str, repeat: int, phase: str, ready_depth: int, + background_cap: int, worker_reserve: int, helper_background: int, + tile_background_low: int, tile_background_normal: int, startup_boost: int, + tick_logs: bool, insights: bool) -> dict: + label = (f"{phase}-{stratum.lower()}-rd{ready_depth}-bg{background_cap}" + f"-reserve{worker_reserve}-helperbg{helper_background}" + f"-tilelow{tile_background_low}-tilenormal{tile_background_normal}" + f"-ticklog{int(tick_logs)}" + f"-boost{startup_boost}-{repeat}") + trace = OUT / f"{label}.trace.jsonl" + utrace = OUT / f"{label}.utrace" + abslog = OUT / f"{label}.log" + stdout = OUT / f"{label}.stdout.log" + output = HOST_PLUGIN_SAVED / "output" / label + userdir = HOST_SAVED / "user" / label + zendata = HOST_SAVED / "zen" / label + ddc = HOST_SAVED / "DDC" + for directory in (OUT, output, userdir, zendata, ddc): + directory.mkdir(parents=True, exist_ok=True) + trace.unlink(missing_ok=True) + utrace.unlink(missing_ok=True) + spawn_cm = ",".join(f"{value * 100.0:.3f}" for value in SPAWNS_M[stratum]) + command = [ + str(EDITOR), str(HOST / "VFHost.uproject"), "-nullrhi", "-unattended", + "-nop4", "-nosplash", "-NoSound", "-corelimit=12", "-NoZenAutoLaunch", + "-notraceserver", "-nocrashreports", + "-DisablePlugins=AndroidFileServer", + "-ddc=NoZenLocalFallback", f"-userdir={userdir}", f"-ZenDataPath={zendata}", + f"-LocalDataCachePath={ddc}", "-run=VoxelForgeExplore", "-seed=0", + "-archetype=TunnelNetwork", f"-slot={1 if stratum == 'FI' else 0}", + "-modes=export", "-gameconfig=1", "-coreworldtest=1", + "-ownerfourstrateplan=1", "-corevoxelsizecm=25", "-coreislandtest=0", + "-corefallbacktest=0", "-coreofftest=0", "-coreplantest=0", + "-settingsref=/Game/VoxelForge/DA_Settings.DA_Settings", "-corehybridmesher=0", + f"-out={output}", "-voxel.StreamingCoarseFirst=1", + "-voxel.StreamingDynamicNearReserve=1", + "-voxel.StreamingLegacyPendingResultCap=0", + "-voxel.CoreRegionConcurrentAdmission=1", + "-voxel.StreamingResultBacklogCap=256", + f"-voxel.StreamingReadyQueueDepth={ready_depth}", + f"-voxel.StreamingBackgroundWorkerCap={background_cap}", + f"-voxel.StreamingBackgroundWorkerReserve={worker_reserve}", + f"-voxel.CoreParallelForBackgroundPriority={helper_background}", + f"-voxel.StreamingTileBackgroundLow={tile_background_low}", + f"-voxel.StreamingTileBackgroundNormal={tile_background_normal}", + f"-voxel.TestLogTickFunctions={int(tick_logs)}", + "-voxel.CoreColumnCacheDirectMapped=0", "-voxel.CoreMCGradientMemoization=1", + "-voxel.StreamingCollisionLookahead=1", + f"-voxel.StreamingStartupApplyBoost={startup_boost}", + "-voxel.TestMove=1", "-voxel.TestMoveSpeedCmPerSecond=140.0", + f"-voxel.TestMoveStartSeconds={MOVE_START_S:.1f}", + f"-voxel.TestMoveDurationSeconds={MOVE_DURATION_S:.1f}", + f"-voxel.TestExitSeconds={EXIT_S:.1f}", "-voxel.TestMoveDirection=1,0,0", + "-voxel.StreamingSchedulerPolicy=3", "-voxel.TestExitOnSteady=0", + "-coreworldstreamtrace=1", f"-voxel.StartupTraceSpawnCm={spawn_cm}", + "-voxel.StartupTraceUntilEndPlay=1", "-voxel.StartupTraceThroughCrossing=1", + "-voxel.StartupTraceMaxTiles=16384", f"-voxel.StartupTraceFile={trace}", + f"-abslog={abslog}", + ] + if insights: + command += ["-trace=cpu,task,frame", "-statnamedevents", + f"-tracefile={utrace}", "-tracefiletrunc"] + wrapped = ["taskset", "-c", "0-11", "timeout", f"{RUN_TIMEOUT_S}s", *command] + started = time.monotonic() + with stdout.open("w", encoding="utf-8") as stream: + process = subprocess.Popen(wrapped, cwd=ROOT, stdout=stream, + stderr=subprocess.STDOUT) + peak_rss = None + while process.poll() is None: + observed = sample_unreal_peak_rss() + if observed and (peak_rss is None or observed[0] > peak_rss[0]): + peak_rss = observed + time.sleep(0.25) + exit_code = process.wait() + observed = sample_unreal_peak_rss() + if observed and (peak_rss is None or observed[0] > peak_rss[0]): + peak_rss = observed + + events = read_events(trace) + worker_backend_config = next( + (e for e in events if e.get("type") == "worker_backend_config"), {}) + worker_cap_readback = next( + (e for e in events if e.get("type") == "streaming_worker_cap_readback"), {}) + samples = [e for e in events if e.get("type") == "streaming_sample"] + arrival = [e for e in samples if float(e.get("t_s", 0.0)) < MOVE_START_S] + full = next((e for e in arrival + if sum(e.get("desired_by_lod", [0] * 9)[:3]) > 0 + and sum(e.get("settled_by_lod", [0] * 9)[:3]) + >= sum(e.get("desired_by_lod", [0] * 9)[:3])), None) + first_admission = next((e for e in events if e.get("type") == "scheduler_admission" + and e.get("desired_rebuilt") == 1), None) + request_events = [e for e in events if e.get("type") == "request"] + initial_request_events = [e for e in request_events if e.get("epoch") == 0] + tile_by_ordinal = {e.get("request_ordinal"): e for e in events + if e.get("type") == "tile"} + collision_count = sum(1 for e in events if e.get("type") == "collision_ready" + and e.get("level", 9) <= 2 + and full and e.get("t_s", 1e30) <= full["t_s"]) + initial_tiles = [tile_by_ordinal[r.get("request_ordinal")] for r in initial_request_events + if full and r.get("t_s", 1e30) <= full["t_s"] + and r.get("request_ordinal") in tile_by_ordinal + and tile_by_ordinal[r["request_ordinal"]].get("level", 9) <= 2] + burst_start_s = float(first_admission["t_s"]) if first_admission else -1.0 + burst_end_s = float(full["t_s"]) if full else -1.0 + work_by_frame = {int(e.get("frame", -1)): e for e in events + if e.get("type") == "game_thread_work_frame"} + work_values = [e for e in work_by_frame.values() + if burst_start_s <= float(e.get("t_s", -1.0)) <= burst_end_s] + arrival_frame_ids = {int(e.get("frame", -1)) for e in work_values} + scheduler_by_frame = {int(e.get("frame", -1)): e for e in events + if e.get("type") == "scheduler_admission"} + scheduler_values = [e for e in events if e.get("type") == "scheduler_admission" + and burst_start_s <= float(e.get("t_s", -1.0)) <= burst_end_s] + apply_by_frame = {int(e.get("frame", -1)): e for e in events + if e.get("type") == "apply_frame"} + commandlet_tick_by_frame = {int(e.get("frame", -1)): e for e in events + if e.get("type") == "commandlet_world_tick_frame"} + garbage_collect_by_frame = {int(e.get("frame", -1)): e for e in events + if e.get("type") == "garbage_collect"} + # apply_frame N+1 records the interval containing AVoxelWorld::Tick N. Its apply_ms + # belongs to Tick N+1, so use apply_by_frame[N] for the same-tick apply breakdown. + near_intervals = [apply_by_frame[frame + 1] for frame in arrival_frame_ids + if frame + 1 in apply_by_frame] + rmc_by_frame: dict[int, dict[str, float]] = {} + for event in events: + frame = int(event.get("frame", -1)) + if event.get("type") != "rmc_submit" or frame not in arrival_frame_ids: + continue + values = rmc_by_frame.setdefault(frame, { + "component_acquire_ms": 0.0, + "create_section_group_ms": 0.0, + "create_sections_ms": 0.0, + "collision_handoff_ms": 0.0, + }) + for key in values: + values[key] += float(event.get(key, 0.0)) + collision_completion_by_frame: dict[int, float] = {} + for event in events: + frame = int(event.get("frame", -1)) + if event.get("type") == "collision_completion" and frame in arrival_frame_ids: + collision_completion_by_frame[frame] = collision_completion_by_frame.get(frame, 0.0) \ + + float(event.get("handler_ms", 0.0)) + near_requests = [r for r in request_events if full and r.get("t_s", 1e30) <= full["t_s"] + and r.get("level", 9) <= 2] + initial_lod0_counts = Counter( + tuple(r.get("tile", ())) for r in near_requests if r.get("level") == 0) + repeated_lod0_keys = [ + {"tile": list(tile), "requests": count} + for tile, count in sorted(initial_lod0_counts.items()) if count > 1] + initial_scheduler_frames = [e for e in events + if e.get("type") == "scheduler_admission" and full + and e.get("t_s", 1e30) <= full["t_s"]] + worst_interval = max(near_intervals, key=lambda e: float(e.get("frame_ms", 0.0)), default=None) + worst_interval_frame_id = int(worst_interval.get("frame", -1)) if worst_interval else -1 + worst_work_frame_id = worst_interval_frame_id - 1 + worst_work = work_by_frame.get(worst_work_frame_id, {}) + worst_apply = apply_by_frame.get(worst_work_frame_id, {}) + worst_scheduler = scheduler_by_frame.get(worst_work_frame_id, {}) + worst_rmc = rmc_by_frame.get(worst_work_frame_id, {}) + worst_collision_completion_ms = collision_completion_by_frame.get(worst_work_frame_id, 0.0) + rmc_values = [rmc_by_frame.get(frame, {}) for frame in arrival_frame_ids] + apply_values = [apply_by_frame[frame] for frame in arrival_frame_ids + if frame in apply_by_frame] + commandlet_tick_values = [commandlet_tick_by_frame[frame] for frame in arrival_frame_ids + if frame in commandlet_tick_by_frame] + garbage_collect_values = [garbage_collect_by_frame[frame] for frame in arrival_frame_ids + if frame in garbage_collect_by_frame] + + def work_stat(name: str, fraction: float | None = None) -> float | None: + values = [float(e.get(name, 0.0)) for e in work_values] + return percentile(values, fraction) if fraction is not None else max(values, default=0.0) + + def scheduler_stat(name: str, fraction: float | None = None) -> float | None: + values = [float(e.get(name, 0.0)) for e in scheduler_values] + return percentile(values, fraction) if fraction is not None else max(values, default=0.0) + + def rmc_stat(name: str, fraction: float | None = None) -> float | None: + values = [float(e.get(name, 0.0)) for e in rmc_values] + return percentile(values, fraction) if fraction is not None else max(values, default=0.0) + + def commandlet_stat(name: str, fraction: float | None = None) -> float | None: + values = [float(e.get(name, 0.0)) for e in commandlet_tick_values] + return percentile(values, fraction) if fraction is not None else max(values, default=0.0) + + row = { + "label": label, "stratum": stratum, "repeat": repeat, "phase": phase, + "ready_queue_depth": ready_depth, + "requested_background_worker_cap": background_cap, + "background_worker_cap": worker_cap_readback.get( + "streaming_background_worker_cap", background_cap), + "requested_background_worker_reserve": worker_reserve, + "background_worker_reserve": worker_backend_config.get( + "streaming_background_worker_reserve", worker_cap_readback.get( + "background_worker_reserve", worker_reserve)), + "requested_core_parallel_for_background_priority": helper_background, + "core_parallel_for_background_priority": worker_backend_config.get( + "core_parallel_for_background_priority", helper_background), + "requested_streaming_tile_background_low": tile_background_low, + "streaming_tile_background_low": worker_backend_config.get( + "streaming_tile_background_low", tile_background_low), + "requested_streaming_tile_background_normal": tile_background_normal, + "streaming_tile_background_normal": worker_backend_config.get( + "streaming_tile_background_normal", tile_background_normal), + "effective_tile_slots": worker_backend_config.get("effective_tile_slots"), + "background_worker_threads": worker_backend_config.get("taskgraph_background_threads"), + "startup_apply_boost": startup_boost, "insights": insights, + "engine_tick_logs": tick_logs, + "exit_code": exit_code, + "launch_wrapper": "taskset -c 0-11 timeout 1740s (no address-space limit)", + "peak_rss_kib": peak_rss[0] if peak_rss else None, + "peak_rss_mib": round(peak_rss[0] / 1024.0, 1) if peak_rss else None, + "peak_rss_pid": peak_rss[1] if peak_rss else None, + "endphysics_to_postphysics_gap_ms_max": None, + "endphysics_gap_frame": None, + "wall_s": round(time.monotonic() - started, 3), + "desired_t_s": next((e.get("t_s") for e in events if e.get("type") == "desired_set"), None), + "first_admission_t_s": first_admission.get("t_s") if first_admission else None, + "lod02_full_settled_t_s": full.get("t_s") if full else None, + "near_settle_from_first_admission_s": round( + float(full["t_s"]) - float(first_admission["t_s"]), 4) + if full and first_admission else None, + "desired_by_lod": full.get("desired_by_lod", [])[:3] if full else None, + "settled_by_lod": full.get("settled_by_lod", [])[:3] if full else None, + "near_admitted_tiles": len(near_requests), + "initial_lod0_admissions": sum(initial_lod0_counts.values()), + "repeated_lod0_request_keys_before_settle": repeated_lod0_keys, + "peak_result_backlog_before_settle": max( + (int(e.get("result_backlog", 0)) for e in initial_scheduler_frames), default=0), + "last_admission_t_s": max((e.get("t_s", 0.0) for e in near_requests), default=None), + "near_tile_events": len(initial_tiles), + "near_worker_seconds": round(sum(e.get("worker_task_s", 0.0) for e in initial_tiles), 4), + "near_core_seconds": round(sum(e.get("core_generate_s", 0.0) for e in initial_tiles), 4), + "collision_ready_by_settle": collision_count, + "burst_frame_count": len(near_intervals), + "burst_frame_ms_p95": round(sorted(e.get("frame_ms", 0.0) for e in near_intervals)[ + min(len(near_intervals) - 1, int(.95 * (len(near_intervals) - 1)))], 3) + if near_intervals else None, + "burst_frame_ms_max": round(max((e.get("frame_ms", 0.0) for e in near_intervals), default=0.0), 3), + "burst_full_world_tick_ms_p99": round(commandlet_stat("world_tick_ms", .99) or 0.0, 3), + "burst_full_world_tick_ms_max": round(commandlet_stat("world_tick_ms") or 0.0, 3), + "burst_world_pre_actor_ms_p99": round(commandlet_stat("world_pre_actor_ms", .99) or 0.0, 3), + "burst_world_pre_actor_ms_max": round(commandlet_stat("world_pre_actor_ms") or 0.0, 3), + "burst_world_actor_phase_ms_p99": round(commandlet_stat("world_actor_phase_ms", .99) or 0.0, 3), + "burst_world_actor_phase_ms_max": round(commandlet_stat("world_actor_phase_ms") or 0.0, 3), + "burst_world_post_actor_ms_p99": round(commandlet_stat("world_post_actor_ms", .99) or 0.0, 3), + "burst_world_post_actor_ms_max": round(commandlet_stat("world_post_actor_ms") or 0.0, 3), + "burst_commandlet_readiness_check_ms_max": round( + commandlet_stat("readiness_check_ms") or 0.0, 3), + "burst_commandlet_sleep_ms_p99": round(commandlet_stat("sleep_ms", .99) or 0.0, 3), + "burst_commandlet_sleep_ms_max": round(commandlet_stat("sleep_ms") or 0.0, 3), + "burst_commandlet_loop_ms_max": round(commandlet_stat("loop_ms") or 0.0, 3), + "burst_gc_count": len(garbage_collect_values), + "burst_gc_duration_ms_max": round(max( + (float(e.get("duration_ms", 0.0)) for e in garbage_collect_values), default=0.0), 3), + "burst_max_meshes_per_frame": max((e.get("meshes_applied", 0) for e in apply_values), default=0), + "burst_max_apply_ms": round(max((e.get("apply_ms", 0.0) for e in apply_values), default=0.0), 3), + "burst_max_admit_per_frame": max((sum(e.get("submitted_by_lod", [0] * 9)[:3]) + for e in scheduler_values), default=0), + "burst_frame_timing_count": len(work_values), + "burst_active_generation_workers_p95": round(work_stat("active_generation_workers", .95) or 0.0, 3), + "burst_active_generation_workers_max": round(work_stat("active_generation_workers") or 0.0, 3), + "burst_active_tile_tasks_max": round(work_stat("active_tile_tasks") or 0.0, 3), + "burst_world_tick_ms_p99": round(work_stat("tick_ms", .99) or 0.0, 3), + "burst_world_tick_ms_max": round(work_stat("tick_ms") or 0.0, 3), + "burst_update_chunks_ms_p99": round(work_stat("update_chunks_ms", .99) or 0.0, 3), + "burst_update_chunks_ms_max": round(work_stat("update_chunks_ms") or 0.0, 3), + "burst_pending_drain_ms_p99": round(work_stat("pending_drain_ms", .99) or 0.0, 3), + "burst_pending_drain_ms_max": round(work_stat("pending_drain_ms") or 0.0, 3), + "burst_post_apply_callbacks_ms_p99": round(work_stat("post_apply_callbacks_ms", .99) or 0.0, 3), + "burst_post_apply_callbacks_ms_max": round(work_stat("post_apply_callbacks_ms") or 0.0, 3), + "burst_visibility_reconcile_ms_p99": round(work_stat("visibility_reconcile_ms", .99) or 0.0, 3), + "burst_visibility_reconcile_ms_max": round(work_stat("visibility_reconcile_ms") or 0.0, 3), + "burst_collision_probe_ms_max": round(work_stat("collision_probe_ms") or 0.0, 3), + "burst_visible_apply_probe_ms_max": round(work_stat("visible_apply_probe_ms") or 0.0, 3), + "burst_lod0_ready_ms_max": round(work_stat("lod0_ready_ms") or 0.0, 3), + "burst_steady_trace_ms_max": round(work_stat("steady_trace_ms") or 0.0, 3), + "burst_modification_ms_max": round(work_stat("modification_ms") or 0.0, 3), + "burst_lifecycle_audit_ms_p99": round(work_stat("lifecycle_audit_ms", .99) or 0.0, 3), + "burst_lifecycle_audit_ms_max": round(work_stat("lifecycle_audit_ms") or 0.0, 3), + "burst_collision_gate_ms_p99": round(work_stat("collision_gate_ms", .99) or 0.0, 3), + "burst_collision_gate_ms_max": round(work_stat("collision_gate_ms") or 0.0, 3), + "burst_unload_ms_p99": round(work_stat("unload_ms", .99) or 0.0, 3), + "burst_unload_ms_max": round(work_stat("unload_ms") or 0.0, 3), + "burst_trace_json_ms_p99": round(work_stat("trace_json_ms", .99) or 0.0, 3), + "burst_trace_json_ms_max": round(work_stat("trace_json_ms") or 0.0, 3), + "burst_desired_rebuild_ms_max": round(scheduler_stat("desired_rebuild_ms") or 0.0, 3), + "burst_rmc_create_group_ms_p99": round(rmc_stat("create_section_group_ms", .99) or 0.0, 3), + "burst_rmc_create_group_ms_max": round(rmc_stat("create_section_group_ms") or 0.0, 3), + "burst_rmc_collision_handoff_ms_max": round(rmc_stat("collision_handoff_ms") or 0.0, 3), + "burst_collision_completion_ms_max": round(max( + collision_completion_by_frame.values(), default=0.0), 3), + "worst_frame_breakdown": { + "frame": worst_work_frame_id, + "world_tick_frame": worst_work_frame_id, + "frame_interval_end_frame": worst_interval_frame_id, + "frame_interval_ms": round(float(worst_interval.get("frame_ms", 0.0)), 3) + if worst_interval else None, + "apply_ms": round(float(worst_apply.get("apply_ms", 0.0)), 3) + if worst_apply else None, + "result_drain_ms": round(float(worst_apply.get("drain_ms", 0.0)), 3) + if worst_apply else None, + "outside_world_tick_ms": round(max(0.0, float(worst_interval.get("frame_ms", 0.0)) + - float(worst_work.get("tick_ms", 0.0))), 3) + if worst_interval else None, + "outside_full_world_tick_ms": round(max(0.0, float(worst_interval.get("frame_ms", 0.0)) + - float(commandlet_tick_by_frame.get(worst_work_frame_id, {}).get("world_tick_ms", 0.0))), 3) + if worst_interval else None, + "full_world_tick_ms": round(float(commandlet_tick_by_frame.get( + worst_work_frame_id, {}).get("world_tick_ms", 0.0)), 3), + "world_pre_actor_ms": round(float(commandlet_tick_by_frame.get( + worst_work_frame_id, {}).get("world_pre_actor_ms", 0.0)), 3), + "world_actor_phase_ms": round(float(commandlet_tick_by_frame.get( + worst_work_frame_id, {}).get("world_actor_phase_ms", 0.0)), 3), + "world_post_actor_ms": round(float(commandlet_tick_by_frame.get( + worst_work_frame_id, {}).get("world_post_actor_ms", 0.0)), 3), + "commandlet_readiness_check_ms": round(float(commandlet_tick_by_frame.get( + worst_work_frame_id, {}).get("readiness_check_ms", 0.0)), 3), + "commandlet_sleep_ms": round(float(commandlet_tick_by_frame.get( + worst_work_frame_id, {}).get("sleep_ms", 0.0)), 3), + "commandlet_loop_ms": round(float(commandlet_tick_by_frame.get( + worst_work_frame_id, {}).get("loop_ms", 0.0)), 3), + "gc_duration_ms": round(float(garbage_collect_by_frame.get( + worst_work_frame_id, {}).get("duration_ms", 0.0)), 3), + "world_tick_ms": round(float(worst_work.get("tick_ms", 0.0)), 3), + "active_generation_workers": int(worst_work.get("active_generation_workers", 0)), + "active_tile_tasks": int(worst_work.get("active_tile_tasks", 0)), + "collision_gate_ms": round(float(worst_work.get("collision_gate_ms", 0.0)), 3), + "update_chunks_ms": round(float(worst_work.get("update_chunks_ms", 0.0)), 3), + "desired_rebuild_ms": round(float(worst_scheduler.get("desired_rebuild_ms", 0.0)), 3), + "pending_drain_ms": round(float(worst_work.get("pending_drain_ms", 0.0)), 3), + "post_apply_callbacks_ms": round(float(worst_work.get("post_apply_callbacks_ms", 0.0)), 3), + "collision_probe_ms": round(float(worst_work.get("collision_probe_ms", 0.0)), 3), + "visibility_reconcile_ms": round(float(worst_work.get("visibility_reconcile_ms", 0.0)), 3), + "visible_apply_probe_ms": round(float(worst_work.get("visible_apply_probe_ms", 0.0)), 3), + "lod0_ready_ms": round(float(worst_work.get("lod0_ready_ms", 0.0)), 3), + "unload_ms": round(float(worst_work.get("unload_ms", 0.0)), 3), + "steady_trace_ms": round(float(worst_work.get("steady_trace_ms", 0.0)), 3), + "modification_ms": round(float(worst_work.get("modification_ms", 0.0)), 3), + "lifecycle_audit_ms": round(float(worst_work.get("lifecycle_audit_ms", 0.0)), 3), + "trace_json_ms": round(float(worst_work.get("trace_json_ms", 0.0)), 3), + "trace_archive_serialize_ms": round(float( + worst_work.get("trace_archive_serialize_ms", 0.0)), 3), + "collision_completion_ms": round(worst_collision_completion_ms, 3), + **{key: round(float(worst_rmc.get(key, 0.0)), 3) + for key in ("component_acquire_ms", "create_section_group_ms", + "create_sections_ms", "collision_handoff_ms")}, + }, + "trace": str(trace.relative_to(ROOT)), "utrace": str(utrace.relative_to(ROOT)), + "abslog": str(abslog.relative_to(ROOT)), "stdout": str(stdout.relative_to(ROOT)), + } + stdout_text = stdout.read_text(encoding="utf-8", errors="replace") + gap_ms, gap_frame = endphysics_gap(stdout) + row["endphysics_to_postphysics_gap_ms_max"] = gap_ms + row["endphysics_gap_frame"] = gap_frame + failure = re.search(r"\[VoxelForgeCoreWorldTest\] failed: ([^\r\n]+)", stdout_text) + if failure: + row["world_test_failure"] = failure.group(1) + with (OUT / "runs.jsonl").open("a", encoding="utf-8") as stream: + stream.write(json.dumps(row, sort_keys=True) + "\n") + (OUT / f"{label}.summary.json").write_text(json.dumps(row, indent=2) + "\n", + encoding="utf-8") + print("[L106UnrealArrival] " + json.dumps(row, sort_keys=True), flush=True) + if exit_code: + raise SystemExit(f"Unreal commandlet failed: {stdout}; exit={exit_code}") + if not full or len(arrival) < 8: + raise SystemExit(f"Unreal arrival trace incomplete: {stdout}; exit={exit_code}") + if insights and (not utrace.is_file() or utrace.stat().st_size < 1024): + raise SystemExit(f"Unreal Insights trace missing or empty: {utrace}") + return row + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--stratum", choices=("FI", "SW"), required=True) + parser.add_argument("--repeat", type=int, default=1) + parser.add_argument("--phase", choices=("probe", "before", "after", "snapshot"), default="probe") + parser.add_argument("--ready-depth", type=int, default=64) + parser.add_argument("--background-cap", type=int, choices=(0, 1), default=1) + parser.add_argument("--worker-reserve", type=int, choices=(0, 1, 2, 3), default=0) + parser.add_argument("--helper-background", type=int, choices=(0, 1), default=1) + parser.add_argument("--tile-background-low", type=int, choices=(0, 1), default=1) + parser.add_argument("--tile-background-normal", type=int, choices=(0, 1), default=0) + parser.add_argument("--startup-boost", type=int, choices=(0, 1), default=1) + parser.add_argument("--tick-logs", action="store_true") + parser.add_argument("--insights", action="store_true") + args = parser.parse_args() + if not EDITOR.is_file(): + raise SystemExit(f"missing Unreal commandlet executable: {EDITOR}") + run(args.stratum, args.repeat, args.phase, args.ready_depth, + args.background_cap, args.worker_reserve, args.helper_background, + args.tile_background_low, args.tile_background_normal, args.startup_boost, + args.tick_logs, args.insights) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/contact-sheet.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/contact-sheet.png new file mode 100644 index 0000000..a72bad8 Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/contact-sheet.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-lower-mouth-after-l106.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-lower-mouth-after-l106.png new file mode 100644 index 0000000..e66c556 Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-lower-mouth-after-l106.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-lower-mouth-before-l105.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-lower-mouth-before-l105.png new file mode 100644 index 0000000..e66c556 Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-lower-mouth-before-l105.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-mid-50-after-l106.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-mid-50-after-l106.png new file mode 100644 index 0000000..470075a Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-mid-50-after-l106.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-mid-50-before-l105.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-mid-50-before-l105.png new file mode 100644 index 0000000..470075a Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-mid-50-before-l105.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-upper-mouth-after-l106.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-upper-mouth-after-l106.png new file mode 100644 index 0000000..40992af Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-upper-mouth-after-l106.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-upper-mouth-before-l105.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-upper-mouth-before-l105.png new file mode 100644 index 0000000..40992af Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-0-1-0-upper-mouth-before-l105.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-lower-mouth-after-l106.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-lower-mouth-after-l106.png new file mode 100644 index 0000000..dc231dc Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-lower-mouth-after-l106.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-lower-mouth-before-l105.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-lower-mouth-before-l105.png new file mode 100644 index 0000000..dc231dc Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-lower-mouth-before-l105.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-mid-50-after-l106.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-mid-50-after-l106.png new file mode 100644 index 0000000..fcbfdaf Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-mid-50-after-l106.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-mid-50-before-l105.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-mid-50-before-l105.png new file mode 100644 index 0000000..fcbfdaf Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-mid-50-before-l105.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-upper-mouth-after-l106.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-upper-mouth-after-l106.png new file mode 100644 index 0000000..759be2b Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-upper-mouth-after-l106.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-upper-mouth-before-l105.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-upper-mouth-before-l105.png new file mode 100644 index 0000000..759be2b Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-1-2-0-upper-mouth-before-l105.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-lower-mouth-after-l106.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-lower-mouth-after-l106.png new file mode 100644 index 0000000..d89372f Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-lower-mouth-after-l106.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-lower-mouth-before-l105.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-lower-mouth-before-l105.png new file mode 100644 index 0000000..d89372f Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-lower-mouth-before-l105.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-mid-50-after-l106.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-mid-50-after-l106.png new file mode 100644 index 0000000..178f6a3 Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-mid-50-after-l106.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-mid-50-before-l105.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-mid-50-before-l105.png new file mode 100644 index 0000000..178f6a3 Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-mid-50-before-l105.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-upper-mouth-after-l106.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-upper-mouth-after-l106.png new file mode 100644 index 0000000..09fc90d Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-upper-mouth-after-l106.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-upper-mouth-before-l105.png b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-upper-mouth-before-l105.png new file mode 100644 index 0000000..09fc90d Binary files /dev/null and b/Standalone/Research/WorldAssemblerM3/Linux106/captures/passage-2-3-0-upper-mouth-before-l105.png differ diff --git a/Standalone/Research/WorldAssemblerM3/Linux106/patches/band-column-cache-not-merged.patch b/Standalone/Research/WorldAssemblerM3/Linux106/patches/band-column-cache-not-merged.patch new file mode 100644 index 0000000..e76524c --- /dev/null +++ b/Standalone/Research/WorldAssemblerM3/Linux106/patches/band-column-cache-not-merged.patch @@ -0,0 +1,272 @@ +diff --git a/Standalone/Core/include/VoxelForgeCore/VoxelForgeCore.h b/Standalone/Core/include/VoxelForgeCore/VoxelForgeCore.h +index 972aaf8..94ec19b 100644 +--- a/Standalone/Core/include/VoxelForgeCore/VoxelForgeCore.h ++++ b/Standalone/Core/include/VoxelForgeCore/VoxelForgeCore.h +@@ -139,6 +139,9 @@ namespace VoxelForgeCore + std::uint64_t passageTileRouteSegmentCandidates = 0; + std::uint64_t passageTileMouthSegmentCandidates = 0; + std::uint64_t resolvedBoundaryBandQueries = 0; ++ std::uint64_t resolvedBoundaryBandColumnEvaluations = 0; ++ std::uint64_t resolvedBoundaryBandColumnCacheHits = 0; ++ std::uint64_t resolvedBoundaryBandColumnOffGridFallbacks = 0; + std::uint64_t passageComposerQueries = 0; + std::uint64_t passageIterations = 0; + std::uint64_t passageTileBoundChecks = 0; +diff --git a/Standalone/Core/src/ShapeWorldInternal.h b/Standalone/Core/src/ShapeWorldInternal.h +index 8626d7e..2c53c35 100644 +--- a/Standalone/Core/src/ShapeWorldInternal.h ++++ b/Standalone/Core/src/ShapeWorldInternal.h +@@ -339,6 +339,15 @@ public: + verticalBoundSkipsByCaller{},negativeReachSkipsByCaller{}, + positiveReachSkipsByCaller{}; + }; ++ struct FResolvedBoundaryColumnData ++ { ++ std::int32_t minimumXVoxel=0,minimumYVoxel=0,stepVoxel=0,dimension=0; ++ std::size_t pairCount=0; ++ std::vector pairReady; ++ std::vector bands; ++ std::uint64_t evaluations=0; ++ mutable std::atomic cacheHits{0},offGridFallbacks{0}; ++ }; + FScopedPassageTileQueries(const FStandaloneShapeWorld& shape, + const std::array& minimumSubvoxels, + const std::array& maximumSubvoxels, +@@ -502,6 +511,8 @@ public: + passageCarveReachMetres_(source.passageCarveReachMetres_), + passageSweepData_(source.passageSweepData_), + ownsPassageSweepDiagnostics_(false), ++ resolvedBoundaryColumnData_(source.resolvedBoundaryColumnData_), ++ ownsResolvedBoundaryColumnDiagnostics_(false), + clampCounters_(source.clampCounters_),ownsClampCounters_(false), + projectionCacheSize_(source.projectionCacheSize_), + reuseCounters_(source.reuseCounters_),ownsReuseCounters_(false) +@@ -546,6 +557,7 @@ public: + std::memory_order_relaxed); + } + } ++ copyResolvedBoundaryColumnDiagnostics(); + if(ownsReuseCounters_&&reuseCounters_) + std::printf("[L98PassageReuseTile] lod=%d projection_columns=%llu projection_computes=%llu projection_hits=%llu floor_profile_reads=%llu carve_prepared=%llu support_prepared=%llu capsule_prepared=%llu mouth_prepared=%llu\n", + profileLod_, +@@ -626,6 +638,23 @@ public: + bool capturesCallerDiagnostics() const noexcept{return profileStats_!=nullptr;} + void prepareSweptTubeBase(const FIntVector& origin,std::int32_t step, + std::int32_t cells,std::uint32_t seed); ++ void prepareResolvedBoundaryColumns(const FIntVector& origin, ++ std::int32_t step,std::int32_t cells); ++ const VoxelForgeCore::ResolvedLayoutTransitionBandsV1* cachedTransitionBands( ++ float X,float Y,const VoxelForgeCore::ResolvedLayoutStratumV2& upper, ++ const VoxelForgeCore::ResolvedLayoutStratumV2& lower) const noexcept; ++ void copyResolvedBoundaryColumnDiagnostics() const noexcept ++ { ++ if(!ownsResolvedBoundaryColumnDiagnostics_||!profileStats_ ++ ||!resolvedBoundaryColumnData_)return; ++ profileStats_->resolvedBoundaryBandColumnEvaluations= ++ resolvedBoundaryColumnData_->evaluations; ++ profileStats_->resolvedBoundaryBandColumnCacheHits= ++ resolvedBoundaryColumnData_->cacheHits.load(std::memory_order_relaxed); ++ profileStats_->resolvedBoundaryBandColumnOffGridFallbacks= ++ resolvedBoundaryColumnData_->offGridFallbacks.load( ++ std::memory_order_relaxed); ++ } + const FPassageSweepSample* sweptSample(std::size_t passageIndex, + double qx,double qy,double qz) const; + bool hasSweptTubeBase() const noexcept{return passageSweepData_!=nullptr;} +@@ -917,6 +946,8 @@ public: + double passageCarveReachMetres_=0.0; + std::shared_ptr passageSweepData_; + bool ownsPassageSweepDiagnostics_=true; ++ std::shared_ptr resolvedBoundaryColumnData_; ++ bool ownsResolvedBoundaryColumnDiagnostics_=true; + mutable std::shared_ptr clampCounters_; + bool ownsClampCounters_=true; + std::size_t projectionCacheSize_=0; +@@ -1546,8 +1577,12 @@ public: + static_cast(X)*Owner->CoreVoxelSizeMicrons)); + const auto yMicrometres=static_cast(std::llround( + static_cast(Y)*Owner->CoreVoxelSizeMicrons)); +- const auto bands=ResolvedTransitionBandsAtXY(*transitionUpper, +- *transitionLower,xMicrometres,yMicrometres,skipPassageFade); ++ const auto* cachedBands=tileQueries&&tileQueries->matches(*this) ++ ?tileQueries->cachedTransitionBands(X,Y,*transitionUpper, ++ *transitionLower):nullptr; ++ const auto bands=cachedBands?*cachedBands:ResolvedTransitionBandsAtXY( ++ *transitionUpper,*transitionLower,xMicrometres,yMicrometres, ++ skipPassageFade); + const std::int64_t solidBottom=std::min( + bands.upperBedrockBottomMicrometres,bands.lowerCeilingBottomMicrometres); + const std::int64_t solidTop=std::max( +@@ -1609,8 +1644,11 @@ public: + #if defined(VF_L30_ENABLE_COST_COUNTERS) + if(skipPassageFade)++ThreadClassificationStats().TransitionPassageTileSkips; + #endif +- const auto resolvedBands=ResolvedTransitionBandsAtXY(*upper,*lower, +- xMicrometres,yMicrometres,skipPassageFade); ++ const auto* cachedBands=tileQueries&&tileQueries->matches(*this) ++ ?tileQueries->cachedTransitionBands(X,Y,*upper,*lower):nullptr; ++ const auto resolvedBands=cachedBands?*cachedBands: ++ ResolvedTransitionBandsAtXY(*upper,*lower,xMicrometres,yMicrometres, ++ skipPassageFade); + // Treat the two authored bands as one continuous solid interval. Its + // exterior faces remain the lower ceiling bottom and upper bedrock top; + // measuring signed distance from those faces also clears native generator +@@ -14424,6 +14462,115 @@ private: + inline static thread_local const FPassageCandidateBrick* ActivePassageCandidateBrick=nullptr; + }; + ++inline void FStandaloneShapeWorld::FScopedPassageTileQueries::prepareResolvedBoundaryColumns( ++ const FIntVector& origin,std::int32_t step,std::int32_t cells) ++{ ++ if(!shape_||!mayContainResolvedBoundaryBand_||step<=0||cells<=0 ++ ||!shape_->ResolvedLayout||!shape_->Owner ++ ||shape_->Owner->CoreVoxelSizeMicrons==0)return; ++ ++ constexpr std::int64_t Q=VoxelForgeCore::WorldAssemblySubvoxelsPerVoxel; ++ const auto& layout=*shape_->ResolvedLayout; ++ auto data=std::make_shared(); ++ data->minimumXVoxel=origin.X-step; ++ data->minimumYVoxel=origin.Y-step; ++ data->stepVoxel=step; ++ data->dimension=cells+3; ++ for(const auto& stratum:layout.strates) ++ if(stratum.order>=0) ++ data->pairCount=std::max(data->pairCount, ++ static_cast(stratum.order)+1); ++ if(data->pairCount==0)return; ++ data->pairReady.assign(data->pairCount,0); ++ const double minimumZMicrometres=static_cast(minimumSubvoxels_[2]) ++ *shape_->Owner->CoreVoxelSizeMicrons/Q; ++ const double maximumZMicrometres=static_cast(maximumSubvoxels_[2]) ++ *shape_->Owner->CoreVoxelSizeMicrons/Q; ++ std::size_t activePairCount=0; ++ for(const auto& upper:layout.strates) ++ { ++ if(upper.order<0||static_cast(upper.order)>=data->pairCount ++ ||!shape_->IsResolvedLayoutTransitionActive(upper))continue; ++ const auto* lower=shape_->FindResolvedLayoutStratumByOrder(upper.order+1); ++ if(!lower)continue; ++ const auto maximumDisplacement= ++ shape_->ResolvedTransitionMaximumFaceDisplacementMicrometres(upper,*lower); ++ const double transitionMinimum=std::min(upper.bedrock.bottomMicrometres, ++ lower->ceiling.bottomMicrometres)-maximumDisplacement; ++ const double transitionMaximum=std::max(upper.bedrock.topMicrometres, ++ lower->ceiling.topMicrometres)+maximumDisplacement; ++ if(maximumZMicrometres=transitionMaximum)continue; ++ data->pairReady[static_cast(upper.order)]=1; ++ ++activePairCount; ++ } ++ if(activePairCount==0)return; ++ ++ const std::size_t columnCount=static_cast(data->dimension) ++ *static_cast(data->dimension); ++ if(columnCount>std::numeric_limits::max()/data->pairCount)return; ++ data->bands.resize(columnCount*data->pairCount); ++ data->evaluations=static_cast(columnCount)*activePairCount; ++ const bool skipPassageFade=!mayContainTransitionPassageFade_; ++ for(std::int32_t gy=0;gydimension;++gy) ++ for(std::int32_t gx=0;gxdimension;++gx) ++ { ++ const float x=static_cast(static_cast(data->minimumXVoxel) ++ +static_cast(gx)*data->stepVoxel); ++ const float y=static_cast(static_cast(data->minimumYVoxel) ++ +static_cast(gy)*data->stepVoxel); ++ const std::int64_t xMicrometres=static_cast(std::llround( ++ static_cast(x)*shape_->Owner->CoreVoxelSizeMicrons)); ++ const std::int64_t yMicrometres=static_cast(std::llround( ++ static_cast(y)*shape_->Owner->CoreVoxelSizeMicrons)); ++ const std::size_t column=static_cast(gy) ++ *static_cast(data->dimension)+static_cast(gx); ++ for(const auto& upper:layout.strates) ++ { ++ if(upper.order<0||static_cast(upper.order)>=data->pairCount ++ ||!data->pairReady[static_cast(upper.order)])continue; ++ const auto* lower=shape_->FindResolvedLayoutStratumByOrder(upper.order+1); ++ if(!lower)continue; ++ data->bands[column*data->pairCount+static_cast(upper.order)]= ++ shape_->ResolvedTransitionBandsAtXY(upper,*lower,xMicrometres, ++ yMicrometres,skipPassageFade); ++ } ++ } ++ resolvedBoundaryColumnData_=std::move(data); ++} ++ ++inline const VoxelForgeCore::ResolvedLayoutTransitionBandsV1* ++FStandaloneShapeWorld::FScopedPassageTileQueries::cachedTransitionBands( ++ float X,float Y,const VoxelForgeCore::ResolvedLayoutStratumV2& upper, ++ const VoxelForgeCore::ResolvedLayoutStratumV2& lower) const noexcept ++{ ++ const auto* data=resolvedBoundaryColumnData_.get(); ++ if(!data||upper.order<0||lower.order!=upper.order+1 ++ ||static_cast(upper.order)>=data->pairCount ++ ||!data->pairReady[static_cast(upper.order)])return nullptr; ++ const double xCoordinate=(static_cast(X)-data->minimumXVoxel)/data->stepVoxel; ++ const double yCoordinate=(static_cast(Y)-data->minimumYVoxel)/data->stepVoxel; ++ const auto gx=static_cast(std::llround(xCoordinate)); ++ const auto gy=static_cast(std::llround(yCoordinate)); ++ if(gx<0||gy<0||gx>=data->dimension||gy>=data->dimension ++ ||static_cast(static_cast(data->minimumXVoxel) ++ +gx*data->stepVoxel)!=X ++ ||static_cast(static_cast(data->minimumYVoxel) ++ +gy*data->stepVoxel)!=Y) ++ { ++ if(profileStats_) ++ data->offGridFallbacks.fetch_add(1,std::memory_order_relaxed); ++ return nullptr; ++ } ++ const std::size_t column=static_cast(gy) ++ *static_cast(data->dimension)+static_cast(gx); ++ const std::size_t index=column*data->pairCount ++ +static_cast(upper.order); ++ if(index>=data->bands.size())return nullptr; ++ if(profileStats_)data->cacheHits.fetch_add(1,std::memory_order_relaxed); ++ return &data->bands[index]; ++} ++ + inline void FStandaloneShapeWorld::FScopedPassageTileQueries::prepareSweptTubeBase( + const FIntVector& origin,std::int32_t step,std::int32_t cells,std::uint32_t seed) + { +diff --git a/Standalone/Core/src/VoxelForgeCore.cpp b/Standalone/Core/src/VoxelForgeCore.cpp +index 0a5d451..81a04b1 100644 +--- a/Standalone/Core/src/VoxelForgeCore.cpp ++++ b/Standalone/Core/src/VoxelForgeCore.cpp +@@ -2806,6 +2806,7 @@ namespace VoxelForgeCore + options.profilePassageComposer?&out.diagnostics:nullptr,key.lod, + options.diagnosticPassageSourceFloorVoxels, + options.diagnosticPassageCarveReachMetres); ++ passageFeatureScope.prepareResolvedBoundaryColumns(origin,MeshStep,MeshCells); + passageFeatureScope.prepareSweptTubeBase(origin,MeshStep,MeshCells, + static_cast(world.Seed)); + static std::atomic l98ProjectionProbeStarted{false}; +@@ -3367,6 +3368,7 @@ namespace VoxelForgeCore + out.diagnostics.meshAssemblyMilliseconds=std::chrono::duration( + std::chrono::steady_clock::now()-MeshAssemblyStart).count(); + CopyIslandCacheMetrics(); ++ passageFeatureScope.copyResolvedBoundaryColumnDiagnostics(); + return out; + } + +diff --git a/Standalone/Source/StreamingWorld.h b/Standalone/Source/StreamingWorld.h +index 3019e39..3f706f9 100644 +--- a/Standalone/Source/StreamingWorld.h ++++ b/Standalone/Source/StreamingWorld.h +@@ -1538,7 +1538,7 @@ private: + static_cast(D.islandSharedCacheConflictReplacements), + static_cast(D.islandSharedCacheLockWaitNanoseconds)/1000.0); + if(D.passageComposerProfiled) +- std::printf("[PassageComposerTile] key=(%d,%d,%d,lod%d) worker_ms=%.3f samples=%llu native_field_calls=%llu features=%llu route_candidates=%llu mouth_candidates=%llu boundary_band_queries=%llu queries=%llu passage_iterations=%llu tile_bound_checks=%llu tile_bound_rejects=%llu tile_bound_skipped_iterations=%llu upper_mouth_checks=%llu column_candidates=%llu column_distance_calls=%llu negative_bounds=%llu negative_rejects=%llu positive_bounds=%llu positive_rejects=%llu carve_segment_visits=%llu smooth_carve_joins=%llu support_segment_visits=%llu smooth_support_joins=%llu support_capsule_checks=%llu later_bore_checks=%llu other_capsule_checks=%llu mouth_support_calls=%llu mouth_route_segment_checks=%llu relief_field_calls=%llu smoothstep_evaluations=%llu sweep_segments=%llu sweep_pairs=%llu sweep_outside_pairs=%llu sweep_min_updates=%llu sweep_base_samples=%llu sweep_detail_samples=%llu sweep_grid_samples=%llu sweep_no_influence_samples=%llu sweep_lookups=%llu sweep_cache_hits=%llu sweep_base_hits=%llu sweep_outside_hits=%llu sweep_missing=%llu\n", ++ std::printf("[PassageComposerTile] key=(%d,%d,%d,lod%d) worker_ms=%.3f samples=%llu native_field_calls=%llu features=%llu route_candidates=%llu mouth_candidates=%llu boundary_band_queries=%llu boundary_column_evaluations=%llu boundary_column_hits=%llu boundary_column_offgrid=%llu queries=%llu passage_iterations=%llu tile_bound_checks=%llu tile_bound_rejects=%llu tile_bound_skipped_iterations=%llu upper_mouth_checks=%llu column_candidates=%llu column_distance_calls=%llu negative_bounds=%llu negative_rejects=%llu positive_bounds=%llu positive_rejects=%llu carve_segment_visits=%llu smooth_carve_joins=%llu support_segment_visits=%llu smooth_support_joins=%llu support_capsule_checks=%llu later_bore_checks=%llu other_capsule_checks=%llu mouth_support_calls=%llu mouth_route_segment_checks=%llu relief_field_calls=%llu smoothstep_evaluations=%llu sweep_segments=%llu sweep_pairs=%llu sweep_outside_pairs=%llu sweep_min_updates=%llu sweep_base_samples=%llu sweep_detail_samples=%llu sweep_grid_samples=%llu sweep_no_influence_samples=%llu sweep_lookups=%llu sweep_cache_hits=%llu sweep_base_hits=%llu sweep_outside_hits=%llu sweep_missing=%llu\n", + Event.Key.Coord.X,Event.Key.Coord.Y,Event.Key.Coord.Z,Event.Key.Level, + Event.WorkerMilliseconds,static_cast(Event.DensitySamples), + static_cast(D.fieldCalls), +@@ -1546,6 +1546,9 @@ private: + static_cast(D.passageTileRouteSegmentCandidates), + static_cast(D.passageTileMouthSegmentCandidates), + static_cast(D.resolvedBoundaryBandQueries), ++ static_cast(D.resolvedBoundaryBandColumnEvaluations), ++ static_cast(D.resolvedBoundaryBandColumnCacheHits), ++ static_cast(D.resolvedBoundaryBandColumnOffGridFallbacks), + static_cast(D.passageComposerQueries), + static_cast(D.passageIterations), + static_cast(D.passageTileBoundChecks),