diag: passage candidate compiler census (L90, opt-in); falsifier failed at 10 m reach
54,324 sparse 4 m bricks, uncapped carve/tread/clearance/mouth lists, built in 26-28 ms, ~12 MiB. Weighted over 30.7M near-LOD sample hits: active carve p90 13, mouth p90 13; reach fallback keeps 18.3% of passage time (FI 12%, TN 19%) - all because the density-dependent reach (-d*Q + 0.24 m) exceeds 10 m for deeply negative source samples. Not wired into production. R41 notes (Unreal LOD0-2 gap: ~0.7 s pre-request in BeginPlay, LOD2 tail, apply boost reset; L80's 4 missing tiles = re-requested empty collision-support tiles) attached. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016dhxM43dnWvUu2B5Ph1iG1
This commit is contained in:
Fr0zkaandClaude Opus 5.5 committed
1 parent
04cdd197fb
commit
a7a03d42b4
5 files changed
+948
-5
No files matched your search
@@ -0,0 +1,31 @@
|
||||
## Ranked causes for L85
|
||||
|
||||
1. **The first tile request starts about 0.7 s later than headless.** [TRACE] The Unreal clock starts at `BeginPlay` (`0.000 s`), which ends at `0.598 s`. Desired-set construction takes only 3.7 ms at `0.789–0.793 s`; the first tile request is at `0.793 s`. The scheduler still reports `core_not_ready` at `0.772 s`. [CODE] `BeginPlay` synchronously exports and parses the Core snapshot, then launches shape compilation on a `BackgroundNormal` task; `UpdateChunksAroundPosition` returns while Core is not ready. ([trace]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l85/unreal-fi/fi-arrival-l85.trace.jsonl:18), [trace]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l85/unreal-fi/fi-arrival-l85.trace.jsonl:48), [trace]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l85/unreal-fi/fi-arrival-l85.trace.jsonl:52), [VoxelWorld.cpp]( /home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:7646), [VoxelWorld.cpp]( /home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:7782), [VoxelWorld.cpp]( /home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelWorld.cpp:11782).)
|
||||
|
||||
[CODE] Strata/passage work is measured at roughly 76 ms. Snapshot serialization and `FromJson` run synchronously on the `BeginPlay` caller; shape compilation runs asynchronously and overlaps the rest of `BeginPlay`. The trace does not split the remaining `BeginPlay` time. Package or asset loading before `BeginPlay` is outside this trace; the recorded settings asset is transient. [INFERENCE] Headless starts its desired request around 60–80 ms from cold start, while Unreal’s first tile request is at 793 ms. That accounts for roughly 0.7 s of the gap as later pre-request work or a different clock boundary, but does not identify which part is avoidable. ([Linux87 report]( /home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Research/WorldAssemblerM3/Linux87/REPORT.md:9).)
|
||||
|
||||
2. **The remaining difference is in Unreal’s post-request chain, especially the staged LOD2 tail.** [TRACE] Unreal’s LOD0 ring is resident at `2.391 s`; at `3.134 s`, LOD0 and LOD1 are loaded but 240 LOD2 tiles remain missing. All 1,216 LOD0–2 tiles are resident at `3.650 s`. That is 2.86 s from first request to full residency, versus about 2.2 s for the headless run. ([trace]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l85/unreal-fi/fi-arrival-l85.trace.jsonl:1455), [trace]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l85/unreal-fi/fi-arrival-l85.trace.jsonl:2438), [trace]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l85/unreal-fi/fi-arrival-l85.trace.jsonl:3135), [Linux85 report]( /home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Research/WorldAssemblerM3/Linux85/REPORT.md:21).)
|
||||
|
||||
[CODE] Admission is frame-paced: the actor calls `UpdateChunksAroundPosition` once per tick. The trace shows a task cap of 8, ready queue depth 8, and total admission cap 16. At `3.134 s` all 16 slots are occupied and all 8 workers are running. So the cap is active, but this sample does not show idle workers waiting for admission. [INFERENCE] Frame pacing and LOD2 staging plausibly explain some of the post-request excess; the trace does not isolate their exact shares. ([VoxelWorld.cpp]( /home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:7074), [VoxelWorld.cpp]( /home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:13449), [Linux87 report]( /home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Research/WorldAssemblerM3/Linux87/REPORT.md:26).)
|
||||
|
||||
3. **The game-thread apply cap may add some tail latency, but it is not the main measured cost.** [TRACE] The startup boost ends at `3.157 s` because the desired epoch changed; that frame reports the normal limit of four mesh applies and hits the count limit. [CODE] Any desired-epoch change ends the boost. [INFERENCE] Keeping the boost active through the initial epoch churn is a small candidate to test, but the trace does not show that it accounts for a large fraction of 1.4 s. ([trace]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l85/unreal-fi/fi-arrival-l85.trace.jsonl:2458), [trace]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l85/unreal-fi/fi-arrival-l85.trace.jsonl:2466), [VoxelWorld.cpp]( /home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:9505).)
|
||||
|
||||
4. **Worker count, publication, and collision cooking are weak explanations for this gap.** [TRACE] `streaming_background_worker_cap=1` yields 8 effective tile slots, matching the 8 TaskGraph background threads; the setting is a clamp toggle, not a one-worker limit. Headless FI readiness changes by only about 36 ms between W8 and W12 in L87 (124 ms in the L85 matrix), far below this gap. ([trace]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l85/unreal-fi/fi-arrival-l85.trace.jsonl:5), [VoxelWorld.cpp]( /home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:11592), [Linux87 report]( /home/jahni/VoxelM/Plugins/VoxelForge/Standalone/Research/WorldAssemblerM3/Linux87/REPORT.md:38).)
|
||||
|
||||
[TRACE] Unreal’s per-tile summaries show small apply times for LOD0–2; they cover the full trace, so they are indicative rather than an isolated startup breakdown. [CODE] The tile event records worker completion and game-thread application; application does not mean collision-ready. L85 measures residency through `LoadedTiles`, and its later milestone shows collision readiness is still a separate, incomplete count. RMC cooking therefore cannot explain the 3.650 s residency timestamp. ([trace summary]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l85/unreal-fi/fi-arrival-l85.trace.jsonl:12896), [VoxelWorld.cpp]( /home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:9908), [trace]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l85/unreal-fi/fi-arrival-l85.trace.jsonl:5843).)
|
||||
|
||||
## L80’s four missing tiles
|
||||
|
||||
[TRACE] The four are LOD0 `(47,-2,-99)`, `(47,-2,-100)`, `(47,-2,-101)`, and `(47,-2,-102)`. At `45.375 s`, desired LOD0 is 516, resident is 512, pending is zero, and all 516 are visually covered. The trace repeatedly records empty results for those keys and then requests them again. ([trace]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l80/movement/unreal/fast-L80.trace.jsonl:21823), [trace]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l80/movement/unreal/fast-L80.trace.jsonl:21828), [trace sample]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l80/movement/unreal/fast-L80.trace.jsonl:21843).)
|
||||
|
||||
[INFERENCE] They most likely form the pawn collision gate’s four-tile downward support prefetch: the code requests the support tile and three below it. Empty results are marked both loaded and `CollisionNotRequired`; a later desired-set rebuild removes no-collision gate keys, while unload clears both that marker and loaded bookkeeping. The next gate check can then add the same keys again. This fits the repeated four-tile region compiles at the same center, though the trace does not record per-key collision-only membership or unload reasons. ([VoxelWorld.cpp]( /home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:8277), [VoxelWorld.cpp]( /home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:10653), [VoxelWorld.cpp]( /home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForge/Private/VoxelWorld.cpp:14330), [trace]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l80/movement/unreal/fast-L80.trace.jsonl:21841).)
|
||||
|
||||
This is **not a stranded worker task**: the empty tile work completes, and the commandlet ends with pending, cooks, and active tasks all zero. There is also a **readiness-check mismatch**: the commandlet requires every desired key to be in `LoadedTiles`, while its collision-set check accepts `CollisionNotRequiredTiles`. The short resident count is real at that instant, but the coverage count says it is not a visual hole. ([ExploreCommandlet.cpp]( /home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForgeEditor/Private/VoxelForgeExploreCommandlet.cpp:18132), [ExploreCommandlet.cpp]( /home/jahni/VoxelM/Plugins/VoxelForge/Source/VoxelForgeEditor/Private/VoxelForgeExploreCommandlet.cpp:18142), [L80 log]( /home/jahni/VoxelM/Plugins/VoxelForge/Saved/Linux/l80/movement/unreal/fast-L80.log:3542).)
|
||||
|
||||
## Cheapest confirmation and smallest fixes
|
||||
|
||||
- **For the L85 timing:** add trace timestamps around snapshot serialization, `FromJson`, compile launch, compile worker start, and `bReady` publication. The existing trace already records first request, tile worker times, apply, and residency. One matched run would show whether the unexplained pre-request interval is snapshot work or shape compilation.
|
||||
- **Smallest latency change to A/B:** keep the startup apply boost through initial desired-epoch changes, ending it on initial readiness or timeout. The trace shows the current early reset and a count-limited apply frame. It may shorten the LOD2 tail; it will not explain or remove the pre-request delay.
|
||||
- **Smallest L80 logic fix:** retain known-empty collision classification across unloads for the current modification epoch, invalidating it when tile data changes. That should stop re-requesting those support keys; confirm their collision-only role with per-key logging before changing the readiness predicate.
|
||||
|
||||
No files were changed; no builds, tests, or project binaries were run.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Linux 90 — passage candidate census
|
||||
|
||||
Read Astra pass 3 before implementation. The owner-world census compiled 54,324 sparse 4 m bricks with a 10 m reach and four independent, uncapped candidate lists. The density proof used the incoming-sample bound `-d·Q + 0.24 m`; samples whose bound exceeded 10 m explicitly kept the existing passage work. The build emitted 217,072 carve, 24,359 tread, 5,416 clearance, and 57,594 mouth/column references, with zero compiler failures. Build time was 26.3–27.9 ms; estimated memory was 12,667,000 bytes (about 12.1 MiB).
|
||||
|
||||
List-size distributions are weighted across 30,681,829 near-LOD sample hits. Percentiles include zero-candidate samples; the active percentiles include only nonempty lists. The `16+` column is a reporting bucket; the stored lists have no cap.
|
||||
|
||||
| Semantic | Nonempty hits | All-hit p90/p95 | Active p90/p95 | 16+ of hits |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Carve | 52.89% | 11 / 13 | 13 / 14 | 1.65% |
|
||||
| Tread | 33.53% | 3 / 4 | 4 / 4 | 0% |
|
||||
| Clearance | 15.60% | 1 / 2 | 2 / 3 | 0% |
|
||||
| Mouth/column | 41.35% | 11 / 12 | 13 / 14 | 0.88% |
|
||||
|
||||
Measured passage time still retained by explicit reach fallback:
|
||||
|
||||
| Stratum | Fallback share |
|
||||
|---|---:|
|
||||
| S0 | 24.94% |
|
||||
| FI | 12.19% |
|
||||
| TN | 19.03% |
|
||||
| S3 | 42.57% |
|
||||
| Weighted | 18.28% |
|
||||
|
||||
Every final proof counter was zero for omitted carve, tread, clearance, route, column, or bridge candidates. All fallback was due to the 10 m reach limit. The 3,897,959 fallback iterations still visited 19.51M carve segments, 9.61M tread segments, and 3.77M clearance capsules (plus 14,163 mouth checks). Both stop conditions fail: carve and mouth/column active p90s are 13 candidates, and fallback exceeds 10% in every stratum. **Stop here; no junction evaluator prototype was built.**
|
||||
|
||||
## Verification and runs
|
||||
|
||||
- Fast tier: **PASS**, 269 checks and 5 diagnostics in 296.2 s. Pinned hash and determinism checks passed.
|
||||
- Linux `VFHostEditor` after timing: **PASS**, 27/27 actions in 69.13 s with `-NoUBA -MaxParallelActions=4`; no address-space limit. Three existing RealtimeMesh deprecation warnings were emitted.
|
||||
- The first census collection was discarded after review found its column proof recorder read the distance before assignment. The recorder was fixed and the complete four-stratum census rerun; final column proof misses are zero.
|
||||
- No run exceeded 30 minutes. No prototype replay, W8/W12 prototype timing, capture, or version bump applies after the census stop. The compiled lists are opt-in census data and are not wired into production evaluation.
|
||||
|
||||
Source stamp: `c797603e5272850855e448a6177ae32dccdeb20974484a725324ebbc0903106d`. Logs and census data are under `Saved/Linux/l90/`. No commit, push, or stash was made.
|
||||
@@ -0,0 +1,195 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compile and census spatial passage candidates on the four owner strata."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[4]
|
||||
OUT = ROOT / "Saved/Linux/l90/candidate-census"
|
||||
DATA = ROOT / "Standalone/data/owner_world.json"
|
||||
BINARY = ROOT / "Saved/Linux/l90/bin/VoxelForgeCoreHeadless-l90-clang19"
|
||||
STRATA = {
|
||||
"S0": {"id": 0, "camera": (8.0, 16.0, -372.2626953125), "near": 1408, "whole": 2850},
|
||||
"FI": {"id": 1, "camera": (25.8369140625, 3.19921875, -737.0), "near": 1216, "whole": 2249},
|
||||
"TN": {"id": 2, "camera": (19.166015625, -14.2001953125, -753.3740234375), "near": 1344, "whole": 2233},
|
||||
"S3": {"id": 3, "camera": (-20.0625, -5.068359375, -900.353515625), "near": 1344, "whole": 2329},
|
||||
}
|
||||
PIPELINE = re.compile(r"^\[StreamPipeline\] (.*)$", re.M)
|
||||
FIELDS = re.compile(r"([a-zA-Z0-9_]+)=([^ ]+)")
|
||||
NEAR = re.compile(
|
||||
r"^\[WorldAssemblerNearLODReady\] lods=0-2 tiles=(\d+)/(\d+) cold_ready_s=([0-9.]+)$", re.M)
|
||||
STAMP = re.compile(r"^\[VoxelForgeSourceStamp\] sha256=(\S+)$", re.M)
|
||||
BUILD = re.compile(r"^\[L90PassageCandidateBuild\] (.*)$", re.M)
|
||||
CENSUS = re.compile(r"^\[L90PassageCandidateTile\] (.*)$", re.M)
|
||||
HIST_NAMES = ("carve_hist", "tread_hist", "clearance_hist", "mouth_column_hist")
|
||||
|
||||
|
||||
def command(stratum: str) -> list[str]:
|
||||
spec = STRATA[stratum]
|
||||
camera = ",".join(f"{value:.12f}" for value in spec["camera"]) + ",0,0"
|
||||
return [
|
||||
"taskset", "-c", "0-11", "prlimit", "--as=6442450944", "--",
|
||||
"env", "VOXELFORGE_L90_PASSAGE_CANDIDATE_CENSUS=1",
|
||||
"VOXELFORGE_L81_PASSAGE_BOUND_PADDING_VOXELS=0", "timeout", "1740s",
|
||||
str(BINARY), str(DATA), "--world-assembler-prototype", "--stream-benchmark",
|
||||
"--stream-pipeline", "--workers=8", "--clip-radius=3", "--max-clip-level=8",
|
||||
"--max-level=8", f"--camera={camera}", "--profile-lod-work",
|
||||
"--profile-passage-composer", "--stream-scheduler-policy=3",
|
||||
]
|
||||
|
||||
|
||||
def parse_hist(value: str) -> list[int]:
|
||||
items = [int(item) for item in value.split(",")]
|
||||
if len(items) != 17:
|
||||
raise ValueError(f"expected 17 histogram bins, got {len(items)}")
|
||||
return items
|
||||
|
||||
|
||||
def parse_fields(line: str) -> dict[str, str]:
|
||||
return dict(FIELDS.findall(line))
|
||||
|
||||
|
||||
def parse_tuple(value: str, expected: int) -> list[int]:
|
||||
items = [int(item) for item in value.strip("()").split(",")]
|
||||
if len(items) != expected:
|
||||
raise ValueError(f"expected {expected} values, got {value}")
|
||||
return items
|
||||
|
||||
|
||||
def run(stratum: str) -> dict:
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
log = OUT / f"{stratum.lower()}-w8-profile.log"
|
||||
started = time.monotonic()
|
||||
with log.open("w", encoding="utf-8") as stream:
|
||||
result = subprocess.run(command(stratum), cwd=ROOT, stdout=stream,
|
||||
stderr=subprocess.STDOUT, check=False, timeout=1750)
|
||||
output = log.read_text(encoding="utf-8", errors="replace")
|
||||
near, stamp = NEAR.search(output), STAMP.search(output)
|
||||
if result.returncode or near is None or stamp is None:
|
||||
raise RuntimeError(f"run failed or required output is missing: {log}; exit={result.returncode}")
|
||||
expected = STRATA[stratum]["near"]
|
||||
if (int(near.group(1)), int(near.group(2))) != (expected, expected):
|
||||
raise RuntimeError(f"incomplete near set in {log}: {near.group(1)}/{near.group(2)}")
|
||||
builds = [parse_fields(match.group(1)) for match in BUILD.finditer(output)]
|
||||
if not builds:
|
||||
raise RuntimeError(f"candidate compiler build row missing: {log}")
|
||||
|
||||
hist = [[0] * 17 for _ in range(4)]
|
||||
census_rows = 0
|
||||
near_queries = no_brick = fallback_iterations = fallback_ns = query_ns = 0
|
||||
fallback_reach = fallback_shortspan = fallback_invalid = fallback_proof = 0
|
||||
fallback_carve = fallback_tread = fallback_clearance = fallback_mouth = 0
|
||||
proof_checks = [0] * 6
|
||||
proof_misses = [0] * 6
|
||||
for match in CENSUS.finditer(output):
|
||||
fields = parse_fields(match.group(1))
|
||||
if int(fields["lod"]) > 2:
|
||||
continue
|
||||
census_rows += 1
|
||||
near_queries += int(fields["queries"])
|
||||
no_brick += int(fields["no_brick"])
|
||||
fallback_iterations += int(fields["fallback_iterations"])
|
||||
fallback_reach += int(fields["fallback_reach"])
|
||||
fallback_shortspan += int(fields["fallback_shortspan"])
|
||||
fallback_invalid += int(fields["fallback_invalid"])
|
||||
fallback_proof += int(fields["fallback_proof"])
|
||||
fallback_ns += int(fields["fallback_ns"])
|
||||
query_ns += int(fields["query_ns"])
|
||||
fallback_carve += int(fields["fallback_carve"])
|
||||
fallback_tread += int(fields["fallback_tread"])
|
||||
fallback_clearance += int(fields["fallback_clearance"])
|
||||
fallback_mouth += int(fields["fallback_mouth"])
|
||||
proof_checks = [a + b for a, b in zip(proof_checks, parse_tuple(fields["proof_checks"], 6))]
|
||||
proof_misses = [a + b for a, b in zip(proof_misses, parse_tuple(fields["proof_misses"], 6))]
|
||||
for semantic, name in enumerate(HIST_NAMES):
|
||||
values = parse_hist(fields[name])
|
||||
hist[semantic] = [a + b for a, b in zip(hist[semantic], values)]
|
||||
if census_rows == 0 or near_queries == 0:
|
||||
raise RuntimeError(f"near-LOD candidate census rows missing: {log}")
|
||||
pipeline = PIPELINE.search(output)
|
||||
if pipeline is None:
|
||||
raise RuntimeError(f"stream pipeline summary missing: {log}")
|
||||
pipeline_fields = parse_fields(pipeline.group(1))
|
||||
row = {
|
||||
"stratum": stratum, "workers": 8, "exit_code": result.returncode,
|
||||
"wall_s": round(time.monotonic() - started, 3),
|
||||
"lod02_ready_s": float(near.group(3)),
|
||||
"whole_view_ready_s": float(pipeline_fields["whole_view_ready"].removesuffix("s")),
|
||||
"source_stamp": stamp.group(1), "near_tiles": expected,
|
||||
"census_scopes_lod02": census_rows, "near_query_hits": near_queries,
|
||||
"no_brick_hits": no_brick, "fallback_iterations": fallback_iterations,
|
||||
"fallback_reasons": {"reach_overflow": fallback_reach, "short_span": fallback_shortspan,
|
||||
"invalid_compiler": fallback_invalid, "proof_miss": fallback_proof},
|
||||
"fallback_ns": fallback_ns, "query_ns": query_ns,
|
||||
"fallback_share": fallback_ns / query_ns if query_ns else None,
|
||||
"fallback_work": {"carve_visits": fallback_carve, "tread_visits": fallback_tread,
|
||||
"clearance_checks": fallback_clearance, "mouth_checks": fallback_mouth},
|
||||
"proof_checks": dict(zip(("carve", "tread", "clearance", "mouth_route", "columns", "bridges"), proof_checks)),
|
||||
"proof_misses": dict(zip(("carve", "tread", "clearance", "mouth_route", "columns", "bridges"), proof_misses)),
|
||||
"histograms": {name.removesuffix("_hist"): values
|
||||
for name, values in zip(HIST_NAMES, hist)},
|
||||
"build_rows": builds, "log": str(log.relative_to(ROOT)),
|
||||
}
|
||||
print("[L90CandidateCensusRun] " + json.dumps(row, sort_keys=True), flush=True)
|
||||
return row
|
||||
|
||||
|
||||
def aggregate(rows: list[dict]) -> dict:
|
||||
total_hist = [[0] * 17 for _ in range(4)]
|
||||
for row in rows:
|
||||
for semantic, name in enumerate(HIST_NAMES):
|
||||
values = row["histograms"][name.removesuffix("_hist")]
|
||||
total_hist[semantic] = [a + b for a, b in zip(total_hist[semantic], values)]
|
||||
weighted = {}
|
||||
for semantic, name in enumerate(HIST_NAMES):
|
||||
values = total_hist[semantic]
|
||||
total = sum(values)
|
||||
def percentile(p: float, first_bin: int = 0) -> int:
|
||||
population = sum(values[first_bin:])
|
||||
target = max(1, int((p * population) + 0.999999999))
|
||||
seen = 0
|
||||
for index in range(first_bin, len(values)):
|
||||
count = values[index]
|
||||
seen += count
|
||||
if seen >= target:
|
||||
return index
|
||||
return 16
|
||||
nonempty = sum(values[1:])
|
||||
weighted[name.removesuffix("_hist")] = {
|
||||
"queries": total,
|
||||
"histogram_0_to_15_plus": values,
|
||||
"mean_lower_bound": (sum(i * n for i, n in enumerate(values)) / total
|
||||
if total else 0.0),
|
||||
"p50_lower_bound": percentile(0.50) if total else 0,
|
||||
"p90_lower_bound": percentile(0.90) if total else 0,
|
||||
"p95_lower_bound": percentile(0.95) if total else 0,
|
||||
"share_16_plus": values[16] / total if total else 0.0,
|
||||
"nonempty_share": nonempty / total if total else 0.0,
|
||||
"nonempty_mean_lower_bound": (
|
||||
sum(i * n for i, n in enumerate(values)) / nonempty if nonempty else 0.0),
|
||||
"nonempty_p90_lower_bound": percentile(0.90, 1) if nonempty else 0,
|
||||
"nonempty_p95_lower_bound": percentile(0.95, 1) if nonempty else 0,
|
||||
}
|
||||
return {"strata": rows, "weighted_semantics": weighted}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--strata", nargs="+", choices=tuple(STRATA), default=tuple(STRATA))
|
||||
args = parser.parse_args()
|
||||
if not BINARY.is_file() or not DATA.is_file():
|
||||
raise SystemExit(f"missing input: {BINARY} or {DATA}")
|
||||
rows = [run(stratum) for stratum in args.strata]
|
||||
summary = aggregate(rows)
|
||||
(OUT / "summary.json").write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")
|
||||
print("[L90CandidateCensusSummary] " + json.dumps(summary["weighted_semantics"], sort_keys=True), flush=True)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in new issue
Block a user