Commit Graph
57 Commits
Author SHA1 Message Date
Fr0zkaandClaude Opus 5 ee89c4d8c2 perf(gen): 5x fewer candidates, 16x cheaper floor -- and 0% end-to-end. Read the last section.
Owner, after playing 0576e80: "slightly faster, still not on par... I can outrun the
generation and fall." Also, unprompted: "the tunnels seem forced, with a forced
flat -- it technically looks good, but could be a reason." He was right that it
was a reason; it was not THE reason.

WHAT LANDED, all real and all measured

Conservative spatial index (immutable per-chunk uniform bins, baked conservative
bounds, squared rejects, deterministic ordering):

  candidates scanned/sample   25.140 -> 5.192
  candidates evaluated/sample 13.879 -> 2.992
    cave tunnels  11.228/6.069 -> 1.525/0.723
    tunnel core   11.228/5.831 -> 1.216/0.567

The earlier flat AABB attempt failed because loose bounds culled almost nothing
while adding a pass. Technique from GPU Gems 3 ch.32 broad-phase.

The forced-flat floor, which the owner spotted by eye: a floor decision is
inherently per-COLUMN and was being recomputed for every Z sample in that column.
Now resolved once per integer XY column and reused.

  TunnelCorePosts   2.29 us/sample -> 0.138 us/sample   (16x)

The floor is unchanged in behaviour: still applied only after a tunnel passes its
world bounds and walkable projection, still walkable, look untouched.

FIRST MEASUREMENT OF OUR MEMORY FOOTPRINT
The owner: "it shouldn't be pressured on such system, even less when not running
anything." He is right, and it is ours:

  total resident worker cache   431,162,116 B  =  411 MiB
  15 workers                    27.41 MiB each
  average valid entry           278,704 B
  largest entry                 541,344 B
  static slot storage             2.4 MiB   (dynamic: 428.6 MB)

Cutting 4096 -> 128 slots only saved static storage. The weight is 272 KB PER
CACHED CHUNK STATE, ~102 live per worker. Nobody has yet asked why one cached
chunk state needs 272 KB. That is the open question, not the slot count.

⚠️ AND THE PART THAT MATTERS MOST: THIS BOUGHT NOTHING END-TO-END

  128^3 TunnelNetwork, seed 1337 slot 1, same region, unprofiled:
    before this commit   2.623 s
    after, 3 runs        2.705 / 2.678 / 2.682 s

Flat. A 5x candidate reduction and a 16x cheaper floor moved real generation time
by zero.

The reason is visible in the numbers and I should have caught it earlier: the
PROFILED path reports 14.74 us/sample while the real unprofiled cost is
1.28 us/voxel. **The instrument inflates by ~11x.** Every bucket we have been
ranking and optimising for two rounds was measured under that inflation, so the
ranking does not reflect where real time goes. We optimised what the profiler
exaggerated.

⇒ Before any further optimisation: make the profiler cheap enough that profiled
and unprofiled totals agree within a small factor, or switch to sampling. An
alarming number is the instrument until proven otherwise -- three for three in
this project, and now four.

Still true: ~21x slower than the pre-op-stack build, and generation still loses
to a walking player. Geometry changed by 40 triangles (153,692 -> 153,732), which
is allowed; two full exports hash identically (37E00BEE..0A7DF). Build 42/42.
No CPU-feature dispatch. Nothing outside Plugins/VoxelForge was touched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012aheo2crrZmHuYtPpdY2Cb
2026-09-09 03:05:40 +02:00
Fr0zkaandClaude Opus 5 0576e800c4 perf(gen): the landing work was per-voxel -- 342s to 2.6s, same world
The owner played 64f5d61 and reported the truth: "generation still takes ages...
TunnelNetwork doesn't even finish generating, as big as the surface world" while
SurfaceWorld completed in under 10s, and stopping PIE hung for 30+ seconds.

e08b49a's 21x was real and irrelevant. It fixed per-CHUNK state rebuilds, which
dominate the commandlet export path. The game is bound by per-VOXEL evaluation.
Same symptom, different bottleneck, and nobody had measured the second one.

WHY PIE HUNG
bShuttingDown was never checked inside VoxelMarchingCubesMesher.cpp -- zero
occurrences. A worker inside GenerateMesh could not abort, so 64f5d61's correct
unconditional EndPlay wait was waiting on tiles physically unable to stop. The
hang was not a teardown bug; it was evidence that one tile took tens of seconds.
Added shutdown polling per slab/row in the mesher and through GenerateTileResult;
aborted results carry Epoch + bAborted and ProcessQueue drops them. EndPlay still
waits unconditionally -- the 3s mark only logs now.

  PIE stop   3.409s / 44.759s  ->  4 ms teardown, 38 us worker drain

WHERE THE PER-VOXEL COST WAS
The owner's own comparison was the sharpest instrument available. Profiled both:

  GetDensityAt          TunnelNetwork 12.64 us/sample   SurfaceWorld 1.32
  tunnel candidates evaluated      7.088 per sample                 0
  tunnel-core candidates evaluated 7.201 per sample                 0
  room candidates evaluated        1.000 per sample                 0

  largest buckets: RoomGraphSource 2.57, TunnelCorePosts 0.76,
                   PassageCarveOp 0.22, PassageStructuralPosts 0.20

The three regressing commits map cleanly, and one is innocent:
  7a20eee drainage tree  -- cache-build work, NOT per-voxel
  413c15d landings       -- repeated full passage modification per sample
  55bf807 landing rooms  -- room SDF + tunnel/core evaluation per sample

Fixes: FPassageCarveOp carves only, landing/support moved to the structural MC
post; tunnel core support and SDF fused into one pass; room graph caches shared
per worker on complete deterministic keys; tunnel density cache uses open
addressing with 4096 slots. Room cache rebuilds fell to 9 per 42,875 samples, so
what remains is genuine O(local cached room/tunnel count) work.

  TunnelNetwork 128^3, seed 1337 slot 1, same region:
    342.2 s / 163.161 us/vox   ->   2.623 s / 1.251 us/vox     (130x)

  Geometry SHA-256 aaf95559..066b4, IDENTICAL to the pre-optimisation run at the
  same roughness. 153,692 triangles either way. Verified independently of the
  agent's own report.

NOT FIXED, and the first one is the next real target:
- L4 tiles still cost 16.93 s. Hit rate falls 99.9% (L0) -> 23.7% (L4), but
  1024 -> 4096 slots only moved 18.39s -> 16.93s and the 1,927 misses ARE the
  distinct key count. That is low reuse, not eviction. A bigger cache will not
  save it.
- TunnelNetwork is still ~10x SurfaceWorld (0.354s vs 0.028s on a matched 32^3).
- The live PIE tile table is SurfaceWorld: this checkout's PIE map is configured
  that way. TunnelNetwork went through the same mesher and density path via the
  commandlet, but no TunnelNetwork tile has been timed in an actual game session.
  Export numbers are not presented as game numbers.

Still outstanding, untouched here: the inter-strate passage has intersections it
should not have, routes through open passage when it should not, and should
descend from a low point of the current strate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012aheo2crrZmHuYtPpdY2Cb
2026-09-08 23:47:53 +02:00
Fr0zkaandClaude Opus 5 e08b49a183 perf(gen): a thrashing one-slot cache cost 18.7x -- and the world is unchanged
TunnelNetwork density evaluation was 45x more expensive than Maze for identical
work: 2,097,152 voxels, 64 tiles, same mesher, same machine. Profiling killed
every hypothesis I brought to it -- box classification skipping 0/64 tiles was
true but irrelevant (export bypasses classification), near-surface early-outs
fired at 49.7%, the room cache built 340 entries for 2.74M samples, and the
per-room overrides resolved once per sample rather than once per modifier.

The cause was the wrapper, not the stack. A mesher tile samples an EXPANDED grid
(-1..Cells+1), so one worker crosses several exact chunk coordinates while
meshing a single tile. The generator's cache held only the last coordinate, so
every boundary crossing rebuilt the resolved parameters, the whole 22-operator
stack and the tunnel-core cache. On one representative tile: 3,675 refetches,
38.0s. The entire operator stack was 18.5s of a 4,272s total -- 0.43%.

Replaced with a 128-slot direct-mapped thread_local cache keyed on owner, chunk,
layout version, params fingerprint and generator type. Collisions force a safe
rebuild, never a stale read. No shared mutable state, no order-dependent
reduction.

  TunnelNetwork  344.5s / 164.267 us/vox  ->  18.4s / 8.763 us/vox   (18.7x)
  Maze             7.4s /   3.535 us/vox  ->   1.08s / 0.513 us/vox  ( 6.9x)
  same tile        38.0s                  ->   0.51s ; refetches 3675 -> 27

Output is BYTE-IDENTICAL, before vs after and across repeats, both archetypes:
TunnelNetwork 50d6b4d0..f23126, Maze 364f7489..2fbe99. The permission to change
the field was not needed.

Also in this commit, from the two preceding tasks:

- Explorer renders from the MESH instead of raymarching. It meshes the region
  once through the canonical UVoxelMarchingCubesMesher, then CPU-rasterises via
  a BVH. A viewpoint went from ~16 minutes to 0.155s; 8 views now cost 1.25s of
  drawing. No render pixel samples the density field.
- One sampled grid and player-fit mask per run, shared by render/walk/export,
  instead of the same 368s walk computed twice.
- A wall-clock budget (25 min default, hard 30 max) that stops cleanly and
  records truncated/truncated_during/completed_modes rather than overshooting.
- ParallelFor across the measurement sampling; the suite was ~10% of a 16-thread
  machine.
- -surfaceroughness= on the commandlet, and a surface-relief metric in export
  mode (edge lengths, dihedral distribution, face-vs-smooth normal divergence)
  so "does this look like rock" is a number, not a screenshot argument.
- Smooth vertex normals in the rasteriser: FVoxelMeshData::Normals is the
  density-gradient normal the engine shades with, and the raster path was
  recomputing a face normal over it.
- ScaleDiagnosis: baked candidate indices are now derived from the current
  corpus, so a legitimate field change stops breaking the test.

NOT verified, and not claimed:
- TunnelNetworkSpineEquivalence exceeded the 30-minute cap and was stopped.
  Inconclusive, not passing. The identical hashes are strong evidence the
  op-stack/legacy equivalence holds, but the test has not confirmed it.
- CrossPlatformDigest passes but reports 2 samples within 1e-6 of the
  isosurface; FMath::Sin/Cos leaves Windows/Linux equivalence unproven.
  Pre-existing, not caused by this change.
- The cache's memory footprint (128 entries x ~16 workers) is unmeasured.

What the roughness sweep settled, for the record: SurfaceRoughness 0 -> 8 moves
cost by 0.3% (163.2 / 163.2 / 163.7 us), so roughness was never the cost. It
does reach the surface (dihedral p50 1.1 -> 2.2 -> 5.9), which kills my claim
that the density-channel operator contributed nothing. TunnelNetwork's hard
seams are present at roughness 0 (4.4% of edges past 40 deg vs Maze's 1.0%), so
they are structural -- in how primitives are unioned -- not a roughness
deficiency. More roughness makes them worse, not better.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012aheo2crrZmHuYtPpdY2Cb
2026-09-08 20:31:56 +02:00
Fr0zkaandClaude Opus 5 d97373c5a8 feat(gen): Jahni's playtest corrections -- rooms flattened, roads deleted
From walking a TunnelNetwork preset at (0,0). Four corrections, and two of
them were the same mistake.

2a. The (0,0) landing room is sized to host a FUTURE straight vertical
    shaft from the surface, with flooring around the origin itself. The
    room is the landing for a shaft the player digs later.
2b. The inter-strate descent no longer starts at the spine -- placed by
    deterministic hash within a radius around (0,0). The spine room is for
    the surface shaft; the descent is elsewhere.
2c. ⛔ DELETED the landing->hub connector roads from 413c15d. Jahni:
    "its straight up making other tunnels straightly through everything...
    they have nothing to do here." Zero references remain.
2d. ⭐ FLATTEN THE ROOM JOINS instead. Where two spherical rooms intersect
    and leave a lip above head height, the join is brought down to a
    walkable level.

⭐ 2c AND 2d ARE ONE CORRECTION. I solved "rooms do not connect" by
TUNNELLING BETWEEN THEM when the answer was MAKING THEIR FLOORS MEET. The
81-run sweep had already said so -- zero narrow-gap events, every failure a
vertical severance, renders showing "a sharp diagonal ledge descending into
black open space". A radial corridor to the origin could never have fixed
that; flattening the joins is aimed at exactly it.

2e. Descent tunnel 1.5x wider, per "1.5x bigger wouldn't be too much".
2f. Corridors WANDER. Jahni, from the renders: "its not a maze, its still a
    latice made straight paths." The spanning tree changed WHICH edges
    exist, not what an edge looks like -- every corridor was a straight
    axis-aligned capsule, every junction 90 degrees. Now hash-jittered
    chains, seam-safe like the tree.

RENDERER: 192x128 -> 512x288, and cameras are now seeded from the
player-fit set -- "a density-checked pose from the exact player-fit walk
mask". Previously they sat INSIDE ROCK looking at back-faces, which Jahni
spotted instantly and no metric could. First genuinely legible pictures of
this world.

ParallelFor landed at 6 sites in VoxelStrateMeasure before I stopped the
run; the rest of that work continues next.

⚠️ AND THE RENDERER IS ARCHITECTURALLY WRONG, per Jahni: "why, in engine,
does it take a minute to have clear view of thousands of angles, but in our
renderer it takes ages?" Because the engine MESHES ONCE and rasterises --
~32k density calls per chunk, then every angle is free. I RAYMARCH: tens of
millions of density calls PER IMAGE, nothing reused between frames. We
already have the mesher and an export mode. Next task renders from the
mesh.

Also from him: runs must stay under 30 minutes. This one hit 33 and
produced 2 images.

Build clean. Committed as a checkpoint -- the task was stopped mid-Part-3,
not finished.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012aheo2crrZmHuYtPpdY2Cb
2026-09-08 14:14:29 +02:00
Fr0zkaandClaude Opus 5 55bf807f60 feat(descent): landing ROOMS and a walkable tunnel -- per Jahni's correction
Jahni corrected the design: "the connection between level is a tiny tunnel
connecting both strate, not vertical, it needs to be walkable" and "'the
spine' shouldn't be carved except optionally from the top first strate,
then it's used to have a landing room".

COMPOSER-NOTES §6.8 had said the spine "provides a clean landing space, not
a continuous open shaft" -- and the implementation carved a vertical
cylinder anyway. A vertical cylinder has no walkable floor except at its
bottom, so a carve guaranteeing open AIR guaranteed nothing STANDABLE.
That one omission is behind five of my dead hypotheses this week.

Now:
  origin spine   a finite rounded ROOM per strate, no vertical bore.
                 radius 14 -> 8 m wide, 7.5 m tall
  descent        a walkable tunnel, single-file by design: 3 m mouth,
                 2.5 m waist, gradient capped at a NAMED 15 deg
  exotic styles  VerticalShaft / SpiralDescent / CascadingDrops /
                 CrackCrevice preserved, no longer the default

⚠️ THE GEOMETRY BITES, and it is worth Jahni's eye: at 15 deg a 32 m drop
needs 119.4 m of horizontal run -- about 59.7 m per switchback leg. That is
a long walk between strates. It is the honest consequence of "walkable",
not a tuning artifact, and the alternative is a steeper number.

⭐ AND THE ROOT CAUSE GOT NAMED A THIRD TIME, by Codex mid-run: "the round
tube was hollow air but had no authored support strip, so 'open' still did
not mean 'standable'." Tunnels now carry a floor band, and the root join
terminates in the finite origin room rather than an annular shaft hub.

Measured, commandlet sweep, 8 archetypes x 8 seeds, 64/64 valid:
  origin column has player-fit cells   17/64 -> 63/64
  origin column reachable               0/64 ->  7/64
  arrival landing probes            315/320 fit AND in the arrival component
  tunnel audit    5,916/5,916 floor and capsule samples pass; zero gradient,
                  floor, capsule, geometry or seal failures
  narrow-gap events 0        box verdicts 14 checked, 0 violations

⚠️ arrival->departure is 6/64. The landings and tunnels are now provably
walkable; what remains is the INTRA-STRATE vertical connectivity problem
found in 6b91669 -- floors at different levels inside a strate with nothing
joining them. Separate and larger. Not tuned toward.

⚠️ PROCESS: this took 5.7 hours, of which Codex used 4 SECONDS of CPU. The
rest was polling the 73-minute automation suite at ~10% machine load. The
commandlet exists to avoid exactly that. New rule recorded: iterate with the
commandlet, full suite ONCE at the end.

Editor + Shipping builds clean. Automation 32/32, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012aheo2crrZmHuYtPpdY2Cb
2026-09-08 02:14:37 +02:00
Fr0zkaandClaude Opus 5 6b916699af test(explore): 81 runs in an afternoon -- every failure is a DROP, not a squeeze
First real use of the commandlet, and it paid for itself immediately: 64
sweep runs + failure renders + origin probes, which at the old 73-minutes-
per-automation-run would have been days.

⭐ FINDING 1: THE WORLD IS FRAGMENTED VERTICALLY, NOT PINCHED.
Across all 64 runs: ZERO narrow-gap events. Every failed run has separate
arrival/departure fit components with a positive gap, 12.75 m to 72.98 m.
So corridors are not too tight anywhere -- the space is severed.

And for the first time in this project we LOOKED at a failure instead of
inferring from aggregates:
  FlatPlain/42   "a broad flat floor sheet with a sharp diagonal ledge
                  descending into black open space" -- a drop, gap 63.9 m
  SurfaceWorld/7 "terrain shelves and a large black void between them.
                  There is no connecting floor" -- gap 51.8 m
  Maze/42        "a massive solid rock wall/overhang fills the separation"
                  -- gap 27.7 m
I checked the images myself. FlatPlain shows two floor LEVELS 27 m apart
in Z with nothing joining them.

⇒ The missing thing is VERTICAL CONNECTION between floor levels. Ramps,
slopes, ledges, stairs. Every archetype builds floors; nothing builds a way
between them.

⭐ FINDING 2: THE ORIGIN HUB IS NOT REACHABLE, which contradicts 413c15d.
  agent's reachable set included the origin column:  0 / 64 runs
  origin column had ANY player-fit cell:            17 / 64 runs
  Maze slot 4, all 8 seeds:                          0 fit cells, unreachable
Even in the 17, the origin cells were not in the arrival component.

The reason is structural and obvious in hindsight: the origin spine is a
VERTICAL CYLINDER. A vertical shaft has no walkable floor except at its
bottom, so a scalar-density carve that guarantees open air guarantees
nothing standable. The landings join a hub you cannot stand in.

That is a finding about the §6.5 hub join as exposed to the player-fit
layer -- not walk tuning. Recorded rather than patched.

Also from Jahni, looking at the renders -- two things numbers missed:
- The render cameras sit INSIDE ROCK, viewing tunnel surfaces from the
  solid side. Cameras must be seeded from the player-fit set.
- ⚠️ THE MAZE IS STILL A LATTICE OF STRAIGHT PATHS. The spanning tree
  changed WHICH edges exist, not what an edge LOOKS like: every corridor is
  still a straight axis-aligned capsule between cell centres, every junction
  90 deg. So 15.9% dead ends and mean degree 2.504 measured maze-ness of the
  TOPOLOGY while the eye sees a sparser grid. Sixth time a metric answered a
  question adjacent to the one that mattered.

Two tool issues found: PowerShell needs -modes="render,walk" quoted or it
parses as render-only; and "showcase slot" is ambiguous (cards use 1/6, the
Maze diagnosis uses 4).

Determinism spot-checked: repeated arguments give identical SHA-256 JSON.
Measurement-only changes; no generation, parameter or threshold edits.
Editor build succeeded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012aheo2crrZmHuYtPpdY2Cb
2026-09-07 18:33:15 +02:00
Fr0zkaandClaude Opus 5 bd547c35c4 feat(explore): a commandlet -- 91x faster, and the first real PICTURE
Every experiment in this project cost a build + full-suite cycle. The
landing task took 52 of them over ~9 hours. And nobody could SEE the world:
every visual judgement all session was made on density SLICES -- diagrams,
not pictures. That is how a floor heaving 1.4 m every two metres got
described as "roughness visible but modest" while Jahni spotted it in
thirty seconds by walking it.

-run=VoxelForgeExplore, editor-only module, three modes, one build many runs:

  render  raymarched PNGs, fixed 0.25-voxel step + 10 bisection iterations
          (NOT sphere tracing -- the op stack yields density, not a signed
          distance, so a long step tunnels through walls). Three views, each
          with a 1.76 m human marker in frame, because every scale mistake
          here came from numbers with no body beside them.
  walk    an agent using the EXISTING VF player-fit stencil, never its own
          movement model -- two definitions of "can a player stand here"
          would drift and we could not tell which was lying.
  export  OBJ through the canonical UVoxelMarchingCubesMesher. No second
          mesher; a forked meshing invariant is the bug this project keeps
          paying for. 23.2 MB, 106,762 verts, 200,842 tris for 128^3.

⭐ THE NUMBER THAT JUSTIFIES IT: the full automation run takes 4,396 s
(73m16s). The commandlet does all three modes in 48.1 s wall, 5.9 s of
which is engine startup. ~91x faster. And I had guessed the suite cost
~90 s -- it is 73 MINUTES. The iteration loop was far more expensive than
I knew.

First real walk metrics on the Maze (seed 0, slot 4):
  distance 214 m · tortuosity 4.98 · dead ends 0.935/100 m
  narrow-gap events 0 · reachable fit fraction 3.01%
  result NotConnectedAtThisResolution
Tortuosity ~5 is a genuinely winding route, which is right for a maze.
3% reachable is not.

Editor Development build clean. SHIPPING build clean with the receipt
containing VoxelForge and NOT VoxelForgeEditor -- the instrument cannot
reach a packaged game. Automation 32/32, 0 failed.

⚠️ Codex initially shipped this UNBUILT, and correctly: I omitted the
## Validation section from that spec, so it fell back to CLAUDE.md's
"NEVER build or compile". 100+ compile errors from three causes -- bare
TCondensedJsonPrintPolicy and TJsonWriter needing their template
arguments, and a missing UObject/Package.h. That include is the SECOND
miss this session, so CODEMAP §6 now carries the gotcha.

⚠️ Unrelated, found in passing: VoxelForge.uplugin does not declare
RealtimeMeshComponent although the runtime module uses it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K8V9my1PL68pmQb2gutZZV
2026-09-07 04:07:07 +02:00