tools(profile): the instrument lied by 11.5x -- now it checks itself, and batching kills startup cost

Overnight task 1 of the owner's mandate: "better diagnostic, profiler and tools,
so we can batch test way more quicker."

WHY THIS CAME BEFORE ANY MORE OPTIMISATION
ee89c4d delivered a 5x candidate reduction and a 16x cheaper support floor for
0% end-to-end gain, because the profiler inflated by ~11.5x (14.74 us/sample
profiled vs 1.28 us/voxel real). Two rounds of bucket rankings were artefacts of
FPlatformTime::Cycles64() wrapped around ~100 ns operations -- the timer pair
calibrates at ~0.1 us, so it was measuring itself.

PROFILER
Deterministic count sampling plus amortised block timing every 64 density calls.
Coarse counters (calls, candidates) stay always-on -- they were never the problem.
Timing-only scopes are opt-in behind -profiledensityfull. UE CPU trace scopes added.

  before   profiled 14.74 us/sample  vs  1.28 us/voxel real   ~11.5x
  after    profiled 0.463050 s       vs  0.558470 s unprofiled   -17.09%
           geometry 235AC2DC in both

⭐ Every profiled report now prints profiled total, unprofiled total, discrepancy,
ratio and geometry equality, and EXITS CODE 2 if the comparison fails. An
intermediate run hit -22.6% and failed loudly, which is the point: this class of
bug must be impossible to miss rather than something noticed three rounds later.

⚠️ The -17% residual is run-to-run noise, so ~17% is our measurement floor.
Improvements smaller than that cannot be claimed from a single pair of runs.

BATCH EXPLORER
One process, many cases, one JSON. Startup was ~25-30 s of engine boot for ~3 s of
work, which is what made every iteration expensive.

  3 cases in 6.59 s of case time
  standalone vs batched, same binary: hash 235AC2DC, 2,461 tris, 343,000 tunnel
  lookups, 554,381 passage evaluations -- identical on both

-out= is enforced absolute under the plugin's own Saved/.

CACHE BREAKDOWN (the 272 KB/entry question)
The slot table is small. The weight is spatial-index bins and candidate arrays,
room arrays, support-column arrays and tunnel-world indices -- i.e. much of the
411 MiB is the spatial index ee89c4d added, which also bought 0% end-to-end.
Worth revisiting on those terms.

RESEARCH (recorded, not acted on) -- CODEX-TASK-1-PROFILER-BATCH-RESEARCH.md
Ranked follow-ups: deterministic empty-space early-out, density-grid reuse, task
granularity (ParallelForWithExistingTaskContext + MinBatchSize), SIMD MC (~2x over
optimised scalar, but cross-machine determinism is the risk), sparse hierarchy,
RMC partial updates, Transvoxel LOD seams. Sources cited in the doc.
⚠️ Upstream RMC is 5.3.2; the owner's copy identifies as 5.4 with custom
modifications. No upgrade is safe. Left untouched.

PUSHBACK, accepted: "buckets must sum to their parent" needs a parallel-time
definition -- summing worker CPU is not wall time. Reports now separate worker CPU
sum from wall-clock estimate.

End-to-end unchanged, as expected for a tooling change: 2.746 / 2.691 s on the
128^3 seed 1337 region (was 2.678-2.705). Build clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012aheo2crrZmHuYtPpdY2Cb
This commit is contained in:
Fr0zkaandClaude Opus 5 committed 2026-09-09 05:02:21 +02:00
1 parent ee89c4d8c2
commit e63f016835
10 files changed
+1494 -124

No files matched your search

+13 -5
View File
@@ -18,6 +18,7 @@
#include "VoxelHeightOp.h" // IVoxelBiomeField — the adapter below implements it
#include "VoxelStats.h"
#include "VoxelDensityProfile.h"
#include "ProfilingDebugging/CpuProfilerTrace.h"
#if WITH_EDITOR
#include "VoxelStrateComposer.h"
@@ -535,6 +536,7 @@ namespace
uint64 EntryBytes = 0;
uint64 LargestEntryBytes = 0;
uint64 ValidEntries = 0;
VoxelDensityProfile::FCacheMemoryBreakdown Breakdown;
for (int32 Index = 0; Index < TunnelDensityCacheSlotCount; ++Index)
{
const FTunnelNetworkDensityCacheEntry& Entry = Entries[Index];
@@ -547,6 +549,8 @@ namespace
const uint64 StackBytes = static_cast<uint64>(Entry.OpStack.GetAllocatedSize());
const uint64 TunnelCoreBytes = static_cast<uint64>(
Entry.TunnelCore.Cache.GetAllocatedSize());
Breakdown.OpStackBytes += StackBytes;
Breakdown += Entry.TunnelCore.Cache.GetAllocatedSizeBreakdown();
const uint64 EntryDynamicBytes = StackBytes + TunnelCoreBytes;
const uint64 FullEntryBytes = static_cast<uint64>(
sizeof(FTunnelNetworkDensityCacheEntry)) + EntryDynamicBytes;
@@ -554,6 +558,8 @@ namespace
EntryBytes += FullEntryBytes;
LargestEntryBytes = FMath::Max(LargestEntryBytes, FullEntryBytes);
}
Breakdown.SlotStorageBytes = static_cast<uint64>(sizeof(FTunnelNetworkDensityCacheEntry))
* TunnelDensityCacheSlotCount;
VoxelDensityProfile::SetWorkerTunnelCacheFootprint(
TunnelDensityCacheSlotCount,
@@ -563,7 +569,8 @@ namespace
DynamicBytes,
EntryBytes,
LargestEntryBytes,
ValidEntries);
ValidEntries,
Breakdown);
}
FORCEINLINE int32 TunnelDensityCacheSlot(const FIntVector& Chunk)
@@ -852,6 +859,7 @@ void UVoxelGenerator::InitializeSettings(const UVoxelSettings* Settings)
float UVoxelGenerator::GetDensityAt(float WorldX, float WorldY, float WorldZ) const
{
TRACE_CPUPROFILER_EVENT_SCOPE(VoxelForge_GetDensityAt);
VoxelDensityProfile::FScopedTimer DensityProfileTimer(
VoxelDensityProfile::EBucket::GetDensityAt);
VoxelDensityProfile::FScopedTimer DensityPrologueTimer(
@@ -998,7 +1006,7 @@ float UVoxelGenerator::GetDensityAt(float WorldX, float WorldY, float WorldZ) co
if (!bIsGapChunk)
{
if (VoxelDensityProfile::IsEnabled())
if (VoxelDensityProfile::AreCountersEnabled())
{
VoxelDensityProfile::AddCounter(
VoxelDensityProfile::ECounter::TunnelCacheLookup);
@@ -1009,7 +1017,7 @@ float UVoxelGenerator::GetDensityAt(float WorldX, float WorldY, float WorldZ) co
TunnelDensityCacheEntryForWrite = CachedEntryPtr;
if (bCacheKeyMatches)
{
if (VoxelDensityProfile::IsEnabled())
if (VoxelDensityProfile::AreCountersEnabled())
{
VoxelDensityProfile::AddCounter(
VoxelDensityProfile::ECounter::TunnelCacheHit);
@@ -1035,7 +1043,7 @@ float UVoxelGenerator::GetDensityAt(float WorldX, float WorldY, float WorldZ) co
ActiveTunnelCoreCache = &CachedEntry.TunnelCore;
bLoadedTunnelDensityCache = true;
}
else if (VoxelDensityProfile::IsEnabled())
else if (VoxelDensityProfile::AreCountersEnabled())
{
VoxelDensityProfile::AddCounter(
VoxelDensityProfile::ECounter::TunnelCacheMiss);
@@ -1765,7 +1773,7 @@ float UVoxelGenerator::GetDensityWithParams(float WorldX, float WorldY, float Wo
if (bNeedRebuild)
{
if (VoxelDensityProfile::IsEnabled())
if (VoxelDensityProfile::AreCountersEnabled())
{
VoxelDensityProfile::AddCounter(
VoxelDensityProfile::ECounter::SdfCacheBuild);