VoxelForge/Standalone/SHARED_CPP_LIBRARY_PLAN.md

VoxelForgeCore shared-library plan

This plan describes the extracted shape-world pipeline and the next Unreal integration. The standalone implementation is under Standalone/Core; no file in the plugin's Source tree was changed.

Current layout and build

• Core/include/VoxelForgeCore/VoxelForgeCore.h is the public C++20 API. It includes only standard C++ headers and value types.
• Core/src/VoxelForgeCore.cpp adapts the copied density evaluator and marching-cubes mesher to the public API. ShapeWorldInternal.h owns shape compilation, density/detail evaluation, edits and audits. FarShellsInternal.h owns far-parent repairs and strata/connector shells.
• Core/core_sources.txt is the translation-unit manifest. Build.ps1 reads it for both -Target Core and -Target All; the latter links the viewer against Build/VoxelForgeCore/VoxelForgeCore.lib. Use -OutputName to keep another executable open. Builds use /std:c++20 /O2 /arch:AVX2 /fp:precise /MP8, BelowNormal priority and affinity 0xFF00.
• Source/StandaloneShapeWorld.h and Source/StandaloneResearchTunnels.h are forwarding headers for old standalone include sites. StreamingWorld.h remains host code: it owns worker threads, queued jobs, caches, cancellation, D3D upload state and per-frame upload budgets. When a core world is supplied, it gets desired tiles and their priority from the library and submits each tile to World::generateTile. Shape-world --stream-benchmark, --stream-pipeline, edit, audit, and --benchmark-level runs use the Core world. Explicit evaluator and owner-comparison modes retain the compatibility generator only as the comparison side of those diagnostics.
• The build compiles the copied CPU generator and mesher with the private compatibility headers under Compat. Those headers and their Unreal-shaped stand-ins are implementation details. They are absent from the public include tree and public API.

Implementation namespace and Unreal link safety

The public VoxelForgeCore.h surface contains only standard C++ declarations. The copied generator, mesher, and compatibility implementation compile under vfcore::detail, so engine symbols such as FVector, TArray, FMath, and UObject-shaped stand-ins do not acquire the same linker names as Unreal's definitions. Core/src/VoxelForgeCore.cpp owns the public API boundary; it opens vfcore::detail only around its private implementation includes, then defines the VoxelForgeCore API outside that namespace.

For each non-boundary source in Core/core_sources.txt, Build.ps1 writes a one-source shim under Build/VoxelForgeCore/wrappers. The shim includes CoreNamespacePreamble.h before opening vfcore::detail, then includes the canonical source path from the manifest. The preamble loads the standard/platform headers and stateless JSON parser before namespace entry, preventing system declarations from being accidentally nested in vfcore::detail. The VoxelForgeCore.cpp row is compiled directly because it sets up its own public/private namespace boundary. Keep one implementation source per shim.

Compat still defines Unreal-shaped types and a few Unreal-named macros for the copied code, but those declarations live in private implementation translation units only. The UBT module must disable unity builds and PCHs for these shims, and must not force-include Unreal headers into them. This keeps compatibility macros local to one translation unit. Normal plugin adapter translation units may include CoreMinimal.h and the public VoxelForgeCore.h together; the public header does not include any compatibility header or Unreal declaration.

Build.ps1 runs Core/CheckCoreSymbols.ps1 on the completed static library. The check invokes dumpbin /SYMBOLS, derives copied Unreal-style type names from Compat, Source, and Core/src, and fails on an unqualified global occurrence. It also fails closed if dumpbin returns too little symbol data to inspect. Rebuilding the 557f6aa manifest directly and running the same checker found 16,255 unqualified Unreal-style type symbols; the namespaced build checks 177,569 symbol-table lines against 370 type names with no violations. The baseline and current logs are retained in Core/verification/link_safety_557f6aa_final.log and Core/verification/link_safety_final.log. Core/verification/UnrealHeaderCoexistence.cpp was also compiled against the installed Unreal 5.7 CoreMinimal.h and the public header; see Core/verification/unreal_header_smoke_final.log.

API and ownership

ResolvedWorldSnapshot::FromJson takes the normalized UTF-8 schema-v1 JSON produced by the standalone owner export. It resolves the input into owned read-only data. World owns the compiled shape field and edit state for that snapshot. See Core/API.md for signatures, units, and examples.

The host supplies a JobScheduler. The core calls its synchronous parallel_for during shape compilation; the host chooses the pool, fibers, task graph or serial execution. Core code starts no threads. After compilation, hosts may run independent tile generation and query calls on their own workers. Serialize compilation and edit application against reads/generation of the same World; there is no internal lock or cancellation policy.

Tile meshes own positions, normals, UVs, colors and indices. Positions are centimetres; query and tile coordinates are voxels. Group counts expose the ground index run, ceiling index run, and skirt triangle subset. Density/sign, point-solid, segment-solid, and tile-classification queries support host gameplay and collision checks without returning engine objects.

Edit transactions contain an authority-supplied increasing sequence and an ordered brush list. The core hashes those exact values, accepts an identical replay idempotently, rejects sequence reuse with different data and out-of-order transactions, and returns affected desired tiles grouped by LOD. The host schedules generateTile for those keys and drops results made stale by its own cancellation/version policy. Every client must apply the same sequence and brush order.

Unreal integration: UBT module

Add a VoxelForgeCore runtime module to the plugin and make the existing VoxelForge module depend on it. Its Build.cs sets C++20, exports only Standalone/Core/include, and keeps the Standalone root plus these private include roots available to the core implementation:

• Standalone
• Standalone/Core/src
• Standalone/Compat
• Standalone/Source
• Standalone/Source/VoxelForge/Public
• Standalone/Source/VoxelForge/Private

Set PCHUsage = PCHUsageMode.NoPCHs and bUseUnity = false for the Core module. For example:

string PluginRoot = Path.GetFullPath(Path.Combine(ModuleDirectory, "../.."));
string Standalone = Path.Combine(PluginRoot, "Standalone");
PublicIncludePaths.Add(Path.Combine(Standalone, "Core", "include"));
PrivateIncludePaths.AddRange(new[] {
    Standalone,
    Path.Combine(Standalone, "Core", "src"),
    Path.Combine(Standalone, "Compat"),
    Path.Combine(Standalone, "Source"),
    Path.Combine(Standalone, "Source", "VoxelForge", "Public"),
    Path.Combine(Standalone, "Source", "VoxelForge", "Private")
});
CppStandard = CppStandardVersion.Cpp20;
PCHUsage = PCHUsageMode.NoPCHs;
bUseUnity = false;

UBT discovers translation units under a module's Source tree. Add one thin .cpp include shim per row of Core/core_sources.txt, with each shim including exactly one canonical source file by a path rooted at the private Standalone include path. For every implementation row except Core/src/VoxelForgeCore.cpp, the shim includes CoreNamespacePreamble.h, opens namespace vfcore::detail, then includes that one source. The special API row gets a shim that includes Core/src/VoxelForgeCore.cpp directly; that source establishes its own namespace boundary. Keeping one included source per shim preserves the translation-unit boundaries used by Build.ps1. A manifest check in the UBT module should fail when a row lacks a shim, a shim names a different file, or an extra shim exists. Do not copy implementation files into the module. The standalone and UBT builds then compile the same canonical source bytes.

The existing VoxelForge module owns the adapter and calls the new module. Its adapter runs on the game thread, resolves soft references and asset defaults, preserves array order, and serializes settings, biome and terrain-op records, strata layouts, passages, seed, voxel size, clip limits and viewer start into the exact schema-v1 document accepted by FromJson. It must not serialize UObject pointers or depend on unordered map iteration. Snapshot creation returns a descriptive error for unresolved or unsupported assets. Once created, workers only see the immutable core snapshot and standard C++ results.

The plugin supplies a JobScheduler backed by its chosen task system. The adapter submits shape build work and tile jobs to that scheduler, observes generation IDs for cancellation, and discards stale results. It copies completed TileMesh data into FVoxelMeshData or the plugin's current mesh payload on the host side, preserving centimetre positions, normals, UVs, colors, indices and ground/ceiling/skirt group counts. UObject creation, collision cooking, render-resource changes, game-thread application, residency and upload budgets remain in the plugin.

What the plugin pipeline replaces

The current plugin's AVoxelWorld::GenerateTileResult worker path performs proofs and UVoxelGenerator::ClassifyTile, calls UVoxelMarchingCubesMesher::GenerateMesh, builds RealtimeMesh streams with BuildTileStreamSet, then publishes an epoch-tagged FChunkResult for game-thread upload/collision handling. AVoxelWorld::CarveAtPosition, CarveBox, and CarveCapsule route soft FVoxelModification brushes through UVoxelDiffLayer::ApplyModification and remesh the affected chunks. The adapter replaces those CPU-field and edit operations as follows:

Current call or state	Core API replacement	Keep in the plugin
Shape interval/detail-band build and any active shape-world dispatch	World::compile with explicit bounds/detail options	Asset resolution, build admission and cancellation
GetDensityAt, lattice sign tests, and point/segment checks in meshing/gameplay	densityAt, densityAtLattice, isSolidAt, isSolidAlongSegment	Gameplay policy and physics objects
ClassifyTile / tile proof	classifyBox	Clipmap residency and tile lifetime
GenerateMesh plus shape-world far-parent repair and shell stitching	generateTile	Convert owned result to FVoxelMeshData, call BuildTileStreamSet, and upload it
VoxelClipmapDesiredTiles::Build tile-set calculation	desiredTiles and prioritizeTiles	Decide when to request, cancel, retain or evict jobs
UVoxelDiffLayer::ApplyModification and affected-chunk scan	applyEditTransaction with densityOffset brushes carrying signed strength and falloff; regenerate returned LOD keys	Brush budgets, authority sequence, replication, and result acceptance

Delete plugin-side active-world dispatch when routing these calls; the World context is an explicit object passed to each operation. The plugin's epoch/cancellation checks still reject stale results, but cannot interrupt an executing core call. Because edit application replaces field edit state, the host must drain in-flight core calls before applying a transaction and then submit the affected tiles under a new epoch. Do not move worker admission, collision cooking, actor or material creation, or rendering into the library.

Acceptance record

The 557f6aa commit is the strict output baseline. The rebuilt namespaced Core matches all 11 full-detail, whole-view and LOD0–8 hash rows at W1 and W8, with W1/W8 parity. The fresh records are Core/verification/last_standard_hash_w1.stdout.log, Core/verification/last_standard_hash_w8.stdout.log, and Core/verification/final_rebuilt_hash_parity.log.

The look and shape audits pass, and walkability remains 5,642/5,642. The dedicated passage-camera detail audit passes at 13,271 passage samples with a detail ratio of 1.152. Far survival passes with 7,200 surfaces, zero missing surfaces or holes, and 10,976 far connectors. The damaged-shell control removes 512 triangles, reports 16 missing boundary surfaces and 16 holes, and exits 14. The final rebuilt logs are Core/verification/final_rebuilt_shape_audit.stdout.log, Core/verification/final_rebuilt_passage_camera_audit.stdout.log, Core/verification/final_rebuilt_far_survival.stdout.log, and Core/verification/final_rebuilt_bad_shell_control.stdout.log.

The six fixed 1280×800 views were recaptured from the final rebuilt viewer and compare byte-for-byte with the prior Core captures. Their images are Core/verification/core_final_rebuild_*.png; the SHA-256 results are in Core/verification/final_rebuilt_capture_parity.log. The single-tile --shape-world --benchmark-level=0 path reports generator=VoxelForgeCore.

The link-safety check demonstrates the failure before the change and the pass after it. Running it on the 557f6aa library finds 16,255 unqualified Unreal-style symbols; the rebuilt Core library passes after checking 177,569 dumpbin /SYMBOLS lines against 370 copied Unreal-style type names. Core/verification/link_safety_557f6aa_final.log and Core/verification/link_safety_final.log retain the outputs. A smoke translation unit that includes the installed Unreal 5.7 CoreMinimal.h and the public VoxelForgeCore.h also compiles; see Core/verification/unreal_header_smoke_final.log.

Speed gate: not met. The no-hash, cold W8 --stream-pipeline --shape-world --force-generate check was run in four interleaved baseline/Core pairs, each generating 2,226 tiles and the full 110.90 MiB mesh payload with no cache hits. whole_view_ready averaged 0.89175 s for the exact 557f6aa executable and 0.92050 s for the namespaced build, a 28.8 ms (3.2%) regression. Stream wall time was effectively unchanged at 0.46000 s and 0.46025 s; the difference was in shape compilation, including intervals+signs averaging 95.7 ms on baseline and 114.9 ms on Core. The raw cold runs are in Core/verification/final_rebuilt_pipeline_*.stdout.log. API-only /Ob3 and /Ob1 trials also retained hash parity but were slower, so the build is restored to /O2 /MP8. The required “no slower” acceptance condition remains outstanding.