Culling

This unit is three files — `render/culling.hpp`, `shaders/compute/gpu_cull.comp`, `shaders/compute/hiz_generate.comp` — and only the header runs. The CPU frustum in `culling.hpp` is built once per frame inside the GBuffer pass's command recording, from the same view-projection that pass is about to rasterize with:

That is the only construction of a `Frustum` in the tree, and `isAABBVisible` its only test with a caller — `isSphereVisible`, `isPointVisible` and `getPlane` have none. Both compute shaders compile anyway (`shaders/CMakeLists.txt` globs every `.comp` recursively) but nothing in the C++ tree loads either `.spv`. Their histories differ, and that difference is the substance of this unit: `gpu_cull.comp` never had a working dispatcher, `hiz_generate.comp` lost one. `shaders/compute/light_culling.comp` sits in the same limbo, compiled and unreferenced, but it is a tile-lighting design rather than part of this one.

Hardware face culling is a separate mechanism, and the GBuffer pipeline turns it off:

so the GBuffer pass rasterizes every back-facing triangle in the scene. The one pipeline in the deferred path that culls faces is the cascade shadow pass, and it culls the *front* ones:

recording shadow depth on the back surface of each caster — the usual trade of acne for peter-panning, on top of the depth bias the same pipeline enables. A single-sided mesh that rasterizes correctly into the GBuffer can therefore vanish from the shadow map.

The extraction is Gribb-Hartmann, worth deriving because the code looks transposed. A point $\mathbf{p}$ is inside the left clip boundary when its clip-space $x$ satisfies $-w \le x$. Writing $M = P V$ with rows $\mathbf{m}_0 \ldots \mathbf{m}_3$, so that $x_{clip} = \mathbf{m}_0 \cdot \mathbf{p}$ and $w = \mathbf{m}_3 \cdot \mathbf{p}$, that inequality is:

The left plane's $(\mathbf{n}, d)$ is thus a sum of two *rows* of the view-projection — no inverse, no decomposition, six adds. GLM stores column-major, so row 3 is `m[0][3], m[1][3], m[2][3], m[3][3]`, which is why the indices look swapped:

Near and far are $\mathbf{m}_3 \pm \mathbf{m}_2$, and that is the line with a fuse in it. $\mathbf{m}_3 + \mathbf{m}_2$ is the near plane only when clip-space $z$ runs $-w \ldots w$ — the OpenGL convention. OHAO's camera builds its projection with plain `glm::perspective`, and the tree never defines `GLM_FORCE_DEPTH_ZERO_TO_ONE`:

so the extraction is correct as written. Define that macro — the obvious "make GLM Vulkan-native" cleanup — and exactly one plane changes. Far survives, since $z_{clip} \le w$ is the far boundary in both conventions; left, right, top and bottom never involved $z$. Near does not. Under zero-to-one depth the near boundary is $z_{clip} \ge 0$, i.e. $\mathbf{m}_2$ alone, whose zero set for `glm::perspective(fovy, aspect, n, f)` is $z = -n$. The row *sum* the code computes is $(0,\,0,\,-1-\tfrac{f}{f-n},\ -\tfrac{fn}{f-n})$, whose zero set is:

That is a plane at half the near distance — *closer* to the camera than the true one, not behind it. Everything nearer than $n/2$ is still rejected; only the shell between $n/2$ and $n$ stops being. Nothing renders wrong, since the rasterizer still clips; the cull just quietly gets weaker in a region nobody puts geometry in.

One perturbation reaches the matrix before extraction. `DeferredRenderer::setCameraData` negates a single element for Vulkan's Y-down NDC:

which is a negation of all of row 1, because every other entry of that row in a `glm::perspective` matrix is zero. `BOTTOM` is extracted as $\mathbf{m}_3 + \mathbf{m}_1$, the $y_{clip} \ge -w$ boundary — the bottom of the image in OpenGL's Y-up NDC, the *top* of it in the Y-down NDC this matrix now targets. Both tests loop over all six planes without caring which is which, so only the labels lie: `getPlane(Frustum::TOP)` hands back the plane bounding the bottom of the image, and the only reason that is harmless is that nothing calls it.

The matrix the GBuffer pass receives is `jitteredProj`, not `m_proj`:

Usually those are the same matrix. TAA is off by default, and `getJitterOffset` short-circuits to `glm::vec2(0.0f)` while it stays off:

The only site in the tree that enables it is `examples/model_viewer.cpp` on the `--deferred` path; four others — the three dense inverse-rendering fits and the differentiable forward pass — disable it explicitly, and `cornell_box`, `interactive`, `env_demo` and `turntable` never touch it. When jitter *is* live it is added to two entries of the projection's third column:

shearing the cull frustum sub-pixel each frame. That is right — it then matches the geometry actually rasterized — and worth knowing before someone "fixes" it by passing `m_proj`. It also costs the Y-flip its symmetry: `[2][1]` is a row-1 entry, so under active jitter `TOP` and `BOTTOM` are still interchanged but no longer exact mirrors.

The box test never touches a corner. For a plane $(\mathbf{n}, d)$ and a box with centre $\mathbf{c}$ and half-extents $\mathbf{e}$:

$r$ is the projected radius — the half-width of the box's shadow on the plane normal. For an axis-aligned box it is exact: the extreme corner along $\mathbf{n}$ is the one whose sign pattern matches $\mathbf{n}$ componentwise, so the $|n_k|$-weighted sum of extents *is* that corner's offset.

The test is invariant to plane scale — both $\mathbf{n}\cdot\mathbf{c}+d$ and $r$ scale linearly with the plane vector — so `normalize()` is not needed by `isAABBVisible` at all. It is needed by `isSphereVisible`, whose radius arrives in world units, and that function has no callers. The normalization is paid for a consumer that does not exist.

The world-space box comes from Arvo's transform: transformed centre, plus half-extents pushed through the absolute value of the upper 3×3.

Under rotation that is conservative rather than tight — the safe direction for a rejection test.

Per frame, per actor, the GBuffer pass rebuilds the local bounding box by walking every vertex of the mesh:

Nothing is cached; the box is recomputed whether or not the transform moved. That is $O(V)$ of CPU work per actor to skip that actor's draws — plural: the loop emits one call per material range, so culling an actor saves as many calls as it has ranges, not one:

The scan is guarded by `if (model && !model->vertices.empty())`, so actors without a model are never tested and always draw. At a handful of actors that is invisible; it is also why the technique does not scale, and what `gpu_cull.comp` was drafted to replace.

The scan needs `model->vertices` to stay CPU-resident after GPU upload, and it is not the only thing that does — picking walks the same array twice per ray, once for the world AABB and once for the triangles:

so `AABB` has two users beyond the cull: that ray-box rejection, and the engine test suite, which builds one by hand to exercise `isValid`, `center` and `size`.

The shadow pass makes the opposite call and filters only on the component visibility flag:

That is correct rather than an oversight: an object outside the camera frustum can still cast a shadow into it, so camera-frustum culling of shadow casters deletes shadows. Culling for CSM needs the *light's* frustum per cascade, which is not implemented.

`gpu_cull.comp` is a 64-wide pass over mesh descriptors: optional distance rejection, then a bounding-sphere test whose world radius is scaled by the largest of the three model-matrix column lengths — conservative, and visibly too large for a non-uniformly stretched mesh.

The stricter `aabbInFrustum` in the same file, which picks the p-vertex per plane instead of projecting a radius, has no caller anywhere. Occlusion is a declared flag bit with a commented-out body, and that comment is the only mention of Hi-Z in the file — an intention, not a wiring:

The dispatcher this shader was written for did exist, as `GpuCullPass` in `ohao/render/deferred/_disabled/indirect_draw_buffer.cpp`, deleted in commit `e8eb9d1` — and it never ran. `createPipeline()` built a pipeline layout, logged "shader loading deferred" and returned without creating a pipeline, so `execute()` bailed out on a null `m_cullPipeline`. This shader is a staged design, not a regression.

Three things must change before it can dispatch, none of them obvious from a skim:

• The push-constant block is 192 bytes under std430 (`mat4` + `vec4` + 6×`vec4` + four

`uint`s); Vulkan guarantees only 128. The `mat4 viewProj` is never read — the frustum arrives pre-extracted in `frustumPlanes` — so deleting those 64 bytes puts the block exactly on the guaranteed floor.

• Each surviving mesh writes `firstInstance = meshIdx` so the vertex stage can index

instance data. A non-zero `firstInstance` in an *indirect* draw requires `drawIndirectFirstInstance`; the device is created with `multiDrawIndirect` enabled and that feature absent. {{cite ohao/gpu/vulkan/device_setup.cpp "deviceFeatures2.features.multiDrawIndirect = VK_TRUE;"}}

• Commands are appended with an atomic bump, so surviving draws land in an order that

changes frame to frame — harmless for opaque depth-tested geometry, flicker for any blended pass. The counter must also be host-zeroed before every dispatch, since the shader only increments. {{cite shaders/compute/gpu_cull.comp "uint drawIdx = atomicAdd(drawCount, 1);"}}

Instance data is indexed as `instances[meshIdx]`, baking in a 1:1 mesh-to-instance mapping even though `DrawInstance` carries a `meshIndex` field for the general case.

`hiz_generate.comp` is not the occlusion companion to `gpu_cull.comp`; its own header says otherwise:

and the only shader in the tree that has ever sampled it is screen-space reflections:

which starts at the coarsest level, drops a mip and steps back when a hit looks possible, and climbs one when the ray is clear.

That reader was not a sketch. Until commit `e8eb9d1` ("refactor: Delete dead passes"), `SSRPass` built a compute pipeline from `compute_hiz_generate.comp.spv`, dispatched it once per mip at the top of its `execute`, and `PostProcessingPipeline::execute` called that every frame SSR was on. The pyramid ran; the code that ran it was deleted. Today's SSR pass is a different shader, marching the full-resolution depth buffer directly with no mip chain and no Hi-Z binding:

So the pyramid has no reader by regression, not by design — and which reader it lost decides what its reduction should have been. Each invocation takes the maximum of a 2×2 block into an `r32f` storage image, 8×8 per workgroup:

The GBuffer depth test is `VK_COMPARE_OP_LESS`, so smaller is nearer and the maximum over a footprint is the *furthest* surface in it.

For occlusion culling that is the conservative direction: a max can only make the occluder look further away, so such a test errs toward "visible". For a hierarchical ray march it is the wrong one. The march treats "ray depth exceeds the stored depth" as a possible hit and refines; the else branch calls the block clear and takes a bigger step at a coarser mip.

Being nearer than a block's *furthest* surface says nothing about the surfaces in front of it, so that coarse step can jump over a real intersection. The minimum is the safe reduction here — nearer than the block minimum, nothing in the footprint can be hit. The pyramid and its consumer disagree about which extreme is conservative, and the disagreement is invisible because the consumer is disabled.

The reduction is also a strict 2×2 with no odd-size fallback:

Real pyramids gather 3×3 whenever a source dimension is odd, or the last row or column is never sampled and the level below summarizes a smaller footprint than it claims. A 1080-tall chain halves to 540, 270, 135 and on to 67, 33, 16, 8, 4, 2, 1; the three odd sources in it — 135, 67 and 33 — each drop their bottom row, and the even ones drop nothing. The two sampling branches differ only in how the level is named: mip 0 uses `texture()`, later mips `textureLod(..., srcMip)`. A compute dispatch has no fragment-quad derivatives for an implicit LOD to come from, and glslang lowers the implicit form here to an explicit `Lod 0` anyway — so the first branch would be clearer, and compile the same, as `textureLod(..., 0.0)`.

• `extractFromViewProj` assumes OpenGL clip space ($-w \le z \le w$). Defining

`GLM_FORCE_DEPTH_ZERO_TO_ONE` anywhere in the build moves the near plane from $n$ to about $n/2$, weakening the cull with no error and no visual artifact. The other five planes are unaffected.

• The frustum must come from the *same* matrix the pass rasterizes with, jitter included.
• Plane labels (`TOP`/`BOTTOM`) are inverted relative to the screen after the Vulkan

Y-flip. Future code reading `getPlane(index)` by name — cascade fitting, portal tests — inherits the swap.

• The GBuffer cull depends on `model->vertices` staying CPU-resident. Freeing host vertex

data after upload turns every actor into an untested actor, and picking, which guards on the same array, stops hitting anything.

• `ohao::AABB` (`render/culling.hpp`) is constructed from `(min, max)`;

`ohao::physics::math::AABB` (`physics/utils/physics_math.hpp`) has identical fields and an identical two-`vec3` signature but takes `(center, halfExtents)`. Confusing them compiles cleanly and yields a wrong-sized box.