Blank depth buffer from ARSCNView in DRAW_QUAD pass of SCNTechnique

Question

I'm trying to do depth-testing in a post-processing step of an AR SceneKit demo. I would need the depth map of the rendere ARSCNView for that. It seems impossible to get it using an SCNTechnique.

I keep getting blank (full of 1.0-s) depth buffers when trying to use the depth from a DRAW_SCENE pass as input in a DRAW_QUAD pass in an SCNTechnique. I've followed the guides at SCNTechnique and named the depth target. Is this a bug in the implementation of SCNTechnique, or am I missing something in the configuration?

The color buffer is chained correctly, and the example from https://github.com/lachlanhurst/SCNTechniqueTest/tree/pixelate works.

Here's the debug view of the metal technique, as you can see, the depth buffer is completely white.

Here's the technique plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>passes</key>
    <dict>
        <key>pixelate_scene</key>
        <dict>
            <key>draw</key>
            <string>DRAW_SCENE</string>
            <key>inputs</key>
            <dict/>
            <key>outputs</key>
            <dict>
                <key>color</key>
                <string>color_scene</string>
                <key>depth</key>
                <string>depth_scene</string>
            </dict>
            <key>colorStates</key>
            <dict>
                <key>clear</key>
                <true/>
                <key>clearColor</key>
                <string>sceneBackground</string>
            </dict>
        </dict>
        <key>resample_pixelation</key>
        <dict>
            <key>draw</key>
            <string>DRAW_QUAD</string>
            <key>program</key>
            <string>doesntexist</string>
            <key>metalVertexShader</key>
            <string>pixelate_pass_through_vertex</string>
            <key>metalFragmentShader</key>
            <string>pixelate_pass_through_fragment</string>
            <key>inputs</key>
            <dict>
                <key>colorSampler</key>
                <string>color_scene</string>
                <key>depthSampler</key>
                <string>depth_scene</string>
            </dict>
            <key>outputs</key>
            <dict>
                <key>color</key>
                <string>COLOR</string>
            </dict>
        </dict>
    </dict>
    <key>sequence</key>
    <array>
        <string>pixelate_scene</string>
        <string>resample_pixelation</string>
    </array>
    <key>targets</key>
    <dict>
        <key>color_scene</key>
        <dict>
            <key>type</key>
            <string>color</string>
        </dict>
        <key>depth_scene</key>
        <dict>
            <key>type</key>
            <string>depth</string>
        </dict>
    </dict>
    <key>symbols</key>
    <dict/>
</dict>
</plist>

The shaders look like this:

#include <metal_stdlib>
#include <metal_geometric>
using namespace metal;


#include <SceneKit/scn_metal>

struct custom_vertex_t
{
    float4 position [[attribute(SCNVertexSemanticPosition)]];
};

constexpr sampler s = sampler(coord::normalized,
                              address::repeat,
                              filter::nearest);

struct out_vertex_t
{
    float4 position [[position]];
    float2 uv;
};

vertex out_vertex_t pixelate_pass_through_vertex(custom_vertex_t in [[stage_in]], constant SCNSceneBuffer& scn_frame [[buffer(0)]])
{
    out_vertex_t out;
    out.position = in.position;
    out.uv = float2((in.position.x + 1.0) * 0.5 , (in.position.y + 1.0) * -0.5);
    return out;
};



fragment half4 pixelate_pass_through_fragment(out_vertex_t vert [[stage_in]], texture2d<float, access::sample> colorSampler [[texture(0)]], texture2d<float, access::sample> depthSampler [[texture(1)]])
{
    float4 fragment_color = colorSampler.sample( s, vert.uv);
    float ar_depth = depthSampler.sample(s, vert.uv).r;

    return half4(fragment_color * 0.5 + float4(ar_depth) * 0.5);
};

Ecclesiastical

Did you ever find a way to get the depth buffer of an ARSCNView frame in the SCNTechnique shaders? – Venule

Answer 1

In your fragment shader the type for the depthSampler should be depth2d instead of texture2d. Setting up the pipeline correctly using SCNTechnique can be somewhat tricky, but it works once all the peaces of the puzzle are right.

Mercenary