Unity function to access the 2D box immediately from the 3D pipeline?

Question

In Unity, say you have a 3D object,

Of course, it's trivial to get the AABB, Unity has direct functions for that,

(You might have to "add up all the bounding boxes of the renderers" in the usual way, no issue.)

So Unity does indeed have a direct function to give you the 3D AABB box instantly, out of the internal mesh/render pipeline every frame.

Now, for the Camera in question, as positioned, that AABB indeed covers a certain 2D bounding box ...

In fact ... is there some sort of built-in direct way to find that orange 2D box in Unity??

Question - does Unity have a function which immediately gives that 2D frustrum box from the pipeline?

(Note that to do it manually you just make rays (or use world to screen space as Draco mentions, same) for the 8 points of the AABB; encapsulate those in 2D to make the orange box.)

I don't need a manual solution, I'm asking if the engine gives this somehow from the pipeline every frame?

Is there a call?

(Indeed, it would be even better to have this ...)

My feeling is that one or all of the

• occlusion system in particular
• the shaders
• the renderer

would surely know the orange box, and perhaps even the blue box inside the pipeline, right off the graphics card, just as it knows the AABB for a given mesh.

We know that Unity lets you tap the AABB 3D box instantly every frame for a given mesh: In fact does Unity give the "2D frustrum bound" as shown here?

Darbee

This is an interesting question... – Extraditable

@Extraditable I think so. – Kisner

Me too. It's evident that (1) the overall rendering engine and/or (2) the shaders and/or (3) the occlusion system, must in fact instantly know this information inside every frame about every object. – Darbee

Indeed an interesting question. I would personally approach it by getting the mesh bounds and instancing a square based on those bounds, that looks at the camera at all times. However if it is already present in the engine I think it will most likely be here: github.com/Unity-Technologies/UnityCsReference – Disaster

( @Disaster - that algorithm actually doesn't work! think of when the object rotates. you immediately face the main chore of expanding/contracting the sides to encompass the breaking of the local frustrum section. A very difficult problem.) – Darbee

Probably this is a slow (and manual) solution, but what about using mesh-model info? You have every vertex on the model, so if you have the 4 farthest vertex in every direction (N, S, E, W), having the camera as a reference, it's easy to build a 2D box (tight, as you want on the blue box). – Noah

hey @Noah , sure, as mentioned over and over, manual solutions are not the issue here.check out any of the well-known texts on 3D mathematics to explore existing well-known approaches! amazon.com/Eric-Lengyel-Mathematics-Programming-Hardcover/dp/… – Darbee

(Just FWIW note that "furthest" vertex has no relationship (unfortunately!) to position in 3D frustrum.) – Darbee

finally just FWIW, @Noah - what's wanted is the "flat-camera-box" for the AABB. ie the "orange square". it's relatively easy to do that manually (as I explained) using the 8 extremes. the question is whether unity does this (likely on the metal) built-in. (My addendum about the blue box is quite different; but who knows, maybe Unity supplies that also.) – Darbee

Afaik there is no such public API function. (p.s. if done manually, I'd probably opt to go via the Camera's WorldToScreenPoint for all 8 AABB corners and then computing the bounds). – Wallraff

you know, you should possible put that in as an answer, @bart ! sometimes a negative fact is useful – Darbee

Answer 1

As far as I am aware, there is no built in for this.

However, finding the extremes yourself is really pretty easy. Getting the mesh's bounding box (the cuboid shown in the screenshot) is just how this is done, you're just doing it in a transformed space.

1. Loop through all the verticies of the mesh, doing the following:
2. Transform the point from local to world space (this handles dealing with scale and rotation)
3. Transform the point from world space to screen space
4. Determine if the new point's X and Y are above/below the stored min/max values, if so, update the stored min/max with the new value
5. After looping over all vertices, you'll have 4 values: min-X, min-Y, max-X, and max-Y. Now you can construct your bounding rectangle

You may also wish to first perform a Gift Wrapping of the model first, and only deal with the resulting convex hull (as no points not part of the convex hull will ever be outside the bounds of the convex hull). If you intend to draw this screen space rectangle while the model moves, scales, or rotates on screen, and have to recompute the bounding box, then you'll want to do this and cache the result.

Note that this does not work if the model animates (e.g. if your humanoid stands up and does jumping jacks). Solving for the animated case is much more difficult, as you would have to treat every frame of every animation as part of the original mesh for the purposes of the convex hull solving (to insure that none of your animations ever move a part of the mesh outside the convex hull), increasing the complexity by a power.

Inerasable

hey Draco, game engines calculate such things right off the chipset as part of (as explained in the question) three processes - rendering, shader, occlusion. In fact, we know that Unity exposes the AABB calculation (hence it would be insane to calculate this yourself, right? :) ). The question here is whether the frustrum box is exposed in Unity (much as the AABB stream is, every frame). – Darbee

I have deleted my own comments and edited the answer. – Inerasable

It sees that "As far as I am aware, there is no built in for this." is in fact correct - so that's that! – Darbee

Answer 2

3D bounding box

1. Get given GameObject 3D bounding box's center and size
2. Compute 8 corners
3. Transform positions to GUI space (screen space)

Function GUI3dRectWithObject will return the 3D bounding box of given GameObject on screen.

2D bounding box

1. Iterate through every vertex in a given GameObject
2. Transform every vertex's position to world space, and transform to GUI space (screen space)
3. Find 4 corner value: x1, x2, y1, y2

Function GUI2dRectWithObject will return the 2D bounding box of given GameObject on screen.

Code

public static Rect GUI3dRectWithObject(GameObject go)
{

    Vector3 cen = go.GetComponent<Renderer>().bounds.center;
    Vector3 ext = go.GetComponent<Renderer>().bounds.extents;
    Vector2[] extentPoints = new Vector2[8]
    {
            WorldToGUIPoint(new Vector3(cen.x-ext.x, cen.y-ext.y, cen.z-ext.z)),
            WorldToGUIPoint(new Vector3(cen.x+ext.x, cen.y-ext.y, cen.z-ext.z)),
            WorldToGUIPoint(new Vector3(cen.x-ext.x, cen.y-ext.y, cen.z+ext.z)),
            WorldToGUIPoint(new Vector3(cen.x+ext.x, cen.y-ext.y, cen.z+ext.z)),
            WorldToGUIPoint(new Vector3(cen.x-ext.x, cen.y+ext.y, cen.z-ext.z)),
            WorldToGUIPoint(new Vector3(cen.x+ext.x, cen.y+ext.y, cen.z-ext.z)),
            WorldToGUIPoint(new Vector3(cen.x-ext.x, cen.y+ext.y, cen.z+ext.z)),
            WorldToGUIPoint(new Vector3(cen.x+ext.x, cen.y+ext.y, cen.z+ext.z))
    };
    Vector2 min = extentPoints[0];
    Vector2 max = extentPoints[0];
    foreach (Vector2 v in extentPoints)
    {
        min = Vector2.Min(min, v);
        max = Vector2.Max(max, v);
    }
    return new Rect(min.x, min.y, max.x - min.x, max.y - min.y);
}

public static Rect GUI2dRectWithObject(GameObject go)
{
    Vector3[] vertices = go.GetComponent<MeshFilter>().mesh.vertices;

    float x1 = float.MaxValue, y1 = float.MaxValue, x2 = 0.0f, y2 = 0.0f;

    foreach (Vector3 vert in vertices)
    {
        Vector2 tmp = WorldToGUIPoint(go.transform.TransformPoint(vert));

        if (tmp.x < x1) x1 = tmp.x;
        if (tmp.x > x2) x2 = tmp.x;
        if (tmp.y < y1) y1 = tmp.y;
        if (tmp.y > y2) y2 = tmp.y;
    }

    Rect bbox = new Rect(x1, y1, x2 - x1, y2 - y1);
    Debug.Log(bbox);
    return bbox;
}

public static Vector2 WorldToGUIPoint(Vector3 world)
{
    Vector2 screenPoint = Camera.main.WorldToScreenPoint(world);
    screenPoint.y = (float)Screen.height - screenPoint.y;
    return screenPoint;
}

Reference: Is there an easy way to get on-screen render size (bounds)?

Vendue

Answer 3

refer to this

It needs the game object with skinnedMeshRenderer.

Camera camera = GetComponent(); 
SkinnedMeshRenderer skinnedMeshRenderer = target.GetComponent(); 
// Get the real time vertices 
Mesh mesh = new Mesh(); 
skinnedMeshRenderer.BakeMesh(mesh); 
Vector3[] vertices = mesh.vertices; 
for (int i = 0; i < vertices.Length; i++) 
{ 
    // World space 
    vertices[i] = target.transform.TransformPoint(vertices[i]); 
    // GUI space 
    vertices[i] = camera.WorldToScreenPoint(vertices[i]); 
    vertices[i].y = Screen.height - vertices[i].y; 
} 
Vector3 min = vertices[0]; 
Vector3 max = vertices[0]; 
for (int i = 1; i < vertices.Length; i++) 
{ 
    min = Vector3.Min(min, vertices[i]); 
    max = Vector3.Max(max, vertices[i]); 
} 
Destroy(mesh); 
// Construct a rect of the min and max positions 
Rect r = Rect.MinMaxRect(min.x, min.y, max.x, max.y); 
GUI.Box(r, "");

Bellman

hi Haoyu! It's relatively easy to do it manually ourselves. As it says in the question, "Note that to do it manually you just make rays for the 8 points of the AABB; encapsulate those in 2D to make the orange box." The question here is whetyher the internal, gpu, process that does this, is, exposed. – Darbee

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Most