Trying to get mouse location in world while in scene, HandleUtility.GUIPointToWorldRay not helpful

Question

I want to make a tile editor so I need to get where my cursor is pointing at in the world in the scene window. This is what I tried:

public class MyEditor : EditorWindow
{
    static void TilemapEditor()
    {
        EditorWindow.GetWindow( typeof(RoomEditor) );
    }
    public void OnInspectorUpdate()
    {
        Repaint();
    }
    void OnGUI()
    {
        //other stuff here, not relevant to my issue
        Debug.Log( HandleUtility.GUIPointToWorldRay(Event.current.mousePosition) );
    }
}

HandleUtility.GUIPointToWorldRay simply doesn’t work, console error being "Unable to convert GUI point to world ray if a camera has not been set up!". Any way to fix that or a different way of doing what I want?

Steddman

Same problem here...

– Metrical

Answer 1

The problem here is (afaik) that during OnGUI you don’t have an “active camera”, as EditorWindow’s aren’t at all connected with the SceneView.
You might be able to call

Vector3 mousePos = Event.current.mousePosition;

The mouse position must be flipped into ScreenSpace because we only have access to the ScreenPointToRay-function, which is similar to the GUIPointToWorldRay, except it takes use of screenspace:

mousePos.y = SceneView.lastActiveSceneView.camera.pixelHeight - mousePos.y;

Then we can create the ray:

Ray ray = SceneView.lastActiveSceneView.camera.ScreenPointToRay(mousePos);

if you however want to incorporate more GUI/Handles into the sceneview itself for the EditorWindow, I reccoment to do this:

public class SceneInteractingWindow : EditorWindow
{    
   private void OnEnable()
   {
      //register that OnSceneGUI of this window 
      //should be called when drawing the scene.
      SceneView.onSceneGUIDelegate += OnSceneGUI;
   }
   private void OnDisable()
   {
      //cleanup: when the window is gone
      //we don't want to try and call it's function!
      SceneView.onSceneGUIDelegate -= OnSceneGUI;
   }
   private void OnSceneGUI(SceneView sv)
   {
      //DO SceneGUI stuff here...
      //for example something like this:
      Vector3 mousePos = Event.current.mousePosition;
      mousePos.y = sv.camera.pixelHeight - mousePos.y;
      Ray ray = sv.camera.ScreenPointToRay(mousePos);
      RaycastHit hit;
      if (Physics.Raycast(brushRay, out hit))
      {
         Handles.color = Color.red;
         Handles.DrawWireDisc(hit.point,hit.normal, 5);
         sv.Repaint();
      }
   }
}

Retake

Answer 2

The Handles are mainly meant for the usage within the SceneView so those methods are usually used inside any OnSceneGUI callbacks. If you want to work with handles in your own editor window you have to use Handles.SetCamera to provide a reference camera view.

GUIPointToWorldRay simply does this:

public static Ray GUIPointToWorldRay(Vector2 position)
{
	Ray result;
	if (!Camera.current)
	{
		Debug.LogError("Unable to convert GUI point to world ray if a camera has not been set up!");
		result = new Ray(Vector3.zero, Vector3.forward);
	}
	else
	{
		Vector2 v = HandleUtility.GUIPointToScreenPixelCoordinate(position);
		Camera current = Camera.current;
		result = current.ScreenPointToRay(v);
	}
	return result;
}

Keep in mind that the resulting Ray is not a “position” but just a ray in worldspace. Like “stektpotet” said if you want to actually work in the sceneview you want to execute your code in the sceneview callback where the sceneview camera is the active one by default. Keep in mind that there could be more than one sceneview open at the same time. Inside the SceneView gui function you can use

HandleUtility.GUIPointToWorldRay(Event.current.mousePosition)

just fine.

Ildaile