Using MediaPipe Hand Tracking with Unity

Question

We are working on a project for a Mobile (Android) Mixed-Reality (MR) game using only hand gestures for interacting with the virtual objects. We've researched for frameworks and api's that we could use for the project. Currently, we think the best way is to use Google Media Pipe for Multihand tracking and Vuforia for creating the MR environment. But since Vuforia is used with Unity and MediaPipe does not have a Unity port yet, we are not sure about how we are going to combine all of these.

Do you have any recommendations for other frameworks we could use for this purpose or could you give any ideas on how to proceed?

Levelheaded

Did you make any progress on this yet? We're looking for a very similar use case here. We're C++ developers - no real experience with Unity. Our approach would involve running MediaPipe in a separate process and forwarding the hand skeleton data to Unity. We have successfully used this technique in a previous project - just not hand tracking data. I'd be happy to help where I can - Would be awesome if somebody with unity experience could help on the other side. – Abixah

@JoelBodenmann Thanks for your interest in my post. We are senior university students trying to implement mediapipe into a unity project as a senior project for a mobile application. However, we could not make any progress trying to do what you have done in an your project. We have three developers in our team with experience working with unity for up to two years, a year of those being on site. I am not sure if we are experienced enough to solve your problems, but we will try our best if you would like us to help. Looking to hear back from you. – Levelheaded

Answer 1

In case someone is still interested in the problem, there's the MediaPipeUnityPlugin which you could use to directly connect Unity and MediaPipe.

Bombast