I’m starting work on a 360 VR video player, aimed at mobile devices (iOS/Android Cardboard + GearVR) and I’m evaluating whether it is possible to get video quality from 1080p to 4k in Unity or if I should instead just do the project natively (aka, OpenGL).
I’ve been trying to find information on the state of video in Unity but I haven’t arrived to a clear answer. I’m going to explain the info I’m running on as of now, hopping that someone can add to this or provide an answer or at least show that I am erring on the side of ignorance 
Assume a video in mp4, H264 at 4K resolution (equirectangular mapping):
OpenGL Approach:
If I were to do this in OpenGL, a (somewhat simplified) workflow would entail decoding each video frame, updating a GL_TEXTURE_2D with the pixel contents, and rendering to a sphere with glCullFace(GL_FRONT) as if it where a regular skybox.
With the above setup, the limiting factors would be:
-
Does the GPU in my device support 4K textures? what marketing calls 4k is usually 3840 horizontal, and, an iPhone6 for instance, returns 4096 for
glGetIntegerv(GL_MAX_TEXTURE_SIZE,...). So I’d assume the iPhone6 can load textures as big as required for 4K. -
Fill rate: i.e.: my iPhone6’s GPU might be capable of storing 4K Textures, but it might not be fast enough to fill up a 4K frame buffer at an acceptable frame rate. This unfortunately I think the only way to know is by testing. There is also the added overhead of the Cardboard rendering which means 2 passes per frame.
Unity Approach:
Now for a Unity approach, this is what I did:
- Empty scene with a sphere and cardboard camera at the origin.
- For the sphere, I created a material based on the Unlit shader with one texture and set the cull face to front.
- imported the video and assigned it as the material’s texture.
My observations from the above test are:
UPDATE: I now realize that my test was using Unity’s Move Texture, which is not available on mobile. So if I still pursuit this project in Unity I will anyway have to either do my own native Unity plugin and write the OpenGL code to map the textures to the geometry or purchase an asset like AVPRO
- When importing images, Unity also allows texture sizes of 4096 so 4K should be possible.
- Importing a video however, takes a ridiculous amount of time. I believe this is because Unity actually uses Ogg Theora and so it takes forever to transcode to Ogg. As a matter of fact, sometimes I was able to import and sometimes Unity would just stall.
- When importing a video, I was not given any mmap, filtering or size options as if I was importing an image. So I’m not really sure what Unity is doing to the video. All I get is a quality / compression slider. The end video is still 3840x1920, which was the original resolution.
- The rest of the experiment works ok, as in, yes, the video playbacks in the sphere. But the framerate is not great.
So at this point my conclusions are:
- Is it possible to do 4K in Unity? Kind of, In the sense that you can import 4K videos and reproduce them without any external assets.
- Is it really 4K quality? Probably not. mp4 is lossy and so is ogg, So I assume my video went a second round of lossy compression when it was transcoded to Ogg.
- Is it practical? Not sure. So far, video importing takes an impractical amount of time. Also, 4K in general might be too much for a mobile device rending in stereo, regardless of Unity or doing it straight in OpenGL.
As a side note and to complicate things even more, I was thinking of allowing both downloading and streaming video in the app. But from the above it seems that would be out of the question performance wise. Maybe I have to stick to 1080p or even 720p?
I was currently trying out the demo for AVPro, substantially more expensive than Easy Movie Texture. So you are saying Easy Movie is enough to play 4K? Also, yes, in the meantime I discovered that if I import to the streaming folder I skip the processing time since the files are left intact, like you said. And do you think it might be worth exploring me just doing a native plugin instead? Have you tried this approach? The potential would lie on me making sure only the necessary GL commands are issued for just this use case, but it might just not be significantly more performant. Thanks!
– LeucopoiesisDoes easy movie texture work for pc releases too? All I find are android examples but I'd like to play a 4k Video for the HTC VIVE.
– EmotionHi @Suttles i am using Easy-Movie-Texture. But even if I play 4k video (for VR project), it seems of less quality. Do you have any idea why is it so? The video is playing in the 4k supported devices, but it doesn't have the quality.
– Radiation