Render website as texture in Unity3D on Android

Question

Quick Summary:

I'm working on a VR application, and we want to render a website in 3D space in our VR scene, preferably on a texture. There are many ways to accomplish this on desktop (PC/Mac), but not on Android devices.

Details:

We have a working prototype on Windows that relies on Zen Fulcrum's Embedded Browser plugin, but we want to support Android devices such Google DayDream, Gear VR, and eventually the Android-based standalone headsets that will be released within the next year.

Here's a list of all the Unity webpage rendering solutions I've found so far:

• Embedded Browser by Zen Fulcrum - supports Windows and OSX
• UniWebView by Yumigi - They do not support rendering webpages to textures. Webpages are rendered on a flat layer on top of the graphics engine's rendering.
• In-App Web Browser by Piotr Zmudzinski - Appears to have the same limitation
• Webkit for iOS by Chestnut Games - Only works for iOS, their company website doesn't seem to work.
• HTML Engine for NGUI & Unity GUI by ZHing - Doesn't appear to be a real web browser. It just seems like someone made some scripts that convert HTML into UI elements.
• EasyWebViewTexture For Android by JaeYunLee - I know some devs who used this but the project was mysteriously taken down and discontinued.
• Awesomium for Unity (Beware: dead/suspicious link) by Khrona Software - Their wiki articles for Unity are mysteriously blank both on their wiki site and on their GitHub. The entire awesomium website has been shut down as well, and so has the Krhona Software company website. They have a GitHub Repo for their Unity integration but their readme says it only supports Windows & Mac.
• uWebKit3 by Mythos Labs - No longer available. Their original website no longer exists. Previous versions of uWebKit claimed to support Android. Their announcement on the Unity forums about them closing down was very mysterious and abrupt. I found something on GitHub that claims to be uWebKit3, but the readme claims to only support PC and Mac.
• Unreal Engine 4's Web Browser Widget - This is an experimental widget that's being built into UE4, and on desktop it renders webpages in the 3D world with little problems. Unfortunately, even though it claims to support Android, when you actually deploy to a device you'll find that the browser just gets rendered flat on top of the game engine rendering, not in the 3D world.

I really can't find any info on why this is no longer supported on Android when it used to be supported. Maybe something changed on the Android stack? If I was to try messing with Chromium myself and get it to work on Android, would I just run into whatever dead-ends killed all these projects?

The Android SDK offers the native WebView component, which is rendered as an independent on-screen element that we can't really hook into. Google just released a preview for the Chrome VR browser in Daydream, but it's very early in development and I really don't think they plan to provide a solution for VR devs to use anytime soon. Oculus has the experimental Carmel Browser but it seems more focused on rendering WebVR than providing tools for VR devs to hook into. I've been in touch with someone from the Oculus Web Browser team (met him at PAX, lol), and they plan to release a feature to make it easier for devs to launch web pages that open in Gear VR's built-in Oculus Web Browser. But that's an app-switching scenario, it doesn't let you render a webpage in your own 3D scene.

One possibility that I'm considering exploring: what if we got a server to render webpages for users and stream that content back to their devices as texture data? It'd kinda be like what OnLive did with videogames, except you could tolerate more latency at times. It could use Selenium (or some other webdev visual regression testing tool) to handle the rendering... Eh, it sounds like a total pain to make, though, not sure if our company can afford to spend months on something like that. -_-

Any suggestions? Thanks!

Petuu

Are you sure "Awesomium for Unity" is only for win/mac? I couldn't find such statement in read.me which says so. – Silin

That's a good point. I just saw how the ReadMe kept talking about PC & Mac and didn't mention other platforms, so I assumed it was PC & Mac only. There's also a line about how the .dll for PC & Mac were included in the /bin/ folder and that sounded like no mobile support to me. Maybe I should try it and investigate further. – Petuu

I believe it was talking about editor platforms (Not runtime platforms) as you would use unity editor only on Mac/Windows. – Silin

Did you ever resolve this? – Pete

I haven't. My client decided to pivot away from this problem. – Petuu

Answer 1

I looked around for webview plugins that support video and the Unity 3D webview plugin for Android is the only one I could find that does. I've been using it for a while now and would recommend it. Here's an example of using it:

using UnityEngine;
using Vuplex.WebView;

class SceneController : MonoBehaviour {

    // I set this in the editor to reference a webview in the scene
    public WebViewPrefab webPrefab;

    void Start() {
        webPrefab.Init(1.5f, 1.0f);
        webPrefab.Initialized += (sender, args) => {
            // load the video so the user can interact with it
            webPrefab.WebView.LoadUrl("https://www.youtube.com/watch?v=dQw4w9WgXcQ");
        };
    }
}

Graeme

Answer 2

UPDATE: I updated the repo to support video and is a fully functioning 3D browser based on the GeckoView browser engine. It relies on the OVROverlay from Oculus to render frames generated in an Android plugin onto a Unity3D texture.

I'm not sure if this question is a duplicate or the newer one is, but this is a repo I made in the hopes we can implement a nice in-game browser. It's a bit buggy/slow but it works (most of the time).

It uses a Java plugin that renders an Android WebView to a Bitmap by overriding the view's Draw method, converts that to a png and passes it to a unity RawImage. There is plenty of work to do so feel free to improve it!

How to use it:

At the repo you can find the plugin (unitylibrary-debug.aar) you need to import to Assets/Plugins/Android/ and BrowserView.cs and UnityThread.cs which you can use to convert an Android WebView to a texture that Unity's RawImage can display. Fill BrowserView.cs's public fields appropriately. Make sure your API level is set to 25 in Unity's player settings.

Code samples

Here's overriding the WebView's Draw method to create the bitmap and PNG, and init-ing the variables you need:

public class BitmapWebView extends WebView{

    private void init(){
        stream = new ByteArrayOutputStream();
        array = new ReadData(new byte[]{});
        bm = Bitmap.createBitmap(outputWindowWidth,
             outputWindowHeight, Bitmap.Config.ARGB_8888);
        bmCanvas = new Canvas(bm);
   }


    @Override
    public void draw( Canvas ){
        // draw onto a new canvas  
        super.draw(bmCanvas);
        bm.compress(Bitmap.CompressFormat.PNG, 100, stream);

        array.Buffer = stream.toByteArray();
        UnityBitmapCallback.onFrameUpdate(array,
            bm.getWidth(),
            bm.getHeight(),
            canGoBack,
            canGoForward );
        stream.reset();

    }
}  
// you need this class to communicate properly with unity
public class ReadData {
    public byte[] Buffer;
    public ReadData(byte[] buffer) {
        Buffer=buffer;
    }
}

Then we pass the png to a unity RawImage. Here's the Unity receiving side:

// class used for the callback with the texture
class AndroidBitmapPluginCallback : AndroidJavaProxy
{
    public AndroidBitmapPluginCallback() : base("com.unityexport.ian.unitylibrary.PluginInterfaceBitmap") { }
    public BrowserView BrowserView;

    public void onFrameUpdate(AndroidJavaObject jo, int width, int height, bool canGoBack, bool canGoForward)
        {
        AndroidJavaObject bufferObject = jo.Get<AndroidJavaObject>("Buffer");
        byte[] bytes = AndroidJNIHelper.ConvertFromJNIArray<byte[]>(bufferObject.GetRawObject());
        if (bytes == null)
            return;
        if (BrowserView != null)
        {
            UnityThread.executeInUpdate(()=> BrowserView.SetTexture(bytes,width,height,canGoBack,canGoForward));
        }
        else
            Debug.Log("TestAndroidPlugin is not set");

    }
 }


public class BrowserView : MonoBehaviour {
// Browser view needs a RawImage component to display webpages


   void Start () {
        _imageTexture2D = new Texture2D(Screen.width, Screen.height, TextureFormat.ARGB32, false);
        _rawImage = gameObject.GetComponent<RawImage>();
        _rawImage.texture = _imageTexture2D;

        #if !UNITY_EDITOR && UNITY_ANDROID
        // Get your Java class and create a new instance
        var tempAjc = new AndroidJavaClass("YOUR_LIBRARY.YOUR_CLASS")
        _ajc = tempAjc.CallStatic<AndroidJavaObject>("CreateInstance"); 
        // send the callback object to java to get frame updates
        AndroidBitmapPluginCallback androidPluginCallback = new AndroidBitmapPluginCallback {BrowserView = this};
        _ajc.Call("SetUnityBitmapCallback", androidPluginCallback);
        #endif

    }
   // Android callback to change our browser view texture
    public void SetTexture( byte[] bytes, int width, int height, bool canGoBack, bool canGoForward)
    {

        if (width != _imageTexture2D.width || height != _imageTexture2D.height)
            _imageTexture2D = new Texture2D(width, height, TextureFormat.ARGB32, false);

        _imageTexture2D.LoadImage(bytes);
        _imageTexture2D.Apply();
        _rawImage.texture = _imageTexture2D;
    }
}

Martyrdom

Generally, links to a tool or library should be accompanied by usage notes, a specific explanation of how the linked resource is applicable to the problem, and some sample code. – Durning

Awesome Repo! @palebone. Quick question; is it possible to use this without OVROverlay? – Twerp

As well, I am somehow unable to use "extends" in my C# script. and how to i include the "WebView" ? – Twerp

Only works with the OVROverlay unfortunately. Not sure about your second question, maybe open an issue with more detail on the repo? – Martyrdom

Why convert to a PNG first? – Wetterhorn

@Wetterhorn check out the update, the old post is no longer really relevant. – Martyrdom