Non-blocking loading and copying of large Texture2D's in C# for Unity
Asked Answered
R

3

15

I'm building a Unity app for Android which deals with loading a lot of large textures dynamically (all images are over 6MB in size as png's). These textures can either come from an Amazon S3 server, in which case they arrive as a stream, or from the user's device itself.

In both cases I'm able to get hold of the raw data or texture asynchronously without a problem. In the first I query the server and get a callback with the stream of data, and in the second I use the WWW class to get hold of the texture making use of the "file://" protocol.

The problem happens as soon as I want to copy this data into a Texture2D to some place I can make use of, such as onto a Texture2D private member.

With the stream I convert it into a byte[] and try calling LoadImage(), and with the WWW class I simply try copying it with myTexture = www.texture. Both times I get a massive frame out as the texture is loaded or copied. I want to eradicate this frame out because the App is simply un-shippable with it.

using (var stream = responseStream)
{
   byte[] myBinary = ToByteArray(stream);
   m_myTexture.LoadImage(myBinary);  // Commenting this line removes frame out
}

...

WWW www = new WWW("file://" + filePath);
yield return www;
m_myTexture = www.texture;  // Commenting this line removes frame out

Unfortunately Unity doesn't seem to like running these operations on a separate thread from the main thread and throws an exception when I try.

Is there any way to perhaps chunk up these operations so that it takes multiple frames? Or do some sort of fast memcopy operation that won't stall the main thread?

Thanks in advance!

PS: I've created a working example of the problem in the following repo: https://github.com/NeoSouldier/Texture2DTest/

Ramtil answered 6/11, 2016 at 15:10 Comment(1)
At what point are you doing this? Could you hide it behind a loading screen or intro?Massicot
R
4

Eventually this problem was solved by creating a C++ Plugin (built through Android Studio 2.2) that makes use of "stb_image.h" for loading the image, and OpenGL to generate textures and map a set of scanlines onto the texture over multiple frames. The texture is then handed over to Unity through Texture2D.CreateExternalTexture().

This method does not make the work asynchronous but spreads the loading cost over multiple frames removing the synchronous block and subsequent frame out.

I wasn't able to make the texture creation asynchronous because in order for the OpenGL functions to work you are required to be running the code from Unity's main Render Thread, so functions must be called through GL.IssuePluginEvent() - Unity's docs use the following project to explain how to make use of this functionality: https://bitbucket.org/Unity-Technologies/graphicsdemos/

I've cleaned up the test repo I was working on and written instructions in the README to make it as easy as possible to understand the final solution I came to. I hope that it will be of use to someone at some point and that they won't have to spend as long as I've done to solve this problem! https://github.com/NeoSouldier/Texture2DTest/

Ramtil answered 15/1, 2017 at 21:29 Comment(0)
H
14

The www.texture is known to cause hiccups when large Texture is downloaded.

Things you should try:

1.Use the WWW's LoadImageIntoTexture function which replaces the contents of an existing Texture2D with an image from the downloaded data. Keep reading if problem is still not solved.

WWW www = new WWW("file://" + filePath);
yield return www;
///////m_myTexture = www.texture;  // Commenting this line removes frame out
www.LoadImageIntoTexture(m_myTexture);

2.Use the www.textureNonReadable variable

Using www.textureNonReadable instead of www.texture can also speed up your loading time. I'be seen instances of this happening from time to time.

3.Use the function Graphics.CopyTexture to copy from one Texture to another. This should be fast. Continue reading if problem is still not solved.

//Create new Empty texture with size that matches source info
m_myTexture = new Texture2D(www.texture.width, www.texture.height, www.texture.format, false);
Graphics.CopyTexture(www.texture, m_myTexture);

4.Use Unity's UnityWebRequest API. This replaced the WWW class. You must have Unity 5.2 and above in order to use this. It has GetTexture function that is optimized for downloading textures.

using (UnityWebRequest www = UnityWebRequest.GetTexture("http://www.my-server.com/image.png"))
{
    yield return www.Send();
    if (www.isError)
    {
        Debug.Log(www.error);
    }
    else
    {
        m_myTexture = DownloadHandlerTexture.GetContent(www);
    }
}

If the three options above did not solve the freezing problem, another solution is copying the pixels one by one in a coroutine function with the GetPixel and SetPixel functions. You add a counter and set when you want it to wait. It spaced the Texture copying over time.

5.Copy Texture2D pixels one by one with the GetPixel and SetPixel functions. The example code includes 8K texture from Nasa for testing purposes. It won't block while copying the Texture. If it does, decrease the value of the LOOP_TO_WAIT variable in the copyTextureAsync function. You also have option to provide a function that will be called when this is done copying the Texture.

public Texture2D m_myTexture;

void Start()
{
    //Application.runInBackground = true;
    StartCoroutine(downloadTexture());
}

IEnumerator downloadTexture()
{
    //http://visibleearth.nasa.gov/view.php?id=79793
    //http://eoimages.gsfc.nasa.gov/images/imagerecords/79000/79793/city_lights_africa_8k.jpg

    string url = "http://eoimages.gsfc.nasa.gov/images/imagerecords/79000/79793/city_lights_africa_8k.jpg";
    //WWW www = new WWW("file://" + filePath);
    WWW www = new WWW(url);
    yield return www;

    //m_myTexture = www.texture;  // Commenting this line removes frame out

    Debug.Log("Downloaded Texture. Now copying it");

    //Copy Texture to m_myTexture WITHOUT callback function
    //StartCoroutine(copyTextureAsync(www.texture));

    //Copy Texture to m_myTexture WITH callback function
    StartCoroutine(copyTextureAsync(www.texture, false, finishedCopying));
}


IEnumerator copyTextureAsync(Texture2D source, bool useMipMap = false, System.Action callBack = null)
{

    const int LOOP_TO_WAIT = 400000; //Waits every 400,000 loop, Reduce this if still freezing
    int loopCounter = 0;

    int heightSize = source.height;
    int widthSize = source.width;

    //Create new Empty texture with size that matches source info
    m_myTexture = new Texture2D(widthSize, heightSize, source.format, useMipMap);

    for (int y = 0; y < heightSize; y++)
    {
        for (int x = 0; x < widthSize; x++)
        {
            //Get color/pixel at x,y pixel from source Texture
            Color tempSourceColor = source.GetPixel(x, y);

            //Set color/pixel at x,y pixel to destintaion Texture
            m_myTexture.SetPixel(x, y, tempSourceColor);

            loopCounter++;

            if (loopCounter % LOOP_TO_WAIT == 0)
            {
                //Debug.Log("Copying");
                yield return null; //Wait after every LOOP_TO_WAIT 
            }
        }
    }
    //Apply changes to the Texture
    m_myTexture.Apply();

    //Let our optional callback function know that we've done copying Texture
    if (callBack != null)
    {
        callBack.Invoke();
    }
}

void finishedCopying()
{
    Debug.Log("Finished Copying Texture");
    //Do something else
}
Haulm answered 7/11, 2016 at 4:44 Comment(28)
Wow, this is an incredible answer @Programmer! Thanks a lot for your help with getting me to ask the correct question, and with your well thought out answer. So unfortunately I tried the first 4 methods but had no luck. I'm trying the 5th method, which seems very simple, clean and promising, but I'm running into a weird issue. Simply iterating and calling www.texture.GetPixel() is causing the App to crash after only a few iterations (no more than 8, if I stick some Debug.Log()'s it crashes sooner) - I've commented SetPixel() to avoid confusion =/Ramtil
I'm able to stop the app from crashing if I either comment out www.texture.GetPixel() (which obviously is not an actual solution), or if instead of calling www.texture.GetPixel(), I try doing a copy before like: "tempTexture = www.texture; ... tempTexture.GetPixel();" - but this is obviously not a solution either because it re-introduces the original "myTexture = www.texture;" problem. So I believe somehow the combination of calling GetPixel() on www.texture on Android may somehow be the problem here, although I have no idea why =/. Did you try running your code on Android? Thanks a lot btw!Ramtil
Oh and also, when I say "www.texture.GetPixel()" instead of "source.GetPixel()", its because I've joined your downloadTexture() and copyTextureAsync() functions all into one coroutine rather than splitting them into two. I did also try splitting them in two as in your example, just in case, but it led to the same crash. Thanks!Ramtil
I will make a comment but I want you to start replying as soon as possible. You usually reply a day after. Hold on for a sec.Haulm
The #5 code I provided is working %100 on my side. It's more readable if you leave it in 2 separate code like I did. Please test it just like it is in my answer. Don't change anything and say it doesn't work. There is no place in that code that I did tempTexture = www.texture; so please don't add that. The GetPixel(); and GetPixel(); replaces that. After copying the pixels one by one, m_myTexture.Apply(); is called to apply the changes to the destination Texture. You do not need tempTexture = www.texture; as this replaces it.Haulm
Try it on the Editor just like it is then try it on your Android without modifying it. The only thing you should change is the url. Other than that, just it like it is then let me know what happens in the Editor and Android. Also make a standalone build for PC and let me know the outcome. I expect it to work. It should because it has been working for me.Haulm
Ok! So I can report back that with a http request it does indeed work, both on PC and on Android! Seems like your solution works perfectly until I try using grabbing the file from the device itself, in other words when I use 'WWW www = new WWW("file://" + filePath)'. This odd crash only happens when the file comes from the device itself. I guess that's a bug in Unity then, wouldn't you agree?Ramtil
Apologies for not replying earlier, my laptop ran out of battery and I had to head home.Ramtil
I wouldn't jump into bug yet. If it fails when you try to read it from the device, what is the value filePath? Where did you place the image you are trying the read from the device and what is the file size of the image in MB?Haulm
The filepath is something like this "/storage/emulated/0/DCIM/Gear 360/SAM_100_0096.jpg" - this picture in particular was taken from a 360 camera, and it doesn't crash through the original method. It has a size of approximately 6MB... I've been testing a few things and in the end I was able to get your 2 function method to work without crashing. However, there was still a frame out when the second coroutine got called, I assume this is because a copy operation happens behind the scenes to pass the Texture2D into the second function.Ramtil
To remove that frame out I once again tried embedding copyTextureAsync() into downloadTexture() but it went back to crashing immediately after a few calls into GetPixel(), even with a http request. It feels like the variable www.texture doesn't like having GetPixel() called on it, you need to copy it into another Texture2D before calling GetPixel() works. Don't feel pressured to reply immediately @Programmer, take your time. I really appreciate the help! Thanks!Ramtil
In fact I think I know the problem, its probably due to this in the Unity manual " Each invocation of texture property allocates a new Texture2D ", so my loop is allocating new textures every iteration =O hence why it crashes. And with the two function method, the large allocation on copying into the copyTextureAsync() function must be what's making it frame out...Ramtil
In other words, the simple act of checking the www.texture variable causes a frame out (if the texture is big enough I guess)! What baffles me is why Unity doesn't just allow this to be run on a separate thread...Ramtil
Hi @Programmer, I still haven't been able to solve this problem, would you have any other ideas on what I might do? ThanksRamtil
I thought you fixed this. I don't know where to begin to naswer fro after reading your comments. That function has been tested on my side and is working 100%. I think that the only way to help you is if you create a simple project and try to replicate this problem there. After that, upload the project to another website and provide the link. I will take a look a at this and see what's going on. Without access to the project, we will both be wasting out time here.Haulm
Hey @Programmer, that sounds like a good idea. I'll make a small project for you in the next couple of days to replicate the problem and put it on my github account! Once I've done it I'll let you know!Ramtil
Ok. Take your time.Haulm
Done! I've edited my question and added the link to my repo as a "PS" at the bottom! Its an incredibly paired back version of the full code and app that I'm building so there might be some unrequired complexity in the code or scene. Check it out and please ask me any questions if anything is unclear. Thanks!Ramtil
Ok. Please describe to me what exactly the problem is and how to replicate it. I will do it tomorrow.Haulm
@Programer the README file should explain it. If anything's unclear, please let me know!Ramtil
Hey @Programmer, have you had any chance to look at that github repo link that I posted? Thanks in advance, ArthurRamtil
I did and went through it carefully and I suggest you file a bug report with Unity about Texture2D.LoadImage freezing. It is a bug. You can also request a sync version of this functionHaulm
Awesome, thanks for your time @Programmer! Yeah, I don't see why there's no async way of performing this operation. I'll let you know what they say!Ramtil
May I jump in ans ask what is the purpose of #5? The first method has the texture, passes it to a second method to copy it in small chunks. Ok b ut you already have the full texture already why would copy it again? Isn't it just doing twice the same thing? What am I missing here?Freedman
@Freedman In the old days, accessing www.texture creates new Texture and returns it and that can cause a small freeze as descried by OP in the question because that's not an asyn operation. The idea of that question is to copy the pixels slowly over multiple frames so that the freeze won't happen but after it did not solve OP's problem, I now believe that accessing any property or function of ``www.texture` will cause WWW to still create a new texture so that's still unavoidable. This was before and can't tell if it changed.Haulm
Because this question is still gaining views, I plan on updating it with a complete plugin solution when I have time. The biggest problem is testing plugin on all platforms to make sure that it worksHaulm
Yep coz how I understand, www.texture already returns the texture via the WWW, and that should fix the hickup problem since it is done in async mode already. Copying it by chunk is only making a second iteration. One way that is now available through C#6 would be to use await/async to download the byte array and figure out a way to pass the array to main thread.Freedman
@Freedman No. Downloading the file is not the problem. The problem is Unity automatically creating texture with the downloaded data when www.texture is accessed which I think that process is using LoadImage to do that. This causes the freezing. Read the comment in his code from his question that says "Commenting this line removes frame out"Haulm
R
4

Eventually this problem was solved by creating a C++ Plugin (built through Android Studio 2.2) that makes use of "stb_image.h" for loading the image, and OpenGL to generate textures and map a set of scanlines onto the texture over multiple frames. The texture is then handed over to Unity through Texture2D.CreateExternalTexture().

This method does not make the work asynchronous but spreads the loading cost over multiple frames removing the synchronous block and subsequent frame out.

I wasn't able to make the texture creation asynchronous because in order for the OpenGL functions to work you are required to be running the code from Unity's main Render Thread, so functions must be called through GL.IssuePluginEvent() - Unity's docs use the following project to explain how to make use of this functionality: https://bitbucket.org/Unity-Technologies/graphicsdemos/

I've cleaned up the test repo I was working on and written instructions in the README to make it as easy as possible to understand the final solution I came to. I hope that it will be of use to someone at some point and that they won't have to spend as long as I've done to solve this problem! https://github.com/NeoSouldier/Texture2DTest/

Ramtil answered 15/1, 2017 at 21:29 Comment(0)
P
1

The problem is, Unity will always load the entire image in memory when you create a Texture2D, no matter the method used. This takes time, and there's no way to avoid it. It won't parse the file and get bits of the image data, or load slowly per frame. This happens with any instantiation of anything in Unity, be it images, terrain, objects created by Instantiate(), etc.

If you require only the image data for some processing, I would suggest using a library like libjpeg or libpng (in a C# rendition of it) to get the data in another thread (you can use another thread as long as you don't invoke Unity methods), but if you have to display it, I don't see a way you're going to stop the lag.

Plata answered 6/11, 2016 at 19:9 Comment(8)
Not True, there are ways you can use to avoid lagHaulm
What for example?Plata
Check back on my answer #4. It will let load the texture over time.Haulm
Sorry, can't find, you can link?Plata
Look at the #5 in my answer. It loads the image over time.Haulm
Hi @V.Borodinov, it seems like you might be right about Unity doing the instantiation and loading the Image all in a single frame =/ . Do you know how would might an open-world game dynamically streaming in textures be able to handle this without framing out then...?Ramtil
@Programmer, isn't the _myTexture.Apply() the most expensive operation? You can even set a single pixel every frame, but calling Apply() will lag.Mintamintage
@RuslanL. Apply is expensive but not the most expensive operation. I am not calling it every frame. I am calling it after all the SetPixel loop is done which is fine.Haulm

© 2022 - 2024 — McMap. All rights reserved.