Flutter Dart Dio Get Request is so slow

Question

I am hosting a space in digital ocean - it is basically Amazon S3 equalivant of digital ocean. My problem with dio is, I am making a get request with dio to a file of 10MB size. The request takes around 9 seconds on my phone but 3 seconds in my browser. I also had this issue in my custom backend. Get requests made with dio (which uses http module of dart) seems to be extremely slow. I need to solve this issue as I need to transfer 50MB of data to user from time to time. Why is dio acting slow on GET requests?

I suspect this might be the underlying cause check here

await Dio().get(
        "Remote_Url_I_can_not_share",
        onReceiveProgress: (int downloaded, int total) {
          listener
              .call((downloaded.toDouble() / total.toDouble() * metadataPerc));
        },
        cancelToken: _cancelToken,
      ).catchError((err) => throw err);

Superscription

Does that happen on both debug/release modes? – Oatmeal

@MiguelRuivo Yes it does – Superscription

@Uni If this was a server problem than why does my browser is always at least 2x, 2.5x faster than dio? – Superscription

flutter clean and flutter channel stable Try this.. – Sibyl

@VinayHP how would flutter channel stable help? Could you elaborate? – Superscription

https://www.flutterclutter.dev/flutter/basics/flutter-channels/2020/860/ Please read the article. – Sibyl

@VinayHP I'm already on stable channel – Superscription

Any Improvements? on Stable Channel.. – Sibyl

@VinayHP nope still the same – Superscription

Not just GET request, I have seen questions with POST and PUT also with similar issue. Just try searching but the common suggestion is to use http instead of Dio.. – Sibyl

@VinayHP i also tried http but its the same thing, slow. Do you think that it would be a good idea to write a plugin to do a get request on native side? – Superscription

Answer 1

I believe that reason for this; buffer size is limited to 8KB somewhere underlying.

I tried whole day to increase it. Still no success. Let me share my experience with that buffer size.

Imagine you're downloading a file which is 16 mb.

Please consider that remote server has also higher speed than your download speed. (I mean just forget about server load, etc.. )

If buffersize:

128 bytes, downloading 16mb file takes : 10.820 seconds

1024 bytes, downloading 16mb file takes : 6.276 seconds

8192 bytes, downloading 16mb file takes : 4.776 seconds

16384 bytes, downloading 16mb file takes : 3.759 seconds

32768 bytes, downloading 16mb file takes : 2.956 seconds

------- After This, If we increase chunk size, download time is also increasing

65536 bytes, downloading 16mb file takes : 4.186 seconds

131072 bytes, downloading 16mb file takes : 5.250 seconds

524288 bytes, downloading 16mb file takes : 7.460 seconds

So somehow, if you can set that buffersize 16k or 32k rather than 8k, I believe download speed will increase.

Please feel free to test your results (I got 3 tries and got average of them for the timings)

package dltest;

import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;

public class DLTest 
{
    public static void main(String[] args) throws Exception 
    {

        String filePath = "http://hcmaslov.d-real.sci-nnov.ru/public/mp3/Metallica/Metallica%20'...And%20Justice%20For%20All'.mp3";

        URL url = new URL(filePath);
        URLConnection uc = url.openConnection();
        InputStream is = uc.getInputStream();

        long start = System.currentTimeMillis();

        int downloaded = 0;
        int total = uc.getContentLength();
        int partialRead = 0;

        // byte chunk[] = new byte[128];
        // byte chunk[] = new byte[1024];
        // byte chunk[] = new byte[4096];
        // byte chunk[] = new byte[8192];
           byte chunk[] = new byte[16384];
        // byte chunk[] = new byte[32768];
        // byte chunk[] = new byte[524288];

        while ( (partialRead = is.read(chunk)) != -1)
        {
                // Print If You Like..
        }

        is.close();
        long end = System.currentTimeMillis();

        System.out.println("Chunk Size ["+(chunk.length)+"] Time To Complete :  "+(end - start));
    }
}

Gaily

I believe this is the case too. Dart developers should allows us to adjust the buffer size. It's very painful to make a request with dart rn. I guess you have to use platform channels and use native requests to ensure full speed – Superscription

Answer 2

My experience with DigitalOcean Spaces has been a very fluctuating one. DO Spaces is, in my opinion, not production ready. I was using their CDN feature for a website, and sometimes the response times would be about 20ms, but sometimes they would exceed 6 seconds. This was in AMS3 datacenter region.

Can you confirm this happens with other S3/servers as well? Such as gstatic, or Amazon CloudFront CDN?

This fluctuating behaviour happened constantly, which is why we transferred all our assets to Amazon S3 + CloudFront. It provides much more consistent results.

It could be that the phone you are testing on uses a very unoptimized traceroute to the DigitalOcean datacenters. That's why you should try different servers.

Abradant

This is very valuable information. Kind regards. I used to have a Strapi backend on DO and I had the same exact issue I mentioned above. Browser downloads in 3 seconds, dio does it around 10. While your opinion is very valuable, I want to be ensured that its not dio that ruins the performance. I will be considering moving to Amazon S3 + CloudFront I used DO because its very cheap. I will try this with s3 and write back – Superscription

Depending on your data needs, the free plan of Amazon AWS is actually very reasonable. You can try creating a bucket + cloudfront in under 15 minutes and test it out. Let me know if this was the issue, I'm very curious. – Abradant

I tested it, with AWS S3. I tried to download a 10MB file. With dio the download took around 20 to 35 seconds. Sometimes it takes longer around 35 seconds mark, I dont know why. I tried this with Android native Volley library and it takes around 25 seconds all the time. Its extremely consistent. I tested it for 15 times and every download took around 25 seconds where as dio fluctuated between 20 to 35, median around 29 seconds. – Superscription

I am not sure about your download speeds, S3/CloudFront setup, etc, but based on your estimates I would say that dio varying between 20-35 with a 29 second median sounds pretty normal if compared with Android Native volley. Again I am not an expert with dio, but this sounds like a better test, considering AWS' + volley's stability. I think this might be a performance trade-off of dio. But I don't know that, really. Don't want to jump to conclusions. Good luck @csguy (my personal opinion: this sounds like a pretty good deal +- 4 seconds avg/med for non-native application). – Abradant

Additionally, @csguy, what browser times did you get with AWS S3? If the browser tests come out to 25 - 35 seconds too, then the 3x amplification (9 seconds vs 3 seconds on your initial issue) was definitely a DigitalOcean issue. – Abradant

I did not check it with the browser sadly, however I might write a plugin to do the download. Dio seems okay but I need the download to be as stable and fast as it can be and native Volley seems super consistent. As for my question I think it was just the digital ocean messing things up :) – Superscription

I actually checked it with browser but I only tried it for like 2 or 3 times. The downloads were around 22 to 25 second as far as I remember. I said I didn't as my data is very vague – Superscription