403. That’s an error - Rate-limit exceeded That’s all we know - Image from YouTube Data API

Question

I've created a website that uses the YouTube data API to get YouTube data for a chosen user. This has always worked fine for me, but the amount of traffic is now at a level that embedding YouTube images is no longer a 100% solid.

I use the YouTube Data API V3 to get the image URL of YouTube banner images, which are then added as the src attribute for an existing <img> element.

Because the website has pretty high amounts of traffic (mostly 500-3000 realtime users) the YouTube image servers (like this one: https://yt3.ggpht.com) started returning a 403 error when my server sends to many (image) GET requests. The error looks like this:

---

403. That’s an error.

Your client does not have permission to get URL https://yt3.ggpht.com/Xs3rhrNjuGoNnd6aXPdZqdfiOIp5EbvN9-PYSaP8xTY1cx6Ced5No6sHNZupNrQmsidY2W-X=w2560-fcrop64=1,00005a57ffffa5a8-nd-c0xffffffff-rj-k-no from this server. (Client IP address: xx.xx.xxx.xxx)

Rate-limit exceeded That’s all we know.

---

From the moment this limit is exceeded, my website shows no images (from the YouTube image server) for a while and after x minutes when the exceeded limit is reset the images work again. I want to prevent this from happening, because the YouTube bannerimage is important for the UX of my website.

My first question: Does anyone know what the exact numbers are for this rate limit on YouTube images? How many requests can be done in X time?

My second question: Does anyone know a solution to prevent triggering this error and still serve YouTube images to every user?

I have some theories of my own but they would have a lot of impact on my servers:

• Since the rate limit is bound to an IP address, I could run my website on multiple IP's with some kind of load balancer.
• I could save the images to my server when it is requested for the first time as a caching mechanism. When the image would get requested again I could serve it from my own server, which prevents limits from being exceeded. To prevent updated YouTube images from not showing I would have to delete the images every X hours.

I'd rather not do this because both of these theories would have big impact on the server load.

Glynnis

Same problem here. I noticed that SocialBlade loads images in the same way buy without problems. I do not understand why that happens... Any suggestion? – Ethridge

Edit: SocialBlade does really experience the same problem as well – Ethridge

Answer 1

Both your theories would work in practice but as you mentioned, it's a lot of work. You can add the referrerPolicy="no-referrer" to your img tag and it should make the request without a referrer header, making the request not return a 403 anymore.

My assumption is that you, like myself were running on a local server with a popular port and Google/Youtube's image server calculates rate limits based on the referrer header (Although a very poor method, this seems logical based on the HTTP interaction between the image request getting a 200 VS a 403 being decided based on the referer header). This means everyone who is accessing YouTube channel logos using localhost:3000 would be sharing the same rate limit due to sharing the same referer header value and whatever the rate limit is being exceeded. This is only my theory though as it's impossible to tell what YouTube is doing on their backend.

Sundberg

This was exactly my issue when accessing user avatar images as well! It seems like it's wise to add the no-referrer property whenever fetching an image from Google API services. – Connected

I have the same implementation and sometimes hit this error without any idea why. I think your assumption makes sense, although as you said, we can't validate it. Your referrerPolicy attribute immediately worked. I moved one image at a time; those without the policy were still blocked. Those with the policy set showed up. Brilliant! Thank you. – Gentleman

This approach would probably also work for images hosted on Google Drive that are embedded in Jupyter Notebook Markdown cells but only if you add referrerpolicy to allowedAttributes in sanitizer.ts or patch the corresponding JavaScript in site-packages/notebook/static. Haven't been able to verify that though because in my case the error occurs only occasionally. – Reorganization

Answer 2

I performed some experiments to understand better the problem.

It seems like YouTube images servers perform a kind of matching on referer (and probably on IP too) to limit the requests from external environments

As far as I've understood, the problem lies in HTTP referer. Indeed, when you experience a 403 error you can try to modify the HTTP request, changing the referer, and it works like a charm.

From the technical point of view, I think it is not possibile doing that via Javascript, you could set up a proxy to process all the img requests and change the referer as you like. The drawback is that all requests pass through your proxy and come from the same IP.

From the YouTube policy point of view, I do not know if that solution could be fine. Probably not (otherwise why that limitations?).

Ethridge