How often does python-requests perform dns queries

Question

We are using Locust to for load testing rest api services behind elastic load balancing. I came across this article regarding load balancing and auto scaling, which is something we are testing.

Locust is using python-requests which is using urllib3, so my question is if python-requests does a dns query for every connect, and if not, is it configurable?

Scrannel

Urllib3 is probably using socket.getaddrinfo which should be using the getaddrinfo of the OS you are using according to this answer on another SO question. So it should cache the results depending on the OS for each subsequent request to the same hostname. – Wreckfish

And according to this question: https://mcmap.net/q/152752/-dns-caching-in-linux-closed caching is disabled on most Linux configurations. – Scrannel

Answer 1

Locust is using python requests that is using urllib3 that is using socket.getaddrinfo which has DNS caching disabled according to this SO thread (given that your test machine runs linux).

Scrannel

Answer 2

python-requests does a dns query for every connect.

To disable this you can use a dns cache.

Now you can enable systemd-resolved with systemctl enable systemd-resolved

more info - https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html

Centroid

Answer 3

Yes, the Python requests lib perform a DNS query at every request;

However, you can improve that behavior using requests-cache.

requests-cache is a persistent HTTP cache that provides an easy way to get better performance with the python requests library.

It's really easy to use, 1 minute, 2 lines of code and you're ready to go.

import requests
import requests_cache

requests_cache.install_cache('my_simple_cache')

Your subsequent python requests calls should automagically use the cache now. And there are more fine-grained options available if you want, like customizing expire time, etc.

(This solved a problem I was having trying to batch process something that suddenly stopped working after 10K calls, not because the service was unavailable, but because the DNS requests would get refused at my intranet DNS server.)

Velour

After various attempts with systemd-resolved, this helps reduce the DNS calls substantially. requests_cache.install_cache(backend='memory', expire_after=600) – Deviationism

AFAICT, requests-cache is a cache for the payload, not a DNS cache. – Corroborate

@Corroborate Well, then now you know more! It does provide hostname-to-ip dns caching as well. Like I described in my answer, and like sp1111 mentioned in his comment. – Marine