Slow wget speeds when connecting to https pages

S

5

3

I'm using wget to connect to a secure site like this:

wget -nc -i inputFile

where inputeFile consists of URLs like this:

https://clientWebsite.com/TheirPageName.asp?orderValue=1.00&merchantID=36&programmeID=92&ref=foo&Ofaz=0

This page returns a small gif file. For some reason, this is taking around 2.5 minutes. When I paste the same URL into a browser, I get back a response within seconds.

Does anyone have any idea what could be causing this?

The version of wget, by the way, is "GNU Wget 1.9+cvs-stable (Red Hat modified)"

Satang answered 18/9, 2008 at 16:17 Comment(0)

C

9

I know this is a year old but this exact problem plagued us for days. Turns out it was our DNS server but I got around it by disabling IP6 on my box.

You can test it out prior to making the system change by adding "--inet4-only" to the end of the command (w/o quotes).

Crystalcrystalline answered 4/11, 2009 at 19:22 Comment(1)

Thank you very much for this response. My browser was working perfectly, but commands such as wget on the same domains took two minutes to connect. I disabled IPv6 on all my network interfaces, and wget is back to speed as are my scripts engaging in HTTP requests. – Mattoid 15/12, 2022 at 21:23

D

8

Try forging your UserAgent

-U "Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-GB; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1"

Disable Ceritificate Checking ( slow )
```
--no-check-certificate
```
Debug whats happening by enabling verbostity
```
-v
```
Eliminate need for DNS lookups:

Hardcode thier IP address in your HOSTS file
```
/etc/hosts 
123.122.121.120  foo.bar.com
```

Devon answered 18/9, 2008 at 16:20 Comment(3)

In this situation using a hosts file seems like overkill because the poster says a browser loads the url instantly. None the less, if the poster is having problems, it is one more possible solution to try. – Pinprick 6/10, 2008 at 16:40

Browsers have a habit of caching DNS sometimes :) – Devon 7/10, 2008 at 0:4

For my case, it works with --no-check-certificate. Thanks! – Passing 26/10, 2015 at 7:53

F

1

Have you tried profiling the requests using strace/dtrace/truss (depending on your platform)?

There are a wide variety of issues that could be causing this. What version of openssl is being used by wget - there could be an issue there. What OS is this running on (full information would be useful there).

There could be some form of download slowdown being enforced due to the agent ID being passed by wget implemented on the site to reduce the effects of spiders.

Is wget performing full certificate validation? Have you tried using --no-check-certificate?

Firewarden answered 18/9, 2008 at 16:30 Comment(0)

E

0

Is the certificate on the client site valid? You may want to specify --no-certificate-check if it is a self-signed certificate.

HTTPS (SSL/TLS) Options for wget

Efficiency answered 18/9, 2008 at 16:22 Comment(0)

F

0

One effective solution is to delete https:\\. This accelerate my download for around 100 times.

For instance, you wanna download via:

wget https://data.keithito.com/data/speech/LJSpeech-1.1.tar.bz2

You can use the following command alternatively to speed up.

wget data.keithito.com/data/speech/LJSpeech-1.1.tar.bz2

Fecund answered 2/12, 2021 at 6:11 Comment(0)

Recommended topics

Hot tags