Is wget or similar programs always available on POSIX systems?

Question

Is there an HTTP client like wget/lynx/GET that is distributed by default in POSIX or *nix operating systems that could be used for maximum portability?

I know most systems have wget or lynx installed, but I seem to remember installing some Ubuntu server systems using default settings and they had neither wget or lynx installed in the base package.

I am writing a shell script for Linux (and probably Mac) to install a piece of software onto the computer. To prevent having to distribute a couple of large files, I would like to fetch these files from the internet instead of packaging in with the installer. Currently, the install script is to be distributed in a single file created from Makeself.

I'd like to avoid having the install script be over 100 MB which it would be if the files were included, and also they may not be required if the person is upgrading or re-installing the software. Maybe the most portable thing to do is include the files in the pacakage.

Right now I am just thinking of having the script check for wget, lynx, and GET, in that order and it will use whichever one it can for downloading, but I could avoid this altogether if there was a way I could download the files that would work on all systems.

EDIT:

Does anyone know much about lwp-request (GET) and its availability? This seems to be readily available on several of the systems I have checked so far, and I remember this always being around 10+ years ago going back to RedHat.

Methodology

You could also use curl or (extreme example) netcat (often aliased as nc). However, I think wget should be the most universally available. If this isn't available, chances are that all the others won't be as well – Insensate

True curl is a possibility too, I figured that one would be less common than wget though. I didn't think about nc, so that could be a possibility too, I know its been around a long time. – Methodology

By the way, you can also use bash itself to make the HTTP request. I guess bash is even more commonly available than wget (although that particular feature might not so commonly be enabled). – Insensate

Cool find on the Bash /dev/tcp thing. If you wanna add an answer about nc and that bash trick, I will probably accept that. I'm starting to think that maybe I am over-analyzing this particular piece and could always just tell them to install one of those packages in the worst case. – Methodology

I've done that, but sarnold's answer is even a bit more informative :) – Insensate

Since you mentioned Mac: OS X comes with curl (and ftp, sftp, nc, and of course bash) but not wget. – Cauda

Answer 1

Edit in 2019-11-04: I'm rewriting my answer to reflect the importance of ensuring that a transfer isn't tampered with while in flight. I'll leave my original answer below the rule.

I suggest using rsync over ssh to transfer your files. rsync's interface may look overwhelming, but most users may be able to pick rsync -avzP, and if you need more flexibility, rsync can adapt. Using ssh will provide integrity, authenticity, and privacy to your connection.

curl is the de facto standard for http transfers; if plain http or https are preferred, curl or tools based on curl are probably a good choice.

---

In my experience, tools are available about in this order:

• wget
• curl
• sftp
• ftp
• GET (I use HEAD all the time and often forget it is just one tool in the suite)
• tftp
• nc (not as common as I wish)
• socat (even less common)

The bash /dev/tcp tool is available on most systems I've used (some used dash or pdksh instead), but using echo with bash, nc, or socat is going the long-way-around for HTTP access -- you'll have to handle headers somehow, which reduces its elegance.

Mainsheet

Many ftp clients, ncftp, lftp, etc., may or may not install ftp symlinks; Debian-derived systems will at least populate /etc/alternatives/ftp with the system-preferred ftp client, and /usr/bin/ftp will point to that. – Mainsheet

I've found wget is available on most systems I've used however curl has generally always been something I've compiled by hand. By contrast however because SFTP is based on having SSH access, I've found that it's generally also always available. So I'd rank that higher – Sedberry

@Mainsheet it might be helpful if you give some kind of clue about what kind of OS's your experience samples. I imagine it might be mostly unix and linux systems? But I just noticed that, on my macbook, curl is installed but not wget, which might indicate that's typical for macos users, which would mean your list would look quite different if it included the mac world (which is way bigger than the linux world). Just guessing on details for the most part here, of course. Also I just noticed your answer is from 2012 so I shouldn't hold you to anything you've said here anyway :-) – Morna

@DonHatch, thanks for the ping. I'll update this answer a little. – Mainsheet

Answer 2

Official list of POSIX 7 utilities

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/contents.html

The following are not present in the list:

• wget
• curl
• ftp

The same goes for the LSB which essentially only guarantees the POSIX utilities.

But I do think that the POSIX C is enough to implement most of netcat functionality, so it is really a missed opportunity. E.g.: How to make an HTTP get request in C without libcurl?

Likely it is because network protocols like HTTP were deemed too specific/didn't exist at the time POSIX was still evolving, and POSIX basically froze forever. Notably, HTTPS encryption is likely not trivial to implement.

Yardley

Answer 3

Curl is probably even more common than wget. At least in my experience merely because more other tools depend on it. But both curl and wget are a super simple install and will be available on any system.

Hollington

I agree, but when it comes to being pre-installed on a system, I don't think it is commonly installed "out of the box". Especially when we are talking about 5+ year old linux distros. – Methodology

You're right, I don't know what I'm thinking. I don't think it will be going on anything that old. Thanks for the comments – Methodology

@ManfredMoser to give an example: Debian (Stretch) comes with wget pre-installed but not with curl (and I doubt that it was any different in 2012). In my experience, wget is more often installed than curl (I wish it would be different). – Katiakatie

@Katiakatie I'm not sure "In my experience, wget is more often installed than curl" is that meaningful of a statement, other than as a statement of what subculture you happen to hang out in :-) I say that because I see that, although wget (and not curl) is installed on my offshoot-of-ubuntu linux machine, curl (and not wget) is installed on my macbook. I'm guessing the latter implies that most people in the mac world (which is way bigger than the linux world) would say "In my experience, curl is more often installed than wget". – Morna

@DonHatch you are probably correct and it depends on the subculture. A few days after I wrote that comment, I saw the situation on macOS too and was quite surprised. And it is true, lately, I didn't switch a lot between distributions, so even if curl was kinda rare (in default installations) 15 years ago that situation might have changed too. – Katiakatie

Answer 4

gnu awk (gawk) has built in TCP handeling even on non-linux systems http://www.gnu.org/software/gawk/manual/gawkinet/html_node/

Octennial

Answer 5

I suggest using ftp or wget, as they are the most common in Linux distributions. The best practice might be to have your script look to see if a command is available, if not go to the next command.

Saskatchewan

Answer 6

No, but you'll almost always have wget, busybox-wget, or curl.

Ready-made (auto-detect) script

I wrote a Posix-compliant Shell function, which can also be used as a command:

http_get

#!/bin/sh
## Usage: http_get <url> [filepath]
## Copyright (C) 2022 AJ ONeal <[email protected]>
## Permission to copy and modify is granted under the CC0-1.0 license
set -e
set -u

http_get() { (
    http_get_url="${1}"
    http_get_filepath="${2:-"$(basename "${http_get_url}")"}"
    if [ -z "${http_get_url}" ]; then
        echo >&2 "Usage: http_get <url> [filepath]"
        return 1
    fi

    if command -v wget > /dev/null; then
        # wget supports resumable downloads
        if ! wget -q -c "${http_get_url}" -O "${http_get_filepath}.part"; then
            echo >&2 "failed to download ${http_get_url} to ${http_get_filepath}.part"
            return 1
        fi
    elif command -v curl > /dev/null; then
        if ! curl -fsSL "${http_get_url}" -o "${http_get_filepath}.part"; then
            echo >&2 "failed to download ${http_get_url} to ${http_get_filepath}.part"
            return 1
        fi
    fi

    # move downloaded file into file location
    mv "${http_get_filepath}.part" "${http_get_filepath}"
); }

http_get "${1:-}" "${2:-}"

Benefits

• Safe for command or sourced function use (variables in a subshell, and namespaced)
• Runs in "strict mode" (clean exit on error)
• shellcheck-linted (Posix-compliant, no bashisms)
• Safe for batch mode (silences progress bar)
• Safe for partial & resumable downloads (prefers wget when available)

Extra features

There's an extended version that also detects if the shell is interactive (to enable the progress bar) at https://therootcompany.com/blog//posix-shell-http-request-curl-vs-wget/

Addend