How do I mirror a directory with wget without creating parent directories?

Question

I want to mirror a folder via FTP, like this:

wget --mirror --user=x --password=x ftp://ftp.site.com/folder/subfolder/evendeeper

But I do not want to create a directory structure like this:

ftp.site.com -> folder -> subfolder -> evendeeper

I just want:

evendeeper

And anything below it to be the resulting structure. It would also be acceptable for the contents of evendeeper to wind up in the current directory as long as subdirectories are created for subdirectories of evendeeper on the server.

I am aware of the -np option, according to the documentation that just keeps it from following links to parent pages (a non-issue for the binary files I'm mirroring via FTP). I am also aware of the -nd option, but this prevents creating any directory structure at all, even for subdirectories of evendeeper.

I would consider alternatives as long as they are command-line-based, readily available as Ubuntu packages and easily automated like wget.

Specify

With lftp: lftp -c mirror <url> – Labiche

Answer 1

For a path like: ftp.site.com/a/b/c/d

-nH would download all files to the directory a/b/c/d in the current directory, and -nH --cut-dirs=3 would download all files to the directory d in the current directory.

Aboutface

Thanks, --cut-dirs was the silver bullet I was looking for. – Specify

I would change to --cut-dirs=4 if you need all files from directory d – Corroboree

-nH would only ignore the host directory for me. You can set --cut-dirs=100 for preventing almost all the pre directorys -- tick – Watercool

It's fairly obvious with an URL like ftp.site.com/a/b/c/d/ one only wants the content in d/ and its subfolders. Is there an auto-cut option or something similar for wget, that autodetects the cut depth? – Sarcous

@Sarcous I was hoping the --cut-dirs=100 trick suggested by @Watercool (and in Tony TCG's answer below) would achieve this, but alas, it works like -nd, i.e. it flattens the entire hierarchy, including subdirectories below the one you're downloading. Looks like there's no automatic solution using wget alone... – Selfsatisfied

Answer 2

I had a similar requirement and the following combination seems to be the perfect choice:

In the below example, all the files in http://url/dir1/dir2 (alone) are downloaded to local directory /dest/dir

wget  -nd -np -P /dest/dir --recursive http://url/dir1/dir2

Thanks @ffledgling for the hint on "-nd"

For the above example:

wget -nd -np --mirror --user=x --password=x ftp://ftp.site.com/folder/subfolder/evendeeper

Snippets from manual:

   -nd
   --no-directories
       Do not create a hierarchy of directories when retrieving recursively.  With this option turned on, all files will get saved to the current directory, without clobbering (if a name shows up more than once, the
       filenames will get extensions .n).


   -np
   --no-parent
       Do not ever ascend to the parent directory when retrieving recursively.  This is a useful option, since it guarantees that only the files below a certain hierarchy will be downloaded.

Darkroom

Answer 3

-np (no parent) option will probably do what you want, tied in with -L 1 (I think, don't have a wget install before me), which limits the recursion to one level.

EDIT. ok. gah... maybe I should wait until I've had coffee.. There is a --cut or similar option, which allows you to "cut" a specified number of directories from the output path, so for /a/b/c/d, a cut of 2 would force wget to create c/d on your local machine

Agog

You mean: --cut-dirs=2 — Also, -L is --relative, so you probably meant -l or --level=1 – Ambrose

Answer 4

Instead of using:

-nH --cut-dirs=1

use:

-nH --cut-dirs=100

This will cut more directories and no folders will be created.

Note: 100 = the number of folders to skip creating. You can change 100 to any number.

Delindadelineate

Why not just use -nd instead then? – Mafalda