`os.symlink` vs `ln -s`

Question

I need to create a symlink for every item of dir1 (file or directory) inside dir2. dir2 already exists and is not a symlink. In Bash I can easily achieve this by:

ln -s /home/guest/dir1/* /home/guest/dir2/

But in python using os.symlink I get an error:

>>> os.symlink('/home/guest/dir1/*', '/home/guest/dir2/')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 17] File exist

I know I can use subprocess and run ln command. I don't want that solution.

I'm also aware that workarounds using os.walk or glob.glob are possible, but I want to know if it is possible to do this using os.symlink.

Rid

I think os.symlink is a mere wrapper of the respective system call (well, more or less) and therefore wouldn't provide the same semantics you get from a full-fledged utility using that system call. – Marguerite

What is the reason for the "I want to do this using os.symlink" requirement? If the right thing to do is a for loop over a glob, why don't you want to do the right thing? – Odontoblast

@Odontoblast I will do that if I have to. I wanted to know if there is a way to do it with os.symlink. I thought it would provide everything ln -s does, but turns out it does not. – Rid

@jurgenreza: You're still missing the key point: what you're looking for isn't something ln -s does, it's something the shell does. – Odontoblast

@0xC0000022L: It's a tiny bit more complicated (see [the source](5f4d7/Modules/posixmodule.c#l7199)), but only a tiny bit; basically, it ultimately calls symlink, symlinkat, or CreateSymbolicLinkW. And the semantics are clearly designed to match POSIX symlink even if that's not what it ends up using. – Odontoblast

@abarnert: yep, that's what my "well, more or less" was aimed at. – Marguerite

@Odontoblast oh ok, I think I got it now. Thanks for your answer. – Rid

@0xC0000022L: I wasn't correcting you, so much as taking away any doubt left by your "I think" prefix. – Odontoblast

Answer 1

os.symlink creates a single symlink.

ln -s creates multiple symlinks (if its last argument is a directory, and there's more than one source). The Python equivalent is something like:

dst = args[-1]
for src in args[:-1]:
    os.symlink(src, os.path.join(dst, os.path.dirname(src)))

So, how does it work when you do ln -s /home/guest/dir1/* /home/guest/dir2/? Your shell makes that work, by turning the wildcard into multiple arguments. If you were to just exec the ln command with a wildcard, it would look for a single source literally named * in /home/guest/dir1/, not all files in that directory.

The Python equivalent is something like (if you don't mind mixing two levels together and ignoring a lot of other cases—tildes, env variables, command substitution, etc. that are possible at the shell):

dst = args[-1]
for srcglob in args[:-1]:
    for src in glob.glob(srcglob):
        os.symlink(src, os.path.join(dst, os.path.dirname(src)))

You can't do that with os.symlink alone—either part of it—because it doesn't do that. It's like saying "I want to do the equivalent of find . -name foo using os.walk without filtering on the name." Or, for that matter, I want to do the equivalent of ln -s /home/guest/dir1/* /home/guest/dir2/ without the shell globbing for me."

The right answer is to use glob, or fnmatch, or os.listdir plus a regex, or whatever you prefer.

Do not use os.walk, because that does a recursive filesystem walk, so it's not even close to shell * expansion.

Odontoblast

ln is ln [target] [link], how can there be several targets as you describe? I think you mean several links linking to one target? – Dissonant

Answer 2

* is a shell extension pattern, which in your case designates "all files starting with /home/guest/dir1/".

But it's your shell's role to expand this pattern to the files it matches. Not the ln command's.

But os.symlink is not a shell, it's an OS call - hence, it doesn't support shell extension patterns. You'll have to do that work in your script.

To do so, you can use os.walk, or os.listdir. As indicated in the other answer, the appropriate call will depend on what you want to do. (os.walk wouldn't be the equivalent of *)

---

To convince yourself: run this command on an Unix machine in your terminal: python -c "import sys; print sys.argv" *. You'll see that it's the shell that's doing the matching.

Clorindaclorinde

+1 for the experiment about the globbing. Keep in mind, however, that thw Windows shell (cmd.exe) does not do this before passing it to the invoked program. – Marguerite

@0xC0000022L: Well, to be fair, Windows doesn't come with ln. And MSVCRT (their libc) comes with a hack to simulate shell globbing on argv either before your main starts, or when you explicitly ask it to, and I'm pretty sure most ports of Unix programs will either use that or something equivalent (e.g., whatever Cygwin does). – Odontoblast

@Marguerite I must confess checking whether this worked on Windows didn't cross my mind - will update! – Clorindaclorinde

@abarnert: well, they call it mklink, just like they call what's known in the *nix world as mkdir simply md in the Windows world. – Marguerite

@0xC0000022L: But mklink doesn't have the same syntax as ln. In particular, mklink cannot take a list of sources and link them all into a directory; it takes a single source and a single target. – Odontoblast

@abarnert: true point. Thought you were referring to the lack of a tool as a whole. – Marguerite

Answer 3

As suggested by @abarnert it's the shell that recognizes * and replaces it with all the items insside dir1. Therefore I think using os.listdir is the best choice:

for item in os.listdir('/home/guest/dir1'):
    os.symlink('/home/guest/dir1/' + item, '/home/guest/dir2/' + item)

Rid

I suggest using os.path.join rather than +. – Dramaturge