Python can't open symlinked file

Question

Python is unable to open my simlinked file. I made sure the file exists and I can access it. I was under the impression that symlinks are resolved on the OS level so Python would never know about it.

therold@therold:~/Programming/ML/deeptank/dataset/inc (master)$ ls -lisa  /Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png
7870541 8 lrwxr-xr-x  1 therold  staff  46 13 Mai 16:44 /Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png -> ../training_data/matilda-iv/matilda-iv_689.png

OSX is clearly able resolve the symlinked image file. However the Python open() method fails me:

therold@therold:~/Programming/ML/deeptank/dataset/inc (master)$ python
Python 2.7.10 (default, Sep 23 2015, 04:34:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.72)] on darwin
>>> open("/Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory: '/Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png'
>>>

Any ideas what I am doing wrong? Help is much appreciated.

Venus

The symlink might be broken. Even if OS can read the destination where the link points to, it doesn't mean that the destination does exist. Can you verify that? – Tapdance

It seems that the symlink was created as a relative link. Hence the global Python installation is unable to open the file after resolving the relative reference. – Venus

Try ls -lL /Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png to confirm the existence of the file. – Erythroblast

Is there a way to convert relative to absolute symlinks? – Venus

There may be, but your logic is flawed. Python, along with every other program, is perfectly capable of opening relative symbolic links. Relative symbolic links are always resolved relative to the directory containing the link. – Erythroblast

@Robᵩ No such file or directory It seems the symlink is broken after all. I still don't understand why I was able to resolve it in the ls statement in my question. – Venus

You weren't able to resolve it in the ls statement in your question. ls -l simply displays the linked-to path. ls -lL attempts to resolve the link. – Erythroblast

@Robᵩ Ah. That explains a lot. Thanks for pointing this out to me. I wasn't aware of this. Feel free to turn this comment into an answer so I can approve if you like. – Venus

Answer 1

Any ideas what I am doing wrong?

The target of the symbolic link doesn't exist.

I don't understand why I was able to resolve it in the ls statement in my question.

You weren't.

The ls command by default operates on the link itself, not on the target of the link. Absent the -L option, ls never attempts to resolve the symbolic link.

Consider a directory with these two files:

$ ls -l
-rw-rw-r-- 1 me me 6 May 13 11:58 a
lrwxrwxrwx 1 me me 3 May 13 12:00 b -> ./a

a is a text file containing the six bytes 'hello\n'. b is a link file containing the three bytes to its target path: './a'. ls is able to describe the properties of the link without dereferencing the link itself.

In contrast, use the -L option:

$ ls -lL
-rw-rw-r-- 1 me me 6 May 13 11:58 a
-rw-rw-r-- 1 me me 6 May 13 11:58 b

Now ls has resolved the link in b, and displays information about the linked-to file. With -L, ls now claims that b is also a six-byte text file.

Finally, consider this:

$ rm a
$ ls -l
lrwxrwxrwx 1 me me 3 May 13 12:00 b -> ./a
$ ls -lL
ls: cannot access b: No such file or directory
l????????? ? ? ? ?            ? b

Now b is a link that resolves to a file that no longer exists. Since ls -l never attempts the resolve the link, its output is unchanged from the previous test. (b is a link file, 3 bytes long, contents './a'.)

But ls -lL attempts to resolve the link, fails, and displays the failure information.

Lahdidah

Thank you again for the detailed answer. – Venus

Answer 2

... don't forget to use complete paths. On Raspberry Pi with Python 3.7.3 I had to include the whole paths for a working link:

os.symlink("Desktop/a.txt","Desktop/a_link.txt")  # WRONG !

python3 did not recognize the file a.txt because the paths of the standard shell are not active.

os.symllink("/home/pi/Desktop/a.txt","/home/pi/Desktop/a_link.txt")

works well.

Reasonable

are you sure the problem with the first version isn't that the script's [print] working directory isn't currently set to /home/pi? use os.getcwd() to confirm use os.chdir() to change the working directory (with relative or absolute paths) – Pyromania

Answer 3

I'm logging an answer because I came to this question with the same problem, but the target of my symlink did exist.

Trying to open a file across a symlink in pandas gave an unhelpful not-found message:

⇒  ipython.exe  # this is a clue to the answer below...

In [1]: import pandas as pd

In [2]: codes = pd.read_excel('codes-no-agg-children.xlsx', ...)
---------------------------------------------------------------------------
<...lots of traceback stuff...>
FileNotFoundError: [Errno 2] No such file or directory: 'codes-no-agg-children.xlsx'   

Since the target did exist, the answer above didn't help me, but the question prompted me to try a regular python open instead of through pandas:

In [3]: import os                                                                                                                                                                                                                                         
In [4]: os.system('cp ./nvivo-win/good-extracts/codes-no-agg-children.xlsx .')                                               
\\wsl$\Ubuntu\home\sigfried\ccvs\analysis'                                                                                  
CMD.EXE was started with the above path as the current directory.                                                            
UNC paths are not supported.  Defaulting to windows directory.                                                               
cp: cannot stat './nvivo-win/good-extracts/codes-no-agg-children.xlsx': No such file or directory

I don't know what all that means, but I assume the cause is:

Python for windows can't follow Ubuntu (WSL 2) symlinks (or, at least it can't open Ubuntu symlinks back to a windows file.

Hope this helps someone.

Ringlet

I saw someone had upvoted my answer, so I looked back through it. I think my conclusion is correct, and the joke at the end is slightly funny, but the code blocks make no sense at all. I must have pasted in the wrong bits of code or something. Sorry about that! – Ringlet

@sigfied just as an FYI, Windows can do its own symlinks with mklink. – Flax

this seems correct in my system. – Cavan

Answer 4

I had the same issue-Fedora 32 system, with Jupyter's Notebook folder containing my files having symlinks to json files. I beat my head against the proverbial wall for a few days, googling came up short. In the end, copying the files over fixed the issue (though there are other options like hardlinks).

Bothy

Answer 5

I hit something similar today: when symlinking to a file, and reading lines into an array, some strings of digits would get broken into separate list elements??

This behaviour disappears when opening the original file directly (instead of via the symlink):

Original file -> Python list
123456  ->  '1', '23', '456', ''
123456789  ->  '1234567', '89'

Hex editor must be following symlinks properly

$ head rockyou.txt | xxd
00000000: 3132 3334 3536 0a31 3233 3435 0a31 3233  123456.12345.123
00000010: 3435 3637 3839 0a70 6173 7377 6f72 640a  456789.password.
00000020: 696c 6f76 6579 6f75 0a70 7269 6e63 6573  iloveyou.princes
00000030: 730a 3132 3334 3536 370a 726f 636b 796f  s.1234567.rockyo
00000040: 750a 3132 3334 3536 3738 0a61 6263 3132  u.12345678.abc12
00000050: 330a                                     3.
$ head rockyou.txt.short | xxd
00000000: 3132 3334 3536 0a31 3233 3435 0a31 3233  123456.12345.123
00000010: 3435 3637 3839 0a70 6173 7377 6f72 640a  456789.password.
00000020: 696c 6f76 6579 6f75 0a70 7269 6e63 6573  iloveyou.princes
00000030: 730a 3132 3334 3536 370a 726f 636b 796f  s.1234567.rockyo
00000040: 750a 3132 3334 3536 3738 0a61 6263 3132  u.12345678.abc12
00000050: 330a                                     3.
$ ls -lah rock*
lrwxr-xr-x  1 u  g    17B Apr 16 10:56 rockyou.txt -> rockyou.txt.short
-rw-r--r--  1 u  g   945B Apr 16 11:39 rockyou.txt.short

MacOS Catalina, Python 3.7, 3.8

Pyromania

Did you ever figure this out? Also, rather seems more suited to a question of its own rather than as an answer to the OP. – Flax

I have not got to the bottom of this, nor is it an issue anymore, now that I know to use the full path, rather than a symlink. – Pyromania

Answer 6

Note that given a file_path,

os.path.realpath(file_path)

will interpret symlinks and give you the real path to the actual file. So if you are thinking its a problem of the symlink, using realpath will resolve the issue.

I seemed to have the same problem, but for an "existing" file. Turns out this is a question that may have multiple answers, as another option is that the filename string is somehow malformed (extra space, wrong file separator etc.)...

In my case e.g. from jupyter:

!ls -alh {file_path}
!cat -alh {file_path}

were finding the file, so I was thinking it was python's not following the symlink.

What I realized, was that I had an extra space in the filename e.g.:

file_path = '/foo/bar/file.txt '
                              ^notice the extra space here

As a result, ls etc. were working fine, but I could not find the file from python... :D

Cavan