execve file not found when stracing the very same file!

Question

someone i know encountered a problem when running 'lmutil' so i asked them to strace -f lmutil. Why is execve failing with "No such file"!!! It makes no sense, since I am straceing the very same file!! What exactly is going on here???

strace -f /home/tabitha/Starprogram/FLEXlm_11.7/linux-x86_64-2.3.4/bin/lmutil

Output:

execve("/home/tabitha/Starprogram/FLEXlm_11.7/linux-x86_64-2.3.4/bin/lmutil", ["/home/tabitha/Starprogram/FLEXlm"...], [/* 38 vars */]) = -1 ENOENT (No such file or directory)
dup(2)                                  = 3
fcntl(3, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd7cb8b0000
lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
write(3, "strace: exec: No such file or di"..., 40strace: exec: No such file or directory
) = 40
close(3)                                = 0
munmap(0x7fd7cb8b0000, 4096)            = 0
exit_group(1)                           = ?

ldd output

$ ldd ./lmutil
        linux-vdso.so.1 =>  (0x00007fffcd5ff000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x00007fe40ebbe000)
        libm.so.6 => /lib/libm.so.6 (0x00007fe40e93b000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007fe40e724000)
        libc.so.6 => /lib/libc.so.6 (0x00007fe40e3a1000)
        libdl.so.2 => /lib/libdl.so.2 (0x00007fe40e19d000)
        /lib64/ld-lsb-x86-64.so.3 => /lib64/ld-linux-x86-64.so.2 (0x00007fe40edf5000)

$ find . -name lmutil -exec file {} \;
./bin.linux.x86_64/lmutil: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), for GNU/Linux 2.4.0, stripped
./bin.linux.x86/lmutil: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), for GNU/Linux 2.2.5, stripped
./lmutil: Bourne shell script text executable

Spermiogenesis

Just to be sure, the ldd output is for …/linux-x86_64-2.3.4/bin/lmutil, right? What OS is this (for Linux: what distribution), what release, and what architecture? – Kerianne

well.. i'm not sure about that and now they are trying with CentOS (Qemu).. The OS was Linux, Ubuntu latest version on AMD (but I'm not absolutely sure) anyway, I told them to check the architecture (32bit vs 64bit, intel/amd/sparc, linux/fbsd) carefully to make sure that wasn't the problem. – Spermiogenesis

Just to make it clear, I only want to know why strace was giving that error (File not found).. don't really care about fixing the user problem (getting lmutil to execute). *** Also IGNORE the "find . -name lmutil -exec file {} \; and the relevant output. Apologies for this! Apparently that command wasn't run by the user. Wish I could edit that out but i don;t know how***** – Spermiogenesis

If you didn't write lmutil, this isn't a programming-related question, in which case it's off-topic here and I suggest requesting a migration to Unix Stack Exchange. – Kerianne

Answer 1

The file you're trying to execute (…/lmutil) exists but its “loader” doesn't exist, where

• the loader of a native executable is its dynamic loader, for example /lib/ld-linux.so.2;
• the loader of a script is the program mentioned on its shebang line, e.g., /bin/sh if the script begins with #!/bin/sh.

From the name of the directory, there's a good chance that lmutil is an amd64 Linux binary, looking for /lib64/ld-linux-x86-64.so.2 as its loader, but you have an amd64 Linux kernel running a 386 (i.e. 32-bit) userland. You need to get suitable binaries for your platform.

I consider this situation to be Unix's most misleading error message. Unfortunately fixing it would be hard: the kernel can only report a numeric error code to the caller of the program, so it only has room for “command not found” (ENOENT) and not for the name of the loader it's looking for. This is one of these rare cases where strace doesn't help.

Unseal

well the libs exist - ldd /whatever/lmutil was okay. It is also a binary file - file /whatever/lmutil. The package was installed by the user on his laptop so it's not a remoteFS. – Spermiogenesis

ignore find . -name lmutil -exec file {} \; (that output wasn't generated by user) it's an error and needs to be edited out. – Spermiogenesis

@paleywiener: And what about ldd ./lmutil, was it run on the same executable you're trying to strace? The lack of a loader is the only explanation I can think of for executing an existing file to fail with ENOENT, so I'm inclined to cast doubt on that part of the report too. – Kerianne

@Gilles: you just saved my day! This was exactly my problem (I was trying to run a 32-bits executable on a 64-bits Ubuntu 12.04 installation without realizing it), and just issuing sudo apt-get install ia32-libs made the problem go away. – Flavorous

@Flavorous On Ubuntu 12.04, you can use multiarch, which lets you use more libraries than the old-fashioned ia32-libs way. See help.ubuntu.com/community/MultiArch – Kerianne

@Gilles: thanks but I don't see from that page how it lets me handle more libraries, and they say ia32-libs is a transitional package anyway. However I'll bear it in mind if I run into this problem again. – Flavorous

It would still be useful if the kernel reported missing loader error with something other than ENOENT. – Earlap

I ran into the same problem and it's due to binary incompatibility issue. This answer is impresively insightful without more specific logs! – Doordie

Answer 2

Your ldd output refers to /lib64/ld-lsb-x86-64.so.3, but this loader may not actually exist unless (on Ubuntu) you've installed lsb-core package. The postinst script for the package creates the relevant symbolic links in /lib* directories.

Dyeline

This solved it for me: the "ldd" output shows "/lib64/ld-lsb-x86-64.so.3 => /lib64/ld-linux-x86-64.so.2" but I didn't have "/lib64/ld-lsb-x86-64.so.3" on my system (Ubuntu 16.04). Installing the "lsb-core" package created this symlink and then lmutil worked. – Mn

Answer 3

Just a bit of speculation, but my first question would be if the user who is having this problem can run the executable by itself without strace.

Also the execve manual page says that ENOENT will occur if either the file or a required script interepreter or shared library cannot be found. (I notice there is 64-bit-ness involved here. Are all the right libraries available?)

Is the file a native executable or could it be a script of some sort?

This looks like a licensing manager - any chance it has made itself intentionally hard to debug?

Speaking of users, is 'tabitha' in whose directory the executable resides the user having the problem? Or are we looking at a possible complication of trying to run a program installed by another ordinary user rather than in a normal system-wide fashion by root?

Misogamy

If it can't be run by the OS, strace can't do anything with it either, which is what you are seeing here. You probably need to check read and execute permissions for that user, and then perhaps library issues. – Misogamy

Answer 4

You can use readelf (any readelf should do, you don't need one from a special crosscompiler toolchain) to check which loader is expected by the dynamically loaded or executable.

$ readelf -l <filename> |grep -i interp
...
[Requesting program interpreter: /system/bin/linker]

Mariellamarielle

+1 for actually providing an actionable way to debug this problem for ELF executables. While technically answering a separate question, it's a very relevant and natural follow-up question. – Pintail

Answer 5

From the execve manpage:

On success, execve() does not return, on error -1 is returned, and errno is set appropriately.

strace is assuming that -1 means "file not found" as the errno value ENOENT is -1 and strace doesn't make a distinction.

Essentially, then, you can ignore this: the -1 just means that some error occurred. the strace output doesn't tell you what the value of errno is.

I write this as a warning call to not jump to conclusions with strace and return values, even though it may turn out that errno is ENOENT here anyway.

Gonococcus

thank you! Also found this article: people.redhat.com/alikins/old_docs/debug.txt – Spermiogenesis

Ooh, that's a good document. Also, if correct, it means that my answer was wrong. :) – Gonococcus

No, you're reading the strace output wrong. execve returns -1, and ENOENT is the value of errno after calling execve. – Kerianne

@Gilles: It's almost like you didn't read my previous comment. – Gonococcus

@Tomalak: If you've realized your answer is wrong, edit it to fix it. A comment saying “my answer was wrong” is not helpful to readers: the answer is supposed to stand on its own, and even if they do read the comment, as I write this, they won't know what is wrong unless they read my comment too. – Kerianne

@Tomalak: I'd have to remove everything, since all you're saying is that paleywiener is misinterpreting the strace output, which he isn't. I explain the usual cause of the original symptoms in my answer, but the ldd output isn't consistent with this cause. – Kerianne

@Gilles: I guess we both ought to delete our answers then. – Gonococcus

Answer 6

You can see execve return ENOENT . Then search ENOENT in execve man.

The file pathname or a script or ELF interpreter does not exist.

The lmutil file exist, and is a ELF file. So check whether ELF interpreter exist or not. According the man again:

If the executable is a dynamically linked ELF executable, the interpreter named in the PT_INTERP segment is used to load the needed shared objects. This interpreter is typically /lib/ld-linux.so.2 for binaries linked with glibc .

We can find the ELF interpreter path with readelf as @auselen said:

$ readelf -l <filename> |grep -i interp
...
[Requesting program interpreter: /system/bin/linker]

The ELF interpreter should not exist here.

Karol