Find original owning process of a Linux socket

Question

In Linux and other UNIX-like operating systems, it is possible for two (or more) processes to share an Internet socket. Assuming there is no parent-child relationship between the processes, is there any way to tell what process originally created a socket?

Clarification: I need to determine this from "outside" the processes using the /proc filesystem or similar. I can't modify the code of the processes. I can already tell what processes are sharing sockets by reading /proc/<pid>/fd, but that doesn't tell me what process originally created them.

Kaitlin

Answer 1

You can use netstat for this. You should look in the columns 'Local Address' and 'PID/Program name'.

xxx@xxx:~$ netstat -tulpen
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 127.0.0.1:4005          0.0.0.0:*               LISTEN      1000       68449       7559/sbcl       
tcp        0      0 0.0.0.0:6000            0.0.0.0:*               LISTEN      0          3938        -               
tcp6       0      0 :::6000                 :::*                    LISTEN      0          3937        -               
udp        0      0 0.0.0.0:68              0.0.0.0:*                           0          4528        -               

Aussie

Answer 2

doesn't 'lsof -Ua' help?

Magnificence

Good answer with added information: It does work though I believe -a is unnecessary as results seem the same with and without -a, but, on some Linux, it appears 'lsof' is largely neutered and requires elevated permissions to find this information. For example, I wanted to know the PID of /tmp/ssh-W1Tl4i8HiftZ/agent.21283, but 'lsof -Ua /tmp/ssh-W1Tl4i8HiftZ/agent.21283' shows nothing at all - not even an error message. On the other hand, 'sudo lsof -Ua /tmp/ssh-W1Tl4i8HiftZ/agent.21283' was very revealing and did show a PID. Sadly, sudo or elevated permissions are not viable for many tasks. – Headwaters

Answer 3

You can likely find the shared sockets by parsing /proc/net/tcp (and similar "files" for other protocols). There's some docs on /proc/net/tcp here.

You would need to find the socket (perhaps by its IP addresses/port numbers ?) and parse out the inode number. Once you have the inode, you can search through all of /proc/*/fd/* , calling stat for every link and inspect the st_ino member of struct stat until you find a match.

The inode number should match between the 2 processes, so when you've gone through all /proc/*/fd/* you should have found them both.

If what you do know is the process id and socket fd of the first, you might not need to go through /proc/net/tcp, all you need to do is stat the /proc/<pid>/fd/<fd> and search the rest of /proc/*/fd/* for a matching inode. You'd need /proc/net/tcp if you want to fetch the ip addresses/port number though - which you can find if you know the inode number

Teletype

Maybe I didn't explain myself well. I already know how to find out which processes are sharing sockets. I do that using exactly the method you described. But this doesn't help me determine which process owned the socket originally. Perhaps I need to make a guess based on the age of each process. – Kaitlin

It would be very helpful if you updated your question to state what you are looking for, and that you are doing this from outside any of the programs involved. There is however no direct information on the original owner or creator anywhere. The closest thing you will get is calling lstat on the links in /proc/pid/fd/fd and guess that the oldest timestamp(ctime likely) was the original creator. – Teletype

Updated the question to clarify. – Kaitlin

@Rob H You cannot figure out what process originally created a socket, there is no information about that anywhere in the OS. You can only figure out which processes that have a reference(file descriptor) to a socket right now. – Teletype

Answer 4

For purposes creating a test case, consider a situation where multiple ssh-agent processes are running and have open sockets. I.e. A user runs ssh-agent multiple times and loses the socket/PID information given when the agent started:

$ find /tmp -path "*ssh*agent*" 2>/dev/null
/tmp/ssh-0XemJ4YlRtVI/agent.14405
/tmp/ssh-W1Tl4i8HiftZ/agent.21283
/tmp/ssh-w4fyViMab8wr/agent.10966

Later, the user wants to programmatically determine the PID owner of a particular ssh-agent socket (i.e. /tmp/ssh-W1Tl4i8HiftZ/agent.21283):

$ stat /tmp/ssh-W1Tl4i8HiftZ/agent.21283
  File: '/tmp/ssh-W1Tl4i8HiftZ/agent.21283'
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 805h/2053d      Inode: 113         Links: 1
Access: (0600/srw-------)  Uid: ( 4000/ myname)   Gid: ( 4500/   mygrp)
Access: 2018-03-07 21:23:08.373138728 -0600
Modify: 2018-03-07 20:49:43.638291884 -0600
Change: 2018-03-07 20:49:43.638291884 -0600
Birth: -

In this case, because ssh-agent named its socket nicely as a human onlooker can guess that the socket belongs to PID 21284, because the socket name contains a numeric component that is one-off from a PID identified with ps:

$ ps -ef |  grep ssh-agent
myname   10967     1  0 16:54 ?        00:00:00 ssh-agent
myname   14406     1  0 20:35 ?        00:00:00 ssh-agent
myname   21284     1  0 20:49 ?        00:00:00 ssh-agent

It seems highly unwise to make any assumption that the PIDs will be so reliable as to always only be off by one, but also, one might suppose that not all socket creators will name the sockets so nicely.

@Cypher's answer points to a straightforward solution to the problem of identifying the PID of the socket owner, but is incomplete as lsof actually can only identify this PID with elevated permissions. Without elevated permissions, no results are forthcoming:

$ lsof /tmp/ssh-W1Tl4i8HiftZ/agent.21283
$

With elevated permissions, however, the PID is identified:

$ sudo lsof /tmp/ssh-W1Tl4i8HiftZ/agent.21283
COMMAND     PID    USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
ssh-agent 21284 myname     3u  unix 0xffff971aba04cc00      0t0 1785049 /tmp/ssh-W1Tl4i8HiftZ/agent.21283 type=STREAM

In this case, the owner of the PID (myname) and socket was the one doing the query, so it seemed elevated permissions should not be needed. Furthermore, the task performing the query was not supposed to be able to elevate permissions, so I looked for another answer.

This led me to @whoplisp's answer proposing netstat -tulpen as a solution to the OP's problem. While it may have been effective for the OP, the command line is too restrictive to serve as a general purpose command and was completely ineffective in this case (even with elevated permissions).

$ sudo netstat -tulpen | grep -E -- '(agent.21283|ssh-agent)'
$

netstat, however, can come close if a different command-line is used:

$ netstat -ap | grep -E -- '(agent.21283)'
(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.)
unix  2      [ ACC ]     STREAM     LISTENING     1785049  -                    /tmp/ssh-W1Tl4i8HiftZ/agent.21283

Sadly, here too, the PID is elusive without elevated permissions:

$ sudo netstat -ap | grep -E -- '(agent.21283|ssh-agent)'
unix  2      [ ACC ]     STREAM     LISTENING     1765316  10967/ssh-agent      /tmp/ssh-w4fyViMab8wr/agent.10966
unix  2      [ ACC ]     STREAM     LISTENING     1777450  14406/ssh-agent      /tmp/ssh-0XemJ4YlRtVI/agent.14405
unix  2      [ ACC ]     STREAM     LISTENING     1785049  21284/ssh-agent      /tmp/ssh-W1Tl4i8HiftZ/agent.21283

Of the two solutions, however, lsof clearly wins at the races:

$ time sudo netstat -ap | grep -E -- '(agent.21283|ssh-agent)' >/dev/null

real    0m5.159s
user    0m0.010s
sys     0m0.019s
$ time sudo lsof /tmp/ssh-W1Tl4i8HiftZ/agent.21283 >/dev/null

real    0m0.120s
user    0m0.038s
sys     0m0.066s

Yet another tool exists according to the netstat man page:

$ man netstat | grep -iC1 replace
NOTES
       This program is mostly obsolete.  Replacement for netstat is ss.  Replacement for netstat -r is ip route.  Replacement for netstat -i
       is ip -s link.  Replacement for netstat -g is ip maddr.

Sadly, ss also requires elevated permissions to identify the PID, but, it beats both netstat and lsof execution times:

$ time sudo ss -ap | grep -E "(agent.21283|ssh-agent)"
u_str  LISTEN     0      128    /tmp/ssh-w4fyViMab8wr/agent.10966 1765316               * 0                     users:(("ssh-agent",pid=10967,fd=3))
u_str  LISTEN     0      128    /tmp/ssh-0XemJ4YlRtVI/agent.14405 1777450               * 0                     users:(("ssh-agent",pid=14406,fd=3))
u_str  LISTEN     0      128    /tmp/ssh-W1Tl4i8HiftZ/agent.21283 1785049               * 0                     users:(("ssh-agent",pid=21284,fd=3))

real    0m0.043s
user    0m0.018s
sys     0m0.021s

In conclusion, it might seem that for some PID identification, it appears that elevated permissions are required.

Note: Not all operating systems require elevated permissions. For example, SCO Openserver 5.0.7's lsof seemed to work just fine without elevating permissions.

Caveat: This answer may fail with respect to the OP's qualification for finding "the original creator" of the socket. In the example used, no doubt PID 21283 was the originator of the socket's creation as this PID is identified in the socket name. Neither lsof nor netstat identified PID 21283 as the original creator, though clearly PID 21284 is the current maintainer.

Headwaters

I was looking for exactly that: how to find the pid of the maintainer of an ssh-agent socket. Thank you for these findings. Still wondering why I need elevated permissions to investigate my own sockets and processes, especially considering that the information is not hidden without them: grep "ssh.*agent" /proc/$(pidof ssh-agent)/net/unix 00000000fba3c9f9: 00000002 00000000 00010000 0001 01 783851 /tmp/ssh-XXXXXXQKem0I/agent.4142346, just difficult to find. – Idolize