Take a look at samba for an example of this can be done. The samba daemon runs as root but forks and assumes the credentials of a normal user as soon as possible.
Unix systems have two separate sets of credentials: the real user/group ids and the effective user/group ids. The real set identifies who you actually are, and the effective set defines what you can access. You can change the effective uid/gid as you please if you are root—including to an ordinary user and back again—as your real user/group ids remain root during the transition. So an alternative way to do this in a single process is to use seteuid/gid to apply the permissions of different users back and forth as needed. If your server daemon runs as root or has CAP_SETUID then this will be permitted.
However, notice that if you have the ability to switch the effective uid/gid at whim and your application is subverted, then that subversion could for example switch the effective uid/gid back to 0 and you could have a serious security vulnerability. This is why it is prudent to drop all privileges permanently as soon as possible, including your real user uid/gid.
For this reason it is normal and safer to have a single listening socket running as root, then fork off and change both the real and effective user ids by calling setuid. Then it cannot change back. Your forked process would have the socket that was accept()ed as it is a fork. Each process just closes the file descriptors they don't need; the sockets stay alive as they are referenced by the file descriptors in the opposite processes.
You could also try and enforce the permissions by examining them individually yourself, but I hope it is obvious that this is potentially error-prone, has lots of edge cases and more likely to go wrong (eg. it won't work with POSIX ACLs unless you specifically implement that too).
So, you have three options:
- Fork and
setgid()/setuid() to the user you want. If communication is required, use pipe(2) or socketpair(2) before you fork.
- Don't fork and
seteuid()/setegid() around as needed (less secure: more likely to compromise your server by accident).
- Don't mess with system credentials; do permission enforcement manually (less secure: more likely to get authorisation wrong).
If you need to communicate with the daemon, then although it might be harder to do it down a socket or a pipe, the first option really is the proper secure way to go about it. See how ssh does privilege separation, for example. You might also consider if can change your architecture so instead of any communication the process can just share some memory or disk space instead.
You mention that you considered having a separate process run for each user, but need a single listening TCP port. You can still do this. Just have a master daemon listen on the TCP port and dispatch requests to each user daemon and communicate as required (eg. via Unix domain sockets). This would actually be almost the same as having a forking master daemon; I think the latter would turn out to be easier to implement.
Further reading: the credentials(7) manpage. Also note that Linux has file system uid/gids; this is almost the same as effective uid/gids except for other stuff like sending signals. If your users don't have shell access and cannot run arbitrary code then you don't need to worry about the difference.
