What's the expected behavior of open(name, O_CREAT|O_DIRECTORY, mode)?

Question

Despite a careful read of the related standard documentation, I can't understand what's the expected behavior in POSIX compliant systems when a open system call is invoked with flags including O_CREAT|O_DIRECTORY.

The standard specifies that

If O_CREAT and O_DIRECTORY are set and the requested access mode is neither O_WRONLY nor O_RDWR, the result is unspecified.

However it does not specify the behavior of the system with neither (O_CREAT|O_DIRECTORY|O_WRONLY) nor (O_CREAT|O_DIRECTORY|O_RDWR). Indeed (as far as I can understand) the behavior on EISDIR only apply to existing directories.

In the section related to O_CREATE, the standard specifies that, when the named file does not exist,

if O_DIRECTORY is not set the file shall be created as a regular file; [...]

but again it does not specify what will happen if O_DIRECTORY is set too.

I've looked the manual pages of both NetBSD (which notoriously cares a lot about POSIX compliance) and Linux (which is a widely used system, despite not actually a POSIX one) but I can't find any clarification.

Is it correct to say that the use of both flags is unspecified? And if so, what's the most common behavior?

Is open(name, O_CREAT|O_DIRECTORY, mode) equivalent to mkdir on any POSIX compliant OS?

Muchness

Note that you should include one of O_RDONLY, O_WRONLY, and O_RDWR in the open flags (or possibly O_EXEC or O_SEARCH). Historically, omitting those is equivalent to specifying O_RDONLY (it is normally 0; O_WRONLY is normally 1, and O_RDWR is normally 2), but these are not bit values. (The O_EXEC and O_SEARCH options specified by POSIX complicate life, but neither Linux (Ubuntu 16.04 LTS) nor macOS 10.12.6 supports either of them.) I see now where I got my 'faulty' code from — I copied your open() command without realizing I wasn't strictly abiding by the rules. – Wishful

In the rationale section of the POSIX specification for open(), it says: In addition, the open() function refuses to open non-directories if the O_DIRECTORY flag is set. This avoids race conditions whereby a user might compromise the system by substituting a hard link to a sensitive file (e.g., a device or a FIFO) while a privileged application is running, where opening a file even for read access might have undesirable side-effects. – Wishful

Answer 1

NetBSD itself contains the following in vn_open:

if ((fmode & (O_CREAT | O_DIRECTORY)) == (O_CREAT | O_DIRECTORY))
        return EINVAL;

so any combination with these 2 is straight up rejected.

In Linux, it's a little bit more hairy, but any trivial test will show you that the directory is not created either, but you can end up with a file.

For kicks, I also checked FreeBSD, which never ends up creating anything with O_DIRECTORY in the first place.

If what you are looking for is a mkdir which gives you back the fd, I'm afraid there is nothing of the sort. On the other hand, you should be able to safely open with O_DIRECTORY anything you mkdir'ed.

Elouise

Note that if you create a directory using mkdir(path_to_dir, 077), you will not be able to open the directory with open(path_to_dir, …|O_DIRECTORY|…) because you created the directory with no access rights for you. It's obscure — and the 077 permissions are most likely a typo for 0777, but… – Wishful

Answer 2

I think you have misunderstood what O_DIRECTORY is meant for. It's not to create a directory but to ensure the "file" opened by open(2) is a directory.

O_DIRECTORY
If path resolves to a non-directory file, fail and set errno to [ENOTDIR].

That's precisely how POSIX documents it (quoted above).

However it does not specify the behavior of the system with neither (O_CREAT|O_DIRECTORY|O_WRONLY) nor (O_CREAT|O_DIRECTORY|O_RDWR)

The behaviour of O_CREAT|O_DIRECTORY|O_WRONLY and O_CREAT|O_DIRECTORY|O_RDWR is equivalent to O_CREAT|O_WRONLY and O_CREAT|O_RDWR respectively provided the pathname (first argument to open(2)) is a directory. The presence of O_DIRECTORY is to ensure the file being opened is a directory - it doesn't affect anything else.

Is it correct to say that the use of both flags is unspecified? And if so, what's the most common behavior?

It means the behaviour of the specific combination O_CREAT | O_DIRECTORY isn't specified; doesn't mean using the individual flags (with or without other flags) is unspecified.

Is open(name, O_CREAT|O_DIRECTORY, mode) equivalent to mkdir on any POSIX compliant OS?

Not at all. That's why it's left as unspecified. On Linux, it's definitely not - a regular file is created:

When both O_CREAT and O_DIRECTORY are specified in flags and the file specified by pathname does not exist, open() will create a regular file (i.e., O_DIRECTORY is ignored).

To create a directory, you'd use mkdir(2).

Calender

Answer 3

The answers so far are not correct. Implementations are free to support directory creation via O_CREAT|O_DIRECTORY|O_WRONLY or O_CREAT|O_DIRECTORY|O_RDWR, but they are not required to. This was clarified in 2014, as you can see in the bug tracker of the Austin group:

open("/path/to/existing/directory", O_CREAT) should fail with EISDIR, but implementations should be allowed to support creating and opening a directory via open("/path/to/directory", O_CREAT|O_DIRECTORY) as an extension

with the following rationale:

The standard does not specify the behavior when open() is called with O_CREAT|O_RDONLY (or O_CREAT|O_SEARCH) on an existing directory. Additionally, some systems want to allow creation of directories using the open() function. This should be an allowed, but not required, extension.

The wording you cited from the current version of the standard

If O_CREAT and O_DIRECTORY are set and the requested access mode is neither O_WRONLY nor O_RDWR, the result is unspecified.

was precisely one of the changes made to allow it.

One of reasons for allowing this, as described by Rich Felkner, is, that it would provide an interface for atomically creating and opening a directory.

However, I don't know, if any POSIX implementation out there does actually provide directory creation via open.

Masseur

"The answers so far are not correct." - please be specific about what is wrong with previous answers. – Calender

It seems unfortunate that O_CREAT|O_DIRECTORY is not mandated to create a directory as subfd = openat(dirfd, "name", O_CREAT|O_DIRECTORY, mode) seems like it would be highly useful. – Erepsin