os.mkdir(path) returns OSError when directory does not exist

Question

I am calling os.mkdir to create a folder with a certain set of generated data. However, even though the path I specified has not been created, the os.mkdir(path) raises an OSError that the path already exists.

For example, I call:

os.mkdir(test)

This call results in OSError: [Errno 17] File exists: 'test' even though I don't have a test directory or a file named test anywhere.

NOTE: the actual path name I use is not "test" but something more obscure that I'm sure is not named anywhere.

Help, please?

Trichome

Answer 1

Greg's answer is correct but doesn't go far enough. OSError has sub-error conditions, and you don't want to suppress them all every time. It's prudent to trap just expected OS errors.

Do additional checking before you decide to suppress the exception, like this:

import errno
import os

try:
    os.mkdir(dirname)
except OSError as exc:
    if exc.errno != errno.EEXIST:
        raise
    pass

You probably don't want to suppress errno.EACCES (Permission denied), errno.ENOSPC (No space left on device), errno.EROFS (Read-only file system) etc. Or maybe you do want to -- but that needs to be a conscious decision based on the specific logic of what you're building.

Greg's code suppresses all OS errors; that's unsafe just like except Exception is unsafe.

As others have pointed out, newer versions of Python provide os.makedirs() that attempts to create the dir only if it doesn't exist, equivalent to mkdir -p from a unix command line.

Twoply

This is the only correct answer. The "look before you leap" approach using os.path.exists that others have suggested creates a race condition: it's entirely possible for the file/dir to be created between the exists and mkdir/makedirs calls, which would generate an unhandled exception. – Ameline

This is not the "only correct answer" since the race condition is not a relevant problem in all (perhaps even most) use cases. – Rurik

@oseiskar, when the right way is as easy or easier than doing it the wrong way, to not do it the right way every time is irresponsible. If you do it the right way every time, you don't have to waste time & energy thinking about it. Good developers focus on developing sound habits. – Twoply

@ChrisJohnson true, but in this case the "as easy" is questionable in terms of readability. The correct balance between these different aspects depends on your use case. Sacrificing readability for resistance to race conditions that never happen in your environment is premature optimization. – Rurik

@oseiskar, what I showed is the normal approach for a Python developer. If it's not readable, then whoever's reading it doesn't understand Python very well. Unless you're running on a single-threaded OS, the potential for this particular race condition always exists. – Twoply

Two criticisms: 1) the pass is superfluous and 2) I'd rather the raise be bare (preserving the traceback - maybe unimportant here, but I've had problems with losing tracebacks due to well-meaning coders raising a new error, losing the traceback, and thus my clue for what to fix, before.) – Coen

Aaron, I agree. The pass is just a harmless habit, but the raise has a real effect. I'll revise that. – Twoply

Seems it is a kind of “Ask forgiveness not permission” #12265951 – Motorbus

@akif yes exactly. – Twoply

Answer 2

In Python 3.2 and above, you can use:

os.makedirs(path, exist_ok=True)

to avoid getting an exception if the directory already exists. This will still raise an exception if path exists and is not a directory.

Overspend

Using exist_ok=True in makedirs would still raise FileExistsError if target path exists and it is not a directory (file, block device, ...) :( – Shipowner

@Shipowner That's a good point, this isn't a drop-in replacement for catching FileExistsError, although often you do want an exception when you're trying to create a directory on top of a file. – Sighted

Answer 3

Just check if the path exist. if not create it

import os    
if not os.path.exists(test):
    os.makedirs(test)

Breeches

This answer is incorrect and dangerous. It creates a race condition, since the directory could be created by another process after the exists call and before the makedirs call. See my answer for a proper solution. – Twoply

hehe how many miliseconds do you think there will be between the two statements :) but yes you are correct :) – Breeches

It will happen just often enough to drive you crazy trying to debug it :) – Twoply

in was kind of environment are you working? and how many processes are there? – Breeches

Someone famous once said: A one in a million chance is next Tuesday. – Greengrocery

Seems like the scheduler likes to slice between OS calls because this particular case is more likely than you'd think. I just ran into it today with two threads calling into the same library that has basically this code in a @property & with a hardcoded directory name. – Pix

Just to play devils advocate - if your code is the only thing on the system that would or could create that folder then your race condition will never occur. Still, I'd prefer errno.EEXIST as then you don't even have to ask that question. – Nativeborn

Race conditions are good to eliminate, but this thread seems overly focused on the race condition in the example code, when a zillion other more common issues can arise and there is no exception handling here at all. Permission errors? Folder in which you are creating the new folder doesn't exist, or worse, is actually a file and not a folder? Out of drive space? Folder name contains invalid characters or is too long? Networked drive is not accessible? Disk is corrupt? Out of memory? Every one of these and many more are not handled. – Lascivious

Answer 4

Happened to me on Windows, maybe this is the case:

Like you I was trying to :

os.mkdir(dirname)

and got OSError: [Errno 17] File exists: '<dirname>'. When I ran:

os.path.exists(dirname)

I got false, and it drove me mad for a while :)

The problem was: In a certain window I was at the specific directory. Even though it did not exists at that time (I removed it from linux). The solution was to close that window \ navigate to somewhere else. Shameful, I know ...

Harrold

Thank you for bringing my sanity back after 3 hours of desperate debugging! From one terminal I had 'cd mydir', then ssh'ed to another machine, and there was no way I could see that I was still inside 'mydir'... – Rosewater

Answer 5

I also faced the same problem, specially, when the string 'test' contains the multiple directory name. So when 'test' contains the single directory -

if not os.path.exists(test):
    try:
        os.makedir(test)
    except:
        raise OSError("Can't create destination directory (%s)!" % (test))  

If the 'test' contains multiple directory like '\dir1\dir2' then -

if not os.path.exists(test):
    try:
        os.makedirs(test)
    except:
        raise OSError("Can't create destination directory (%s)!" % (test))  

Chump

The if serves no purpose. – Twoply

This if statement protects from overwriting the existing directory with the same name. – Chump

No it doesn't. The operating system will prevent the overwriting whether you check before or not. Unless you are running on a single-threaded operating system, another process can create the directory after the time you check the directory doesn't exist and the time you try to create it. The whole point of catching the exception is to deal with what happens if the directory already exists, whether it came into existence last week or 1 microsecond ago. The if serves no purpose. The try/except is the only reliable approach. – Twoply

Answer 6

You have a file there with the name test. You can't make a directory with that exact same name.

Clarion

Answer 7

Simple answer that does not require any additional import, does not suppress errors such as "permission denied", "no space left on device" etc., yet accepts that the directory may already exist:

import os

try:
    os.mkdir(dirname)
except FileExistsError :
    pass
except :
    raise

Axum

Answer 8

I don't know the specifics of your file system. But if you really want to get around this maybe use a try/except clause?

try:
    os.mkdir(test)
except OSError:
    print "test already exists"

You can always do some kind of debugging in the meanwhile.

Signification

The error message is incorrect. The OSError exception can happen for many reasons, e.g. insufficient permissions or a read-only file system. You can't conclude that the directory already exists. You need to check the value of exc.errno to determine the cause of the exception. – Twoply

Answer 9

Maybe there's a hidden folder named test in that directory. Manually check if it exists.

ls -a

Create the file only if it doesn't exist.

if not os.path.exists(test):
    os.makedirs(test)

Chalfant

This answer is incorrect and dangerous. It creates a race condition, since the directory could be created by another process after the exists call and before the makedirs call. See my answer for a proper solution. – Twoply

If it's hidden, it can't possibly be called test, right? It must be .test which is a different string – Klutz

Answer 10

For python 2.7

from distutils.dir_util import mkpath
mkpath(path)

Aguilera

Answer 11

Didn't see any other answers or comments about this particular solution. But make sure if you are actively managing folder permissions that you sign out and sign back in before running your code. I banged my head against the wall for about an hour trying to figure out what my issue was, and this was it.

Canebrake