How can I make my python script wait for a shutil.move to be actually completed?

Question

A python script I'm using is at one point moving a list of files from one directory to the other. I have implemented it with shutil.move:

for a_file in list_of_filenames:
    src = os.path.join(src_dir, a_file)
    dest = os.path.join(dest_dir, a_file)
    shutil.move(src, dest)

Immediately after that, I am writing this list of filenames in a named pipe, which is plugged in my main program (written in C). It then proceeds to reading those files, assuming they have reached their destination directory. The problem is, if I don't tell my python script to wait a couple of seconds before writing to the pipe, my main program chokes saying one of the file does exist.

From my research so far, and the limited understanding I have of the problem, it seems that the OS can notify my script that the move is complete when in fact it is not yet physically done.

Now waiting a couple of seconds does not sound that bad, but what if I have to move 100, 1000 or even 10000 files ? Will it be sufficient, or do I have to wait longer? How do I actually make sure that my files have been moved before processing them ?

My idea so far would be something like this:

was_moved = [False for _ in range(len(list_of_files))]
while not all(was_moved):
    for i, a_file in enumerate(files):
        if was_moved[i]:
            continue

        try:
            # try to open the file in read mode to see if it actually exists
            open_file = open(os.path.join(dest_dir, a_file), "r")
        except FileNotFoundError:
            continue

        open_file.close()
        was_moved[i] = True

This feels like awkward programming though, and I'm not even sure the open is properly testing for the file, or if the time taken to run the loop is what makes the move successful. Any insight or better idea to achieve this would be welcome.

Aviator

I don't think shutil.move would return until it had moved the files. See Is python's shutil.move() atomic on linux? – Writein

You can check if a file exists with os.path.exists or os.path.isfile – Writein

@PeterWood Yes I usually use either of those but I'm affraid it's checking for the file handle only and not the actual content. I'm not sure open performs better if you don't try a read too though... – Aviator

Concerning your first comment, here is what made me concerned about this issue: reddit.com/r/Python/comments/7ekdwd/… and bugs.python.org/issue22024 – Aviator

Answer 1

You can use subprocess.call() to invoke a command line action to accomplish what you want. It won't return until the subprocess is complete, meaning that your files have moved:

on linux:

import subprocess
for a_file in list_of_filenames:
    src = os.path.join(src_dir, a_file)
    dest = os.path.join(dest_dir, a_file)
    subprocess.call('mv ' + src + ' ' + dest)

on windows:

import subprocess
for a_file in list_of_filenames:
    src = os.path.join(src_dir, a_file)
    dest = os.path.join(dest_dir, a_file)
    subprocess.call('move ' + src + ' ' + dest, shell=True)

Shakira

Not an optimal solution but a working one I guess, thanks for you answer. – Aviator

Indeed an optimal solution. Works perfectly well – Saylor

Answer 2

I know this is an old question, but I ran into this problem recently myself. To make sure the files are actually moved, the changes need to be synced to disk. Python has a neat call for it since Python 3.3.

os.sync()

For older versions of python you could use:

from subprocess import check_call
check_call(['sync'])

Spokeshave

os.sync() is only available on Unix and not Windows per docs.python.org/3/library/os.html?highlight=sync#os.sync and my own trial and error. – Gerri