Create new directory in Julia for each run

Question

I'm running Julia code which generates a plot and a text file. There exists an "Output" folder in the same folder where the code in question is located. For the first run, I create a "Run_1" folder, with "Plots" and "Data" subfolders:

fig_path = @__DIR__
mkdir(fig_path*"/Output/Run_1/")
mkdir(fig_path*"/Output/Run_1/Plots/")
mkdir(fig_path*"/Output/Run_1/Data/")

After plotting, I save the figure to "Plots":

fig_name = "test_figure"
savefig(fig_path*"/Output/Run_1/Plots/"*fig_name*".pdf")

and the output file (contained within "output_matrix") is saved to "Data":

outfile_1 = fig_path*"/Output/Run_1/Data/test_data.txt"
open(outfile_1, "w") do f1    
    writedlm(f1,output_matrix)    
end

However, I want to run this code multiple times. Each time it runs, it should create a new "Run" folder in the "Output" folder, i.e. on the first run its Run_1, the second run it's Run_2, and so on. All folders from previous runs are NOT deleted. In each Run folder, there's a "Plots" and a "Data" folder, and I save the plot and data to their respective folders in each run. How can I have Julia update the file name in such a manner?

Hackney

As an aside, you could use joinpath to build paths in an more portable way – Mulley

Also, don't hesitate to take a look at DrWatson: it's a Julia package specifically designed to help managing various aspects of scientific projects. Among which: naming simulations in order to consistently derive paths from simulation parameters (in a deterministic, reproducible way) – Mulley

Answer 1

The ispath function checks whether a file or directory exists in the filesystem.

If you want to keep a naming convention like Run_1...Run_N, something like this could help:

function mk_output_dir()
    i = 1
    while true
        dir_name = joinpath(@__DIR__, "Output", "run_$i")
        if !ispath(dir_name)
            mkpath(dir_name)
            return dir_name
        end
        i += 1
    end
end

This produces:

# First run
julia> top_dir = mk_output_dir()
"/tmp/Output/run_1"

julia> mkdir(joinpath(top_dir, "Plots"))
"/tmp/Output/run_1/Plots"

julia> mkdir(joinpath(top_dir, "Data"))
"/tmp/Output/run_1/Data"

# Second run
julia> top_dir = mk_output_dir()
"/tmp/Output/run_2"

julia> mkdir(joinpath(top_dir, "Plots"))
"/tmp/Output/run_2/Plots"

julia> mkdir(joinpath(top_dir, "Data"))
"/tmp/Output/run_2/Data"

Be aware that race conditions could occur if you start two instances of your program at the same time.

---

Alternatively, I personally tend to use naming conventions involving timestamps when creating directory structures like this. Here would be a minimal example:

using Dates
function mk_output_dir()
    timestamp = Dates.format(now(), "YYYYmmdd-HHMMSS")
    dir_name = joinpath(@__DIR__, "Output", "run_$timestamp")
    @assert !ispath(dir_name) "Somebody else already created the directory"
    mkpath(dir_name)
    return dir_name
end

which produces something like this:

julia> top_dir = mk_output_dir()
"/tmp/Output/run_20201229-210835"

julia> mkdir(joinpath(top_dir, "Plots"))
"/tmp/Output/run_20201229-210835/Plots"

julia> mkdir(joinpath(top_dir, "Data"))
"/tmp/Output/run_20201229-210835/Data"

Augmentation

Answer 2

Maybe something like this:

function mkresultdir(fig_path)
    for i=1:1000
        rundir = joinpath(fig_path, "run_$i")
        if !isdir(rundir)
            mkdir(rundir)
            return rundir
        end
     end
     error("too many results on disk, time for a cleanup!")
 end

res_dir_1 = mkresultdir("/home/my_user/results")
res_dir_2 = mkresultdir("/home/my_user/results")

Farrish