Temporary directory persist across program runs

Question

I need a temporary directory, but I want full control over its creation and deletion.

I will use this directory to place git repositories which I want to monitor for new commits, so I need to store them somewhere permanently.

Therefore I want to avoid /tmp dir, since it can be cleared by user(?). What is the best practice for this?

Kansu

Answer 1

https://pypi.python.org/pypi/platformdirs is a Python package that offers a cross-platform user_cache_dir function.

Despoil

Wow, thanks. I guess this is the way to go. I'm just wondering user_cache_dir vs site_data_dir, which is more suitable for my case. – Kansu

It seems appdirs has been deprecated. Enter platformdirs. – Morpheus

Answer 2

tempfile.mkdtemp will create a temp dir for you and return its name. It will create it in /tmp by default (on Unix-like systems), but "in the most secure manner possible" and with read/write/list permissions only for the caller's user id.

>>> d = tempfile.mktemp()
>>> with open(os.path.join(d, "secret")) as output:
...     output.write("Ha, you can't read this!")

(Btw., on a Unix/Linux system with default settings, users can't just edit or remove each others' files from /tmp.)

Absenteeism

Hehum, I was looking at this. So, I can't rely that once created tempdir will always be there? Also, I need to remember once created dir path and use it on every app run, right? Do you recommend creating one top level tmpdir for all my subdirs (git repos) or to separate each in it's own tempdir? Thanks. – Kansu

@umpirsky: do you want the temp dir to persist across program runs? – Absenteeism

This will most likely just create a directory in /tmp which is the default for tempfile.tempdir. – Askja

Maybe I put my question wrong, sorry. I prefer to persist data accross program runs, and computer reboots if possible. So it would be better to put it somewhere inside app install dir os.path.dirname(__file__), '../data/cache'). – Kansu

@umpirsky: that would require write access to the installation dir. Why not put it in the user's home dir? os.getenv("HOME"). – Absenteeism

@larsmans You are right, permissions. Hehum, will HOME var be always defined? I would like to somehow hide this caching from app users. – Kansu

@umpirsky: HOME is practically always defined. If it's not, the user's setup is so broken that you should just kill the program. – Absenteeism

Windows and MacOSX as well? Where do you recommend to put is inside HOME? – Kansu

@umpirsky: on Mac OS X yes, not sure about Windows. – Absenteeism

not in CPython standard lib. Third-party toolkits that are more desktop-oriented, like QT, have their own cross-platform functions to find the right directory. See my answer below for the best approach in a pure-python world. – Disorganization

"will HOME var be always defined?" in the Nix sandbox, HOME is defined as /homeless-shelter but does not exist. mkdir $HOME throws permission denied. so you could try xdg.xdg_cache_home() and fallback to /tmp – Guggle

Answer 3

I'd say that the best practice is to use tempfile.mkdtemp.

If you don't wan to use /tmp then, you can take advantage of the prefix parameter:

import tempfile
tempfile.mkdtemp(prefix=<your_preferred_directory>)

Edit: Regarding what's the most appropriate directory to sotre your application configuration, cache data, etc. If you're using linux, please have a look at the XDG Base Directory Specification.

Vaso

Answer 4

If it's really temporary, follow larmans' advice and use mkdtemp.

If it's some sort of semi-permanent cache that must survive reboots, then you should use the local application directory, as defined by your OS (%APPDATA%, ~/.local/ etc); some toolkits (e.g. Qt) provide functions to look that folder up in a cross-platform manner.

Edit: from Wikipedia:

• HOME (Unix-like) and USERPROFILE (Microsoft Windows) - indicate where a user's home directory is located in the file system.
• HOME/{.AppName} (Unix-like) and APPDATA{DeveloperName\AppName} (Microsoft Windows) - for storing application settings. Many open source programs incorrectly use USERPROFILE for application settings in Windows - USERPROFILE should only be used in dialogs that allow user to choose between paths like Documents/Pictures/Downloads/Music, for programmatic purposes use APPDATA (roaming), LOCALAPPDATA or PROGRAMDATA (shared between users)

So you should look up os.environ['APPDATA'] or os.environ['HOME'], depending on platform (see sys.platform) and then append your app name, and then you can store there anything you want.

mydir = os.path.join( ".myAppName", "cache")
homeVar = 'HOME'  # default for all *nix variants
if sys.platform == 'win32': 
   homeVar = 'APPDATA'
mydir = os.path.join( os.environ[homeVar], mydir)

Disorganization

Answer 5

Usually programs use a ~/.progname directory to store data that should be persistent but should stay "out of the way" of the user.

Askja

Answer 6

Just a though: You may want to look into git commit hooks. That way, instead of monitoring a tmp directory for new commits (sounds strange: who would commit into a tmp directory with limited permissions?) the repo informs you about commits, or, more specifically, automatically runs a script whenever a commit occurs..

Rhapsodist

I am building some kind of git monitor, which will notify about changes in master branch. Sth like githubs RSS feed. In order to detect changes, I need to fetch repo somewhere. That's why I need tmp dir. Thx, – Kansu

Wouldn't you want to have it in a permanent directory then, so you don't have to clone the entire repo every time you want to check for changes? – Rhapsodist