Where is the proper place to put Python virtual environments according to the Linux Filesystem Hierarchy Standard?

Question

As the title asks, what is the technically proper location to store Python virtual environments on Linux operating systems according to the Linux FHS?

Stated another way that allows for a clear answer: Is it "technically correct" to separate the location of a Python virtual environment from the data files you are serving?

Note: This question differs from the closest, already-asked question I could find, as virtual-environments contain libraries, binaries, header files, and scripts.

As an added complication, I tend to write code that supports internet-accessible services. However, I don't see this as substantially differentiating my needs from scenarios in which the consumers of the service are other processes on the same server. I'm mentioning this detail in case my responses to comments include "web dev"-esque content.

For reference, I am using the following documentation as my definition of the Linux FHS: http://www.pathname.com/fhs/pub/fhs-2.3.html

I do not believe the popular virtualenv-wrapper script suggests the correct action, as it defaults to storing virtual environments in a user's home directory. This violates the implicit concept that the directory is for user-specific files, as well as the statement that "no program should rely on this location."

From the root level of the file system, I lean towards /usr (shareable, read-only data) or /srv (Data for services provided by this system), but this is where I have a hard time deciding further.

If I was to go alongside the decision of my go-to reverse proxy, that means /usr. Nginx is commonly packaged to go into /usr/share/nginx or /usr/local/nginx, however, /usr/ is supposed to be mounted read-only according to the FHS. I find this strange because I've never worked on a single project in which development happened so slowly that "unmount as read-only/remount with write, unmount/remount as read-only" was considered worth the effort.

/srv is another possible location, but is stated as the "location of the data files for particular service," whereas a Python virtual environment is more focused on libraries and binaries for what provides a service (without this differentiation, .so files would also be in srv). Also, multiple services with the same requirements could share a virtual environment, which violates the "particular" detail of the description.

I believe that part of the difficulty in choosing a correct location is because the virtual environment is an "environment," which consists of both binaries and libraries (almost like its own little hierarchy), which pushes my impression that somewhere under /usr is more conventional:

virtual-env/
├── bin          ~= /usr/local : "for use by the system administrator when installing software locally" 
├── include      ~= /usr/include : "Header files included by C programs"
├── lib          ~= /usr/lib : "Libraries for programming and packages"
└── share        ~= /usr/local

With my assumptions and thoughts stated: consider the common scenario of Nginx acting as a reverse proxy to a Python application. Is it correct to place a virtual environment and source code (e.g. application.py) under /usr/local/service_name/ while using /srv for files that are changed more often (e.g. 'static' assets, images, css)?

edit: To be clear: I know why and how to use virtualenvs. I am by no means confused about project layouts or working in development environments.

Hispanicize

Answer 1

As the title asks, what is the technically proper location to store Python virtual environments on Linux operating systems according to the Linux FHS?

Keep in mind that the Linux FHS is not really a standard, it is a set of guidelines. It is only referred to as a standard by the LSB - which is just a bunch of rules that make supporting Linux easier.

/run, /sys, /proc and /usr/local are all not part of the LFS but you see them in most linux distributions.

For me the clear choice to put virtual environments is /opt, because this location is reserved for the installation of add-on software packages.

However, on most Linux distributions only root can write to /opt, which makes this a poor choice because one of the main goals of virtual environments is to avoid being root.

So, I would recommend /usr/local (if its writable by your normal user account) - but there is nothing wrong with installing it in your home directory.

Stated another way that allows for a clear answer: Is it "technically correct" to separate the location of a Python virtual environment from the data files you are serving?

I'm not sure what you mean by "data files you are serving", but here are the rules for virtual environments:

1. Don't put them in source control.
2. Maintain a list of installed packages, and put this in version control. Remember that virtual environments are not exactly portable.
3. Keep your virtual environment separate from your source code.

Given the above, you should keep your virtual environment separate from your source code.

consider the common scenario of Nginx acting as a reverse proxy to a Python application. Is it correct to place a virtual environment and source code (e.g. application.py) under /usr/local/service_name/ while using /srv for more dynamic files (e.g. 'static' assets, images)?

Static assets are not dynamic files, I think you are confusing terms.

Either way, you should do the following:

1. Create a user account to run that application.
2. Put the application files under a directory that is controlled by that user and that user alone. Typically this is the /home/username directory, but you can make this /services/servicename. Place the virtual environment as a subset of this directory, in a standard naming format. For example, I use env.
3. Put your static assets, such as all media files, css files, etc. in a directory that is readable by your front end server. So, typically you would make a www directory or a public_html directory.
4. Make sure that the user account you create for this application has write access to this asset directory, so that you are able to update files. The proxy server should not have execute permissions on this directory. You can accomplish this by changing the group of the directory to the same as that of the proxy server user. Given this, I would put this directory under /home/username/ or /services/servicename.
5. Launch the application using a process manager, and make sure your process manager switches the user to the one created in step 1 when running your application code.

Finally, I cannot stress this enough DOCUMENT YOUR PROCESS and AUTOMATE IT.

Ocarina

Your comment about opt is exactly in the spirit of why I raised this question. "Static assets are not dynamic files, I think you are confusing terms" -- I can see why you would think this way, but I believe you'd agree that css files change relatively often while not being technically dynamic files. – Hispanicize

And to your point #2, /home/username is an objectively wrong way of doing things as a deployment user can be a system user without a homedir (and you don't want to tie a deployment group down to a single user). These two comments in mind, your statement about the FHS convention being paraded as a standard is a helpful reminder. – Hispanicize

You don't run your applications as the same user which you deploy your application (I assume by deploy you mean, "install on production servers"), because this would require elevated privileges. You don't need a "home directory", you need a standard location, and it can be whatever you chose to be at your organization. CSS files rarely change - not as often as say, images uploaded by users. However, when it comes to web development, static files include CSS files. – Ocarina

I don't see anything particularly terrible about the deploy user binding on a non-privileged port. This entire question revolves around the proper choice for a standard location. And yes, CSS are static files, but I meant in terms of frequency of change -- in my experience (at least), CSS, JavaScript, and image assets change with some regularity. – Hispanicize