How to connect Emacs' Elpy in-buffer python interpreter to docker container?

Question

I am getting started working on a Django app that will run within a Docker container. I would like to be able to use interactive python from within Emacs which executes the code within the context of the Docker container and the Django app. Essentially I want the Python interpreter that is used by Emacs to be running within the Docker container.

I have managed to achieve this for the most part by creating a modified Dockerfile which is only different from my actual Dockerfile container in that instead of calling a Django command it ends with:

CMD ./manage.py shell

This means that when I run that Docker container I get a Django interpreter that is actually running within the container. I have Elpy invoke this when starting a python shell by adding:

(setq python-shell-interpreter "/path_to_my_script/run_shell.sh")

to my .emacs.d/init.el file. Where the "/path_to_my_script/run_shell.sh" script is a script which runs the container.

This is actually working for the most part. I am able to highlight a line of code such as

from django.utils import timezone

and run the Emacs command elpy-shell-send-region-or-buffer and it will execute the code. I can then highlight

print(timezone.now())

and it will print the current time. I have verified that this code is in fact running on my container by running:

import os
print(os.getcwd())

and seeing the home directory for my Docker container's user.

The issue I am still having with this setup is when I try to execute multiple lines of python at the same time using the same elpy-shell-send-region-or-buffer command. When I highlight more than one line and execute that command I get the following error within the python shell:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.5/codecs.py", line 895, in open
    file = builtins.open(filename, mode, buffering)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/py10324GOe'

I have been unable to find any answers to the following questions:

• Why is codecs.py trying to read a tmp file?
• What are all the temporary files created by Python for? (If you have some python code running and look in your /tmp directory you'll probably see lots of file like py271761jE.)
• Why is the /tmp directory empty in my Docker container? It seems like Python is wanting to create and read from files in that directory but is failing.
• How can I debug this?

If anyone has any answers to any of these questions it would be a huge help. I have been unable to turn up any information about this issue. Googling about py* files in the /tmp directory just turns up the tempfile package.

I don't know what's going on but my current theory is that when a region with multiple lines is sent to the python shell it needs to deal with newline characters so it tries to use the codecs file. I think somehow this file relies on temporary files and the docker container is messing that up. I don't know how close this theory is but I don't know how to investigate it further.

Muddleheaded

Answer 1

Why is codecs.py trying to read a tmp file?

I'm pretty sure this is what's happening:

1. Emacs writes code to /tmp/py10324GOe
2. Emacs instructs Python to load /tmp/py10324GOe
3. Python fails because the container cannot access the host's /tmp/. The container has its own, separate /tmp/ directory.

The simplest way to fix this would be to link the host's /tmp/ directory to the container's /tmp/ directory. Some googling leads me to a command like:

docker run ... -v /tmp:/tmp

I haven't tested that code, but it seems like the right direction.

Reciprocate

Answer 2

I had some success by modifying python-shell--save-temp-file function from /usr/local/share/emacs/26.3/lisp/progmodes/python.el.gz

(defun python-shell--save-temp-file (string)
  (let* ((temporary-file-directory
          (if (file-remote-p default-directory)
              (concat (file-remote-p default-directory) "/tmp")
            temporary-file-directory))
         (temp-file-name (make-temp-file "py"))
         (coding-system-for-write (python-info-encoding)))
    ;; vvvv asychronous won't work ;;; change ownership of the tmp file to the user firing python proc                                           
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; inside the container and                                                                                  
    (call-process "docker"  nil nil nil "exec" "-u" "root" "mycontainer" "chown" "pythonuser:"  temp-file-name)
    (call-process "docker"  nil nil nil "exec" "-u" "root" "mycontainer" "chmod" "606"  temp-file-name)
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; my user on the host need also access to that file so o+rw                                                
    (with-temp-file temp-file-name
      (insert string)
      (delete-trailing-whitespace))
    temp-file-name))

Hope someone can give a more clever solution.

Marconigraph

Oh! and as said @BrianMalehorn mounting /tmp is necessary – Marconigraph