How does one handle multiple modules/packages in python?

Question

I've spent hours researching this problem, and I'm still baffled. Please find my ignorance charming.

I'm building a python program that will allow me to pit two AIs against each other in a game of battleship.

Here's my directory structure:

.
├── ais_play_battleship
│   ├── game.py
│   ├── __init__.py
│   ├── player.py
│   └── ship.py
├── LICENSE
├── README.md
└── tests
    └── ship_test.py

2 directories, 7 files

Currently, I'm trying to write ship_test.py, but I cannot seem to import ais_play_battleship.ship. I get the dreaded "ModuleNotFoundError"

Here's what my research has taught me about my problem:

• If you want to import python code from another directory, you should make that directory a package instead of a module. Therefore, I've placed an __init__.py file in the root of ais_play_battleship.
• Python will only search the directory python is launched from as well as the directory of the script you're running. Therefore, I've been trying to launch my tests by running python3 tests/ship_tests.py from the root directory.

Here are my specific questions:

• Why is the error a "ModuleNotFound" error? Shouldn't it be "PackageNotFound"?
• Am I correct to make ais_play_battleship a package?
• How can I keep my tests in a separate directory and still use the code in ais_play_battleship?

Please forgive me, as I'm not very good at asking questions on StackOverflow. Please tell me how I can improve.

Yugoslavia

I suspect that you've neglected to set your path to include ais_play_battleship. – Armlet

Are you suggesting I do something like the following? import sys sys.path.insert(0, '/path/to/application/app/folder') Also, I thought that if I was running the program from the parent directory, I didn't need to. Is this wrong? – Yugoslavia

Answer 1

I am answering my own question, as I haven't yet received a satisfactory answer. The best resource I've found is available here. In summary:

Python does NOT search the directory you run python from for modules. Furthermore, adding an __init__.py file to make a directory a package is not enough to make it visible to an instance of python running in another folder. You must also install that package. Therefore, the only two ways to access a module in another directory are:

1. Install the packaged module in site-packages (this requires the creation of a setup.py file and installation using pip install . More information is available here.
2. Modify path to resolve the module

I ended up settling with the second option, for reasons discussed below.

The first option requires one to reinstall the package at every change to a package. This is difficult on a constantly-changing codebase, but can be made easier by using build automation. However, I'd like to avoid this added complexity.

I shied away from the second option for a long time, because it seemed that modifying the path would require hard-coding the absolute path to my module, which is obviously unacceptable, as every developer would have to edit that path to fit their environment. However, this guide provides a solution to this problem. Create a ./tests/context.py file with the following contents:

import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))

Then, in my ship_test.py module, I imported the context and the module I needed:

import context
import ais_play_battleship.ship
# (I include the submodule ship because ais_play_battleship itself does not have
# any attributes or methods, and the submodule ship is the only one I am testing
# in ship_test.py)

This fits my project better, because it works as expected without having to worry about installing my package (or the method by which my package was installed).

Yugoslavia

re:"The first option requires one to reinstall the package at every change to a package. This is difficult on a constantly-changing codebase, but can be made easier by using build automation. However, I'd like to avoid this added complexity." For development, you can do what's called an "editable" install : pip install -e . – Cohere

Answer 2

To solve this problem without relying on hacking about your sys.path, create a setup.py file and as a build step for your test runner, have it run pip install . first. You might want to use a tool like tox.

In the top level directory:

setup.py

from setuptools import setup

setup(name='ais_play_battleship')

tox.ini

[tox]
envlist = py36, py37

[testenv]
deps=pytest
commands=
    pip install . --quiet
py.test -q

then run your tests (in this example we use tox to do this so that we can also configure how the test environment can be configured) : tox

Cohere

Thanks for your help. I'm not going to worry about tox at this point, I'm just concerned with understanding imports and modules and packages. What does setup.py do and why do I need it? What does pip install . do? – Yugoslavia

If you want a comprehensive answer, I would suggest reading this : python-packaging.readthedocs.io/en/latest Otherwise, the gist of it is that pip is just a python package manager, and setup.py is a magic file that the pip python package manager looks for when it wants to install a directory of code that an author wants to bundle together as a single "python package". You can think of it as configuration for how to setup a set of files and subdirectories from some implicit project root directory declare and make modules available to your python PATH. – Cohere

I read the page you linked to, and it was helpful in solving my problem. Thanks! – Yugoslavia

Okay, I spoke too soon. I'm still having problems. I posted my own answer because I thought I had it figured out, but I was wrong. Then, I tried exploring tox and I haven't got that to work either. Currently the error I'm dealing with reports no module named ais_play_battleship. Here's the full ouput of tox. – Yugoslavia

Answer 3

Run tests/ship_test.py as a module

python -m tests.ship_test

Lueck