Parallel: Import a python file from sibling folder

Question

I have a directory tree

working_dir\
    main.py
my_agent\
    my_worker.py
my_utility\
    my_utils.py

Code in each file is as follows

""" main.py """

import os, sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

from my_agent.my_worker import MyWorker
import ray

ray.init()
workers = [MyWorker.remote(i) for i in range(10)]
ids = [worker.get_id.remote() for worker in workers]
# print(*ids, sep='\n')
print(*ray.get(ids), sep='\n')

""" worker.py """
from my_utility import my_utils
import ray

@ray.remote
class MyWorker():
    def __init__(self, id):
        self.id = id

    def get_id(self):
        return my_utils.f(self.id)

""" my_utils.py """
def f(id):
    return '{}: Everything is fine...'.format(id)

Here's a part of the error message I received

Traceback (most recent call last):

File "/Users/aptx4869/anaconda3/envs/p35/lib/python3.5/site-packages/ray/function_manager.py", line 616, in fetch_and_register_actor unpickled_class = pickle.loads(pickled_class)

File "/Users/aptx4869/anaconda3/envs/p35/lib/python3.5/site-packages/ray/cloudpickle/cloudpickle.py", line 894, in subimport import(name)

ImportError: No module named 'my_utility'

Traceback (most recent call last):

File "main.py", line 12, in print(*ray.get(ids), sep='\n')

File "/Users/aptx4869/anaconda3/envs/p35/lib/python3.5/site-packages/ray/worker.py", line 2377, in get raise value ray.worker.RayTaskError: ray_worker (pid=30025, host=AiMacbook)

Exception: The actor with name MyWorker failed to be imported, and so cannot execute this method

If I remove all statements related to ray, the above code works fine. Therefore, I boldly guess the reason is that ray runs each actor in a new process and sys.path.append only works in the main process. So I add the following code to worker.py

import os, sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))

But it still does not work: the same error message shows up. Now I run out of ideas, what should I do?

Curacy

I consider this os.path.dirname(os.path.dirname(__file__)) better than this os.path.join(os.path.dirname(__file__), '..') – Cadelle

@Cadelle Thanks:-) I guess you suggested sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))). But this does not work either... – Curacy

Have you tried to remove the line if __name__ == '__main__' and __package__ is None:? and simply always add to your path the parent directory? – Cadelle

@Cadelle Yes, the same story goes on – Curacy

@Darkonaut I did what you suggested, still not working. I remembered that python3 has no longer require __init__.py, doesn't it? – Curacy

Yes since Python 3.3 it shouldn't be needed anymore, but my guess was that ray might need it somehow since it's a framework. – Edisonedit

@Edisonedit Thanks. I've added __init__.py to all directory, even the root directory which contains these three folders. It still does not work. – Curacy

Could be a naming conflict. You have utils.py and worker.py and I see ray has these files too. Rename your files with a prefix to avoid the clash. – Edisonedit

@Edisonedit I did what you suggested, and prefix all relative file/class with my,,, still not working... – Curacy

@Edisonedit Yep, I've changed all relative names... The error happens at the print statement. IMHO, if the error is caused by name conflict, it should not happen so late. – Curacy

I cannot speak for ray but at least in standard libs multiprocessing.Pool it's necessary that imports for used functions are made for every distributed task again. I'm not familiar with ray's internals but that line ImportError: No module named 'utils' makes me wonder. It looks like it's trying to import utils from utils.py instead of from the directory. – Edisonedit

@Edisonedit I've updated the code, file names, and the error messages. It can be seen now that the ImportError refers to the folder name my_utility instead of the my_utils.py file. To my best knowledge, I think ray calls from my_utility import my_utils in every new actor process, and that causes this problem. – Curacy

Answer 1

You are correct about what the issue is.

In your example, you modify sys.path in main.py in order to be able to import my_agent.my_worker and my_utility.my_utils.

However, this path change is not propagated to the worker processes, so if you were to run a remote function like

@ray.remote
def f():
    # Print the PYTHONPATH on the worker process.
    import sys
    print(sys.path)

f.remote()

You would see that sys.path on the worker does not include the parent directory that you added.

The reason that modifying sys.path on the worker (e.g., in the MyWorker constructor) doesn't work is that the MyWorker class definition is pickled and shipped to the workers. Then the worker unpickles it, and the process of unpickling the class definition requires my_utils to be imported, and this fails because the actor constructor hasn't had a chance to run yet.

There are a couple possible solutions here.

1. Run the script with something like

   PYTHONPATH=$(dirname $(pwd)):$PYTHONPATH python main.py
   

   (from within working_dir/). That should solve the issue because in this case the worker processes are forked from the scheduler process (which is forked from the main Python interpreter when you call ray.init() and so the environment variable will be inherited by the workers (this doesn't happen for sys.path presumably because it is not an environment variable).

2. It looks like adding the line

   parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
   os.environ["PYTHONPATH"] = parent_dir + ":" + os.environ.get("PYTHONPATH", "")
   

   in main.py (before the ray.init() call) also works for the same reason as above.

3. Consider adding a setup.py and installing your project as a Python package so that it's automatically on the relevant path.

Achondrite

Answer 2

The new "Runtime Environments" feature, which didn't exist at the time of this post, should help with this issue: https://docs.ray.io/en/latest/handling-dependencies.html#runtime-environments. (See the working_dir and py_modules entries.)

Prolate

if you have multi dependencies, use "py_modules": ["path/project_name/module"] – Cortneycorty