ModuleNotFoundError: No module named 'main' but path seems correct

Question

The structure of my_dir is

├── README.md
├── main
│   ├── functions
│   │   ├── __pycache__
│   │   ├── my_function.py
│   ├── pipeline.py
│   ├── options
│   │   └── pipeline_options.py
│   └── transforms
│       ├── __pycache__
│       ├── my_transform.py
├── poetry.lock
├── pyproject.toml
└── tests

On pipeline.py:

from main.functions.my_function import MyFunction

On my_function.py:

import apache_beam as beam

class MyFunction(beam.DoFn):
...

I have read similar questions here, including this one which is the solution I have currently. Also read this on Python imports.

When I run pipeline.py, I get

Traceback (most recent call last):
  File "pipeline.py", line 7, in <module>
    from main.functions.my_function import MyFunction
ModuleNotFoundError: No module named 'main'

Additionally, I'm using VSCode and if I run the file via "Python: Run Python File in Terminal" I get back the error. However, if I run via the debugger the paths are all found and the error is not thrown, which I found odd. Also, VSCode isn't throwing any path warnings.

I know there are similar questions but I haven't been able to figure out what is wrong here and have spent quite some time on this already. Any help/pointer is much appreciated.

Additional info

• I'm using python 3.8.1
• Using poetry run python main/pipeline.py to run the code
• Running from command from my_dir

Ambrogio

Have you tried adding __init__.py files to your directories to indicate they should be treated as packages? – Rittenhouse

in pipeline you should directly call from functions.my_function import MyFunction – Vivi

I had before but I'm using python 3.8.1 and __init__ files are only needed for Python <3.5 – Ambrogio

I have tried that too, it returns the same error and is also flagged by VSCode (unresolved import 'functions.my_function'Python(unresolved-import)) – Ambrogio

@Ambrogio Are you running your Python file from the directory where main directory is located? Or are you running it from main directory? – Padishah

Running it from where main is located (meaning if I run ls, main is in the results) – Ambrogio

Have you tried to do in your pipeline.py script before the imports statments : os.sys.path.append('../main/functions') and after that you only do from my_function import MyFunction – Squirmy