TL;DR - This is a PyCharm remote interpreter question.
Remote libraries are not properly synced, and PyCharm is unable to index properly when using remote interpreter. Everything runs fine.
Following is the entire (currently unsuccessful) debug process
See update section for a narrowing down of the problem
I am using a virtual environment created with python -m venv venv, then pointing to it as I always have using ssh interpreter. The exact same happens with conda as well.
After configuring the interpreter, many of the installed packages are marked red by PyCharm, not giving auto complete, and not knowing these packages.
Here is the requirements.txt file, which is used with pip install -r requirements.txt
--index https:<our_internal_pypi_server>
--extra-index-url <some_external_pypi_server>
algo_api>=2.5.0
algo_flows>=2.4.0
DateTime==4.7
fastapi==0.88.0
imagesize==1.4.1
numpy==1.23.1
opencv_python==4.6.0.66
overrides==6.1.0
pydantic==1.9.0
pymongo==4.1.1
pytest==7.1.2
pytorch_lightning==1.6.4
PyYAML==6.0
scikit_learn==1.1.3
setuptools==59.5.0
tinytree==0.2.1
#torch==1.10.2+cu113
#torchvision==0.11.3+cu113
tqdm==4.64.0
uv_build_utils==1.4.0
uv_python_utils>=1.11.1
allegroai
pymongo[srv]
Here is pip freeze
absl-py==1.3.0
aggdraw==1.3.15
aiohttp==3.8.3
aiosignal==1.3.1
albumentations==1.3.0
algo-api==2.5.0
algo-flows==2.4.0
allegroai==3.6.1
altair==4.2.0
amqp==5.1.1
anomalib==0.3.2
antlr4-python3-runtime==4.9.3
anyio==3.6.2
astunparse==1.6.3
async-timeout==4.0.2
attrs==20.3.0
bcrypt==4.0.1
bleach==5.0.1
boto3==1.26.34
botocore==1.29.34
cachetools==5.2.0
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==2.1.1
clearml==1.8.3
click==8.1.3
commonmark==0.9.1
contourpy==1.0.6
cpu-cores==0.1.3
cryptography==38.0.4
cycler==0.11.0
DateTime==4.7
decorator==5.1.1
deepmerge==1.1.0
dnspython==2.2.1
docker-pycreds==0.4.0
docopt==0.6.2
docutils==0.19
dotsi==0.0.3
efficientnet==1.0.0
einops==0.6.0
entrypoints==0.4
fastapi==0.88.0
ffmpy==0.3.0
fire==0.5.0
Flask==2.2.2
flatbuffers==1.12
focal-loss==0.0.7
fonttools==4.38.0
frozenlist==1.3.3
fsspec==2022.11.0
furl==2.1.3
future==0.18.2
gast==0.4.0
gitdb==4.0.10
GitPython==3.1.29
google-auth==2.15.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
gradio==3.15.0
grpcio==1.51.1
gunicorn==20.1.0
h11==0.14.0
h5py==3.7.0
httpcore==0.16.3
httpx==0.23.1
humanfriendly==9.2
idna==3.4
image-classifiers==1.0.0
imageio==2.23.0
imagesize==1.4.1
imgaug==0.4.0
importlib-metadata==5.2.0
importlib-resources==5.10.1
imutils==0.5.4
inflection==0.5.1
iniconfig==1.1.1
itsdangerous==2.1.2
jaraco.classes==3.2.3
jeepney==0.8.0
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.2.0
jsonschema==3.2.0
keras==2.9.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
keyring==23.13.1
kiwisolver==1.4.4
kmeans1d==0.3.1
kornia==0.6.8
libclang==14.0.6
linkify-it-py==1.0.3
luqum==0.11.0
Markdown==3.4.1
markdown-it-py==2.1.0
MarkupSafe==2.1.1
maskrcnn-benchmark==1.1.2+cu113
matplotlib==3.6.2
mdit-py-plugins==0.3.3
mdurl==0.1.2
ml-distillery==1.0.1
more-itertools==9.0.0
multidict==6.0.3
networkx==2.8.8
numpy==1.23.1
oauthlib==3.2.2
omegaconf==2.3.0
opencv-python==4.6.0.66
opencv-python-headless==4.6.0.66
opt-einsum==3.3.0
orderedmultidict==1.0.1
orjson==3.8.3
overrides==6.1.0
packaging==22.0
pandas==1.5.2
paramiko==2.12.0
pathlib==1.0.1
pathlib2==2.3.7.post1
pathtools==0.1.2
pika==1.3.1
Pillow==9.3.0
pkginfo==1.9.2
pluggy==1.0.0
ply==3.11
promise==2.3
protobuf==3.19.6
psd-tools==1.9.23
psutil==5.9.4
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyclipper==1.3.0.post4
pycocotools==2.0.6
pycparser==2.21
pycpd==2.0.0
pycryptodome==3.16.0
pydantic==1.9.0
pyDeprecate==0.3.2
pydub==0.25.1
pygit2==1.11.1
Pygments==2.13.0
pyhumps==3.8.0
PyJWT==2.4.0
pymongo==4.1.1
PyNaCl==1.5.0
pyparsing==2.4.7
pyrsistent==0.19.2
pytest==7.1.2
python-dateutil==2.8.2
python-multipart==0.0.5
pytorch-lightning==1.6.4
pytz==2022.7
PyWavelets==1.4.1
PyYAML==6.0
qudida==0.0.4
readme-renderer==37.3
requests==2.28.1
requests-oauthlib==1.3.1
requests-toolbelt==0.10.1
rfc3986==1.5.0
rich==12.6.0
rsa==4.9
s3transfer==0.6.0
scikit-image==0.19.3
scikit-learn==1.1.3
scipy==1.9.3
SecretStorage==3.3.3
segmentation-models==1.0.1
sentry-sdk==1.12.1
setproctitle==1.3.2
shapely==2.0.0
shortuuid==1.0.11
six==1.16.0
sklearn==0.0.post1
smmap==5.0.0
sniffio==1.3.0
starlette==0.22.0
tensorboard==2.9.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.9.1
tensorflow-estimator==2.9.0
tensorflow-io-gcs-filesystem==0.29.0
termcolor==2.1.1
threadpoolctl==3.1.0
tifffile==2022.10.10
timm==0.5.4
tinytree==0.2.1
tomli==2.0.1
toolz==0.12.0
torch==1.10.2+cu113
torchmetrics==0.9.0
torchtext==0.11.2
torchvision==0.11.3+cu113
tqdm==4.64.0
twine==4.0.2
typing-utils==0.1.0
typing_extensions==4.4.0
uc-micro-py==1.0.1
urllib3==1.26.13
uv-build-utils==1.4.0
uv-envyaml==2.0.1
uv-python-serving==2.0.1
uv-python-utils==1.12.0
uvicorn==0.20.0
uvrabbit==1.4.1
validators==0.20.0
vine==5.0.0
wandb==0.12.17
webencodings==0.5.1
websockets==10.4
Werkzeug==2.2.2
windshield-grid-localisation==1.0.0.dev5
wrapt==1.14.1
yacs==0.1.8
yarl==1.8.2
zipp==3.11.0
zope.interface==5.5.2
The following minimal test program
import pytest
import uv_python_utils
from importlib_metadata import version as version_query
from pkg_resources import parse_version
import requests
installed_pytest_version = parse_version(version_query('pytest'))
installed_uv_python_utils_version = parse_version(version_query('uv_python_utils'))
installed_importlib_metadata_version = parse_version(version_query('importlib_metadata'))
print(installed_pytest_version)
print(installed_uv_python_utils_version)
print(installed_importlib_metadata_version)
runs with output
7.1.2
1.12.0
5.2.0
but in the IDE, it looks like this:
Here is the support ticket for JetBrains (not sure if visible for everyone or not). They were not able to help yet.
They offered, and I have done all of the following which did not help:
- Delete
~/.pycharm_helperson remote - Go to Help | Find Action... and search for "Registry...". In the registry, search for python.use.targets.api and disable it. Reconfigure your project interpreter.
They looked in "the logs" (not sure which log), coming from Help --> "Collect Logs and Diagnostic Data", and saw the following
at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:92)
2022-12-15 11:14:42,932 [ 478638] WARN - net.schmizz.sshj.xfer.FileSystemFile - Could not set permissions for C:\Users\noam.s\AppData\Local\JetBrains\PyCharm2022.3\remote_sources\-2115534621\.\site-packages__1.zip to 1a4
2022-12-15 11:14:42,986 [ 478692] WARN - net.schmizz.sshj.xfer.FileSystemFile - Could not set permissions for C:\Users\noam.s\AppData\Local\JetBrains\PyCharm2022.3\remote_sources\-2115534621\.\.state.json to 1a4
2022-12-15 11:14:43,077 [ 478783] WARN - net.schmizz.sshj.xfer.FileSystemFile - Could not set permissions for C:\Users\noam.s\AppData\Local\JetBrains\PyCharm2022.3\remote_sources\-2115534621\.\python3.8.zip to 1a4
I could not find any permission irregularities though.
I also tried to purge everything from Pycharm from both local and remote, and reinstall, and this persists.
- Uninstall PyCharm, resinstall an older version that works for a colleague (works on the same remote in the same directory for the colleague, so the problem is local)
- Delete .idea
- Delete
C:\Users\noam.s\AppData\Roaming\JetBrains - Obviously I tried invalidate caches & restart.
The libraries just don't get downloaded to the External Libraries [See update below], as shown in the Project menu, which doesn't agree with pip freeze
In the venv case:
In the conda case, the downloaded remote libraries don't even agree with the Pycharm interpreter screen!
This really makes it hard for me to work and I am not able to find any workaround. Any ideas?
Update - The problem occurs when Pycharm tries to unpack from skeletons.zip.
I found a workaround to avoid the "reds":
- Open the Remote Libraries in explorer
- Delete that folder.
- Manually extract the folder from skeletons.zip
- Reindex pycharm
This gave the folowing warnings:
! Attempting to correct the invalid file or folder name
! Renaming C:\Users\noam.s\AppData\Local\Temp\Rar$DRa30340.29792\756417188\uvrabbit\aux.py to C:\Users\noam.s\AppData\Local\Temp\Rar$DRa30340.29792\756417188\uvrabbit\_aux.py
but allowed me to start working. This is not a valid solution in my opinion though, as it required manual handling, rather then let the IDE do it's one job.
- Why does this happen?
- How to fix it?
- How to avoid it?
