Login/Register
Can't connect to Milvus using Pymilvus inside docker. MilvusException: (code=2, message=Fail connecting to server on localhost:19530. Timeout)
Asked Answered
dockerdocker-composefastapietcdmilvus
B

2

6

I'm trying to connect to a Milvus server using Pymilvus. The server is up and running but I can't connect to it: MilvusException: (code=2, message=Fail connecting to server on localhost:19530. Timeout)

I'm running both using docker compose:

version: "3.5"

services:

  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.0
    networks:
      app_net:
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2022-03-17T06-34-49Z
    networks:
      app_net:
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
  
  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.1.4
    networks:
      app_net:
        ipv4_address: 172.16.238.10
    command: ["milvus", "run", "standalone"]
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    ports:
      - "19530:19530"
    depends_on:
      - "etcd"
      - "minio"


  fastapi:
    build: ./fastapi
    command: uvicorn app.main:app --host 0.0.0.0
    restart: always
    networks:
      app_net:
        ipv4_address: 172.16.238.12
    environment:
      MILVUS_HOST: '172.16.238.10'
    depends_on:
      - standalone
    ports:
      - "80:80"
    volumes:
      - pfindertest:/data/fast
    healthcheck:
      test: ["CMD", "curl", "-f", "http://127.0.0.1:80"]
      interval: 30s
      timeout: 20s
      retries: 3
  


networks:
  app_net:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.16.238.0/24
          gateway: 172.16.238.1
    

volumes:
  pfindertest:

Dockerfile

FROM python:3.8

WORKDIR /code

COPY ./requirements.txt /code/requirements.txt

RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt

COPY ./app /code/app

main.py

from fastapi import FastAPI
import uvicorn
from pymilvus import connections

app = FastAPI()

@app.get("/")
def read_root():
    return {"Hello": "World"}

connections.connect(
  alias="default", 
  host='localhost', 
  port='19530'
)

I'm getting the following error:

milvus-1-fastapi-1  | Traceback (most recent call last):
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 115, in _wait_for_channel_ready
milvus-1-fastapi-1  |     grpc.channel_ready_future(self._channel).result(timeout=3)
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/grpc/_utilities.py", line 139, in result
milvus-1-fastapi-1  |     self._block(timeout)
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/grpc/_utilities.py", line 85, in _block
milvus-1-fastapi-1  |     raise grpc.FutureTimeoutError()
milvus-1-fastapi-1  | grpc.FutureTimeoutError
milvus-1-fastapi-1  | 
milvus-1-fastapi-1  | During handling of the above exception, another exception occurred:
milvus-1-fastapi-1  | 
milvus-1-fastapi-1  | Traceback (most recent call last):
milvus-1-fastapi-1  |   File "/usr/local/bin/uvicorn", line 8, in <module>
milvus-1-fastapi-1  |     sys.exit(main())
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
milvus-1-fastapi-1  |     return self.main(*args, **kwargs)
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
milvus-1-fastapi-1  |     rv = self.invoke(ctx)
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
milvus-1-fastapi-1  |     return ctx.invoke(self.callback, **ctx.params)
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
milvus-1-fastapi-1  |     return __callback(*args, **kwargs)
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/main.py", line 404, in main
milvus-1-fastapi-1  |     run(
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/main.py", line 569, in run
milvus-1-fastapi-1  |     server.run()
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/server.py", line 60, in run
milvus-1-fastapi-1  |     return asyncio.run(self.serve(sockets=sockets))
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
milvus-1-fastapi-1  |     return loop.run_until_complete(main)
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
milvus-1-fastapi-1  |     return future.result()
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/server.py", line 67, in serve
milvus-1-fastapi-1  |     config.load()
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/config.py", line 474, in load
milvus-1-fastapi-1  |     self.loaded_app = import_from_string(self.app)
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/importer.py", line 21, in import_from_string
milvus-1-fastapi-1  |     module = importlib.import_module(module_str)
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
milvus-1-fastapi-1  |     return _bootstrap._gcd_import(name[level:], package, level)
milvus-1-fastapi-1  |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
milvus-1-fastapi-1  |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
milvus-1-fastapi-1  |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
milvus-1-fastapi-1  |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
milvus-1-fastapi-1  |   File "<frozen importlib._bootstrap_external>", line 843, in exec_module
milvus-1-fastapi-1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
milvus-1-fastapi-1  |   File "/code/./app/main.py", line 11, in <module>
milvus-1-fastapi-1  |     connections.connect(
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/connections.py", line 262, in connect
milvus-1-fastapi-1  |     connect_milvus(**kwargs, password=password)
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/connections.py", line 233, in connect_milvus
milvus-1-fastapi-1  |     gh._wait_for_channel_ready()
milvus-1-fastapi-1  |   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 118, in _wait_for_channel_ready
milvus-1-fastapi-1  |     raise MilvusException(Status.CONNECT_FAILED, f'Fail connecting to server on {self._address}. Timeout')
milvus-1-fastapi-1  | pymilvus.exceptions.MilvusException: <MilvusException: (code=2, message=Fail connecting to server on localhost:19530. Timeout)>
milvus-1-fastapi-1 exited with code 1

The Milvus server appears to be working so that's not the problem.

NAME                 COMMAND                  SERVICE             STATUS              PORTS
milvus-1-fastapi-1   "uvicorn app.main:ap…"   fastapi             restarting             0.0.0.0:80->80/tcp
milvus-etcd          "etcd -advertise-cli…"   etcd                running             2379-2380/tcp
milvus-minio         "/usr/bin/docker-ent…"   minio               running (healthy)   9000/tcp
milvus-standalone    "/tini -- milvus run…"   standalone          running             0.0.0.0:9091->9091/tcp, 0.0.0.0:19530->19530/tcp

I'm running Docker on Mac if that is important. I tried using gitpod.io but the error remains.

Breena answered 7/11, 2022 at 13:52 Comment(2)
were you able to find a correct fix for this? I am facing a similar issue, a streamlit app needs to talk to the milvus docker, but it just keeps on giving: Waiting for Milvus: <MilvusException: (code=2, message=Fail connecting to server on milvus-standalone:19530, illegal connection params or server unavailable)> – Shaky
Same problem as @Shaky – Coetaneous
A
1

Using pymilvus==2.2.0 worked for me.

Code

from pymilvus import (
    connections,
    utility,
    FieldSchema,
    CollectionSchema,
    DataType,
    Collection,
)

connections.connect("default", host="localhost", port="19530")

print("connected...")
Amiamiable answered 9/6, 2023 at 3:17 Comment(0)
S
0

For me the following solution worked:

  1. Use the container's IP address:

    • Obtain the Milvus container's IP address using docker inspect milvus_container_name.
    • Modify your Streamlit connection string to use this IP explicitly (e.g., "milvus://<container_ip>:19530").

  2. Docker networking:

    • If your Streamlit and Milvus containers reside on different Docker networks, address this:
      • Default bridge network: Make sure both containers are part of the default bridge network.
      • Custom network: Create a custom network (e.g., docker network create my_network) and connect both containers to it.

Explanation:

Docker containers function within their own network namespaces. These potential causes are addressed as follows:

  • IP address: Using the IP directly sidesteps possible DNS resolution issues within the containerized environment.
  • Networking: Establishing proper communication between containers is essential for distributed applications.
Shaky answered 16/4 at 7:29 Comment(1)
This might be of help. @Coetaneous – Shaky

© 2022 - 2024 — McMap. All rights reserved.