How do I specify the "model_config_file" variable to tensorflow-serving in docker-compose?

Question

I will preface this by saying that I am inexperienced with docker and docker-compose. I am trying to convert my docker run ... command to a docker-compose.yml file, however, I cannot get the models.config file to be found.

I am able to correctly run a tensorflow-serving docker container using the following docker run ... command:

docker run -t --rm \
  tensorflow/serving \
  -p 8501:8501 \
  -v "$(pwd)/models/:/models/" \
  --model_config_file=/models/models.config \
  --model_config_file_poll_wait_seconds=60

This works as expected, and the models.config file is found in the container at /models/models.config as expected.

The tensorflow-serving pages do not mention anything about docker-compose, however, I would much rather use this than a docker run ... command. My attempt at a docker-compose file is:

version: '3.3'
services:
  server:
    image: tensorflow/serving
    ports:
      - '8501:8501'
    volumes:
      - './models:/models'
    environment:
      - 'model_config_file=/models/models.config'
      - 'model_config_file_poll_wait_seconds=60'

Using this docker-compose file, the container runs, however, it seems like the environment variables are completely ignored, so I'm not sure if this is how I should set them. The container image looks in the default location for the models.config file, and it doesn't exist there, and so it does not load the configuration defined in models.config.

So, how can I define these values, or run a tensorflow-serving container, using docker-compose?

I appreciate any help.

Thanks

Osteitis

Try without the quotes or like that : environment: model_config_file: /models/models.config – Crumple

@BatchenRegev I have found a solution - see my posted answer; add those options under a command section. – Osteitis

Answer 1

So I have come across a solution elsewhere that I haven't found on any threads/posts/etc that talk about tensorflow/serving, so I will post my answer here.

Adding those options underneath a command section, as follows, works.

version: '3.3'
services:
  server:
    image: tensorflow/serving
    ports:
      - '8501:8501'
    volumes:
      - './models:/models'
    command:
      - '--model_config_file=/models/models.config'
      - '--model_config_file_poll_wait_seconds=60'

I don't know a lot about docker so I don't know if this is an obvious answer, but I didn't find a solution even after a lot of Google'ing.

Osteitis

Answer 2

The problem is that the option --model_config_file is not an environment variable. It is an argument that you can pass to the command that is set as an entrypoint in the image tensorflow/serving. If we look at the Dockerfile that was used to build the image, we can see :

# Create a script that runs the model server so we can use environment variables
# while also passing in arguments from the docker command line
RUN echo '#!/bin/bash \n\n\
tensorflow_model_server --port=8500 --rest_api_port=8501 \
--model_name=${MODEL_NAME} --model_base_path=${MODEL_BASE_PATH}/${MODEL_NAME} \
"$@"' > /usr/bin/tf_serving_entrypoint.sh \
&& chmod +x /usr/bin/tf_serving_entrypoint.sh

ENTRYPOINT ["/usr/bin/tf_serving_entrypoint.sh"]

This script accept various arguments (you can see them by running docker run -t --rm tensorflow/serving --help)

As far as I know, the only environment variables used by TF serving are MODEL_VERSION and MODEL_NAME. model_config_file and model_config_file_poll_wait_seconds are just arguments to the CLI serving executable.

To achieve the result you are after, you could override the entrypoint of the docker image by setting your own entrypoint in the docker-compose.yml :

version: '3.3'
services:
  server:
    image: tensorflow/serving
    ports:
      - '8501:8501'
    volumes:
      - './models:/models'
    entrypoint:
      - /usr/bin/tf_serving_entrypoint.sh
      - --model_config_file=/models/models.config
      - --model_config_file_poll_wait_seconds=60

(Note : I did not test that docker-compose.yml file)

Proletariat