Is there a way to get the config.pbtxt file from triton inferencing server

Question

Recently, I have come across a solution of the triton serving config file disable flag "--strict-model-config=false" while running the inferencing server. This would enable to create its own config file while loading the model from the model repository.

sudo docker run --rm --net=host -p 8000:8000 -p 8001:8001 -p 8002:8002 \
-v /home/rajesh/custom_repository:/models nvcr.io/nvidia/tritonserver:22.06-py3 \
tritonserver --model-repository=/models --strict-model-config=false

I would like to get the generated config file from the triton inferencing server since we can play around with the batch config and other parameters. Is there a way to get the inbuilt generated config.pbtxt file for the models I have loaded in the server so that I can play around the batch size and other parameters.

Venlo

Answer 1

As per Triton docs (source), the loaded model configuration can be found by curl'ing the /config endpoint:

Command:

curl localhost:8000/v2/models/<model_name>/config

[source]

Venlo

Answer 2

The above answer which the uses curl command would return the json response.

If the results should be in the protobuf format, try loading the model using triton inferencing server with strict model config as false and fetch the results by using the below python script which would return the results in necessary protobuf format. Use this to get the format of the model and edit it easily as per the needs in config pbtxt file instead of cnoverting json to protobuf results.

import tritonclient.grpc as grpcclient

triton_client = grpcclient.InferenceServerClient(url=<triton_server_url>)

model_config = triton_client.get_model_config(model_name=<model_name>, model_version=<model_version>)

Venlo