Serve Tensorflow model without installing Tensorflow

Question

I have a trained model that I want to use in a python application, but I can't see any examples of deploying to production environment without installing TensorFlow or creating gRPC service. Is it possible at all? What is the right approach in this situation?

Sellingplater

"Model" is a plain-text or binary description of a computational graph and a set of pre-computed parameters for this graph. How do you imagine to run the model without a library that implements operations of this graph? – Dicarlo

Maybe with some sort of object serialization? – Sellingplater

Answer 1

How do you want to serve it if not using TensorFlow itself or TensorFlow serving? Do you plan on reimplementing the TensorFlow operations to get the same semantics?

That said, with XLA there is now a way to compile a TensorFlow model into a binary which can be called from C++. See the documentation on tfcompile for an example.

Pliant

Thanks. I didn't know this feature. The links should be tensorflow.org/performance/xla/tfcompile – Sellingplater

Answer 2

You can deploy a tensorflow model without tensorflow by using NVIDIA's TensorRT deep learning inference library, which is now compatible with tensorflow since version 3 of the library. It is tailored for inference so it is a very good choice if you fullfill its requirements.

However, it won't work for you if you plan to do inference on CPU or on a platform that is not supported by TensorRT (e.g. Windows).

Vahe

loading the model needs tf tf.saved_model.load – Sapowith