R
RunPod5mo ago
Aerotune

Inquiry on Utilizing TensorFlow Serving with GPU in Serverless Configuration

Hello Runpod Community, I'm exploring options to utilize TensorFlow Serving with GPU support in a serverless configuration on Runpod. Specifically, I'm interested in whether it's feasible to make requests from a Runpod serverless job to a TensorFlow Serving instance running on the same container or environment. Could anyone clarify if this setup is supported? Additionally, are there alternative recommended approaches for deploying TensorFlow Serving with GPU on Runpod's serverless infrastructure? Thank you in advance for your assistance! Best regards, Sebastian
6 Replies
Encyrption
Encyrption5mo ago
I don't know of anyone doing it today but check out this link, for a starting point: https://www.tensorflow.org/install/docker
nerdylive
nerdylive5mo ago
yes i guess so, if the inference is done via code its easily setup-able in runpod serverless runpod serverless is already inside docker container so you wouldn't want a docker host there
Encyrption
Encyrption5mo ago
There is a template in Explore but it doesn'th have any info: https://www.runpod.io/console/explore/runpod-tensorflow
nerdylive
nerdylive5mo ago
i guess it comes with tensorflow library preinstalled just like pytorch
Aerotune
AerotuneOP5mo ago
Thank you all for the responses and suggestions! I currently have a local setup with a Runpod, TensorFlow and CUDA container that works well but is quite large (~7 GB). I'm also considering using TensorFlow Serving, which could reduce the image size to less than 1 GB. I'll test both approaches and share my findings once I have more details. This might take some time, but I'll keep you posted! Cheers, Sebastian
Encyrption
Encyrption5mo ago
IMHO 7GB is fine size for a serverless worker image, especially if that includes the models. I have some worker images that are ~ 15GB.
Want results from more Discord servers?
Add your server