Testing Endpoint in Local with Docker and GPU
I’m working on creating a custom container to run FLUX and Lora on Runpods, using this Stable Diffusion example as a starting point. I successfully deployed my first pod on Runpods, and everything worked fine.
However, my issue arises when I make code changes and want to test my endpoints locally before redeploying. Constantly deploying to Runpods for every small test is quite time-consuming.
I found a guide for local testing in the Runpods documentation here (https://docs.runpod.io/serverless/workers/development/local-testing). Unfortunately, it only provides a simple example that suggests running the handler function directly, like this:
This does not work for me as it ignores the Docker setup entirely and just runs the function in my local Python environment. I want to go beyond this and test the Docker image end-to-end locally—on my GPU—with the exact dependencies and setup that will be used when deploying on Runpods.
Does anyone know if there’s specific documentation for testing Docker images locally for Runpods, or if there’s a recommended workflow for this kind of setup? Any guidance on how to achieve this local environment would be greatly appreciated.
2 Replies
You can modify your dockerfile abit, then pass those arguments when you run the python file
The arguments like:
in your docker image
then run the docker image locally first
or use the new runpodctl projects, (check out documentations) but it limits you without your dockerfile. since there is a specific configuration file that you must use, inside the config file for Runpod Projects you can use your base image, the handler will be copied inside the development pod later on that uses your base image and the changes made to your handler file will be updated over time to the development pod
i normally just have the changes working locally (literally by having the gpu server running and running
python3 src/rp_handler.py --rp_serve_api --reload) and using the /docs endpoint to trigger a run manually
then once it works, i build the docker image and run it locally to make sure it works
then once the docker image works, i push to to docker hub and update serverless/pod accordingly
it also helps to parameterize as much as you can as environment variables so you dont have to build and deploy a new docker image every time you need something to change
for example, i have an environment parameter for the path of a json file with a bunch of configurations (like default parameters for cfg number, samplers, sampling steps, etc.) and i made the path point to a json file in a network volume
this way, if i want to change the json file, i just spin up the cheapest pod connected to that network volume to add or modify a json file
and the serverless endpoint environment variable can just point to that new file