Serverless git integration rollback
Hi team! I've recently switched over to runpod serverless for production with the git integration.
I have a concern about reliability in a failure scenario.
Firstly, I notice that the builds take roughly 1.5 hours to build out and then for a worker to download + extract the image. Consider a bad release that starts getting rolled out - I can't see a button to stop it.
Now, say it rolled out because I missed the process for whatever reason - I can't see a button to roll back to the previous good version.
The only way I currently see to roll back is to accept 1.5 hours of downtime and revert the git commit, wait for everything to build, upload, download, extract.
Is there a workaround in this sort of scenario or are you working on implementing something?
Thanks
Kristo
4 Replies
We will introduce rollback in the near future since we store your older images as well
for the meantime, i would suggest creating a dev branch and a prod branch and then creating two separate endpoints
@drycoco The #1 reason we implemented github integration was to aim for production use cases.
For a production use case I would organize the github repo with dev and prod branch, and then make an endpoint for each. I would test dev branch and its releases, at some point merge dev into prod branch once dev endpoint has been validated. Currently this should get you closer to a dev > production pipeline.
In future we are planning to introduce the following:
- release rollback
- cancel builds if you can't already
- test cases with builds (builds will fail if test cases fail)
Thanks both! Going for the dev branch approach is exactly what I ended up doing. It's not a deal breaker for now but pretty standard functionality in any CD system.
Oh, one more thing I wanted to ask is whether it is possible to reference runpod secrets in the Dockerfile? My use case is a huggingface token to pull model files.
Currently I've hardcoded it since referencing the secret as an env var did not work.
Cancel run is already available
Rollbacks coming soon
We have env vars and support for private images internally
We need to release the ui but thats coming soon