serverless - lora from network storage
Hi, I have a flux generation serverless setup that's working pretty well.
I bake all models into the docker image so even though the docker image size is pretty large, cold start is pretty reasonable and generations are fast enough.
Now the issue arises with a new workflow where I will train more lora and need to set these available to the serverless workflow.
-if I bake new loras into the docker image, it needs to rebuild and download the entire docker image which takes a long time for active workers and causes a down time until it loads
-I was thinking putting the loras on network storage (lets say there are 100 loras on network storage each is around 20mb), whenever worker is up it copies the lora requested (only 1 lora) from network storage to the serverless storage and runs it from there. would this increase execution time significantly?
basically it will run ALL models on docker baked image + single 20mb lora from network storage
is there any other alternative or best practice for doing this?
3 Replies
-if I bake new loras into the docker image, it needs to rebuild and download the entire docker image which takes a long time for active workers and causes a down time until it loads
no, just create a new release "manage" menu
it will do a rolling release (at least do 5 workers or 3 i guess)
-I was thinking putting the loras on network storage (lets say there are 100 loras on network storage each is around 20mb), whenever worker is up it copies the lora requested (only 1 lora) from network storage to the serverless storage and runs it from there. would this increase execution time significantly?
Maybe, a little bit, people has reported ns are quite slower than container disk ( like you load it from your image )
good approach putting loras in network storage, since they are smaller and you can add more dynamically, and access them in the future
thanks, yes "down" wasnt actually true, what I tried to mean is 1 worker getting new docker build takes around 20-25min. 5 worker takes more than an hour. until then any request to new lora will fail.
I think using lora from network storage is the best option
Yupp