Mandelion
RRunPod
•Created by Mandelion on 2/15/2024 in #⚡｜serverless
Is there any way to do dynamic batching?
Say I have a vision model deployed and I send 5 images within x time is there a way to actually stack the images, pass them through the model and return the 5 responses? I was able to find concurrent handlers etc. but nothing actual batching (other than sending them all in the same request of course)
4 replies