R
RunPod•2mo ago
Maher

hanging after 500 concurrent requests

Hi, I loaded llama 8b in serverless with 1 active worker A100, and 1 idle worker, I wanted to benchmark how many requests I can do at the same time so I can go production. But when I send 500 requests at the same the server just hangs and I don't get an error. What could be the issue? how to know how much load 1 gpu can handle and how to optmize it for max concurrency.
3 Replies
digigoblin
digigoblin•2mo ago
@Alpay Ariyak any idea?
nerdylive
nerdylive•2mo ago
Hangs? What's it's like No output just running?
Alpay Ariyak
Alpay Ariyak•2mo ago
Yeah, could you please expand on how it hangs? Also, Max Job Concurrency is 300 by default, you can change it with MAX_CONCURRENCY env var
Want results from more Discord servers?
Add your server
More Posts
is anyone experiencing a massive delay time when sending jobs to GPUs on serverless?We are sending jobs off to our whisper serverless functions and experiencing massive delay times somUrgent! all our workers not working! Any network issues?Please take a look at our workers in endpoint h16kk1hi79s3t0 or kn0n8ry69jj1t7 All the workers are sContainer Files Missing in Workspace On Pod LaunchWhen launching pods (a40) on both community and server cloud, using a custom image that populates /wStart the pod with a custom command after the pod finishes startupHow can add I command that the pod will execute after it has finished starting up? I tried using ``How do I upload 5 gb file and use it in my pod?I tried to upload it several times but only 1.2 gb is uploaded (error). (I used Jupytor web based inSend Binary Image with Runpods ServerlessIs it possible to send binary images on Runpods Serverless? From what I can see, you can only sendHow to start a pod using command line runpodctl or python sdkWhat are the certifications for each Data Center *URGET*I want to see a list of the certifications for each of the Data centers available when creating a poNew release will re-pull the entire image.It was working in the past, pulling only new layers on the top. But now it is pulling everything agaRequests stuck in IN_QUEUE statusWe deployed a LLaVA-v1.6-34B model on 2xA100SXM infra as a serverless endpoint. When we send a reque