MaxFrax
RRunPod
•Created by kazuph（かずふ）🍙 on 12/4/2024 in #⚡｜serverless
Has anyone experienced issues with serverless /run callbacks since December?
Didn't manage to collect all the material yet, however it seems related to constraining the generation with:
extra_body={"guided_json": json_schema}
https://docs.vllm.ai/en/latest/usage/structured_outputs.html88 replies
RRunPod
•Created by kazuph（かずふ）🍙 on 12/4/2024 in #⚡｜serverless
Has anyone experienced issues with serverless /run callbacks since December?
502 are coming in strong now and my in progress requests seems to be multiplying according to inprogress counter (without aparent reason)
88 replies
RRunPod
•Created by kazuph（かずふ）🍙 on 12/4/2024 in #⚡｜serverless
Has anyone experienced issues with serverless /run callbacks since December?
Are any of you still experiencing issues with serverless vllm? I cannot manage to release a working endpoint. I keep getting 500s and even some 502 bad gateway from cloudflare. I don't even know how to further describe my issues, it's days that I'm banging my head on this problem and I'm losing sanity. I tried to rollback to runpod/worker-v1-vllm:v1.6.0stable-cuda12.1.0, without any luck. Lucklily it seems that my old endpoints created in the past few months are not experiencing visible issues
88 replies