524: A timeout occurred (cloudflare)
Hello everyone.
I have a GPU pod with an API that runs a ml model and the calculation takes quite a long time. It's a long running task and it takes about 2-3 minutes. I saw that runpod.io uses cloudflare (this is in the error response) and cloudflare describes that they have a timeout at 100 seconds. I've seen timeout settings for serverless but not for pods.
Does anyone know how to solve this?
The service uses FastAPI (uvicorn) as the web server. Thanks!
Solution:Jump to solution
Cloudflare is designed as a CDN to serve your traffic from edge locations all over the world, so its more for HTTP/HTTPS traffic that should respond pretty quickly, its not designed for long running requests. And if you are using FastAPI, it sounds like your application will probaby need to scale up at some point and pods do not autoscale, so I suggest using serverless instead then you won't have any issues.
16 Replies
make that an async process, then save into storage
Or
use the public ip directly
There is no setting for pods unfortunately, serverless handles long running requests natively, but pods do not and there is no way you can increase the 100 second timeout.
its pods here btw
Solution
Cloudflare is designed as a CDN to serve your traffic from edge locations all over the world, so its more for HTTP/HTTPS traffic that should respond pretty quickly, its not designed for long running requests. And if you are using FastAPI, it sounds like your application will probaby need to scale up at some point and pods do not autoscale, so I suggest using serverless instead then you won't have any issues.
Yeah sorry I meant no setting for pods, I am sick and meds making me not think straight
ooh haha try taking some rest bro
Thanks a lot @nerdylive and @digigoblin ! Also thanks for the explanation that helped me quite a bit. For now, I'll go with public ip but keep serverless in the back of my mind. Get well!
Ahh I've got a follow-up question. How would I make sure that the external port that is assigned to my exposed port remains the same? Every time I restart my pod I see a different external port.
Yeah using ENV variables
It can't remain the same, it will always change randomly, you have to use the GraphQL to get your ports
Using ENV variables will make them the same (internal / external) but you won't get the same port number each time.
Pod environment variables | RunPod Documentation
Configure and manage your pods with these essential environment variables, including pod ID, API key, host name, GPU and CPU count, public IP, SSH port, data center ID, volume ID, CUDA version, current working directory, PyTorch version, and public SSH key.
like RUNPOD_TCP_PORT_22
Oh I see what you mean, thats the same as using a GraphQL query pretty much, but won't make them the same each time.
yep
Oh yea didn't see saying trying to make the same port
You need rest too 🤣
Lol yeah its late night
just testing streams
in PHP 🙂
Thank you guys! 🙂 Appreciate it!