Inconsistent 400 Bad Response from sending /run and /runSync.
Good evening everyone.
I am getting the following error message when requesting Runpod's serverless:
response error StatusCode: 400, <html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
- My Code runs in AWS Lambda
- I am using /run and /runsync
- It happens very inconsistently, maybe 1/30 requests.
- I retry 5 times with up to 20s delay, but that doesn't fix the problem.
- However, rerunning the failed request locally works 99%+ of the time.
- The lambda might run multiple runpod's serverless /Run requests in parallel if one fails, it does not affect the other runpod requests in the same lambda.
- I checked the request size and the response size, and both are ~1.5 MB, much smaller than the 10 MB /Run limit
- I retried with /runsync after a /run failed with that error that does not seem to fix it.
- replicating the issue locally is much harder than getting it in lambda
- but when it happens, retrying locally doesn't work
- A similar error can be triggered when removing the Host header from the request, but I ensured that it was always set, and the error still happens.
It seems this is a bug.
What surprises me the most is that retrying locally works, but on lambda, it does not.
Maybe there is an issue with the lambda instance the request is running on, such as some IP rate limit.
If anyone has a fix or walkaround, I would appreciate it.
5 Replies
There is rate limiting on IP so I suggest using webhooks and getting a dedicated IP
I am pretty sure the status code for rate limiting is 429 though not 400.
Thanks for your suggestion.
It is 100% 400 Bad Request.
The IP rate limit was just a guess. I think it is unlikely the problem as I sometimes run multiple 100 requests locally for testing, and never had it that one endpoint was locked down because of ratelimits.
My best guess is that runpod doesn't like the lambda ips for some obscure reason.
hey rageus try contacting support on the website
they might know or can check whats happening
https://contact.runpod.io/hc/en-us/requests/new
maybe 400 bad request means request isn't right
body, or etc
Hey nerdylive.
Thanks for your response.
I will check with their support.
It is unlikely the request in this case since rerunning on a different device with the exact same query, body, works fine. (some headers are different, but it usually works)
Oh ya okok