Unexpected Infinite Retries Causing Unintended Charges
I recently ran my serverless workload using my custom Docker image on RunPod, and I encountered an issue that resulted in significant unexpected charges. My application experienced failures, and instead of stopping or handling errors appropriately, it kept retrying indefinitely. This resulted in:
- $166.69 charged by OpenAI due to repeated API calls.
- $14.27 charged on RunPod for compute usage.
This behavior is unexpected, as a serverless service should not continuously retry indefinitely without a stopping mechanism. I had to manually intervene to stop the retries, which should not be required. There should be an automatic failure handling mechanism to prevent excessive resource consumption and unintended expenses.
Given that this was caused by an unexpected behavior in RunPod’s execution handling, I kindly request compensation for these unintended charges. Please investigate this issue and let me know how we can prevent this from happening in the future.
I've attached relevant screenshots for reference.
4 Replies
[RUNPOD] Starting job 08319613-d26f-4567-a69f-b1c22a7c52be-e1 with request_id: 67e6201b40bf680189015c80\n
Hey, thank you for your report.
I'm going to ask you to press the button to submit this as a ticket so I can compensate you for your RunPod related spend. I believe we would also be willing to compensate for your OpenAI expenses too, but I (and the rest of support) can only issue credits - so let me know if that's something you're okay with.
Further, I'll submit this to engineering to see why your request didn't stop because this shouldn't happen. I can follow up on that via Discord or Email (or both!) if you have a preference.
@Talion
Escalated To Zendesk
The thread has been escalated to Zendesk!
Once you do that just let me know, I don't get notified for new support tickets.