Unstable Internet Connection in the Workers
Recently, I've noticed that several serverless jobs are encountering a very unstable internet connection, leading to extremely slow download and upload speeds.
This instability is resulting in connection errors on HTTP requests and the loss of packets. Additionally, the slow connection speed is causing significant delays in downloading from and uploading to S3, even for asset files that are just a few MBs in size, resulting in the consumption of excessive credits.
Furthermore, there are instances where connection errors or timeouts are causing the failure of generated output files to upload, resulting in job failures. This is particularly frustrating as credits were spent generating the output.
It's worth noting that this issue doesn't occur consistently; rather, it happens occasionally on some jobs.
Is anyone else experiencing this issue, or is it just me?
Solution:Jump to solution
Never mind. After using Runpod for over 4 months and spending over a thousand dollars on it, our company has decided to completely drop Runpod and switch to another platform. This decision was made due to Runpod's frequent instability and lack of timely and adequate support.
During our time with Runpod, we encountered numerous issues, including a significant "throttle disaster" two weeks ago, problems with webhooks, network issues, and more.
These incidents have resulted in financial losses for us, with some customers becoming upset and leaving. We can no longer tolerate these challenges....
5 Replies
Hi @n8tzto - do you happen to have the endpoint IDs and the time of day that this occurs?
Endpoint ID:
1pzws8rhbpku7g
I'm not exactly sure what time the issue occurred, but I observed significant instability in the connection just recently (on March 14, 2024, between approximately 2pm and 4pm UTC). While it's currently showing signs of improvement compared to earlier, the speed is still notably slow.
Things have gotten worse lately. Our jobs are failing or taking way too long, which means our credits are burning and our clients are getting frustrated.
Any updates on this?for speed to s3, try to select regions closer to your s3 region, that helps al ot
We've transitioned to Cloudflare R2 and are utilizing its
auto region, but the problem persists: Downloading from its public HTTP URL and uploading via boto are excessively slow, with frequent packet losses.
Note that this issue doesn't always occur; it's intermittent. Some tasks are processed quickly and without any problems. However, we've noticed a higher frequency of occurrences over the past few days.Solution
Never mind. After using Runpod for over 4 months and spending over a thousand dollars on it, our company has decided to completely drop Runpod and switch to another platform. This decision was made due to Runpod's frequent instability and lack of timely and adequate support.
During our time with Runpod, we encountered numerous issues, including a significant "throttle disaster" two weeks ago, problems with webhooks, network issues, and more.
These incidents have resulted in financial losses for us, with some customers becoming upset and leaving. We can no longer tolerate these challenges.
We appreciate the creation of Runpod, and it can still be useful for testing, development, or personal purposes.🙏
However, it is not suitable for use in production environments.