nevermind
nevermind
PostsComments
RRunPod
Created by Space Burger on 12/30/2024 in #⛅｜pods
Is there a limit in the number of threads?
I think I faced this problem at 1024 conc processes (means threads). You can always test it with thread swarming on pod
18 replies
RRunPod
Created by Space Burger on 12/30/2024 in #⛅｜pods
Is there a limit in the number of threads?
Yes, it has limit, I faced same issue
18 replies
RRunPod
Created by Ben on 12/20/2024 in #⛅｜pods
Throttled download speed from container registry while still being billed
Mb Runpod needs to host their own registry? If you ask me, we would pay an extra per hour just to launch pods faster (or mb buy some premium for an extra XX$). Other providers give such opportunity, and it seems Runpod experiences lack of this feature 😦
15 replies
RRunPod
Created by nevermind on 12/17/2024 in #⛅｜pods
Whats happening to runpod rn
It was given us with runpod's automation I believe
17 replies
RRunPod
Created by nevermind on 12/17/2024 in #⛅｜pods
Whats happening to runpod rn
And this host was really bad, we had realtime service on this pod, it was frustrating to obtain the host that slow... Could you do something to prevent these problems?
17 replies
RRunPod
Created by nevermind on 12/17/2024 in #⛅｜pods
Whats happening to runpod rn
We'd like to receive alerts for pod outages and we have all the infra for it
17 replies
RRunPod
Created by nevermind on 12/17/2024 in #⛅｜pods
Whats happening to runpod rn
Also, could you tell me the GQL query/mutation to obtain the outage description?
17 replies
RRunPod
Created by nevermind on 12/17/2024 in #⛅｜pods
Whats happening to runpod rn
Redeploy on another host
17 replies
RRunPod
Created by nevermind on 12/17/2024 in #⛅｜pods
Whats happening to runpod rn
Could you provide a way to disable pod revival after this kind of outages? MB some kind of flag, it's crucial for us to redeploy ourselves with our autoscaler, not with Runpod's automation
17 replies
RRunPod
Created by nevermind on 12/17/2024 in #⛅｜pods
Whats happening to runpod rn
{
  "download": 6775599.51883442,  # bits!!
  "upload": 48738976.982727684,  # bits!!
  "ping": 116.211,  # ms!!!
  "server": {
    "url": "http://speedtest1.nova.is:8080/speedtest/upload.php",
    "lat": "64.1333",
    "lon": "-21.9333",
    "name": "Reykjavik",
    "country": "Iceland",
    "cc": "IS",
    "sponsor": "Nova Ehf",
    "id": "3684",
    "host": "speedtest1.nova.is:8080",
    "d": 5.0982814200353594,
    "latency": 116.211
  },
  "timestamp": "2024-12-17T21:08:04.427726Z",
  "bytes_sent": 83329024,
  "bytes_received": 8779088,
  "share": null,
  "client": {
    "ip": "157.157.221.29",
    "lat": "64.0875",
    "lon": "-21.9284",
    "isp": "Mila hf",
    "isprating": "3.7",
    "rating": "0",
    "ispdlavg": "0",
    "ispulavg": "0",
    "loggedin": "0",
    "country": "IS"
  }
}
{
  "download": 6775599.51883442,  # bits!!
  "upload": 48738976.982727684,  # bits!!
  "ping": 116.211,  # ms!!!
  "server": {
    "url": "http://speedtest1.nova.is:8080/speedtest/upload.php",
    "lat": "64.1333",
    "lon": "-21.9333",
    "name": "Reykjavik",
    "country": "Iceland",
    "cc": "IS",
    "sponsor": "Nova Ehf",
    "id": "3684",
    "host": "speedtest1.nova.is:8080",
    "d": 5.0982814200353594,
    "latency": 116.211
  },
  "timestamp": "2024-12-17T21:08:04.427726Z",
  "bytes_sent": 83329024,
  "bytes_received": 8779088,
  "share": null,
  "client": {
    "ip": "157.157.221.29",
    "lat": "64.0875",
    "lon": "-21.9284",
    "isp": "Mila hf",
    "isprating": "3.7",
    "rating": "0",
    "ispdlavg": "0",
    "ispulavg": "0",
    "loggedin": "0",
    "country": "IS"
  }
}
17 replies
RRunPod
Created by nevermind on 12/17/2024 in #⛅｜pods
Whats happening to runpod rn
its slower than my mobile internet conn
17 replies
RRunPod
Created by nevermind on 12/17/2024 in #⛅｜pods
Whats happening to runpod rn
No description
17 replies
RRunPod
Created by nevermind on 11/21/2024 in #⛅｜pods
Pod eternal image fetching
It's like 10gb, but on other pods it takes about 2-3min to pull
6 replies
RRunPod
Created by K-Woww on 11/1/2024 in #⛅｜pods
Is there something wrong in US-OR-1?
US-OR-1 is bad. We described the same problem directly to the RunPod team and also supplemented the report with speedtest. It is usually 3x slower than US-TX-3 with x6 ping
18 replies
RRunPod
Created by nevermind on 9/26/2024 in #⛅｜pods
Urgent: {'message': 'Something went wrong. Please try again later or contact support.'}
We have a bunch of these errors in Grafana, so we can provide any additional information.
7 replies
RRunPod
Created by nevermind on 9/26/2024 in #⛅｜pods
Urgent: {'message': 'Something went wrong. Please try again later or contact support.'}
First three occurred while podFindAndDeployOnDemand, Last one - query myPods
7 replies
RRunPod
Created by nevermind on 9/26/2024 in #⛅｜pods
Urgent: {'message': 'Something went wrong. Please try again later or contact support.'}
We get it with python httpx, not UI
7 replies
RRunPod
Created by nevermind on 9/26/2024 in #⛅｜pods
Urgent: {'message': 'Something went wrong. Please try again later or contact support.'}
Additionally, during this time we experience read timeouts (requests taking more than 90 seconds to process). I think these problems are interrelated.
7 replies
RRunPod
Created by nevermind on 9/21/2024 in #⛅｜pods
SOS pod gpu errors
Alright, if we face the same issue again - I'll report to this thread and try my best to lock the pod
8 replies
RRunPod
Created by nevermind on 9/21/2024 in #⛅｜pods
SOS pod gpu errors
We actually use multiple instances of this configuration and sometimes pods just "die" during its lifetime (it affects our flow control).
8 replies