RunPod•9mo ago

Network issues with 3090 pods

Pods tfc6texf3xrkip and 33laj8z8yzm0du both have borked networking. The download speeds are very slow, and I get issues like these:

Collecting fairseq@ git+https://github.com/pzelasko/fairseq@ba2f4bae68107c9d8a838f19611f951e718577b4 (from -r requirements.txt (line 60))
  Cloning https://github.com/pzelasko/fairseq (to revision ba2f4bae68107c9d8a838f19611f951e718577b4) to /tmp/pip-install-wt38ex46/fairseq_32d4b5f22eec428196d0a086873b7d52
  Running command git clone --filter=blob:none --quiet https://github.com/pzelasko/fairseq /tmp/pip-install-wt38ex46/fairseq_32d4b5f22eec428196d0a086873b7d52
  error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet.
  error: 7392 bytes of body are still expected
  fetch-pack: unexpected disconnect while reading sideband packet
  fatal: early EOF
  fatal: fetch-pack: invalid index-pack output

Collecting fairseq@ git+https://github.com/pzelasko/fairseq@ba2f4bae68107c9d8a838f19611f951e718577b4 (from -r requirements.txt (line 60))
  Cloning https://github.com/pzelasko/fairseq (to revision ba2f4bae68107c9d8a838f19611f951e718577b4) to /tmp/pip-install-wt38ex46/fairseq_32d4b5f22eec428196d0a086873b7d52
  Running command git clone --filter=blob:none --quiet https://github.com/pzelasko/fairseq /tmp/pip-install-wt38ex46/fairseq_32d4b5f22eec428196d0a086873b7d52
  error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet.
  error: 7392 bytes of body are still expected
  fetch-pack: unexpected disconnect while reading sideband packet
  fatal: early EOF
  fatal: fetch-pack: invalid index-pack output

8 Replies

Madiator2011•9mo ago

From what I see I do not see any errors on host side. Have you tried restart pod?

SmoldaOP•9mo ago

yeah, removed and restarted both of these pods multiple times fwiw download speeds from S3 are also slow, like 60 kbps

Madiator2011•9mo ago

mind giving try https://github.com/justinwlin/Runpod-Tips-and-Tricks/tree/main/SpeedTest

GitHub

Runpod-Tips-and-Tricks/SpeedTest at main · justinwlin/Runpod-Tips-a...

Runpod Tips and tricks repository. Contribute to justinwlin/Runpod-Tips-and-Tricks development by creating an account on GitHub.

SmoldaOP•9mo ago

haha it has issues even downloading the benchmark, i'll let you know once i have something useful currently it hangs at

Downloading and making executable the speedtest-cli Python script...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 65334  100 65334    0     0   490k      0 --:--:-- --:--:-- --:--:--  494k

Downloading and making executable the speedtest-cli Python script...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 65334  100 65334    0     0   490k      0 --:--:-- --:--:-- --:--:--  494k

SmoldaOP•9mo ago

@Papa Madiator

message.txt

SmoldaOP•9mo ago

seems like speed is fine but there is some heavy packet loss

Madiator2011•9mo ago

so far this is what I can de I reported that machines to the team

SmoldaOP•9mo ago

thanks, appreciated do you have any time approximation on the fix?

Gaming

Programming

Network issues with 3090 pods