theblackcat102
theblackcat102
PostsComments
RRunPod
Created by enfys on 12/23/2024 in #⛅｜pods-clusters
Extremely slow network storage
anything which involve network storage was congested, local machine storage is okay
31 replies
RRunPod
Created by enfys on 12/23/2024 in #⛅｜pods-clusters
Extremely slow network storage
Same problem in EU-SE-1 today for some reason starting new pod the problem still persist ( slightly faster but same instability issue ), moving files within the storage also have the same problem 
rsync -ah --info=progress2 outputs/model-out-stage2/checkpoint-150/*.safetensors outputs/stage2_results/
          3.25G   3%    2.19MB/s   11:37:31  rsync: [receiver] write failed on "/workspace/data/outputs/stage2_results/model-00001-of-00021.safetensors": Input/output error (5)
rsync error: error in file IO (code 11) at receiver.c(380) [receiver=3.2.7]
rsync -ah --info=progress2 outputs/model-out-stage2/checkpoint-150/*.safetensors outputs/stage2_results/
          3.25G   3%    2.19MB/s   11:37:31  rsync: [receiver] write failed on "/workspace/data/outputs/stage2_results/model-00001-of-00021.safetensors": Input/output error (5)
rsync error: error in file IO (code 11) at receiver.c(380) [receiver=3.2.7]
I already submit a ticket for this
31 replies