`runpodctl send` crawling at <1MB speeds
Hi there! I'm a big fan of RunPod for training SDXL, and have spent a bunch of time (and money!) iterating on fine-tuning models on RunPod using on-demand secure cloud servers. However, I keep running into a blocker: unexpectedly slow speeds with
runpodctl send. Sometimes it works well, with 40MB/s speeds; other times, it drops down to <1MB/s speeds for no apparent reason, and can take hours to download a single 6GB file.
I'll be honest: paying $4.69/hr for 3-4 hours to train a model is much less appealing when I know it might take me another hour of frustration afterwards just to download the results. Is there a faster / more reliable way to download the results of a training?2 Replies
Ah
Runpodctl sucks right now
https://discord.com/channels/912829806415085598/1194711850223415348
Here is a guide I wrote on how to get better transfer speeds; you can actually just use the underlying library they are using called
croc to do the file transfer and use public relays croc provides rather than Runpod's private relays. I think their relays are probably bottlenecked by the amt of data flowing through.
I think besides croc, I also wrote about SSH/SCP which would be a lot more reliable too!Thanks, @justin! This was very useful 🙏