Slow upload speeds with runpodctl?
Hey there. Since yesterday I had to upload files to my instance twice and it's super slow (at the moment it's 650 kB/s which is not feasible really). It usually starts of way faster and slows down after a while. It doesn't seem to depend on my network because upload to other servers is significantly faster and it doesn't really seem to matter from which network (work, home) I upload, the speeds are way lower than e.g. a month ago. What's going on with the uploads and anything I can do about it?
Solution:Jump to solution
You can try using croc, then you don't need to use the RunPod relays.
https://github.com/schollz/croc...
GitHub
GitHub - schollz/croc: Easily and securely send things from one com...
Easily and securely send things from one computer to another :crocodile: :package: - GitHub - schollz/croc: Easily and securely send things from one computer to another :package:
56 Replies
Solution
You can try using croc, then you don't need to use the RunPod relays.
https://github.com/schollz/croc
GitHub
GitHub - schollz/croc: Easily and securely send things from one com...
Easily and securely send things from one computer to another :crocodile: :package: - GitHub - schollz/croc: Easily and securely send things from one computer to another :package:
WOW this is great to know! Amazing! 😮
Yeah could be that runpodctl relay sucks.
And if you want, can use:
https://discord.com/channels/912829806415085598/1194711850223415348
Which would give u direct SSH / SCP connection to your runpod instead if from local computer to pod, and get rid of middle man
I wouldn't necessarily say that the runpodctl relays suck, just maybe overloaded with too much data. By the way, the file send/receive of runpodctl is actually based on croc.
lol i was about to say i was reading their runpodctl code earlier so that i could make a function xD
that i saw croc there haha, if it meant they choose their own custom relay
Interesting though yea
But interesting, i wonder what the difference between connecting two pods directly vs a relay is. Ill need to look into this later
Edit: nvm, i learnt that croc just provides a public relay server i think lol. so always a relay server to get around firewall stuff
Learning so many new toolings every week
🤯
I'll take a look at croc
I think there was a tool called "wormhole" back in the day that did the same
However, I just sticked to runpodctl because it's advertised/implicitly encouraged by runpod.io
just curious, this from ur local to the pod? why not also just scp?
yea i agree, but i guess from my time here, ive seen many ppl talk about its slow performance 🤔
runpodctl is usually baked into most templates so its typically more convenient than croc, but my templates include both
@JM Is slower performance with runpodctl potentially just due to data bottlenecks with everyone moving stuff through runpod's relay? Like into a Kb range seems very slow
I didn't use scp out of laziness haha, that one's on me
I just went through some of the help threads here and apparently I'm not the only one
Yeah i think this is a common thing, why i wrote something on easy ssh + also wrote a wrapper for SCP
Also everything else is kind of slow today with my pods
I still need to write some tests to get my merge accepted... x.x
are u using community or secure? just curious
Secure
U know, i noticed an uptick in ppl complaining so spent last night for fun writing this:
https://discord.com/channels/912829806415085598/1207175605255147571
If u ever want to give it a try
lol
🤔
I wanted to be like, am i crazy; or is runpod the one hahah
Fun thing, but just proved is possible using @Madiator2011 [EU] 's pip python package:
https://discord.com/channels/912829806415085598/1202555381126008852
I set up two SSH on two runpod, and was able to connect to it, so I assume, if you wanted to do a direct connection between two pods, can ssh and scp stuff over too
God bless croc lol, waiting 15 minutes instead of 3 hours
Too lazy to test SCP right now xD need to slide back to work hhh
woooottt
awesome to know
in the future ill also recommend this haha
croc / scp
I don't know why runpodctl shat the bed the last two days. Again, a month ago or so I was using it and it was working perfectly fine.
The best way is probably just using scp
Yeah, and can use my wrapper ;D tho i havent tried it with passwords yet from madiator's stuff, but with the normal ssh setup public / private key stuff; but good to know madiator's template also would allow ssh between two pods. prob would be even faster for in between connections 🤯 for two pods
I think RunPod relay servers bandwidth is maxed out and they probably need to add more relay servers, but thats just a guess.
Also thank both of you for your help!
I'm going to read why/how runpodctl and croc work in the first place haha
ltierally started searching on perplexity and chatgpt bc of this too xD; but i think is essentially get around firewall stuff; if ur thing maybe doesnt allow normal networking; could be wrong lol
Looks like there are only 6 relay servers and the relay list hasn't been updated since 23 Feb last year, I am pretty sure they need more relay servers with the amount of growth since then:
https://github.com/runpod/runpodctl/blob/main/cmd/croc/relays.json
GitHub
runpodctl/cmd/croc/relays.json at main · runpod/runpodctl
🧰 | RunPod CLI for pod management. Contribute to runpod/runpodctl development by creating an account on GitHub.
Unfortunately I have no idea about networking and how stuff works in that regard haha
This is what I meant with "Wormhole" @ashleyk @justin
https://github.com/magic-wormhole/magic-wormhole
GitHub
GitHub - magic-wormhole/magic-wormhole: get things from one compute...
get things from one computer to another, safely. Contribute to magic-wormhole/magic-wormhole development by creating an account on GitHub.
It seems to be the same thing
I've used it a few times ages ago
Usually both runpodctl and croc try to connect user directly relay only is used if one of users is behind NAT
Oh I see, thanks, aren't all RunPod pods behind NAT though?
on pods wih public ip you could run relay to get true P2P. I usually use own relay server
I assume it defaults to RunPod relays though doesn't it?
yes for runpodctl
Just wondering, what is the difference between relay servers vs scp between two pods? Is there a reason to avoid ssh/scp? or is it just that using something like croc / relay server is easier / faster to do
(/ avoid like any weird networking limitations?)
Like to me it seems the natural thing is if u can, just ssh/scp between two computers. but i guess u said that croc does try to do it automatically if it can
croc is simpler to setup than sftp
sftp is awesome for technical people but many RunPod users don't know their way around a terminal/command line and have great difficulty trying to get ssh to work properly
I have a question out of curiosity. If I use croc or a comparable service (runpodctl, the wormhole thing, ...) - can I send it from one device and put the "receive" command in multiple devices and it will download to more than one "target" device?
nah i think is a first come first serve
Also SFTP requires tcp port exposed
since its a one time code? i think
ahhh good point
so i guess essentially if u can't ssh, then use croc / runpodctl; otherwise if ssh/tcp is avaliable can use scp or sftp protocols
Huh, when i get a chance, Im curious to set up two pods + a cli wrapper using ur package lol. wonder if i can get it easier to setup an SCP / SFTP. but yeah, i also think that what u said @ashleyk makes sense.. that a terminal.. can scare ppl lol
I'm going to make some PRs to runpodctl
Oooo, 👁️ with ur package that u said u were porting the functionality over to?
yup so it'
honestly, i wish i could make a PR into runpodctl xD, i just dont know Go well enough to test on haha
yay
lol
The last time i touched go was like.. 4 years ago.. for a distributed computing course.. and all i remember was its a pain for me lol
My last program I made I have written in brainfuck language
xD
do u just like to torture urself haha
Brainfuck
Brainfuck is an esoteric programming language created in 1993 by Urban Müller. Notable for its extreme minimalism, the language consists of only eight simple commands, a data pointer and an instruction pointer.Brainfuck is an example of a so-called Turing tarpit: it can be used to write any program, but it is not practical to do so, because Brai...
yea haha, i think one the most popular troll turing complete language lol.
interesting lol
i guess if at some point i want to learn to make PR to runpodctl, ill brush back up on go
👁️
so runpodctl is more for pod management, and runpod python is more for sdk? stuff
runpod python is dedicated for serverless
AH got it. Okay.
I have a quick question w.r.t. runpod. Doesn't a service like that cause a ton of traffic?
what is wrt / by service are u talking about runpodctl?
w.r.t. = with reference/respect/regards to
Also, great info in here, I will take croc for a spin if I can lock down a GPU. Thanks for sharing that
yea croc is a great thing to have learnt. Ill prob try croc + also ssh between two pods later and do a speed comparison, im curious
yea their runpodctl is prob decently popular, but unsure how often it actually is used, wouldnt know without seeing their data. but i mean runpod as ashelyk said has grown rapidly, 🤯, i didnt even know a year ago they were barely up lol and had like ~5 ppl lol
This thread is closed but I'm getting about 100 MB/s and climbing with croc. Highly recommend. Woof
@nathaniel lets support custom ip for
runpodctl send that will help avoid needing croc since its the samething, we plan to add more relays
one other strategy is allow users to run their own relay as well using runpodctl, so everything can be local and even fasterWill be great for both, too many non technical users won't be able to run their own.