Topics

RunPod•6mo ago

remote-ssh broken

Remote-SSH on VSCode is broken on community cloud instances, due to there not being PTY support on the ssh client (because it's not a public IP?). This was asked before, but not sure if it was fixed

Solution:

You can try this:

pip install OhMyRunPod
OhMyRunPod --setup_ssh

pip install OhMyRunPod
OhMyRunPod --setup_ssh

...

Jump to solution

48 Replies

Madiator2011•6mo ago

For community cloud you need to select public ip filter.

HarishOP•6mo ago

I selected - but seems like it still doesn't work as expected. Chose an instance with a 3090 Is there any sign before deployment I can check if it does have a public IP address? Just tried again @Papa Madiator with an RTX A4000. Made sure to select the public IP filter before deploying, same result It is a custom docker image, but I think that shouldn't matter right?

digigoblin•6mo ago

It does matter, if it doesn't start ssh service you can't ssh

Solution

digigoblin•6mo ago

You can try this:

pip install OhMyRunPod
OhMyRunPod --setup_ssh

pip install OhMyRunPod
OhMyRunPod --setup_ssh

HarishOP•6mo ago

@digigoblin Worked like a charm 🙂 thank you! Quick follow up - I notice the connection instructions now say to use root@<ip address>. Is there any way of setting this to a permanent address (like .runpod.io is used for the other version of connection) so that I don't have to change in my ssh config each time?

digigoblin•6mo ago

The TCP ports are randomly assigned so unfortunately not

HarishOP•6mo ago

@digigoblin Sorry to keep pinging you - connection works now, but I'm not seeing my secrets show up as environment variables when I SSH in and run something. Is there something special I need to do there? They were showing up properly when I just SSH'ed in with the other method

digigoblin•6mo ago

I assume you added your secrerts as environment variables to your pod?

HarishOP•6mo ago

Yep

digigoblin•6mo ago

Sorry I just saw now you said you did, my bad

HarishOP•6mo ago

Nw

digigoblin•6mo ago

You basically have to do something like this to make them accessible: https://github.com/runpod/containers/blob/main/container-template/start.sh#L67-L71

GitHub

containers/container-template/start.sh at main · runpod/containers

🐳 | Dockerfiles for the RunPod container images used for our official templates. - runpod/containers

digigoblin•6mo ago

Except yours may not begin with RUNPOD_

HarishOP•6mo ago

I need to put this in my docker image is that right? Or run it on the pod when it's up

digigoblin•6mo ago

Yeah, usually in your docker image start script

HarishOP•6mo ago

Oh in the start script okay

digigoblin•6mo ago

Yep

HarishOP•6mo ago

bash -c 'apt update;DEBIANFRONTEND=noninteractive apt-get install openssh-server -y;mkdir -p ~/.ssh;cd $;chmod 700 ~/.ssh;echo "$PUBLIC_KEY" >> authorized_keys;chmod 700 authorizedkeys;pip install OhMyRunPod; OhMyRunPod --setup-ssh; printenv | grep -E '^RUNPOD|^PATH=|^_=' | awk -F = '{ print "export " $1 "="" $2 """ }' >> /etc/rp_environment; echo 'source /etc/rp_environment' >> ~/.bashrc; service ssh start;sleep infinity' Something like this in totality? (Haven't tried adding the OhMyRunPod --setup-ssh as part of the start script, I know it works after I ssh in but hoping it works in the start script itself)

Madiator2011•6mo ago

I might add injecting secrets as part of OhMyRunPod

digigoblin•6mo ago

How will it read them though?

Madiator2011•6mo ago

Though when I waited up. Same as start script 😅

digigoblin•6mo ago

But start script has them exposed as environment variables when the container starts. If someone SSH into the pod and they aren't exported by the container, they are no longer available.

Madiator2011•6mo ago

Btw OhMyRunPod auto install ssh server if it’s missing 😅

digigoblin•6mo ago

Yeah saw that, it helped @Harish

HarishOP•6mo ago

Tryna figure out how to escape the ‘’ characters in the awk command for the start script

digigoblin•6mo ago

Try this:

bash -c 'apt update; DEBIANFRONTEND=noninteractive apt-get install openssh-server -y; mkdir -p ~/.ssh; cd $; chmod 700 ~/.ssh; echo "$PUBLICKEY" >> authorized_keys; chmod 700 authorized_keys; pip install OhMyRunPod; OhMyRunPod --setup-ssh; printenv | grep -E '^RUNPOD|^PATH=|^_=' | awk -F = '\''{ print "export " $1 "=" "\\""$2"\\""" }'\'' >> /etc/rp_environment; echo '\''source /etc/rp_environment'\'' >> ~/.bashrc; service ssh start; sleep infinity'

bash -c 'apt update; DEBIANFRONTEND=noninteractive apt-get install openssh-server -y; mkdir -p ~/.ssh; cd $; chmod 700 ~/.ssh; echo "$PUBLICKEY" >> authorized_keys; chmod 700 authorized_keys; pip install OhMyRunPod; OhMyRunPod --setup-ssh; printenv | grep -E '^RUNPOD|^PATH=|^_=' | awk -F = '\''{ print "export " $1 "=" "\\""$2"\\""" }'\'' >> /etc/rp_environment; echo '\''source /etc/rp_environment'\'' >> ~/.bashrc; service ssh start; sleep infinity'

By the way, RunPod automatically injects a variable called PUBLIC_KEY into your pod, so thats usually the convention and not PUBLICKEY.

Madiator2011•6mo ago

service restart As ohmy auto starts service

digigoblin•6mo ago

Probably don't need to start or restart it then because no changes are made after OhMyRunPod starts it

Madiator2011•6mo ago

Probably going to add inject public key and other env should not be hard Btw OhMy now also supports tailscale

HarishOP•6mo ago

Results in: awk: 1: unexpected character '' awk: line 1: runaway string constant " } ...

digigoblin•6mo ago

Probably need to ask ChatGPT or something ¯\_(ツ)_/¯ I would rather just make a custom docker image than to go to all this effort personally

HarishOP•6mo ago

Yeah lol I have a custom docker image but I'd rather not add runpod-specific stuff into it because I use it across platforms But if the thing claude gave me doesn't wokr might as well just do that

Madiator2011•6mo ago

bash -c ' pip install OhMyRunPod && \ OhMyRunPod --setup-ssh && \ mkdir -p ~/.ssh && \ echo "$PUBLIC_KEY" >> ~/.ssh/authorizedkeys && chmod 700 -R ~/.ssh && \ printenv | grep -E "^RUNPOD|^PATH=|^_=" | awk -F = '''{ print "export " $1 "="" $2 """ }''' >> ~/rp_environment && \ echo '''source ~/rp_environment''' >> ~/.bashrc && \ sleep infinity '

HarishOP•6mo ago

Hm ok so I got the escaping to work But seems like the actual command messes up my path somewhere Full command: bash -c 'apt update;DEBIANFRONTEND=noninteractive apt-get install openssh-server -y;mkdir -p ~/.ssh;cd $;chmod 700 ~/.ssh;echo "$PUBLIC_KEY" >> authorized_keys;chmod 700 authorizedkeys; pip install OhMyRunPod; OhMyRunPod --setup-ssh; echo "Exporting Env Variables..."; printenv | grep -E "^RUNPOD|^PATH=|^=" | awk -F = '"'"'{ print "export " $1 "="" $2 """ }'"'"' >> /etc/rpenvironment; echo "Finished exporting env variables. Variables from parent:"; printenv | grep -E "^RUNPOD|^PATH=|^="; echo "source /etc/rp_environment" >> ~/.bashrc; service ssh start;sleep infinity' Container logs: 2024-06-11T01:52:00.108484438+02:00 Exporting Env Variables... 2024-06-11T01:52:00.110996684+02:00 Finished exporting env variables. Variables from parent: 2024-06-11T01:52:00.112944521+02:00 RUNPOD_CPU_COUNT=9 2024-06-11T01:52:00.112962141+02:00 RUNPOD_POD_ID=<redacted> 2024-06-11T01:52:00.112966701+02:00 RUNPOD_MEM_GB=50 2024-06-11T01:52:00.112969881+02:00 RUNPOD_PUBLIC_IP=<redacted> 2024-06-11T01:52:00.112972671+02:00 RUNPOD_GPU_COUNT=1 2024-06-11T01:52:00.112975351+02:00 RUNPOD_POD_HOSTNAME=<redacted> 2024-06-11T01:52:00.112977971+02:00 RUNPOD_GPU_NAME=NVIDIA+GeForce+RTX+3070 2024-06-11T01:52:00.112980611+02:00 RUNPOD_TCP_PORT_22=<redacted> 2024-06-11T01:52:00.112983191+02:00 RUNPOD_APIKEY=<redacted> 2024-06-11T01:52:00.112985921+02:00 PATH=/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-11T01:52:00.112989081+02:00 =/usr/bin/printenv But then after ssh, I get the following as my first messages in console: bash: uname: No such file or directory bash: sed: No such file or directory And then no commands other than cd work @Papa Madiator @digigoblin Is this the right script to run? Going to remove the grep for PATH and see if that fixes it Okay removing the PATH grep worked - but it's not copying over the right env variables now 😢

digigoblin•6mo ago

bash -c '
    pip install OhMyRunPod && \
    OhMyRunPod --setup-ssh && \
    mkdir -p ~/.ssh && \
    echo "$PUBLICKEY" >> ~/.ssh/authorized_keys && chmod 700 -R ~/.ssh && \
    printenv | grep -E "^RUNPOD_|^PATH=|^_=" | awk -F = "{ print \"export \" \$1 \"=\" \$2 }" >> /etc/rp_environment && \
    echo "source ~/rp_environment" >> ~/.bashrc && \
    sleep infinity
'

bash -c '
    pip install OhMyRunPod && \
    OhMyRunPod --setup-ssh && \
    mkdir -p ~/.ssh && \
    echo "$PUBLICKEY" >> ~/.ssh/authorized_keys && chmod 700 -R ~/.ssh && \
    printenv | grep -E "^RUNPOD_|^PATH=|^_=" | awk -F = "{ print \"export \" \$1 \"=\" \$2 }" >> /etc/rp_environment && \
    echo "source ~/rp_environment" >> ~/.bashrc && \
    sleep infinity
'

HarishOP•6mo ago

Unfortunately this still didn't work. I had to actually explicitly grep for the variables I wanted to keep - this makes sense to me too, as the grep for RUNPOD_ and PATH= didn't have the variables I wanted anyways. Could you confirm this is still a secure way of injecting the variables? @digigoblin @Papa Madiator Here is what I did: echo "Exporting Env Variables..."; env | grep -E "^(HF|RUNPOD|AWS|WANDB)" | sed "s/^(.)$/export \1/">> /etc/rpenvironment; echo "Finished exporting env variables. Variables from parent:"; env | grep -E "^(HF|RUNPOD|AWS|WANDB)" Tried multiple versions of RUNPOD and PATH= to no avail. Here's the full command now: bash -c 'apt update;DEBIANFRONTEND=noninteractive apt-get install openssh-server -y;mkdir -p ~/.ssh;cd $;chmod 700 ~/.ssh;echo "$PUBLIC_KEY" >> authorized_keys;chmod 700 authorized_keys; pip install OhMyRunPod; OhMyRunPod --setup-ssh; echo "Exporting Env Variables..."; env | grep -E "^(HF|RUNPOD|AWS|WANDB)" | sed "s/^(.)$/export \1/">> /etc/rp_environment; echo "Finished exporting env variables. Variables from parent:"; env | grep -E "^(HF|RUNPOD|AWS|WANDB)"; echo "source /etc/rp_environment" >> ~/.bashrc; service ssh start;sleep infinity'

digigoblin•6mo ago

Works for me ¯\_(ツ)_/¯ Its wasted too much of my time, can't help anymore especially since it worked when I ran it exactly like the last code I gave you

HarishOP•6mo ago

How would it work though? The injected environment variables don't actually start with either RUNPOD_ or PATH or just _ I'm assuming that security is fine though since in essence this is the same thing, just actually finding the right env variables to copy

digigoblin•6mo ago

not sure why you changed awk to sed I got it working with awk You can just add the others in here:

grep -E "^RUNPOD|^PATH=|^_="

grep -E "^RUNPOD|^PATH=|^_="

HarishOP•6mo ago

sed is what claude gave me lol, also don't have to do the escape characters thing But yeah adding the others is basically what I'm doing, just want to make sure that's right and secure

digigoblin•6mo ago

I think Claude gave you a wrong answer, awk is working with the solution I gave above, I am not in the mood to change it to sed

HarishOP•6mo ago

Also unrelated - the python env doesn't load when I ssh in with root@<ip>, but it does when I ssh in with <user>@runpod No no, that part is working! 🙂 Thank you for your help awk or sed doesn't matter, it is finding and inserting the env variables in correctly

digigoblin•6mo ago

What kind of python env? conda or venv or what?

HarishOP•6mo ago

The question I was asking was more along the lines of is it okay to inject environment variables with this command in the start script

digigoblin•6mo ago

Yeah it should be fine

HarishOP•6mo ago

conda. Am assuming that it's because I'm not exporting PATH though, will figure out how to inject PATH in the same command without overwriting the existing path Wait nvm scratch that, it does work on root now Think I just had to restart the terminal Ok should be all good then, thank you both for your help 🙂 really appreciate it

Madiator2011 (Work)•6mo ago

conda = how to complicate your life and waste time on fixing conda bugs

justin•6mo ago

+1 lol, learning about virtualenv is the best thing ever. i used to use conda all the time, but virtualenv just tends to be a lot more replicable i find

We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!

14KMembers

View on Discord

Want results from more Discord servers?

Add your server