remote-ssh broken
Remote-SSH on VSCode is broken on community cloud instances, due to there not being PTY support on the ssh client (because it's not a public IP?). This was asked before, but not sure if it was fixed
48 Replies
For community cloud you need to select public ip filter.
I selected - but seems like it still doesn't work as expected. Chose an instance with a 3090
Is there any sign before deployment I can check if it does have a public IP address?
Just tried again @Papa Madiator with an RTX A4000. Made sure to select the public IP filter before deploying, same result
It is a custom docker image, but I think that shouldn't matter right?
It does matter, if it doesn't start ssh service you can't ssh
Solution
You can try this:
@digigoblin Worked like a charm π thank you!
Quick follow up - I notice the connection instructions now say to use root@<ip address>. Is there any way of setting this to a permanent address (like .runpod.io is used for the other version of connection) so that I don't have to change in my ssh config each time?
The TCP ports are randomly assigned so unfortunately not
@digigoblin Sorry to keep pinging you - connection works now, but I'm not seeing my secrets show up as environment variables when I SSH in and run something. Is there something special I need to do there?
They were showing up properly when I just SSH'ed in with the other method
I assume you added your secrerts as environment variables to your pod?
Yep
Sorry I just saw now you said you did, my bad
Nw
You basically have to do something like this to make them accessible:
https://github.com/runpod/containers/blob/main/container-template/start.sh#L67-L71
GitHub
containers/container-template/start.sh at main Β· runpod/containers
π³ | Dockerfiles for the RunPod container images used for our official templates. - runpod/containers
Except yours may not begin with RUNPOD_
I need to put this in my docker image is that right?
Or run it on the pod when it's up
Yeah, usually in your docker image start script
Oh in the start script okay
Yep
bash -c 'apt update;DEBIANFRONTEND=noninteractive apt-get install openssh-server -y;mkdir -p ~/.ssh;cd $;chmod 700 ~/.ssh;echo "$PUBLIC_KEY" >> authorized_keys;chmod 700 authorizedkeys;pip install OhMyRunPod; OhMyRunPod --setup-ssh; printenv | grep -E '^RUNPOD|^PATH=|^_=' | awk -F = '{ print "export " $1 "="" $2 """ }' >> /etc/rp_environment; echo 'source /etc/rp_environment' >> ~/.bashrc; service ssh start;sleep infinity'
Something like this in totality?
(Haven't tried adding the OhMyRunPod --setup-ssh as part of the start script, I know it works after I ssh in but hoping it works in the start script itself)
I might add injecting secrets as part of OhMyRunPod
How will it read them though?
Though when I waited up. Same as start script π
But start script has them exposed as environment variables when the container starts. If someone SSH into the pod and they aren't exported by the container, they are no longer available.
Btw OhMyRunPod auto install ssh server if itβs missing π
Yeah saw that, it helped @Harish
Tryna figure out how to escape the ββ characters in the awk command for the start script
Try this:
By the way, RunPod automatically injects a variable called
PUBLIC_KEY into your pod, so thats usually the convention and not PUBLICKEY.service restart
As ohmy auto starts service
Probably don't need to start or restart it then because no changes are made after OhMyRunPod starts it
Probably going to add inject public key and other env should not be hard
Btw OhMy now also supports tailscale
Results in:
awk: 1: unexpected character ''
awk: line 1: runaway string constant " } ...
Probably need to ask ChatGPT or something Β―\_(γ)_/Β―
I would rather just make a custom docker image than to go to all this effort personally
Yeah lol
I have a custom docker image but I'd rather not add runpod-specific stuff into it because I use it across platforms
But if the thing claude gave me doesn't wokr might as well just do that
bash -c '
pip install OhMyRunPod && \
OhMyRunPod --setup-ssh && \
mkdir -p ~/.ssh && \
echo "$PUBLIC_KEY" >> ~/.ssh/authorizedkeys && chmod 700 -R ~/.ssh && \
printenv | grep -E "^RUNPOD|^PATH=|^_=" | awk -F = '''{ print "export " $1 "="" $2 """ }''' >> ~/rp_environment && \
echo '''source ~/rp_environment''' >> ~/.bashrc && \
sleep infinity
'
Hm ok so I got the escaping to work
But seems like the actual command messes up my path somewhere
Full command:
bash -c 'apt update;DEBIANFRONTEND=noninteractive apt-get install openssh-server -y;mkdir -p ~/.ssh;cd $;chmod 700 ~/.ssh;echo "$PUBLIC_KEY" >> authorized_keys;chmod 700 authorizedkeys; pip install OhMyRunPod; OhMyRunPod --setup-ssh; echo "Exporting Env Variables..."; printenv | grep -E "^RUNPOD|^PATH=|^=" | awk -F = '"'"'{ print "export " $1 "="" $2 """ }'"'"' >> /etc/rpenvironment; echo "Finished exporting env variables. Variables from parent:"; printenv | grep -E "^RUNPOD|^PATH=|^="; echo "source /etc/rp_environment" >> ~/.bashrc; service ssh start;sleep infinity'
Container logs:
2024-06-11T01:52:00.108484438+02:00 Exporting Env Variables...
2024-06-11T01:52:00.110996684+02:00 Finished exporting env variables. Variables from parent:
2024-06-11T01:52:00.112944521+02:00 RUNPOD_CPU_COUNT=9
2024-06-11T01:52:00.112962141+02:00 RUNPOD_POD_ID=<redacted>
2024-06-11T01:52:00.112966701+02:00 RUNPOD_MEM_GB=50
2024-06-11T01:52:00.112969881+02:00 RUNPOD_PUBLIC_IP=<redacted>
2024-06-11T01:52:00.112972671+02:00 RUNPOD_GPU_COUNT=1
2024-06-11T01:52:00.112975351+02:00 RUNPOD_POD_HOSTNAME=<redacted>
2024-06-11T01:52:00.112977971+02:00 RUNPOD_GPU_NAME=NVIDIA+GeForce+RTX+3070
2024-06-11T01:52:00.112980611+02:00 RUNPOD_TCP_PORT_22=<redacted>
2024-06-11T01:52:00.112983191+02:00 RUNPOD_APIKEY=<redacted>
2024-06-11T01:52:00.112985921+02:00 PATH=/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-06-11T01:52:00.112989081+02:00 =/usr/bin/printenv
But then after ssh, I get the following as my first messages in console:
bash: uname: No such file or directory
bash: sed: No such file or directory
And then no commands other than cd work
@Papa Madiator @digigoblin
Is this the right script to run?
Going to remove the grep for PATH and see if that fixes it
Okay removing the PATH grep worked - but it's not copying over the right env variables now π’
Unfortunately this still didn't work. I had to actually explicitly grep for the variables I wanted to keep - this makes sense to me too, as the grep for RUNPOD_ and PATH= didn't have the variables I wanted anyways.
Could you confirm this is still a secure way of injecting the variables? @digigoblin @Papa Madiator Here is what I did:
echo "Exporting Env Variables..."; env | grep -E "^(HF|RUNPOD|AWS|WANDB)" | sed "s/^(.)$/export \1/">> /etc/rpenvironment; echo "Finished exporting env variables. Variables from parent:"; env | grep -E "^(HF|RUNPOD|AWS|WANDB)"
Tried multiple versions of RUNPOD and PATH= to no avail. Here's the full command now:
bash -c 'apt update;DEBIANFRONTEND=noninteractive apt-get install openssh-server -y;mkdir -p ~/.ssh;cd $;chmod 700 ~/.ssh;echo "$PUBLIC_KEY" >> authorized_keys;chmod 700 authorized_keys; pip install OhMyRunPod; OhMyRunPod --setup-ssh; echo "Exporting Env Variables..."; env | grep -E "^(HF|RUNPOD|AWS|WANDB)" | sed "s/^(.)$/export \1/">> /etc/rp_environment; echo "Finished exporting env variables. Variables from parent:"; env | grep -E "^(HF|RUNPOD|AWS|WANDB)"; echo "source /etc/rp_environment" >> ~/.bashrc; service ssh start;sleep infinity'
Works for me
Β―\_(γ)_/Β―
Its wasted too much of my time, can't help anymore especially since it worked when I ran it exactly like the last code I gave you
How would it work though? The injected environment variables don't actually start with either RUNPOD_ or PATH or just _
I'm assuming that security is fine though since in essence this is the same thing, just actually finding the right env variables to copy
not sure why you changed awk to sed
I got it working with awk
You can just add the others in here:
sed is what claude gave me lol, also don't have to do the escape characters thing
But yeah adding the others is basically what I'm doing, just want to make sure that's right and secure
I think Claude gave you a wrong answer, awk is working with the solution I gave above, I am not in the mood to change it to sed
Also unrelated - the python env doesn't load when I ssh in with root@<ip>, but it does when I ssh in with <user>@runpod
No no, that part is working! π
Thank you for your help
awk or sed doesn't matter, it is finding and inserting the env variables in correctly
What kind of python env? conda or venv or what?
The question I was asking was more along the lines of is it okay to inject environment variables with this command in the start script
Yeah it should be fine
conda. Am assuming that it's because I'm not exporting PATH though, will figure out how to inject PATH in the same command without overwriting the existing path
Wait nvm scratch that, it does work on root now
Think I just had to restart the terminal
Ok should be all good then, thank you both for your help π really appreciate it
conda = how to complicate your life and waste time on fixing conda bugs
+1 lol, learning about virtualenv is the best thing ever. i used to use conda all the time, but virtualenv just tends to be a lot more replicable i find