Issues with connecting/initializing custom docker image
I've created a custom docker image for quick ocr training; https://hub.docker.com/repository/docker/jeffchen23/paddleocr-image/general
The problem is, everything downloads properly, but then I am unable to connect. When trying to connect, I get
Permission denied (publickey); but the permissions are not an issue for any of my other pods. I think it is because the pod fails to initialize correctly, as it constantly spams messages of Start container.
Can anyone help me pin down this issue? It works on my local machine when I pull it from the web. My local docker command is as follows: docker run -it --runtime nvidia --shm-size 2g --gpus all -v paddleocr-volume:/PaddleOCR paddleocr-image bash
It doesn' t look like I have any direct control over the Docker command from RunPod (from what I can tell), so I'm a little lost.25 Replies
My dockerfile, for reference
You need to keep the pod alive by adding
sleep infinity.Where would I add that?
Also need to add
CMD or ENTRYPOINTHere is an example
https://github.com/ashleykleynhans/fooocus-docker
GitHub
GitHub - ashleykleynhans/fooocus-docker: Docker image for Fooocus: ...
Docker image for Fooocus: Focus on prompting and generating - GitHub - ashleykleynhans/fooocus-docker: Docker image for Fooocus: Focus on prompting and generating
If I do slep infinity, I can't run any bash commands. My goal is to be able to run arbitrary bash commands after it starts?
Use web terminal, SSH etc, you can't run arbitrary bash commands with RunPod.
I would like to be able to do so, would that be via TCP? For some reason, it says that I cannot have any TCP connections to this specific RunPod. For other pods with images (like PyTorch) I am able to connect my VSCode to it
Not possible, RunPod is not normal Docker
What do you mean by that? I'm setting my custom docker image as the template for the pod, the same as the pytorch custom docker images, right?
You have to make your image work the same way as the PyTorch image, you can't run arbitrary bash commands with RunPod as I keep saying, you have to use web terminal or SSH but I am done talking to you now. You think you are too smart to listen so you will get nowhere, good luck wit that.
OK, I have no problem admitting I am confused beyond belief and would love help. I am trying to connect it via ssh.
GitHub
GitHub - justinwlin/AudioCraft-Runpod-Serverless-and-GPU-Pod-NOT-A-...
AudioCraft public example runpod. Contribute to justinwlin/AudioCraft-Runpod-Serverless-and-GPU-Pod-NOT-A-RUNNING-EXAMPLE development by creating an account on GitHub.
Can read this :).
Tldr runpod has some official templates that u can use as ur base image. The reason why is bc under the hood runpod templates does some extra stuff like running a jupyter server / also openssh so u can ssh into the pod.
By using their template as the base image u get a lot of work done for u
RunPod
Customize a Template
A RunPod template is just a Docker container image paired with a configuration. You can choose how deep you want to get into template customization, depending on your skill level.The easiest is to simply start with a RunPod official template or community template and use it as-is.If you want better ...
If u want a custom image just make sure still call their start script
RunPod Blog
DIY Deep Learning Docker Container
Are you tired of using someone else's container, only to find out that they have the wrong versions of your tools installed? Maybe you have just installed everything from scratch every time you wanted to start over and thought to yourself, "this is a waste of time"? I've personally gone
This is a more from scracth method that cna walk u thro what is happening in bg of some the more official stuff
To do this ill say:
1) Just use a runpod template as a base image, probably the pytorch one
2) make sure u still called the cmd command correctly to have the start.sh.
(U can open up a pytorch template to make sure u got the directory correct)
3) Whatever ur cmd + start.sh, ull have the openssh server in the bg to ssh into just like any other official templates
That makes sense, I'll try it now
It worked! Thanks for the help, I couldn't get start.sh to work properly but ended up using this as the docker command:
bash -c "apt update;apt install -y wget;DEBIAN_FRONTEND=noninteractive apt-get install openssh-server -y;mkdir -p ~/.ssh;cd $_;chmod 700 ~/.ssh;echo $PUBLIC_KEY > authorized_keys;chmod 700 authorized_keys;service ssh start;sleep infinity"
I have another question though. My Dockerfile correctly fills the /PaddleOCR folder of the RunPod, but only when the volume mount path is something different. If I try to point the volume mount path also at /PaddleOCR, when I connect, it's empty. I thought that the volume mount path is intended to keep my files, and I want to keep the contents of PaddleOCR in the event of a shutdown?This is a huge issue I had too haha. If you mount a volume it overshadows the local path to that thing. You prob don't need to mount the volume to /PaddleOCR since it will automatically be in the base snapshot. Just save anything you have into the /volume path you have elsewhere.
This is just the way that volume mounting works inherently
So can set your mount to somewhere else
and you'll see your stuff
I also wondered why my stuff kept disappearing before
you can also symlink if you want
so if you got a folder you have in /paddleocr, you can symlink it to wherever you end up mounting
I'll try that
This is so smart, ill keep this in mind
Luckily everything works, just a little unfortunate they can't save it properly. Not such a big deal for me. Thanks again for the help!
Not me lol, I got it from the Ubuntu template base image that runpod provides 😆 Just had the change the environment variable
What you are saying is technically right, but here is the small problem runpod is marketing it as "serverless API" and if I have to ssh into some server to do the same thing what's the point of it? The same spec servers are almost at half price if calculated per hour and I can do the same thing there by buying some cheap domain and making my API live there.
Just a little suggestion to runpod, Add "execute_script "$1" "Running user script..." after post_script execution in start.sh and user can add CMD ["/start.sh", "/user/script/path"] to Dockerfile. It would easily run the user's script after the usual setup.
You can always set an active worker if you don't want to spin up a pod and SSH into the active worker.
CPU pods are significantly cheaper though so I don't recommend doing that.
Actually I am not sure whether CPU pods support network volumes yet though.
Looks like they are only supporte in the RO region.