R
RunPod•2mo ago
Fazil T

Unable to run jupyter on custom docker image

I am basically trying to build my own custom container and run jupyter lab in it. I've copied the configuration from base rundpod image from github. Everything is fine there. The problem occurs when the pod is booted up as it is supposed to start jupyterlab but we get the jupyter command not found issue which is why port 8888 is not ready. First screenshot is from my local machine where i get into the docker and i see jupyter --help works. Second one is from the runpod instance where it complains that jupyter is not found but then when i install in my conda env it only updates certifi version (as you can see from the screenshot) and then after that jupyter command is available again. I'm puzzle by this behaviour. How can a same docker container behave differently in different environments?
No description
No description
19 Replies
nerdylive
nerdylive•2mo ago
Jupyter Lab and its dependencies may not be properly managed or installed in the container's environment, leading to the "jupyter command not found" issue. do you mean you need to reinstall the jupyter everytime you start the pod? hows your dockerfile, and can i see also the script that installs your jupyterlab
Fazil T
Fazil TOP•2mo ago
the first screenshot is from my docker container running locally. As you can see jupyter is installed. it works fine there. The command not found is happening only on runpod
Fazil T
Fazil TOP•2mo ago
I've digged deep into the start.sh that runpod has, it invokes jupyter if JUPYTER_PASSWORD set. https://github.com/runpod/containers/blob/main/container-template/start.sh#L75 However setting environment variables is happening after that: https://github.com/runpod/containers/blob/main/container-template/start.sh#L96 I'm wondering how that lines gets invoked. Right now i have to open a shell into my pod, do conda install -y jupyter and it just barely updates a file called certifi and then jupyter command can be found. so I launch it and connect it.
GitHub
containers/container-template/start.sh at main · runpod/containers
🐳 | Dockerfiles for the RunPod container images used for our official templates. - runpod/containers
Fazil T
Fazil TOP•2mo ago
Perhaps something to do with start jupyter notebook button that default runpod templates have during gpu deployment screen. I don't have that option since I have a custom template.
Fazil T
Fazil TOP•2mo ago
This is the checkbox i mean
No description
nerdylive
nerdylive•2mo ago
Check the pod after deployment does it generate one automatically? I think it does
Fazil T
Fazil TOP•2mo ago
I have the port link but it always says port is not ready. When i click it's HTTP 502 which is why i have to open the web terminal, run 1-2 commands and then it becomes ready Here is my dockerfile for reference: 
ARG MINICONDA_VERSION=24.4.0-0

FROM continuumio/miniconda3:${MINICONDA_VERSION}

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

ENV DEBIAN_FRONTEND=noninteractive
ENV SHELL=/bin/bash

WORKDIR /

RUN mkdir /workspace

# git, wget is installed in the base image
RUN apt-get update --yes && \
    apt-get upgrade --yes && \
    apt install --yes --no-install-recommends curl libgl1 software-properties-common openssh-server nginx && \
    apt-get autoremove -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    echo "en_US.UTF-8 UTF-8" > /etc/locale.gen

COPY requirements/ requirements/

RUN conda env create -n py -f requirements/dev/environment-gpu-full.yml && \
    conda clean -afy

# Activate conda env inside
SHELL ["conda", "run", "-n", "py", "/bin/bash", "-c"]

# This line is necessary to run the container interactively
RUN echo "conda activate py" >> ~/.bashrc

# Remove existing SSH host keys
RUN rm -f /etc/ssh/ssh_host_*

# NGINX Proxy
COPY scripts/runpod/docker/nginx.conf /etc/nginx/nginx.conf
COPY scripts/runpod/docker/readme.html /usr/share/nginx/html/readme.html

# Start Scripts
COPY scripts/runpod/docker/start.sh /
RUN chmod +x /start.sh

COPY . /workspace

# Set the default command for the container
CMD [ "/start.sh" ]
ARG MINICONDA_VERSION=24.4.0-0

FROM continuumio/miniconda3:${MINICONDA_VERSION}

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

ENV DEBIAN_FRONTEND=noninteractive
ENV SHELL=/bin/bash

WORKDIR /

RUN mkdir /workspace

# git, wget is installed in the base image
RUN apt-get update --yes && \
    apt-get upgrade --yes && \
    apt install --yes --no-install-recommends curl libgl1 software-properties-common openssh-server nginx && \
    apt-get autoremove -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    echo "en_US.UTF-8 UTF-8" > /etc/locale.gen

COPY requirements/ requirements/

RUN conda env create -n py -f requirements/dev/environment-gpu-full.yml && \
    conda clean -afy

# Activate conda env inside
SHELL ["conda", "run", "-n", "py", "/bin/bash", "-c"]

# This line is necessary to run the container interactively
RUN echo "conda activate py" >> ~/.bashrc

# Remove existing SSH host keys
RUN rm -f /etc/ssh/ssh_host_*

# NGINX Proxy
COPY scripts/runpod/docker/nginx.conf /etc/nginx/nginx.conf
COPY scripts/runpod/docker/readme.html /usr/share/nginx/html/readme.html

# Start Scripts
COPY scripts/runpod/docker/start.sh /
RUN chmod +x /start.sh

COPY . /workspace

# Set the default command for the container
CMD [ "/start.sh" ]
start.sh, nginx.conf is copied from the runpod github repo.
nerdylive
nerdylive•2mo ago
What commands is it Can you add it on the start script
Fazil T
Fazil TOP•2mo ago
I have done this on runpod (overriding the start cmd) bash -c 'source ~/.bashrc && conda activate py && conda install -y jupyter && ./start.sh' but somehow i still needed webterminal launch jupyter. 🤷‍♂️
nerdylive
nerdylive•2mo ago
Maybe it's not run?, it is better to put it inside your docker file
zfmoodydub
zfmoodydub•3w ago
@Fazil T did you end up figuring this out? im also having the same problem
zfmoodydub
zfmoodydub•3w ago
@Fazil T @nerdylive Start Jupyter Notebook boolean is not even appearing for me on the deployment configuration
No description
zfmoodydub
zfmoodydub•3w ago
added this inside my dockerfile: RUN echo '#!/bin/bash \n\ echo "Starting Jupyter Lab..." \n\ cd / && \n\ conda run -n automasher jupyter lab --allow-root \n\ --no-browser \n\ --port=8888 \n\ --ip=* \n\ --FileContentsManager.delete_to_trash=False \n\ --ContentsManager.allow_hidden=True \n\ --ServerApp.terminado_settings='"'"'{"shell_command":["/bin/bash"]}'"'"' \n\ --ServerApp.token="" \n\ --ServerApp.password="" \n\ --ServerApp.allow_origin=* \n\ --ServerApp.preferred_dir=/workspace &> /workspace/logs/jupyter.log & \n\ echo "Jupyter Lab started" \n\ conda activate automasher \n\ tail -f /dev/null' > /start_jupyter.sh && \ chmod +x /start_jupyter.sh ENTRYPOINT ["/start_jupyter.sh"] i must be doing somethign wrong
nerdylive
nerdylive•3w ago
You don't need to use the start jupyter notebook switch,just hardcode it if you want to start jupyter everytime
Fazil T
Fazil TOP•3w ago
You have to launch the pod as is and port will be available after a while if your configuration is correct. The boolean box is somehow for rundpod template so it won't appear for custom dockers
zfmoodydub
zfmoodydub•3w ago
what does just hardcode it mean? also how do i know if my configuration is correct?
nerdylive
nerdylive•3w ago
Hard code to start your jupyter without any switch or conditions
zfmoodydub
zfmoodydub•3w ago
got it to work, thanks guys
Madiator2011 (Work)
Madiator2011 (Work)•3w ago
code server > jupiter

Did you find this page helpful?