RunPod•5d ago

Unable to run jupyter on custom docker image

I am basically trying to build my own custom container and run jupyter lab in it. I've copied the configuration from base rundpod image from github. Everything is fine there. The problem occurs when the pod is booted up as it is supposed to start jupyterlab but we get the jupyter command not found issue which is why port 8888 is not ready. First screenshot is from my local machine where i get into the docker and i see jupyter --help works. Second one is from the runpod instance where it complains that jupyter is not found but then when i install in my conda env it only updates certifi version (as you can see from the screenshot) and then after that jupyter command is available again. I'm puzzle by this behaviour. How can a same docker container behave differently in different environments?

10 Replies

nerdylive•5d ago

Jupyter Lab and its dependencies may not be properly managed or installed in the container's environment, leading to the "jupyter command not found" issue. do you mean you need to reinstall the jupyter everytime you start the pod? hows your dockerfile, and can i see also the script that installs your jupyterlab

Fazil TOP•5d ago

the first screenshot is from my docker container running locally. As you can see jupyter is installed. it works fine there. The command not found is happening only on runpod

Fazil TOP•5d ago

I've digged deep into the start.sh that runpod has, it invokes jupyter if JUPYTER_PASSWORD set. https://github.com/runpod/containers/blob/main/container-template/start.sh#L75 However setting environment variables is happening after that: https://github.com/runpod/containers/blob/main/container-template/start.sh#L96 I'm wondering how that lines gets invoked. Right now i have to open a shell into my pod, do conda install -y jupyter and it just barely updates a file called certifi and then jupyter command can be found. so I launch it and connect it.

GitHub

containers/container-template/start.sh at main · runpod/containers

🐳 | Dockerfiles for the RunPod container images used for our official templates. - runpod/containers

Fazil TOP•5d ago

Perhaps something to do with start jupyter notebook button that default runpod templates have during gpu deployment screen. I don't have that option since I have a custom template.

Fazil TOP•5d ago

This is the checkbox i mean

nerdylive•5d ago

Check the pod after deployment does it generate one automatically? I think it does

Fazil TOP•5d ago

I have the port link but it always says port is not ready. When i click it's HTTP 502 which is why i have to open the web terminal, run 1-2 commands and then it becomes ready Here is my dockerfile for reference:

ARG MINICONDA_VERSION=24.4.0-0

FROM continuumio/miniconda3:${MINICONDA_VERSION}

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

ENV DEBIAN_FRONTEND=noninteractive
ENV SHELL=/bin/bash

WORKDIR /

RUN mkdir /workspace

# git, wget is installed in the base image
RUN apt-get update --yes && \
    apt-get upgrade --yes && \
    apt install --yes --no-install-recommends curl libgl1 software-properties-common openssh-server nginx && \
    apt-get autoremove -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    echo "en_US.UTF-8 UTF-8" > /etc/locale.gen

COPY requirements/ requirements/

RUN conda env create -n py -f requirements/dev/environment-gpu-full.yml && \
    conda clean -afy

# Activate conda env inside
SHELL ["conda", "run", "-n", "py", "/bin/bash", "-c"]

# This line is necessary to run the container interactively
RUN echo "conda activate py" >> ~/.bashrc

# Remove existing SSH host keys
RUN rm -f /etc/ssh/ssh_host_*

# NGINX Proxy
COPY scripts/runpod/docker/nginx.conf /etc/nginx/nginx.conf
COPY scripts/runpod/docker/readme.html /usr/share/nginx/html/readme.html

# Start Scripts
COPY scripts/runpod/docker/start.sh /
RUN chmod +x /start.sh

COPY . /workspace

# Set the default command for the container
CMD [ "/start.sh" ]

ARG MINICONDA_VERSION=24.4.0-0

FROM continuumio/miniconda3:${MINICONDA_VERSION}

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

ENV DEBIAN_FRONTEND=noninteractive
ENV SHELL=/bin/bash

WORKDIR /

RUN mkdir /workspace

# git, wget is installed in the base image
RUN apt-get update --yes && \
    apt-get upgrade --yes && \
    apt install --yes --no-install-recommends curl libgl1 software-properties-common openssh-server nginx && \
    apt-get autoremove -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    echo "en_US.UTF-8 UTF-8" > /etc/locale.gen

COPY requirements/ requirements/

RUN conda env create -n py -f requirements/dev/environment-gpu-full.yml && \
    conda clean -afy

# Activate conda env inside
SHELL ["conda", "run", "-n", "py", "/bin/bash", "-c"]

# This line is necessary to run the container interactively
RUN echo "conda activate py" >> ~/.bashrc

# Remove existing SSH host keys
RUN rm -f /etc/ssh/ssh_host_*

# NGINX Proxy
COPY scripts/runpod/docker/nginx.conf /etc/nginx/nginx.conf
COPY scripts/runpod/docker/readme.html /usr/share/nginx/html/readme.html

# Start Scripts
COPY scripts/runpod/docker/start.sh /
RUN chmod +x /start.sh

COPY . /workspace

# Set the default command for the container
CMD [ "/start.sh" ]

start.sh, nginx.conf is copied from the runpod github repo.

nerdylive•5d ago

What commands is it Can you add it on the start script

Fazil TOP•5d ago

I have done this on runpod (overriding the start cmd) bash -c 'source ~/.bashrc && conda activate py && conda install -y jupyter && ./start.sh' but somehow i still needed webterminal launch jupyter. 🤷‍♂️

nerdylive•5d ago

Maybe it's not run?, it is better to put it inside your docker file

Gaming

Programming

Unable to run jupyter on custom docker image

Did you find this page helpful?