R
RunPod•5d ago
Justin

Baking model into Dockerimage

Hello, im trying to bake or rather, downlaod the model via vllm directly while building so the image contains the model. I havent found any kind of simple "vllm download" command sadly. The onlny was is either by running vllm and afterwards adding the file to the image which would be too big to host on my registry or let runpod serverless build the image for me with its doing it while building
4 Replies
3WaD
3WaD•5d ago
Why via vllm? You can do this step in the Dockerfile
Justin
JustinOP•5d ago
wdym? I am not sure how i can prewarm vllm viwthout using vllm to download it as its doing more than just downloding it from huggingface. Im not sure if it needs more but im also unsure how exaxctly to do otherwise
3WaD
3WaD•5d ago
If you want to bake a model into the container image you have to do it while building it. You can't do it afterwards without rebuilding it again, which can't be done on RunPod. To download or copy a model and bake it into the image you can do: 
# Use COPY for local files on your disk 
COPY localfolder/model /containerfolder/model
# Use COPY for local files on your disk 
COPY localfolder/model /containerfolder/model
or 
#!/bin/bash
set -Eeuox pipefail

mkdir -p "$1"
cd "$1"
git init
git remote add origin "$2"
git fetch origin "$3" --depth=1
git reset --hard "$3"
#!/bin/bash
set -Eeuox pipefail

mkdir -p "$1"
cd "$1"
git init
git remote add origin "$2"
git fetch origin "$3" --depth=1
git reset --hard "$3"
# Make sure to have git installed!
COPY clone.sh /clone.sh

# Clone selected HuggingFace repo
RUN . /clone.sh /containerfolder/ https://huggingface.co/user/model
# Make sure to have git installed!
COPY clone.sh /clone.sh

# Clone selected HuggingFace repo
RUN . /clone.sh /containerfolder/ https://huggingface.co/user/model
Justin
JustinOP•5d ago
If you want to bake a model into the container image you have to do it while building it. You can't do it afterwards without rebuilding it again, which can't be done on RunPod.
Exactly thats what i mean, my first thought would simply be using vvlm as its building the neccessary fodler structure but i think you can just skip this by using --model and the local path with that you sent right? will test it out thanks ^^

Did you find this page helpful?