worker vllm 'build docker image with model inside' fails
from the page https://github.com/runpod-workers/worker-vllm?tab=readme-ov-file
Option 2: Build Docker Image with Model Inside To build an image with the model baked in, you must specify the following docker arguments when building the image. Prerequisites RunPod Account Docker Arguments: Required MODEL_NAME Optional MODEL_BASE_PATH: Defaults to /runpod-volume for network storage. Use /models or for local container storage. QUANTIZATION WORKER_CUDA_VERSION: 11.8.0 or 12.1.0 (default: 11.8.0 due to a small amount of workers not having CUDA 12.1 support yet. 12.1.0 is recommended for optimal performance). For the remaining settings, you may apply them as environment variables when running the container. Supported environment variables are listed in the Environment Variables section. Example: Building an image with OpenChat-3.5 sudo docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg MODEL_BASE_PATH="/models" .so I cloned the github thing into a folder, then opened a command prompt inside the same folder as the dockerfile, then put in docker build -t toxibunny/RPmixtralAPI:0.1 --build-arg MODEL_NAME="TheBloke/Mixtral-8x7B-MoE-RP-Story-AWQ" --build-arg MODEL_BASE_PATH="/models" but it came back ERROR: "docker buildx build" requires exactly 1 argument. See 'docker buildx build --help'. Usage: docker buildx build [OPTIONS] PATH | URL | - what am I doing wrong?
GitHub
GitHub - runpod-workers/worker-vllm: The RunPod worker template for...
The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - GitHub - runpod-workers/worker-vllm: The RunPod worker template for serving our large language model en...
8 Replies
here's what chatgpt says btw:
The error mentioning docker buildx could be related to the fact that you have Docker Buildx installed on your system. Docker Buildx is an extended version of the Docker CLI with support for multiple builder instances. It might be automatically intercepting your docker build command.
You can try running the docker build command without the Buildx features. You can do this by setting the DOCKER_BUILDKIT=0 environment variable.
I did that with $env:DOCKER_BUILDKIT=0 and after setting my repository name to all lowercase, it's doing something. fingers crossed π
Step 5/13 : RUN --mount=type=cache,target=/root/.cache/pip python3 -m pip install --upgrade pip && python3 -m pip install --upgrade -r /requirements.txt
the --mount option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled
so nah. I need help π
now I'm getting this:
Traceback (most recent call last):
File "/src/download_model.py", line 3, in <module>
from vllm.model_executor.weight_utils import prepare_hf_model_weights
ModuleNotFoundError: No module named 'vllm'
Dockerfile:29
--------------------
28 |
29 | >>> RUN --mount=type=secret,id=HF_TOKEN,required=false \
30 | >>> if [ -f /run/secrets/HF_TOKEN ]; then \
31 | >>> export HF_TOKEN=$(cat /run/secrets/HF_TOKEN); \
32 | >>> fi && \
33 | >>> if [ -n "$MODEL_NAME" ]; then \
34 | >>> python3 /src/download_model.py --model $MODEL_NAME; \
35 | >>> fi
36 |
--------------------
ERROR: failed to solve: process "/bin/sh -c if [ -f /run/secrets/HF_TOKEN ]; then export HF_TOKEN=$(cat /run/secrets/HF_TOKEN); fi && if [ -n "$MODEL_NAME" ]; then python3 /src/download_model.py --model $MODEL_NAME; fi" did not complete successfully: exit code: 1
I am really trying. I give up for tonight though π
@propback
Will check this out today!
Just pushed an update, try it out
Thankyou very much, it's working π
...and I see what you did to fix it. I am learning, lol π
Haha it was my oversight
Thanks for pointing this issue out!