RunPod•12mo ago

How to build worker-vllm Docker Image without a model inside?

I would like to build a worker-vllm with slightly customized vLLM. However, I don't want to build it with a model inside. Basically, it should be the same as your Pre-Built Docker Image that would download the model to the Network Volume. Thanks! https://github.com/runpod-workers/worker-vllm/

GitHub

GitHub - runpod-workers/worker-vllm: The RunPod worker template for...

The RunPod worker template for serving our large language model endpoints. Powered by VLLM. - GitHub - runpod-workers/worker-vllm: The RunPod worker template for serving our large language model en...

7 Replies

kopyl•12mo ago

Just replace this:

export MODEL_BASE_PATH=$MODEL_BASE_PATH && \
export MODEL_NAME=$MODEL_NAME && \
python3.11 /download_model.py --model $MODEL_NAME; \

export MODEL_BASE_PATH=$MODEL_BASE_PATH && \
export MODEL_NAME=$MODEL_NAME && \
python3.11 /download_model.py --model $MODEL_NAME; \

With this

export MODEL_BASE_PATH=$MODEL_BASE_PATH && \
export MODEL_NAME=$MODEL_NAME; \

export MODEL_BASE_PATH=$MODEL_BASE_PATH && \
export MODEL_NAME=$MODEL_NAME; \

in your Dockerfile

antoniogOP•12mo ago

thanks!

kopyl•12mo ago

@antoniog7545 did it work?

antoniogOP•12mo ago

there was another issue, so I couldn't build the image...

kopyl•12mo ago

Which issue?

antoniogOP•12mo ago

I'm getting error when bulding a Docker Image. See below.

1.527   Running command git clone --filter=blob:none --quiet https://github.com/antonioglass/vllm-fork-for-sls-worker.git /src/vllm
4.184   Resolved https://github.com/antonioglass/vllm-fork-for-sls-worker.git to commit 9797380bd9a3eef6a70e899d03b81f6967ecd287
4.192   Installing build dependencies: started
68.88   Installing build dependencies: still running...
131.1   Installing build dependencies: still running...
199.6   Installing build dependencies: still running...
964.9   Installing build dependencies: finished with status 'error'
965.0   error: subprocess-exited-with-error
965.0   
965.0   × pip subprocess to install build dependencies did not run successfully.
965.0   │ exit code: 2

1.527   Running command git clone --filter=blob:none --quiet https://github.com/antonioglass/vllm-fork-for-sls-worker.git /src/vllm
4.184   Resolved https://github.com/antonioglass/vllm-fork-for-sls-worker.git to commit 9797380bd9a3eef6a70e899d03b81f6967ecd287
4.192   Installing build dependencies: started
68.88   Installing build dependencies: still running...
131.1   Installing build dependencies: still running...
199.6   Installing build dependencies: still running...
964.9   Installing build dependencies: finished with status 'error'
965.0   error: subprocess-exited-with-error
965.0   
965.0   × pip subprocess to install build dependencies did not run successfully.
965.0   │ exit code: 2

I'm not sure but it may be related to the small modifications in the vllm-fork-for-sls-worker. I'll try to build the image with the original vllm-fork-for-sls-worker.

antoniogOP•12mo ago

Yes, it's the same issue with the original vllm-fork-for-sls-worker too. I opened an issue: https://github.com/runpod-workers/worker-vllm/issues/25

GitHub

Errors when building the image · Issue #25 · runpod-workers/worker-...

I'm building the image with WORKER_CUDA_VERSION=12.1 on an M1 Mac using command docker buildx build -t antonioglass/worker-vllm-new:1.0.0 . --platform linux/amd64 and getting errors. See below....

Gaming

Programming

How to build worker-vllm Docker Image without a model inside?