JJonahJ
RRunPod
•Created by JJonahJ on 2/13/2024 in #⚡｜serverless
max workers set to 2 but endpoint page shows ‘5 idle’
5 replies
RRunPod
•Created by JJonahJ on 2/9/2024 in #⚡｜serverless
error downloading model? TheBloke/Mixtral-8x7B-MoE-RP-Story-AWQ
2335.9 Traceback (most recent call last):
2335.9 File "/download_model.py", line 48, in <module>
2335.9 tokenizer_folder = download_extras_or_tokenizer(tokenizer, download_dir, revisions["tokenizer"])
2335.9 File "/download_model.py", line 10, in download_extras_or_tokenizer
2335.9 folder = snapshot_download(
2335.9 File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 110, in _inner_fn
2335.9 validate_repo_id(arg_value)
2335.9 File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 164, in validate_repo_id
2335.9 raise HFValidationError(
2335.9 huggingface_hub.utils.validators.HFValidationError: Repo id must use alphanumeric chars or '-',
'', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: ''.
------
Dockerfile:35
--------------------
34 | COPY builder/download_model.py /download_model.py
35 | >>> RUN --mount=type=secret,id=HF_TOKEN,required=false \
36 | >>> if [ -f /run/secrets/HF_TOKEN ]; then \
37 | >>> export HF_TOKEN=$(cat /run/secrets/HF_TOKEN); \
38 | >>> fi && \
39 | >>> if [ -n "$MODEL_NAME" ]; then \
40 | >>> python3 /download_model.py; \
41 | >>> fi
42 |
--------------------
ERROR: failed to solve: process "/bin/sh -c if [ -f /run/secrets/HF_TOKEN ]; then export HF_TOKEN=$(cat /run/secrets/HF_TOKEN); fi && if [ -n "$MODEL_NAME" ]; then python3 /download_model.py; fi" did not complete successfully: exit code: 1
17 replies
RRunPod
•Created by JJonahJ on 2/1/2024 in #⚡｜serverless
quick python vLLM endpoint example please?
…I’ve been on this for 2 hours and the best I can get so far is have a bunch of stuff endlessly ‘queued’.
I’m getting responses from the test thing on the ‘my endpoints’ page but my python script isn’t working… 😅
5 replies
RRunPod
•Created by JJonahJ on 1/28/2024 in #⚡｜serverless
worker vllm 'build docker image with model inside' fails
from the page https://github.com/runpod-workers/worker-vllm?tab=readme-ov-file
Option 2: Build Docker Image with Model Inside To build an image with the model baked in, you must specify the following docker arguments when building the image. Prerequisites RunPod Account Docker Arguments: Required MODEL_NAME Optional MODEL_BASE_PATH: Defaults to /runpod-volume for network storage. Use /models or for local container storage. QUANTIZATION WORKER_CUDA_VERSION: 11.8.0 or 12.1.0 (default: 11.8.0 due to a small amount of workers not having CUDA 12.1 support yet. 12.1.0 is recommended for optimal performance). For the remaining settings, you may apply them as environment variables when running the container. Supported environment variables are listed in the Environment Variables section. Example: Building an image with OpenChat-3.5 sudo docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg MODEL_BASE_PATH="/models" .so I cloned the github thing into a folder, then opened a command prompt inside the same folder as the dockerfile, then put in docker build -t toxibunny/RPmixtralAPI:0.1 --build-arg MODEL_NAME="TheBloke/Mixtral-8x7B-MoE-RP-Story-AWQ" --build-arg MODEL_BASE_PATH="/models" but it came back ERROR: "docker buildx build" requires exactly 1 argument. See 'docker buildx build --help'. Usage: docker buildx build [OPTIONS] PATH | URL | - what am I doing wrong?
13 replies