Serverless Deployment runpod request Issue
Im working on deploying the qwen_2.5_instruct model through RunPod using the vLLM direct deployment method. The qwen_2.5_instruct model is designed to more than one image at a time along with the prompt. However, with the vLLM method, RunPod only allows one image per request.
I need to pass multiple images in the following format:
messages = [...
how can I check the logs to see if my request uses the lora model
I deployed the qwen2-7B model using serverless and want to load the adapter checkpoint.
My environment variable configuration is shown in the figure below, where
LORA_MODULES={"name": "cn_writer", "path": "sinmu/cn-writer-qwen-7B-25w", "base_model_name": "Qwen/Qwen2-7B"}...
Troubles with answers
I use MistralAI 7B Instruct model and i use standart settings in serverless and im getting strange answers. I tried different temperament values, but it didn't help.
Please tell me how to fix it...
Docker login to a specific registry
Hi, I'd like to login to nvcr.io using my API token. However, Runpod only allows me to set a username and password, but not let me specify a registry. How can I set this?
I'm following the instructions here: https://build.nvidia.com/nvidia/audio2face-2d/docker...
fail: timeout ,exporting to oci image format. This takes a little bit of time. Please be patient.
#31 exporting to oci image format. This takes a little bit of time. Please be patient.
Updated workers to 10, now stuck in a loop
Hey, I upgraded the workers from 5 to 10, but now if I want to see how much money do I need for even more workers, I am stuck on this 10 -> 10 loop.
