Runpod Servelerss really unreliable, delay time is way too high sometimes
I'm using a 24 GB vRAM serverless endpoint, the endpoint is way too unstable, 90% of the times the "QUEUE" takes a couple of seconds and then inference of the Omniparser v2 model takes between 3-8 seconds.
This is a replicable result in Google Colab and other GPUs, nonetheless, every once on a while Runpod takes more than 40 seconds ofr even minutes to process a request. This happens when a specific worker bugs and then multiple request goes through it. The worker bugs for no reason and takes multiple minutes to do the job it should do in seconds. This only happens for some workers and when the same worker is used multiple times, it makes no sense and Runpod charges you multiple minutes of DELAY TIME, sometimes it does not even go through, meaning it says "IN_PROGRESS" as seen in the image for multiple minutes without finishing while Runpod charges you every second.
In any other environment and even runpod this process takes seconds, the "IN_PROGRESS" print shows between 3-8 times only. This makes the endpoint highly unstable and way too expensive for a model that does not even use half of the vRAM....
Serverless Deployment runpod request Issue
Im working on deploying the qwen_2.5_instruct model through RunPod using the vLLM direct deployment method. The qwen_2.5_instruct model is designed to more than one image at a time along with the prompt. However, with the vLLM method, RunPod only allows one image per request.
I need to pass multiple images in the following format:
messages = [...
how can I check the logs to see if my request uses the lora model
I deployed the qwen2-7B model using serverless and want to load the adapter checkpoint.
My environment variable configuration is shown in the figure below, where
LORA_MODULES={"name": "cn_writer", "path": "sinmu/cn-writer-qwen-7B-25w", "base_model_name": "Qwen/Qwen2-7B"}...
Troubles with answers
I use MistralAI 7B Instruct model and i use standart settings in serverless and im getting strange answers. I tried different temperament values, but it didn't help.
Please tell me how to fix it...
Docker login to a specific registry
Hi, I'd like to login to nvcr.io using my API token. However, Runpod only allows me to set a username and password, but not let me specify a registry. How can I set this?
I'm following the instructions here: https://build.nvidia.com/nvidia/audio2face-2d/docker...
