how can I check the logs to see if my request uses the lora model
I deployed the qwen2-7B model using serverless and want to load the adapter checkpoint.
My environment variable configuration is shown in the figure below, where
LORA_MODULES={"name": "cn_writer", "path": "sinmu/cn-writer-qwen-7B-25w", "base_model_name": "Qwen/Qwen2-7B"}
My questions are as follows
1. After serverless is deployed, how can I check the logs to see if my request uses the lora model
2. If the logs cannot be viewed, what should be done to verify
Note: Please ignore the method of checking model performance to verify whether the lora model is used
4 Replies
Hi
you can check using openai package
then query all models in your endpoint
it'll return your lora name only if your settings are correct i think
or you can change your model in your request to
cn_writer (use it like a llm model in your openai client)Thanks for your suggestion
The serverless package is runpod packaged and not open to the public. So I can't modify the log printing logic
The model list returned by model list contains lora model
Requesting cn_writer can get the result smoothly.
It's just that the log does not record the use of the model of the request, which makes me unsure whether the lora part is being used.
When deployed in pod mode, each request will be printed and marked with the lora used
Yes I got your message before, you said theres no logs for Lora loaded, you can see the code in vllm-worker repo in github
Runpod-workers/vllm-worker
if it's logged in pod maybe there's a setting for it, try configuring your endpoint (vllm/text) in your quick deploy
Then it'll show you options
or set the env in your endpoint:
DISABLE_LOG_REQUESTS = false
