TanegashimaGunsmith

PostsComments

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

Thank you!

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

At least LMDeploy seems to work for now

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

In the vllm github

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

No solutions there either

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

There are issues on Github as well, where people are stuck on 100% CPU but low GPU utilisation

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

Huh, okay

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

So, it might just be vLLM that's broken

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

I didn't use a A6000 on runpod, but on Hetzner In the meantime I have setup LMDeploy and it seems to fully utilise the A100 on runpod

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

On a different hoster

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

I tried it on an Ada 6000 or so, and it had regular performance which is why I'm surprised

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

I have deleted and recreated it, and it's the sane

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

I'm starting to think it has something to do with the JSON output I'm generating, maybe

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

A100 gpu and Qwen2.5 32B Instruct

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

Thanks for the video!

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

No, tokens per second is very low for me (10-12 tps)

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

but it does show up in nvtop

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

I don't see a process there either, neither for lmdeploy

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

100% CPU and ~50% GPU or 100% GPU?

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

Yes, the same issue with a new pod

82 replies

RRunPod

•Created by Thibaud on 8/13/2024 in #⛅｜pods

vllm seems not use GPU

Yes, after deleting I still have this problem

82 replies