JHenriP
RRunPod
•Created by JHenriP on 2/26/2024 in #⚡｜serverless
Severless 404
@Merrell @Marut @ashleyk ended up being a problem with flash-attn 🙂
With base image runpod/base:0.6.1-cuda12.2.0 and using an A4000 apparently you can't have flash-attn added to the requirements.txt
21 replies
RRunPod
•Created by JHenriP on 2/26/2024 in #⚡｜serverless
Severless 404
Everything worked fine on local deployment
21 replies
RRunPod
•Created by JHenriP on 2/26/2024 in #⚡｜serverless
Severless 404
There's no error actually, simply nothing happens and looking at the worker utilization nothing is ramping up.
Base image: runpod/base:0.6.1-cuda12.2.0"
Requirements:
torch
hf_transfer
accelerate
flash-attn
transformers
runpod
21 replies
RRunPod
•Created by JHenriP on 2/26/2024 in #⚡｜serverless
Severless 404
Still facing the same issue
@Marut
21 replies
RRunPod
•Created by JHenriP on 2/26/2024 in #⚡｜serverless
Severless 404
Haven't tried since then. Will try again later on today
21 replies
RRunPod
•Created by JHenriP on 2/26/2024 in #⚡｜serverless
Severless 404
had to cancel all requests manually
21 replies
RRunPod
•Created by JHenriP on 2/26/2024 in #⚡｜serverless
Severless 404
found it in this repo
I'm also doing STT
21 replies
RRunPod
•Created by JHenriP on 2/26/2024 in #⚡｜serverless
Severless 404
https://github.com/runpod-workers/worker-faster_whisper/blob/main/src/predict.py
a runpod replacement for torch.cuda
21 replies
RRunPod
•Created by JHenriP on 2/26/2024 in #⚡｜serverless
Severless 404
@ashleyk not related but do you happen to know if it's mandatory to use rp_cuda? My worker is getting stuck and I don't see GPU usage ramping up
21 replies
RRunPod
•Created by JHenriP on 2/26/2024 in #⚡｜serverless
Severless 404
you're right, thx @ashleyk
I misunderstood it
21 replies