Arad
RRunPod
•Created by Arad on 10/29/2024 in #⚡｜serverless
Deploying bitsandbytes-quantized Models on RunPod Serverless using Custom Docker Image
Hey everyone 👋
Looking for tips from anyone who's worked with bitsandbytes-quantized models on RunPod's serverless setup. It's not available out of the box with vLLM, and I was wondering if anyone's got it working? Saw a post in the serverless forum about maybe using a custom Docker image for this.
For context: I've fine-tuned LLaMA-3.1 70B-instruct using the unsloth library (which utilizes bitsandbytes for quantization) and am looking to deploy it.
Any insights would be greatly appreciated! 🙏
2 replies