nielsrolf
Starting a pod with runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04 has cuda version 12.6
I am confused what determines the cuda version of a pod I start. I would expect that when I start a docker image with a cuda version in the name that it has this cuda version bundled into the image and when I start the pod that this is the cuda version I see, but this is not the case. How can I start a pod with a predictable cuda version?
6 replies
RRunPod
•Created by nielsrolf on 11/12/2024 in #⚡｜serverless
Incredibly long startup time when running 70b models via vllm
11 replies