codeRetarded
RRunPod
•Created by codeRetarded on 3/12/2024 in #⚡｜serverless
Serverless multi gpu
I have a model deployed on 2 48 GB GPUs and 1 worker. It ran correctly for the first time with cuda distributed. But then fails with this "error_message": "Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument tensors in method wrapper_CUDA_cat)",\n "error_traceback": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.10/dist-packages/runpod/serverless/modules/rp_job.py\\
What can be the issue here?
13 replies
RRunPod
•Created by codeRetarded on 2/7/2024 in #⚡｜serverless
Uploading file to serverless
If I want to upload a file to serverless, then what is the way to do this? Somewhere I saw to use intermediate base64 representation. Setting aside the latency issues for encoding and decoding, what if my file is very large? I have to convert to base64 and then append a very large string to my request, which is not very neat. Also as the file grows larger, we would send large strings, is there any throughput loss compared to multidata-format upload?
6 replies
RRunPod
•Created by codeRetarded on 2/2/2024 in #⚡｜serverless
Docker daemon is not started by default?
In the template I specify docker run command, but the worker cannot execute the container because daemon is not running. docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? I added systemctl start docker command before docker run, but systemctl is not recognised. How to make this container start?
5 replies
RRunPod
•Created by codeRetarded on 1/31/2024 in #⚡｜serverless
Best way to deploy a new LLM serverless, where I don't want to build large docker images
The infrastructure I have come across at runpod, there is not much support for serverless for fast copying of weights of models from a local data centre. Can I get some suggestions on how I should plan my deployment because building large docker images and uploading them on docker file, and then server downloading it at cold start, takes massive time. Help will be appreciated.
27 replies