Irina
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
and if I write
encoded[key] = encoded[key]
without to(self.device), it's working also! (that is logical)
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
I have changed the string
...
translated = self.model.to(self.device).generate( ...
...
added 'to(self.device)', as you advised me. All other strings are without .to(self.device)
And it's working now!!! ☺️ Great thanks!!
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
https://pytorch.org/cppdocs/notes/inference_mode.html about inference mode you can read here 🙂 it's complicated to say in two words)
I'll try! Thanks!
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
here this function
def translate(self, collection, tgt_lang):
texts = [t.strip() for t in collection]
batch_count = len(texts) / BATCH_SIZE
if batch_count < 1:
batch_count = 1
texts_split = np.split(np.array(texts), batch_count)
result = []
with torch.inference_mode():
for batch in texts_split:
encoded = self.tokenizer.batch_encode_plus(list(batch), max_length=self.max_length, truncation=True, return_tensors="pt", padding=True)
for key in encoded.keys():
encoded[key] = encoded[key].to(self.device)
translated = self.model.generate(**encoded, max_length=self.max_length, forced_bos_token_id=self.tokenizer.lang_code_to_id[tgt_lang])
decoded = self.tokenizer.batch_decode(translated, skip_special_tokens=True)
result.extend(decoded)
return [tr if len(tx)>0 else tx for tr, tx in zip(result, texts)]
This function is a method of my class model
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
What makes the error is my question 🙂 I have just found out that the problem in function, that directly uses the library transformers. I can write its code, if it's ok here)
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
I'm making docker image by myself and I check the device (DEVICE:str = 'cuda' if torch.cuda.is_available() else 'cpu')
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
Serverless running
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
It's in the name of the thread: "Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!"
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
I'm simply running facebook/nllb-200-distilled-600M model from Huggingface. That's all)
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
It's working in other environment (Beam, if you ever heart of it), so the problem that it doesn't work in RunPod...This conversation takes me some interesting debugging ideas, so I'll try and will return :))
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
I mean, what's the difference if I run cpu or gpu on totally the same code? Why is it OK with cpu, but it isn't OK with gpu?
Sorry, if I'm writing something strange 🙂
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
I understand this, of cause 🙂 But what does this change? I have device choice in code so gpu is used, when it possible.
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
Ok, then what's the difference between local and serverless(in RunPod) model running? It works locally..
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
Yes, I use facebook/nllb-200-distilled-600M from huggingface
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
Sorry, but I don't have any checkpoint file in my project... or I don't know about it 🙂 I'm a newbie in this field..
70 replies
RRunPod
•Created by ricopella on 1/11/2024 in #⚡｜serverless
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
Hello!
I'm making a Serverless NLLB Endpoint on RunPod.
I have built and pulled a docker image that works perfectly well locally, but when I deploy it on RunPod, it doesn't work with the same error (Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)
Does anybody have some ideas about what's going wrong?
70 replies