R
RunPod•5w ago
Bell Chen

Pytorch Lightening training DDP strategy crashed with no error caught on multi-GPU worker

It looks like serverless worker will crash when spawning new processes from the handler. It crashes after the first process is spawned "Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2". Same code works fine in multi-GPU pod web terminal.
2 Replies
nerdylive
nerdylive•5w ago
what process is that? Python with pytorch lightning code? I tried using multi processes from the handler and it works fine ( I'm sure it's not the same use as yours ) try this : 
# from a bash or sh
unset LOCAL_RANK

# from a python file ( maybe inside your  handler )
del os.environ['LOCAL_RANK']
# from a bash or sh
unset LOCAL_RANK

# from a python file ( maybe inside your  handler )
del os.environ['LOCAL_RANK']
Bell Chen
Bell Chen•5w ago
Oh... yes. I will try
Want results from more Discord servers?
Add your server
More Posts
How to get a public URL?I don't want localhost because I can't access it locally.Quick question about what "Extra workers" areI hit a serverless endpoint a bunch of times in succession, and it spun up two other workers, which Why is the pod still not up even though the "Container is ready"? Using SD 1.9.4 and net volume.Parallel processing images with different promptHi! I am running a1111 on serverless. Is it possible to generate images in parallel with differentSDXL Serverless Worker: How to Cache LoRA modelsIn this code from github sdxl serverless worker repo, how does I cache LoRA models and get there pathow to deploy custom image gen model on serverless?https://blog.runpod.io/custom-models-with-serverless-and-automatic-stable-diffusion/ i read this butGPU Pods in EU-SE-1 unexpectedly die after approximately 30 hoursWe are experiencing many instances of GPU pods (mainly A6000) that stop working after 30 hours losincpu instances don't work> 2024-06-05T20:19:37Z create container runpod/base:0.5.1-cpu > 2024-06-05T20:19:38Z 0.5.1-cpu PulliLoading LLama 70b on vllm template serverless cant answer a simple question like "what is your name"I am loading with 1 worker and 2 GPU's 80g But the model just cant performance at all, it gives gibNetworking Multiple Pods TogetherI'm looking to train a distributed model on runpod. When configuring the torch.distributed or jax.di