Can I Sync Contabo storage
Contabo has a S3 compatible Object Storage same as Amazon S3.
https://docs.contabo.com/docs/products/Object-Storage/s3-connection-settings
I tried to sync using Amazon option but it didn't work since there is no place the change the end point.
...
Frequent GPU problem with H100
Hello,
I've seen that 9 times out of 10, I would get an H100 (PCIe) machine where in Cuda won't work with torch.
For instance, this machine runs Cuda 12.2, but the torch-cuda integration is broken?
@JM or someone from the RunPod team, can you please see since it's happening extremely frequently now?
ID: b6d3hcqct79d7o, runpod/pytorch:2.2.0-py3.10-cuda12.1.1-devel-ubuntu22.04...
Solution:
@Dhruv Mullick H100 PCIe have caused us lots of headaches lately. We are soon releasing a very powerful detection tool for the totality of RunPod servers, which will help us fix these non trivial issues.
It seems it's always around some specific kernel version that might not be compatible even though it's supposed to be. That being said, expect a strong resolution in the near term!...
OSError: [Errno 5] Input/output error
The model training stopped in the middle of the night with I/O error, apparently it is due to physical disk problem, and i tested it is randomly occur. the consequence is that make my pod idling for at least 6 hours, and i paid for it.
1. How to stop it happen again?
2. Can i claim it back for those idling hours?
...
GPU cloud storage GONE + billed for entire month
I'm seriously starting to think you guys have something against me.
This is the second time you've lost my pod since I started casually using your service, but this time it's borderline illegal.
I've been billed for the entire month of Feburary with no indication of when, or if you deleted my pod again. I specifically chose a different setup so it WOULD NOT get deleted like last time. ...
Trying to create a Spot GPU instance leads to 400 response error
Greetings,
I've recently noticed that whenever I try to create a GPU spot instance, I'm getting a 400 responses error.
I'm trying to spin up a spot instance using the RunPod PyTorch 2.2.10 image, with 2x A6000, a network drive, in the Sweden datacenter.
...
Solution:
Yeah we pushed a fix just a little bit ago
Where are all the U.S. network volume data centers?
It's been like this for at least a week. I'm in the U.S. There used to be Kansas options, but now I can only select Canada or Europe.
Is there some kind of data center outage?...
inconsistent speeds--community pod, any tips
Doing some upscales in SD forge--abit intensive 15mb image output, but ETA/UI seems to keep hanging, any troubleshooting tips? are community pods less stable?
2024-03-01T16:08:54.761577365Z [FATAL tini (6)] exec docker failed: No such file or directory Error
Hey folks I'm having trouble running my image on Runpod. My image works properly on a normal root access Docker environment but doesn't work on the Runpod template structure. It's my first time on Runpod, can anyone help out?
We have detected a critical error on this machine which may affect some pods.
Hey all. We're renting a number of H100s as a trial run of Runpod as we are looking for another compute provider. We paid for 24 hours of compute in order to transfer terabytes of data onto the machine, alongside paying for bandwidth and additional storage. We additionally paid our cloud provider egress costs, which is more than we paid for the H100 machine, and rented a disk & network optimized machine in order to transfer the data quickly to the Runpod machine.
After 24 hours, we are getting this error on the Runpod GUI:
We have detected a critical error on this machine which may affect some pods. We are looking into the root cause and apologize for any inconvenience. We would recommend backing up your data and creating a new pod in the meantime.
...
