panos.firbas
I'm pretty sure I've been getting pods where "/" lives on a network disk
Which makes data reading impossible and the pod useless.
I have a little 'see how fast you can read the database' script: It does 6000 iterations/second in my home machine, reading from an SSD. It gets something like 600 iterations/second in "normal" pods, reading my data from /. If I try to read my data from a network storage it gets 2 (!!) iterations per second.
This pod which I just closed cause I don't enjoy paying for useless things also got exactly 2 iterations per second while reading from /. Exactly as if the / exists on the network instead of in the machine itself.
RunPod Pytorch 2.1 ID: wx3xh0bel2o00d 1 x A100 SXM 80GB 16 vCPU 250 GB RAM runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04 On-Demand - Secure Cloud This is unacceptable.
I have a little 'see how fast you can read the database' script: It does 6000 iterations/second in my home machine, reading from an SSD. It gets something like 600 iterations/second in "normal" pods, reading my data from /. If I try to read my data from a network storage it gets 2 (!!) iterations per second.
This pod which I just closed cause I don't enjoy paying for useless things also got exactly 2 iterations per second while reading from /. Exactly as if the / exists on the network instead of in the machine itself.
RunPod Pytorch 2.1 ID: wx3xh0bel2o00d 1 x A100 SXM 80GB 16 vCPU 250 GB RAM runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04 On-Demand - Secure Cloud This is unacceptable.
2 replies
Disk reading unacceptably and mind boggingly slow
I thought I had figured out where to put my data:
pay for extra disk space in /
move the data from /workspace (which is a network drive) to / and from there it can be read fast enough.
But today, I tried the same thing in this pod:
RunPod Pytorch 2.1 ID: yuicz8xifys47n 1 x A100 SXM 80GB 16 vCPU 250 GB RAM runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04 and the reading speed from / was as horrible as reading from /workspace. the pod is down now cause obviously i'm not keen to pay for useless machinery So what are we supposed to do I don't understand how anyone can do ML like this.
RunPod Pytorch 2.1 ID: yuicz8xifys47n 1 x A100 SXM 80GB 16 vCPU 250 GB RAM runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04 and the reading speed from / was as horrible as reading from /workspace. the pod is down now cause obviously i'm not keen to pay for useless machinery So what are we supposed to do I don't understand how anyone can do ML like this.
6 replies
How should I store/load my data for network storage?
Hi,
I've been keeping my data in an sql database which is excruciatingly slow on runpod with a network storage.
But I don't see any obvious alternative.. In what type of file could my data live in on the disk, in order for it to be loaded fast in the network storage scenario of runpod?
But I don't see any obvious alternative.. In what type of file could my data live in on the disk, in order for it to be loaded fast in the network storage scenario of runpod?
142 replies