Network Volume suddenly empty in EU-RO-1
After a restart of a CPU pod that was attached to a network volume located in EU-RO-1 region, the /workspace directory was suddenly empty. No changes were done to the mount path prior to it becoming empty.
Has anyone ever encountered this issue? Is there any way of getting back the data that was on that volume?
Thank you!
12 Replies
I can't reproduce this, works fine for me with GPU as well as CPU pods.
Maybe an issue with the template that your're using that has a conflicting /workspace in WORKDIR
I did go through restarts a few times, for about a week, and it did not happen either.
is there any chance the data on that volume could be recovered?
does the GPU pod still show the volume data? or both gpu and cpu show empty?
I seem to have missed this message, I apologise. Both cpu and gpu show empty
its likely the cpu pod has some code that replaces the data in network volume, you can test that
i've been using it with CPU pod throughout its life. i did start a GPU pod to check the volume there just as an attempt
Which template are you using?
on the CPU pod, i see there is only one template available, Runpod Ubuntu, and I am using that
on the gpu, i was using RunPod Pytorch 2.1
perhaps it is worth noting that throughout the volume's life, it has been used both with CPU and GPU pods, and also with multiple CPU pods at the same time
Thats all fine, not sure why your data would get lost.
Unless there is some issue with the RunPod Ubuntu CPU template.
Just so you know, I decided to forget about that volume and recreate my configs (this time properly backing them up). However, If I can be of any assistance, let me know and I'll help you debug this issue (if you feel like it is worth the time). For safety, I created a second volume, and I intend deleting the previous one in some time
I also increased the size a few times, but i don't think that can be related to the data disappearing
Nah increasing the size doesn't impact your data usually.
size won't change the data, this is the first time hearing of this, trying to think of edge cases where rsync or some utility on boot may wipe it