Runpod storage configuration
Hello, I am working with a dataset off 4 mill files. Normally it takes up about 23gb on my pc. But on runpod its about 10-15x that size. The reason I have found is that runpod takes up 100kb for a file. My files are each around 6-10 kb, but ubuntu which pod is running from is configured to allocate 100kb min. for a file. I don't have permission within the pod to manage that. What can I do about this?
5 Replies
an alternatve: use databases to store those or alike
or maybe ask staff in ticket for this case
For our file system, having a lot of small files can create extra overhead and impact disk usage
sqlite is good
Yup sure depending on your requirements
this will depend on your use case, you can store tar file in network storage, and then extract in local container disk for actual work, that way network storage is looking at 1 or few files instead of millions