Runpod storage configuration
Hello, I am working with a dataset off 4 mill files. Normally it takes up about 23gb on my pc. But on runpod its about 10-15x that size. The reason I have found is that runpod takes up 100kb for a file. My files are each around 6-10 kb, but ubuntu which pod is running from is configured to allocate 100kb min. for a file. I don't have permission within the pod to manage that. What can I do about this?
2 Replies
an alternatve: use databases to store those or alike
or maybe ask staff in ticket for this case
For our file system, having a lot of small files can create extra overhead and impact disk usage