Extremely slow upload to HuggingFace
For the past 10 hours or so I've had issues uploading to HuggingFace on my pod (jhw1d9hmjb8d3v).
speedtest-cli shows acceptable speeds, but specifically uploading to HuggingFace goes below 1MB/s often.6 Replies
try enabling hf transfer, then rerunning it
Use hf_transfer: this is a Rust-based library meant to speed up uploads on machines with very high bandwidth. To use it, you must install it (pip install hf_transfer) and enable it by setting
HF_HUB_ENABLE_HF_TRANSFER=1 as an environment variable.
You can then use huggingface_hub normally.
Disclaimer: this is a power user tool. It is tested and production-ready but lacks user-friendly features like progress bars or advanced error handling. For more details, please refer to this https://huggingface.co/docs/huggingface_hub/v0.27.0.rc1/package_reference/environment_variables#hfhubenablehftransfer
hf_transfer was installed and enabled. I uploaded everything to a S3 bucket and uploaded to HF from another pod. It's quite annoying though, because this can become very costly very quickly.
I see yeah, are you uploading from a network storage?
@nikolai
Escalated To Zendesk
The thread has been escalated to Zendesk!
No, it was a "local" persistent volume. Uploading to S3 was fast in the end, but to HF was very problematic.