BenW
CDCloudflare Developers
•Created by BenW on 4/5/2024 in #workers-help
R2 multipart upload complete function is taking a large amount of cpu time.
When looking at the metrics for our worker that handles file uploads we are noticing that the "complete()" function of multipart upload is spiking the cpu usage, is this expected behavior given that the work should be carried out on the R2 infrastructure not on the worker end?
1 replies
CDCloudflare Developers
•Created by BenW on 3/12/2024 in #workers-help
R2 multipart uploads do not work in local dev mode with long object keys.
When creating multipart uploads an error is thrown when it comes to calling the complete method once all the parts are successfully uploaded:
"One or more of the specified parts could not be found. (10025)"
If it hit the endpoint with shorter object keys the complete method works as expected.
My object keys contain uuids and those are what are pushing it to error despite being well within the documented limits.
(Object key length: 1,024 bytes)
An example key being:
a72726b7-ff9d-4697-8ac5-62c5fea7b8ad/bc6e4e12-010c-45cc-87e4-05ba9d4f02b9/filename.jpg
If I run the worker in remote mode, it works as expected but remote mode is not a viable route for me for local testing as I need 2 way communication between our api with the worker which only works in local mode
Note: using wrangler 2.21.0 as all newer versions do not seem to work yet on our docker containers.
Do let me know if you have come across anything like this and if it is known bug or if there are any workarounds anyone can think of.
Many thanks!
(I wonder if it could be to do with hitting some other limit somewhere, I can see the multipart files are stored here:
/tmp/tmp-26-UDK4cFSR2HZG/.mf/r2/BUCKET/MINIFLARE_INTERNAL/multipart/Qmfxkugic7qV3BuCyZ9CxRuYnFFUAmNLUuzb8Ed5vqOIyXW7-AAspcJCOQRS5TcuFUTIqsh5Klhek-9aQbxTEo5cetz-efvFKV9PzdpPUUOWuXhqxTl-_ghhtbVLPlJOIxb-Gx1ee4jDkw4RTDYakkyXXqykBuXO373Dnq9d63U/a72726b7-ff9d-4697-8ac5-62c5fea7b8ad/bc6e4e12-010c-45cc-87e4-05ba9d4f02b9/filename.jpg
and perhaps this is too long when it included the multi-part id)
FYI my current workaround it to use a very short temp key when using dev and then once the file is complete (all parts stitched together) I then clone the object using the original key which works fine but it a rather annoying hack) - I don't see why the multipart upload even needs a key technically as you get a unique id for it anyway so perhaps a key could just be used for the final step in the multipart complete.
3 replies