Cloudflare Developers•2y ago

Unable to perform multi-part upload for files

Hi all, I'm currently trying to use a worker to perform a multipart upload of a file on my front-end. Since my website is hosted on vercel, I have no choice but to tunnel my request through their APIs in order to reach my worker. As a result, I have a hard 8mb limit on the size of my function body. However, when I try to perform a multi-part upload, I get the following error

Error Encountered!
Error: completeMultipartUpload: Your proposed upload is smaller than the minimum allowed object size.
    at R2MultipartUpload.complete (/Users/ivanleo/Documents/Coding/schulz/tut/node_modules/@miniflare/r2/src/multipart.ts:490:15)
    at Object.fetch (/Users/ivanleo/Documents/Coding/schulz/tut/src/index.ts:99:30)
    at ServiceWorkerGlobalScope.[kDispatchFetch] (/Users/ivanleo/Documents/Coding/schulz/tut/node_modules/@miniflare/core/src/standards/event.ts:385:13)
    at Server.<anonymous> (/Users/ivanleo/Documents/Coding/schulz/tut/node_modules/@miniflare/http-server/src/index.ts:291:20)
POST /54cbb39f-863d-4365-bc33-d2c31f4a523c.mp4?action=mpu-complete&uploadId=LR5UYQuJToSuQ26G5v_pAYg-bfcsSA_kPKmjbGvhAHy2lqX0n9dFXaUnjZYKI3G-SIdYsBjCTg_3I1srvZqh3GYwK9Oi2GKtxEdhm5jZeV-4kc8hs6c8KaDEHV7Mod32Sa-WiThJoFe1IaWTv0cOF0wwebHBVsOooBky-edIwN8 400 Bad Request (42.35ms)

Error Encountered!
Error: completeMultipartUpload: Your proposed upload is smaller than the minimum allowed object size.
    at R2MultipartUpload.complete (/Users/ivanleo/Documents/Coding/schulz/tut/node_modules/@miniflare/r2/src/multipart.ts:490:15)
    at Object.fetch (/Users/ivanleo/Documents/Coding/schulz/tut/src/index.ts:99:30)
    at ServiceWorkerGlobalScope.[kDispatchFetch] (/Users/ivanleo/Documents/Coding/schulz/tut/node_modules/@miniflare/core/src/standards/event.ts:385:13)
    at Server.<anonymous> (/Users/ivanleo/Documents/Coding/schulz/tut/node_modules/@miniflare/http-server/src/index.ts:291:20)
POST /54cbb39f-863d-4365-bc33-d2c31f4a523c.mp4?action=mpu-complete&uploadId=LR5UYQuJToSuQ26G5v_pAYg-bfcsSA_kPKmjbGvhAHy2lqX0n9dFXaUnjZYKI3G-SIdYsBjCTg_3I1srvZqh3GYwK9Oi2GKtxEdhm5jZeV-4kc8hs6c8KaDEHV7Mod32Sa-WiThJoFe1IaWTv0cOF0wwebHBVsOooBky-edIwN8 400 Bad Request (42.35ms)

It seems to me like the individual upload portions succeed but then when it comes to the final step of completing the multipart upload, I get a 400 error because of the error that completeMultiPartUpload throws. I'm a bit confused as to how I might be able to fix this. I've tried to modify my partition function so that the last element is always at least 5MB. I originally started slicing from the front (Eg. 12mb file becomes 5 5 2 and the new algorithm would slice it as 2 5 5 ). Any help would be much appreciated. I am using NextJS with typescript and am happy to share source code if needed.

8 Replies

IvanOP•2y ago

For context, the object I am testing this on is ~60mb. I generated chunks manually using the .slice function on the File object using the following code

export const createChunks = (
  file: File,
  chunkSize = 15 * 1024 * 1024 /* cSize should be byte 1024*1 = 1KB */
) => {
  let offset = 0;
  const originalFileExtension = getFileExtension(file);
  console.log(`Identified Extension as ${originalFileExtension}`);
  const chunks = [];
  while (offset < file.size) {
    const chunkEnd = Math.min(file.size, offset + chunkSize);
    const truncatedBlob = file.slice(offset, chunkEnd);
    const truncatedFile = new File(
      [truncatedBlob],
      `audio-${offset}-${chunkEnd}.${originalFileExtension}`,
      {
        type: file.type,
      }
    );
    chunks.push(truncatedFile);
    offset += chunkSize;
  }
  return chunks;
};

export const createChunks = (
  file: File,
  chunkSize = 15 * 1024 * 1024 /* cSize should be byte 1024*1 = 1KB */
) => {
  let offset = 0;
  const originalFileExtension = getFileExtension(file);
  console.log(`Identified Extension as ${originalFileExtension}`);
  const chunks = [];
  while (offset < file.size) {
    const chunkEnd = Math.min(file.size, offset + chunkSize);
    const truncatedBlob = file.slice(offset, chunkEnd);
    const truncatedFile = new File(
      [truncatedBlob],
      `audio-${offset}-${chunkEnd}.${originalFileExtension}`,
      {
        type: file.type,
      }
    );
    chunks.push(truncatedFile);
    offset += chunkSize;
  }
  return chunks;
};

This yields around 13 chunks which each are 5242880 bytes in size and a last one which is 101232 in size.

kian•2y ago

I originally started slicing from the front (Eg. 12mb file becomes 5 5 2 and the new algorithm would slice it as 2 5 5 ).

You'd want the original slicing

IvanOP•2y ago

So I reverted back to the original algorithm I used

xport const createChunks = (
  file: File,
  chunkSize = 15 * 1024 * 1024 /* cSize should be byte 1024*1 = 1KB */
) => {
  let offset = 0;
  const originalFileExtension = getFileExtension(file);
  console.log(`Identified Extension as ${originalFileExtension}`);
  const chunks = [];
  while (offset < file.size) {
    const chunkEnd = Math.min(file.size, offset + chunkSize);
    const truncatedBlob = file.slice(offset, chunkEnd);
    const truncatedFile = new File(
      [truncatedBlob],
      `audio-${offset}-${chunkEnd}.${originalFileExtension}`,
      {
        type: file.type,
      }
    );
    chunks.push(truncatedFile);
    offset += chunkSize;
  }
  return chunks;
};

xport const createChunks = (
  file: File,
  chunkSize = 15 * 1024 * 1024 /* cSize should be byte 1024*1 = 1KB */
) => {
  let offset = 0;
  const originalFileExtension = getFileExtension(file);
  console.log(`Identified Extension as ${originalFileExtension}`);
  const chunks = [];
  while (offset < file.size) {
    const chunkEnd = Math.min(file.size, offset + chunkSize);
    const truncatedBlob = file.slice(offset, chunkEnd);
    const truncatedFile = new File(
      [truncatedBlob],
      `audio-${offset}-${chunkEnd}.${originalFileExtension}`,
      {
        type: file.type,
      }
    );
    chunks.push(truncatedFile);
    offset += chunkSize;
  }
  return chunks;
};

But it still throws the same error that

Error: completeMultipartUpload: Your proposed upload is smaller than the minimum allowed object size. (10011)
    at async Object.fetch (index.js:52:31) {
  stack: Error: completeMultipartUpload: Your proposed uplo…10011)
    at async Object.fetch (index.js:52:31),
  message: completeMultipartUpload: Your proposed upload is s…ler than the minimum allowed object size. (10011)
}

Error: completeMultipartUpload: Your proposed upload is smaller than the minimum allowed object size. (10011)
    at async Object.fetch (index.js:52:31) {
  stack: Error: completeMultipartUpload: Your proposed uplo…10011)
    at async Object.fetch (index.js:52:31),
  message: completeMultipartUpload: Your proposed upload is s…ler than the minimum allowed object size. (10011)
}

I'm using the same worker provided in the docs too haha

kian•2y ago

How big was that file overall?

IvanOP•2y ago

It's around 60mb Have been using the same file to test But i'm getting the same error on my local wrangler server and my deployed worker File gets broken up into 13 5.24 mb chunks and 1.01 mb chunk.

IvanOP•2y ago

Seems like its around 68.3 mb

kian•2y ago

No idea - my only possible guess would be there's an issue with your actual UploadPart operations and the multipart upload on R2 isn't right

IvanOP•2y ago

Hmm, ok nvm I will try working with the s3 api first and see if that works out Thanks so much @kiannh 🙂

Gaming

Programming

Unable to perform multi-part upload for files