Theo's Typesafe Cult•11mo ago

Image uploading concurrency Prisma

Hello, I am using Cloudflare as CDN and the upload size limit is 100MB. I want to upload multiple images that may be in total bigger than 100MB. I thought of splitting the images to its own requests so 10 images are 10 requests to upload to compatible S3 storage (Backblaze B2). I have an Image model and each user can have a max of 15 images saved in the database. Currently I do transaction like this:

let retries = 0;
while (retries <= MAX_UPLOAD_PROJECT_IMAGE_RETRIES) {
      try {
         return this.prisma.$transaction(
          async (tx) => {
            const project = await tx.project.findUnique({...});
            // If the project has already reached the max amount of images.
            if (project.images.length >= MAX_PROJECT_IMAGES)
              throw new ConflictException('MAX_IMAGES_REACHED');

            // Save the image path in database.
            const uploadedImage = await tx.image.create({
              data: {
                path: fileKey,
                project: {
                  connect: {
                    id: projectId
                  }
                }
              },
              select: {
                id: true,
                path: true
              }
            });
            // Upload the file to S3.
            await this.storageService.uploadFile(
              fileKey,
              fileBuffer
            );
          },
          {
            isolationLevel: 'Serializable',
            timeout: 15000
          }
        )
  } catch (e) {
        if (e.code === 'P2034' && retries < MAX_UPLOAD_PROJECT_IMAGE_RETRIES) {
          retries++;
        } else {
          await this.storageService.deleteFile(fileKey);
          throw e;
        }
      }
}

let retries = 0;
while (retries <= MAX_UPLOAD_PROJECT_IMAGE_RETRIES) {
      try {
         return this.prisma.$transaction(
          async (tx) => {
            const project = await tx.project.findUnique({...});
            // If the project has already reached the max amount of images.
            if (project.images.length >= MAX_PROJECT_IMAGES)
              throw new ConflictException('MAX_IMAGES_REACHED');

            // Save the image path in database.
            const uploadedImage = await tx.image.create({
              data: {
                path: fileKey,
                project: {
                  connect: {
                    id: projectId
                  }
                }
              },
              select: {
                id: true,
                path: true
              }
            });
            // Upload the file to S3.
            await this.storageService.uploadFile(
              fileKey,
              fileBuffer
            );
          },
          {
            isolationLevel: 'Serializable',
            timeout: 15000
          }
        )
  } catch (e) {
        if (e.code === 'P2034' && retries < MAX_UPLOAD_PROJECT_IMAGE_RETRIES) {
          retries++;
        } else {
          await this.storageService.deleteFile(fileKey);
          throw e;
        }
      }
}

But I don't like this way since everyone who tries to upload is waiting because of 1 image. I woud like to lock per user rather than lock whole table.

1 Reply

EQOP•11mo ago

Or maybe something even better if possible

Gaming

Programming

Image uploading concurrency Prisma