Theo's Typesafe Cult•2y ago

Need advice on handling 10,000 images on Lambda

I need advice, I have a SQS that triggers lambda, it's an image processor that takes in 10,000 images and archives them in a zip file then uploads it to S3. I'm getting array buffer allocation on the archiving part. I might need more memory, currently has a max of 2560mb ram. This might be happening because I'm putting all the archived objects in a buffer instead of streaming, but it doesnt not work if i stream on lambda.

archive.finalize();
const buffer = [];
archive.on("data", (data => buffer.push(data)));

archive.on('end', (async () => {
  const data = Buffer.concat(buffer);

  await putObjectToS3(`generator/${userId}/collection.zip`, data);
  console.info(`<${userId}>`, "sent zip file to s3");

  await collection.updateOne({ userId }, { $set: { status: "completed" } });

  resolve(formatResponse({
    statusCode: 200,
    body: { code: "SUCCESS" },
  }))
}))

archive.finalize();
const buffer = [];
archive.on("data", (data => buffer.push(data)));

archive.on('end', (async () => {
  const data = Buffer.concat(buffer);

  await putObjectToS3(`generator/${userId}/collection.zip`, data);
  console.info(`<${userId}>`, "sent zip file to s3");

  await collection.updateOne({ userId }, { $set: { status: "completed" } });

  resolve(formatResponse({
    statusCode: 200,
    body: { code: "SUCCESS" },
  }))
}))

Should I make two lambda functions one for processing the image and uploading everything to s3, and another one for zipping it?

2 Replies

MartinMueller.dev•2y ago

@_typedef you can do some stream pipeline stuff with processing, zipping and uploading. It is a while back when I did that on lambda. I'll try to find the code. nope can't find it anymore. I think was on a client in a private repo.

Web Dev Cody•2y ago

we do something similar at work where we grab a bunch of PDFs, combine them all to a single zip, and stream them to s3. I think you want to use the s3Client.upload method which takes in a stream for the Body so you pass the stream from the archiver directly into s3Client.upload({Body: zipStream}) or something

Gaming

Programming

Need advice on handling 10,000 images on Lambda