Theo's Typesafe Cult•2y ago

Generating 1200x1200 10,000 unique images in lambda.

I'm using lambda for this one, I'm generating 1200x1200 10,000 unique images but it reaches the timeout limit. Any recommendation to make this faster? without removing status updates.

// generate images from layers
      const canvas = createCanvas(dimension.width, dimension.height);
      const ctx = canvas.getContext("2d");
  
      for (let i = 0; i < metadata.length; i++) {
        console.info(`<${userId}>`, `generated image #${i}`);
        ctx.clearRect(0, 0, dimension.width, dimension.height);
  
        const data = metadata[i];
        const traitImages = data.attributes.map((attr) =>
          layerMap.get(attr.trait_type + attr.value),
        );
  
        for (const image of traitImages) {
          ctx.drawImage(image, 0, 0, dimension.width, dimension.height);
        }
  
        archive.append(canvas.toBuffer("image/png"), { name: `images/${i}.${format}` });

        // if divisible by 5 update status
        if (i % 5 === 0) {
          await collection.updateOne({ userId }, { $set: { status: i.toString() } });
        }
      }
      console.info(`<${userId}>`, "generated images");
      await collection.updateOne({ userId }, { $set: { status: "processed" } });

// generate images from layers
      const canvas = createCanvas(dimension.width, dimension.height);
      const ctx = canvas.getContext("2d");
  
      for (let i = 0; i < metadata.length; i++) {
        console.info(`<${userId}>`, `generated image #${i}`);
        ctx.clearRect(0, 0, dimension.width, dimension.height);
  
        const data = metadata[i];
        const traitImages = data.attributes.map((attr) =>
          layerMap.get(attr.trait_type + attr.value),
        );
  
        for (const image of traitImages) {
          ctx.drawImage(image, 0, 0, dimension.width, dimension.height);
        }
  
        archive.append(canvas.toBuffer("image/png"), { name: `images/${i}.${format}` });

        // if divisible by 5 update status
        if (i % 5 === 0) {
          await collection.updateOne({ userId }, { $set: { status: i.toString() } });
        }
      }
      console.info(`<${userId}>`, "generated images");
      await collection.updateOne({ userId }, { $set: { status: "processed" } });

6 Replies

pagwin•2y ago

I don't have any experience with lambdas or js canvas api but just to spitball some ideas 1. you could reimplement it in a faster language 2. WebGL? 3. screw making it faster and just have the lambda call out to a different lambda which generates one image 10,000 times

TypedefOP•2y ago

I think one image per lambda call is crazy. Currently it can generate around 4200 images, almost half, when generating 1200x1200 10,000 images before it reaches the 15 minutes timeout. Let me know your thoughts 🙏🏼

pagwin•2y ago

then just make a version of the lambda that can generate some number specified images and make however many calls to that lambda from the main lambda you need to get everything done fast enough, 2 calls generating 4k each then 1 more generating the last 2k or maybe 5 calls generating 2k

Neto•2y ago

take the first lambda, and spawn N more lambdas do handle the image generations

Neto•2y ago

https://www.youtube.com/watch?v=DEULSRzeMQc

Web Dev Cody

YouTube

This is how we generate thousands of pdfs in seconds at work

📘 T3 Stack Tutorial: https://1017897100294.gumroad.com/l/jipjfm 🤖 SaaS I'm Building: https://www.icongeneratorai.com/ 💬 Discord: https://discord.gg/4kGbBaa 🔔 Newsletter: https://newsletter.webdevcody.com/ 📁 GitHub: https://github.com/webdevcody 📺 Twitch: https://www.twitch.tv/webdevcody 🤖 Website: https://webdevcody.com 🐦 Twitter: https://twitt...

Neto•2y ago

kinda like this

Gaming

Programming

Generating 1200x1200 10,000 unique images in lambda.