Cloudflare Developers•4mo ago

How to use a worker to download an image into R2

Hello, My team and I are using a worker with the itty-router to act as a small server. We have a requirement now where we need to get a list of urls, take those urls and download the images to our R2 storage. We're trying to do it now but when we try it looks as if our fetch doesn't even fire at all. Here is a code snippet.

const imageArrayBuffer = await fetch(
    imageUrl,
    {
        method: "GET",
        redirect: "follow",
    },
);
const temp = await imageArrayBuffer.arrayBuffer();

// Cache the image in our CDN
const r2Object = await this.bucket.put(
    `images/${imageRef}`,
    temp,
);

if (!r2Object) {
    throw new Error("Failed to cache image in R2");
}

const imageArrayBuffer = await fetch(
    imageUrl,
    {
        method: "GET",
        redirect: "follow",
    },
);
const temp = await imageArrayBuffer.arrayBuffer();

// Cache the image in our CDN
const r2Object = await this.bucket.put(
    `images/${imageRef}`,
    temp,
);

if (!r2Object) {
    throw new Error("Failed to cache image in R2");
}

We never get past the fetch. This call is also nested pretty deep within the router. So it's not at the top level index.js but about 3 - 4 functions deep if that's relevant.

6 Replies

James•4mo ago

Your code here looks reasonable assuming imageUrl is set, and is returning what you would expect it to. It would be more optimal to r2.put('key', response.body) instead of loading it all into memory though. If it's not getting past the fetch, that's probably throwing an error. Can you share a small isolated example that can reproduce the issue in entirety?

ShannorOP•4mo ago

Thank you for your response! Saying that it looks right actually help us think of some things we could be doing wrong around this function so we're trying some updates to check including your recommendations! Another question I had that I'm not sure if we'd run into just yet is, if we need to download many images and it takes more than 30 seconds (which will time out the worker) what's the best alternative in those situations? We aren't sure if we'll run into it but we just want to have options if we do.

James•4mo ago

Pushing to a queue in this case to do the downloading would be an option if there’s a lot of images: https://developers.cloudflare.com/queues/

Cloudflare Docs

Overview | Cloudflare Queues

Cloudflare Queues integrates with Cloudflare Workers and enables you to build applications that can guarantee delivery, offload work from a request, send data from Worker to Worker, and buffer or batch data.

ShannorOP•4mo ago

Ok that's very useful! We'll look into that if we need it! Thank you for your help! If I have anything else come up I'll post again. For now we're trying to get it all working based on your feedback.

Kevin•3mo ago

@Shannor, @James get you sorted on the fetch firing?

ShannorOP•3mo ago

Oh yeah he did! Thank you for reminding me. Sorry for not responding again!

Gaming

Programming

How to use a worker to download an image into R2