CD
Cloudflare Developers•4d ago
sh33p

Seeing data loss when serving ReadableStream from worker

I have a generator which returns fixed-size chunks (tested, eg., 16kB and 32kB chunks), and I take those chunks and pass them out via controller.enqueue() calls. Some of those chunks then vanish before reaching the other end of the connection. No errors in the connection itself. It looks like the controller object just misplaces them. Am I doing this right? 
        const generator = pageGenerator(hash, url.pathname);
        const stream = new ReadableStream({
            async pull(controller) {
                const { value, done } = generator.next();
                if (done) {
                    controller.close();
                } else {
                    controller.enqueue(value);
                }
            },
            cancel() {
                generator.return(undefined);
            }
        });
        return new Response(stream, { headers: html_headers });
        const generator = pageGenerator(hash, url.pathname);
        const stream = new ReadableStream({
            async pull(controller) {
                const { value, done } = generator.next();
                if (done) {
                    controller.close();
                } else {
                    controller.enqueue(value);
                }
            },
            cancel() {
                generator.return(undefined);
            }
        });
        return new Response(stream, { headers: html_headers });
1 Reply
sh33p
sh33pOP•3d ago
Problem persists if I turn off compression. Should also mention it's a random chunk in the middle, not first or last. OK, the root of the problem is probably that after I yield from the generator, and the logic above issues controller.enqueue() on that buffer, then by the time the pull() method is called again the output buffer is not yet safe to overwrite. How am I supposed to know when I can re-use the buffer sent out via controller.enqueue()?

Did you find this page helpful?