CD
Cloudflare Developers•3w ago
DanTheGoodman

R2 `Provided readable stream must have a known length` from workers

I would like to stream a file to R2 that's being created by iterating over items in memory and JSON-stringifying them (reduce memory consumption by not double-buffering them in memory) 
const { readable, writable } = new TransformStream<Uint8Array, Uint8Array>()
        const writer = writable.getWriter()

        // start streaming the records to the file
        const writePromise = this.env.StreamData.put(segmentName, readable)
const { readable, writable } = new TransformStream<Uint8Array, Uint8Array>()
        const writer = writable.getWriter()

        // start streaming the records to the file
        const writePromise = this.env.StreamData.put(segmentName, readable)
However I'm getting the error: 
stderr | tests/index.test.ts > Worker > durable object > should create a new stream and produce messages
✘ [ERROR] Uncaught (in promise) TypeError: Provided readable stream must have a known length (request/response body or readable half of FixedLengthStream)

      at async Promise.all (index 1)
      at async StreamCoordinator.writePendingMessagesToLogSegment (file:///Users/dangoodman/code/DurableStreams/.wrangler/tmp/dev-qAfkZE/index.js:2058:5)
      at async StreamCoordinator.flushPendingMessages (file:///Users/dangoodman/code/DurableStreams/.wrangler/tmp/dev-qAfkZE/index.js:2034:5)
      at async StreamCoordinator.alarm (file:///Users/dangoodman/code/DurableStreams/.wrangler/tmp/dev-qAfkZE/index.js:2014:5)
stderr | tests/index.test.ts > Worker > durable object > should create a new stream and produce messages
✘ [ERROR] Uncaught (in promise) TypeError: Provided readable stream must have a known length (request/response body or readable half of FixedLengthStream)

      at async Promise.all (index 1)
      at async StreamCoordinator.writePendingMessagesToLogSegment (file:///Users/dangoodman/code/DurableStreams/.wrangler/tmp/dev-qAfkZE/index.js:2058:5)
      at async StreamCoordinator.flushPendingMessages (file:///Users/dangoodman/code/DurableStreams/.wrangler/tmp/dev-qAfkZE/index.js:2034:5)
      at async StreamCoordinator.alarm (file:///Users/dangoodman/code/DurableStreams/.wrangler/tmp/dev-qAfkZE/index.js:2014:5)
Is there a way I can stream data to R2 without knowing the exact content length ahead of time?
1 Reply
DanTheGoodman
DanTheGoodmanOP•3w ago
this is a local test, so maybe you can't do this with local testing? Even with the s3 api? S3 should support that iirc or if I can pre-calculate the size, can I somehow tell it that? seems this works for telling the length: 
const { readable, writable } = new TransformStream<Uint8Array, Uint8Array>({
            expectedLength: offsets.length * 32 + pendingMessages.length * 32,
        })
const { readable, writable } = new TransformStream<Uint8Array, Uint8Array>({
            expectedLength: offsets.length * 32 + pendingMessages.length * 32,
        })
oooh ill try that

Did you find this page helpful?