Cloudflare Developers•2y ago

With my cloudflare worker I fail with

With my cloudflare worker I fail with the error [mf:err] Cause: RequestContentLengthMismatchError: Request body length does not match content-length header from the following API response:

Note: Unnecessary use of -X or --request, POST is already inferred.
> content-length: 697
> content-type: application/x-www-form-urlencoded
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 400

< server: nginx
< content-length: 28
< strict-transport-security: max-age=63072000
<

* Connection #0 to host api.tinybird.co left intact
Analysis error, invalid data%

Note: Unnecessary use of -X or --request, POST is already inferred.
> content-length: 697
> content-type: application/x-www-form-urlencoded
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 400

< server: nginx
< content-length: 28
< strict-transport-security: max-age=63072000
<

* Connection #0 to host api.tinybird.co left intact
Analysis error, invalid data%

I think this is because there are 2 content-length headers, and the first one is no the correct length. Looking at resources online it seems like when multiple are provided then it should take the last one. This is when streaming the response https://developers.cloudflare.com/workers/learning/using-streams/

12 Replies

DanTheGoodman•2y ago

or potentially if there is a way to ignore this error and just use the last content-length that would be great

Vero•2y ago

Hey @danthegoodman , could you please share a bit of your code to understand this better?

DanTheGoodman•2y ago

Yeah sec, you'd need to make a tinybird account. I've also been in contact with thier support @Vero 🐙 they said they couldnt' recreate it which is weird, let me try to they just got back to me as well AH ok it only happens locally when using wrangler dev where it's hosted on CF it works fine, but when running in local mode it breaks

import { flatten } from 'dansjsonutils/flatten'

export default {
    async fetch(
        request: Request,
        env: Env,
        ctx: ExecutionContext
    ): Promise<Response> {
        console.log(request.headers)
        console.log('working')
        const oldRows = (await request.text()).split("\n")
        const newRows = []
        for (const row of oldRows) {
            newRows.push(JSON.stringify(flatten(JSON.parse(row))))
        }
        const { searchParams } = new URL(request.url)
        const res = await fetch(`${searchParams.get('tb_url')}`, {
            method: 'POST',
            body: newRows.join("\n"),
            headers: request.headers
        })

        let { readable, writable } = new TransformStream()

      res.body!.pipeTo(writable)

        return new Response(readable, res)
    },
};

import { flatten } from 'dansjsonutils/flatten'

export default {
    async fetch(
        request: Request,
        env: Env,
        ctx: ExecutionContext
    ): Promise<Response> {
        console.log(request.headers)
        console.log('working')
        const oldRows = (await request.text()).split("\n")
        const newRows = []
        for (const row of oldRows) {
            newRows.push(JSON.stringify(flatten(JSON.parse(row))))
        }
        const { searchParams } = new URL(request.url)
        const res = await fetch(`${searchParams.get('tb_url')}`, {
            method: 'POST',
            body: newRows.join("\n"),
            headers: request.headers
        })

        let { readable, writable } = new TransformStream()

      res.body!.pipeTo(writable)

        return new Response(readable, res)
    },
};

curl \
    -X POST 'http://localhost:8787?tb_url=https%3A%2F%2Fapi.us-east.tinybird.co%2Fv0%2Fevents%3Fname%3Dworker_test' \
    -H "Authorization: Bearer xxx" \
    -d $'{"hey":"ho","lets":1,"arr":[1,2],"obj":{"this":"key","val":[1,2]},"objar":[{"a":"val","b":2,"c":[1],"d":[{"e":1},{"e":2}],"dstr":[{"e":1},{"e":2},2],"darr":[{"e":1},{"e":2},[2]],"f":{"g":1,"h":[2]}},{"a":"val2","new":"nvale"}],"nestedar":[[1],[2]]}
{"hey":"ho","lets":1,"arr":[1,2],"obj":{"this":"key","val":[1,2]},"objar":[{"a":"val","b":2,"c":[1],"d":[{"e":1},{"e":2}],"dstr":[{"e":1},{"e":2},2],"darr":[{"e":1},{"e":2},[2]],"f":{"g":1,"h":[2]}},{"a":"val2","new":"nvale"}],"nestedar":[[1],[2]]}'

curl \
    -X POST 'http://localhost:8787?tb_url=https%3A%2F%2Fapi.us-east.tinybird.co%2Fv0%2Fevents%3Fname%3Dworker_test' \
    -H "Authorization: Bearer xxx" \
    -d $'{"hey":"ho","lets":1,"arr":[1,2],"obj":{"this":"key","val":[1,2]},"objar":[{"a":"val","b":2,"c":[1],"d":[{"e":1},{"e":2}],"dstr":[{"e":1},{"e":2},2],"darr":[{"e":1},{"e":2},[2]],"f":{"g":1,"h":[2]}},{"a":"val2","new":"nvale"}],"nestedar":[[1],[2]]}
{"hey":"ho","lets":1,"arr":[1,2],"obj":{"this":"key","val":[1,2]},"objar":[{"a":"val","b":2,"c":[1],"d":[{"e":1},{"e":2}],"dstr":[{"e":1},{"e":2},2],"darr":[{"e":1},{"e":2},[2]],"f":{"g":1,"h":[2]}},{"a":"val2","new":"nvale"}],"nestedar":[[1],[2]]}'

{"successful_rows":2,"quarantined_rows":0} is there response body, trying to get the header

> Content-Length: 497
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 202 Accepted
< date: Tue, 03 Jan 2023 16:48:00 GMT
< content-type: application/json
< cf-ray: 783d382ff81c20c9-IAD
< strict-transport-security: max-age=63072000
< cf-cache-status: DYNAMIC
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=sBZpm3IInw01%2BtitM400qwJpBLpqJlafaMBYVKeUJ2GrFKKrseB5lAG3yvQDWdWrd%2FkETpvA6mPEw%2BV1j7HTYH4%2FRgyvB5%2FsgGasAucK%2Bgi6phiSPN0hrBdNnJmbO%2BUypl2%2FtElt1OInHKrg9qOm613cW7uz6qlRYRgU74uAd5d06VUs9gv%2BR4Q%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< vary: Accept-Encoding
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
< server: cloudflare
< Connection: keep-alive
< Keep-Alive: timeout=5
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intact
{"successful_rows":2,"quarantined_rows":0}%

> Content-Length: 497
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 202 Accepted
< date: Tue, 03 Jan 2023 16:48:00 GMT
< content-type: application/json
< cf-ray: 783d382ff81c20c9-IAD
< strict-transport-security: max-age=63072000
< cf-cache-status: DYNAMIC
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=sBZpm3IInw01%2BtitM400qwJpBLpqJlafaMBYVKeUJ2GrFKKrseB5lAG3yvQDWdWrd%2FkETpvA6mPEw%2BV1j7HTYH4%2FRgyvB5%2FsgGasAucK%2Bgi6phiSPN0hrBdNnJmbO%2BUypl2%2FtElt1OInHKrg9qOm613cW7uz6qlRYRgU74uAd5d06VUs9gv%2BR4Q%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< vary: Accept-Encoding
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
< server: cloudflare
< Connection: keep-alive
< Keep-Alive: timeout=5
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intact
{"successful_rows":2,"quarantined_rows":0}%

here is what is passed through when it works (CF hosted test) so seems just to be an issue with miniflare

kian•2y ago

I suspect it's due to reuse of the original request headers. Try request.headers.delete('content-length') before const res = await fetch (...

DanTheGoodman•2y ago

doing

[mf:err] POST /?tb_url=https%3A%2F%2Fapi.us-east.tinybird.co%2Fv0%2Fevents%3Fname%3Dworker_test: TypeError: immutable
    at Headers.delete (/Users/dangoodman/code/tinybird-flattening-worker/node_modules/undici/lib/fetch/headers.js:276:13)
    at Object.fetch (/Users/dangoodman/code/tinybird-flattening-worker/src/index.ts:38:19)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at ServiceWorkerGlobalScope.[kDispatchFetch] (/Users/dangoodman/code/tinybird-flattening-worker/node_modules/@miniflare/core/src/standards/event.ts:385:13)
    at Server.<anonymous> (/Users/dangoodman/code/tinybird-flattening-worker/node_modules/@miniflare/http-server/src/index.ts:298:20)
POST /?tb_url=https%3A%2F%2Fapi.us-east.tinybird.co%2Fv0%2Fevents%3Fname%3Dworker_test 500 Internal Server Error

[mf:err] POST /?tb_url=https%3A%2F%2Fapi.us-east.tinybird.co%2Fv0%2Fevents%3Fname%3Dworker_test: TypeError: immutable
    at Headers.delete (/Users/dangoodman/code/tinybird-flattening-worker/node_modules/undici/lib/fetch/headers.js:276:13)
    at Object.fetch (/Users/dangoodman/code/tinybird-flattening-worker/src/index.ts:38:19)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at ServiceWorkerGlobalScope.[kDispatchFetch] (/Users/dangoodman/code/tinybird-flattening-worker/node_modules/@miniflare/core/src/standards/event.ts:385:13)
    at Server.<anonymous> (/Users/dangoodman/code/tinybird-flattening-worker/node_modules/@miniflare/http-server/src/index.ts:298:20)
POST /?tb_url=https%3A%2F%2Fapi.us-east.tinybird.co%2Fv0%2Fevents%3Fname%3Dworker_test 500 Internal Server Error

kian•2y ago

const headers = new Headers(request.headers)
headers.delete('content-length')
await fetch(..., {
  headers
}

const headers = new Headers(request.headers)
headers.delete('content-length')
await fetch(..., {
  headers
}

DanTheGoodman•2y ago

kian•2y ago

Try that instead Could also make an object from the headers.entries() but I don't know that syntax from the top of my head

DanTheGoodman•2y ago

yep that worked tysm @kiannh !

kian•2y ago

Awesome

Vero•2y ago

you are awesome Kian

DanTheGoodman•2y ago

goated

Gaming

Programming

With my cloudflare worker I fail with