Garbled response / Gzip issue?
I've been having an issue I can't get to the bottom of, so finally asking here.
Context:
- I have one Railway project, with several services: one frontend (NextJS, App Router, mainly SSR), one backend service (Django), a database (postgres)
- Frontend tries to connect to the backend via private URL, and falls back to public URL if the initial connection fails (like when in client side, or during build)
- Frontend calls various APIs, as you'd expect, APIs return JSON
- There is no auth between services
- The backend has
django.middleware.gzip.GZipMiddleware and corsheaders.middleware.CorsMiddleware
- The frontend has Cloudflare DNS, which I use to manage the domain
- I'm using the new proxy
Issue:
- I made a change to remove pagination in one of my APIs, and removed the associated pagination logic on the frontend
- After deploy, I get the attached error from my frontend during build
Things I've tried:
- Calling this API on the cmd line/postman works fine, and I get Content-Encoding = gzip in the response headers
- Disabling GZipMiddleware in Django, get the same result
- Mocking the response - returning hardcoded JSON - get same issue
- When I make the hardcoded JSON smaller, it often works - this makes me think it's a Gzipping issue (which triggers when the response is >200 bytes)
- Trying the legacy proxy - same result
- Setting Accept-Encoding = gzip, deflate, br and Content-Type=application/json on the request in Next
- Adding DEFAULT_RENDERER_CLASSES = rest_framework.renderers.JSONRenderer in Django
- Praying
c18400df-5dee-4fe7-bc56-a98db8d475bd76 Replies
Project ID:
c18400df-5dee-4fe7-bc56-a98db8d475bdAdditional:
Another API is called during build that returns an extremely small response (JSON with ~5 entries) and this works fine.
do you have an endpoint i could curl to reproduce this issue?
is that supported to work?
ha - yes - what are you seeing?
json
yes
can i have an endpoint that reproduces the issue, im not sure what i should be doing with an enpoint that works fine
yes... I mean that's the issue
The endpoint works fine, every time, when called in curl/postman.
But in build it gives the output I shared in the first post. This is what I can't understand.
during build of your FE?
yep
are you able to see my build logs?
how is this url being called?
one sec, getting the code
forgive the terrible loggin
did you say you have disabled GZIP on the BE and still got the same garbage response?
yep - I thought the logs look like printed gzip
so I removed GZipMiddleware, same issue
is GZIP currently disabled (removed)?
no i renabled but can disable it again
lets leave it for now
just trying to parse through these logs
good luck
LMK if I can help
i dont see
content-encoding = gzip anywhere in these FE build logs, does response.headers.entries() strip that out?take a look at the backend logs - I'm logging the headers on inbound requests:
right but
content-encoding = gzip should be returned as a response headeryeah agree it's not there
but why?
misbehaving middleware?
but you said it was the same with the middleware removed?
either way, can you disable / remove the gzip middleware for now?
dude sorry, back at keys
disabling now
for context, curl was showing that it got the gzip response header -
this is what led me to think it might be something to do with the proxy
communicating with the end point directly works
the proxy doesnt touch any of these headers afaik
Railway Help Station
The new proxy is stripping the Accept-Encoding header
On the legacy proxy, requests correctly returned the gzipped response when browsers send the Accept-Encoding: gzip, deflate, br header. However, this does not work on the new proxy. It appears that the Accept-Encoding header is stripped before reaching my server.
aware this is now solved, but wondering if related?
well besides the headers it does touch, you know what i mean lol
btw, I noticed that when I deploy the backend, I have to commit/push to force a rebuild on my frontend, rather than just trigging a
deploy from last commitHello all, we have pushed a change to our proxy, we no longer strip out that header!what happens when you do deploy from last commit?
I know, but just wondering if could be a related thing?
I'll show you
at this time, i dont think so
backend is deployed
frontend deploying now with deploy from last commit
i can confirm with postman the BE response is no longer gzipped, no encoding header and resonse body is larger
agreed, see the same
interesting
ok - frontend built correctly with trigger last commit
but same issue
lets use the private domain then
in build?
yeah
oooh
have you just opted me in
fine if you have
yeah, just trying to move fast here, i will ask next time
no don't worry, just wonderd
didn't remember doing it and I've been battling this for 1.5 days... starting to forget what life was like before this bug
I think it's worked
either way - being able to use private urls in build is a win
shall I renable gzip on the BE?
oh, it worked...
i was kinda hoping it wouldnt
yeah, as is we haven't learnt anything
exactly
just that the new builder works for some reason
enabling gzip
hold on
not yet please
oops
aborted the build
oh wait, this is a production site, can you afford to keep this offline as we debug?
yes
all my user will be ok with it
are you sure? we could spin up a dev env in railway
yes don't worry
alright, sounds good, but yeah lets keep gzip off for now, it eliminates variables
anything I can do to help debug?
I would like to remove the hard-coded response, and replace with DB query
ok for me to try?
I'll leave gzip off
yeah thats fine
that's deployed
redeploying FE
well my fridge doesn't cool anymore, I gotta go AFK while I transfer everything to another fridge outside
good luck
I'm going to try renabling gzip while you're fridging
FYI database connection worked
they didn't before?
ah, it was just one of many things I've been trying to debug the issue - hardcoding the json response in my backend
was looking for special characters/fixing the size of the repsonse, etc
gzip works
okay I have some more things I'd like to try when I'm back
where are we on the fridge outage, Brody? P1? Users affected? Is there an incident channel we should join?
there's an incident? I thought we left off in a working state with the new builder?
was a joke about your fridge, ignore
anyway, new builder working - I'm back to coding, thanks for your help
oh I thought you were serious 😅
rarely
I thought you were like, screw your fridge, help me with this gzip stuff
haha
would never priotise work above food
I'll be back at the computer as soon as I finish eating and moving the food to the other fridge
alright, i am back