504's, 525's, can't get past some access issues
I've just migrated a whole bunch of stuff to a new server, and the domain to access them goes through Cloudflare. Behind Cloudflare is an Nginx Proxy Manager instance, which has LE certificates configured for each subdomain. My router is configured appropriately to forward all requests on associated ports to the correct destination.
I am currently seeing a 504 gateway timeout when requesting one subdomain, but not a different one. Both are configured seemingly identically.
My questions are...
1. Is there some sort of issue with LE certs behind CF? I would expect there may be if requests are passing through Cloudflare.
2. Is there anything specific required for subdomain configuration? I've tried adding explicit records for the subdomain, as well as using a wildcard, but neither seems to be overly happy.
3. Is there anything obvious that stands out to someone more experienced than I?
21 Replies
Oh, one more question... If I see the Cloudflare specific 504 error, does that mean that the issue did not propagate at all into my network? It does it hijack potential downstream 504's to display the Cloudflare one?
that's good to know, at least... i suspect the problem lies behind CF, but i'm also struggling to find evidence of that
wait, no, i am seeing some evidence of that... my access logs show the requests coming in, but it's returning 499's
although that's perhaps a weirdness in and of itself, nginx proxy manager is responding with 499 to legitimate requests, and 504 to "trace" requests
it's as if one server can't talk to another :thonk:
ok, the machine running the reverse proxy for some reason can't communicate with the machine i'm proxying to
499 is nginx saying Cloudflare closed the connection because Cloudflare gave up/timed out, is most likely the sequence there
makes sense
the downside is, i've taken it a bit further, i tweaked some stuff in the proxy manager, and i'm now just getting 521's
users are reporting i'm down, too :PepeHands:
and now i don't see any requests even hitting my reverse proxy, so something is borked on the CF config side of affairs
make sure your ssl/tls mode is Full (Strict) under SSL/TLS -> Overview
oh, i changed that to full, earlier, to test something else
just changed it back though, still 521
ehh Full should still be fine (well, insecure, but wouldn't cause this error), just not Flexible which is sending http/80 to origin
521 is just a generic CF can't talk to your origin
https://developers.cloudflare.com/support/troubleshooting/cloudflare-errors/troubleshooting-cloudflare-5xx-errors/#error-521-web-server-is-down
Cloudflare Docs
Troubleshooting Cloudflare 5XX errors · Cloudflare Support docs
When troubleshooting most 5XX errors, the correct course of action is to first contact your hosting provider or site administrator to troubleshoot and gather data.
You said you moved stuff to a new server, where you using CF Before?
I was, yeah
the only perceivable difference is the new server
makes me wonder if there's some mechanism to reset some CF SSL cache
Well a 521 is just a generic failure to connect, so I would check firewalls/anything you have that could be blocking the connection. You could try going through the front door yourself
curl --resolve example.com:443:<ip> https://example.com/
(can replace <ip> with 127.0.0.1 if in shell on same machine as the web server, or remote ip if you don't have a firewall blocking it)
might need -k if cert isn't trusted
ssl cache?
some cache between cloudflare and my network, whereby it stores the SSL certs and complains if they change before it's refreshed them
connections aren't kept alive for long, and 521 is a failure to establish a connection
wouldn't cause that error
oh wait, the proxy manager is now down
what is happening in my life, how did it all go so wrong
that would do it lol
the ui has been responsive this whole damned time
i know this isn't your thing at all, but if i don't scream here i can't scream anywhere... why tf does this thing not have an ip address 😭
ok, the proxy manager is back online, only this time, the thing that wasn't working earlier now is working, and the thing that was working earlier is now getting a bad gateway :thonk:
@Chaika do you mind me asking if you have some place you're happy to try hit a website?
also, if your DMs are open, i'm sort of uncomfortable posting links to my stuff publicly
as long as you mean "hit" as in just try requesting it sure
yah nothing more
so map "works" and pg 502's, probably returned directly from nginx
gotcha, taking a look internally
i've a feeling i'm being cached out of the real situation locally, hence asking
ok, solved that with the new (old) proxy manager
back online for now, i'll try fix everything again in the morning
thanks for your help, much appreciated ❤️