Error 523 after Oracle Cloud VPS update
I'm not sure if this is the most appropriate place to ask this, but I'll give it a try anyways. I have a website hosted on an Oracle cloud VPS (a Sveltekit app with the node adapter), and it was working fine. As soon as I started updating the machine and restarted some services, the website started to respond with a 523 error. I tried turning the cloudflare proxy on and off, but to no avail. After shutting down the machine, turning it back on and restarting the docker containers, the problem remains. Does anyone know how to solve this?
45 Replies
523 is just a generic origin is unreachable
Error 523 occurs when Cloudflare cannot contact your origin web server. This typically occurs when a network device between Cloudflare and the origin web server doesn’t have a route to the origin’s IP address.If it doesn't work even with proxy disabled, it's an origin issue. I would check the following: 1. Did your VPS's Public IP Change? 2. Can you reach your web server locally within the vps? You can use curl like
curl http://127.0.0.1 or curl --resolve example.com:443:127.0.0.1 https://example.com/ -k replacing example.com with your website name to make a request like would be from CF with right host header/sni
3. If you can reach locally but not externally, check firewalls on your vps and Oracle's VM Firewall.
You could always use Cloudflare Tunnels too: https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/get-started/create-remote-tunnel/, they're a piece of software that runs on your origin connecting out to CF, and then proxies incoming requests over that connection. This bypasses issues with NAT/most firewalls and has no issues if your IP changes, etc1 - no, still the same
2 - first option works, second one gives me the error above
what do you get from the first option? Did you setup your service with ssl/https before, or only ever http?
okay, I rewrote the second option to use port 3001 instead of 443 (since the app is running on that port) and it returns 523
I had https setup, yes
and the first option returned the website's html
were you using an origin rule in CF to override the dest port to 3001?
You'd have to change the port in both places in that command, getting a 523 sounds like you're hitting CF, ex:
curl --resolve example.com:3001:127.0.0.1 https://example.com:3001/ -kthere's actually more than one website: the website associated with the domain (pedrosilvadev.pt) is hosted on port 3000, while the one associated with the subdomain is running on port 3001. curling 127.0.0.1 gives me the html of the 3001 website
ah I see, let me try
but no, I don't think I did that
and it was working before? Guessing you had SSL/TLS -> Overview -> Encryption mode to "Flexible"?
http://127.0.0.1 -> port 80
https://127.0.0.1 -> port443
with no host header (like in the first command) you'd go to the default website
exactly
Flexible is insecure and should never be used
it's lying to your visitors, user -> https/encrypted -> cf -> http/plaintext -> origin
anything between cf -> origin can mitm the connection and steal credentials/respond with anything they want/mitm
well... thankfully the websites were just personal projects, but yeah I should make that change
the apps on 3000/3001 are just http/not ssl? Or did you set them up with certificates?
all of the https/ssl setup was made on cloudflare
unless sveltekit has something that takes care of that automatically
alright, I changed from Flexible to Full
doubt it does. Those are docker apps exposing those ports? If you visit 3000/3001, you see the apps?
using the vps's IP + port from outside? or inside?
inside
curl http://127.0.0.1:3001There's one thing that happened (not sure how): doing this port forwarded the port from the machine to my laptop, so accessing localhost:3001 gives me the site now
so the apps/containers are fine, just running over http
works, 3000 and 3001
The easiest way to expose your origins securely would probably be to just use Cloudflare Tunnels (linked guide above), or you could use CF Origin Certificates (SSL/TLS -> Origin Server) to get certs up to 15 years trusted by CF and hook them up either with your apps directly (config depends) or with nginx in front using the certs and proxying back to the apps.
Tunnel is secure because it runs a piece of software on your origin (cloudflared), so requests go user <- encrypted -> cf <- encrypted tunnel -> tunnel connector <- http over localhost/same machine -> service. Does require you being tied to using CF, and Zero Trust does require a payment method even on free plan. Otherwise you have the option of Origin Certs and setting them up however, or if you wanted to go without CF Proxy, you could go the nginx way and use a Let's Encrypt acme.sh automated stuff to get an auto-renewing certificate. Lots of ways
yeah, changing from flexible to full now gives me a 522 error, probably have to set that up. I'm just not sure why updating the oracle machine prompted this headache
right, I should mention that I'm using nginx as a reverse proxy, no certbot
probably why flexible worked in the first place
a blessing in disguise imo, you've just gotta have everything encrypted end to end these days imo, even on personal websites there's damage to be done by imperonsating you with a mitm attack
If you use curl override on http for your website, what do you see?
curl --resolve example.com:80:127.0.0.1 http://example.com/ -k -vvv
shouldn't have broken over http if you haven't changed anything, might be worth seeing if there's any larger issue there. If you went with the tunnel idea you could just have tunnel be your reverse proxy and connect to 3000/3001 thoughsame result as before, only with some extra info related to dns ("added url to dns cache", request headers, etc.)
hmm, as what before?
as in, you get a successful response with valid html?
I still get the html
correct
with both sites
do you have a linux box that you can try that same command remotely from, replacing 127.0.0.1 with your vps's real ip (like how Cf would connect)?
so using the public ip from another pc? let me try
Thinking it's a firewall change, either from Oracle's side or a fw locally on the machine if you have any configured
careful about leaking your origin's ip
(if you had configured a firewall to only allow CF's IPs, it would also fail on that)
I'm deleting it asap, just letting you see it
I would check over Oracle's firewall stuff to ensure you're properly allowing 80/443
You could also just go with the cf tunnels and bypass all that
Hmmm, I believe I only allow port 80, but it used to work before
no rule for 443
adding that rule gives me a 523 again
yea that looks sane, any firewall on the vps itself like ufw?
there's ufw
maybe those rules got reset after the update? I don't remember changing them but it could be related to that
nevermind, it's inactive
not sure what other firewalls could be installed, it's an ubuntu image
apparantly Oracle's firewall stuff is special and doesn't play well with ufw anyway: https://stackoverflow.com/a/64053781
Stack Overflow
Can't access Oracle Cloud Always Free Compute http port
Yesterday, I was set up my first always free compute VM. I installed ubuntu 18.04 minimal on my VM. For my web service need, I installed Nginx. I'm confused because when I tried to access my public...
It gave a very specific error on trying to connect "No route to host" instead of a standard connection failed/timeout
they become preconfigured with restrictive iptable rules apparently, and you can't use ufw as it messes with them (according to that), that describes how to use iptables to allow 80 in. You can find a lot of info by googling "No route to host oracle"
okay, so before I followed another random stackoverflow page which told me to run iptables -F, and running the command from your link returned a "index of insertion too big" (because the table got wiped). running without the 5 worked
but the 523 error remains
sounds like you flushed all the rules
I still see the same "No route to host"
I would go off connecting over http, that last command you tried externally, and trying to get it so you can connect over port 80/http without getting the no route to host msg
I am the wrong person to help with iptables though, no idea how you'd fix that mess you just did. If you didn't run netfilter-persistent save you could probably reboot and go back to defaults? Or you could try the firewallcmd (second result) there, but there's lots of warning about it breaking docker and other stuff after. You could try switching back CF to Flexible just for now to get it working over http first. I can't help much more with Oracle's firewall behavior and iptables though, most I can do is say there's lots of hits with no route to host and oracle's default configs, strong lead there
you were a big help already. Just one more thing: with cloudflare tunnels, in the guide, they say "Specify a service, for example https://localhost:8000". This localhost is the origin's localhost, right? so I could use localhost:3000 to target one of my websites
yup, you can use it in place of nginx like that
Works like a charm, can't believe it's this easy. thank you so much!
For each website, I should create a new tunnel, right?
each vps should be a new tunnel. For the same vps, just add another public hostname
oh right, I just saw that option. makes sense
so this basically encrypts every request on the origin machine, right? It's safer
and you could use the same tunnel for multiple vps's, as long as they all had the same services available, called tunnel replicas. You just can't easily have more then one tunnel on the same machine (without making your own systemd unit files/services), and you can't use the same tunnel replica on multiple machines which can't service all of the public hostnames for that tunnel
it's way safer then flexible yea, actual encryption between cf -> origin. You also gain the benefit of not needing to poke holes in your firewall
Thank you for bearing with me, I'm glad I went for Cloudflare Tunnels. It's crazy how much less of a hassle it is than fiddling with nginx