Tunnel debugging
Hi! I'm trying to set up a VPS with Postgres via docker compose and then bind it in a worker via Hyperdrive.
I have followed the instructions line by line except when setting up the public hostname: instead of localhost:5432 -> postgres:5432.
When the API calls the Hyperdrive returns 504 (Gateway Timeout) to the client. To make sure the DB is working, I have tried to access the database from the VPS console through postgres:5432 and it works fine.
Any suggestions?
23 Replies
Are you getting any logs on your tunnel? What do you see from your graphql Access logs?
I'm not able to figure out where to find those logs. Should they be in the Cloudflareone/Logs/Access? It looks all empty there.
Tunnel logs are visible where the tunnel is running, you'll need to run it with loglevel=debug to see connections reaching it.
Access logs are probably best seen with just a graphql API call
https://developers.cloudflare.com/analytics/graphql-api/tutorials/querying-access-login-events/
The other thing you might have going on is if you've set up bot fight mode or waf rules or something that are blocking your connection attempts from even getting to your access application
Cloudflare Docs
Querying Access login events with GraphQL | Cloudflare Analytics docs
In this example, we are going to use the GraphQL Analytics API to retrieve logs for an Access login event. These logs are particularly useful for determining why a user received a 403 Forbidden error, since they surface additional data beyond what is shown in the dashboard Access logs.
Ok, I added the loglevel=debug flag to docker compose command, and the debug result of fetching the API that calls the hyperdrive db is the following:
2024-10-20T19:37:46Z DBG GET https://test.myhostname.com/ HTTP/1.1 connIndex=0 content-length=0 event=1 headers={"Accept-Encoding":["gzip, br"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Access-Jwt-Assertion":["areallylongcode"],"Cf-Connecting-Ip":["2606:4700:3030::"],"Cf-Ipcountry":["XX"],"Cf-Ray":["8d5b76de6c0076ef-LHR"],"Cf-Visitor":["{"scheme":"https"}"],"Cf-Warp-Tag-Id":["4eb41b3b-6616-4254-a300-780b2f295693"],"Cookie":["CF_Authorization=areallylongcode"],"Sec-Websocket-Key":["R0M4NHBESkhkTQ=="],"Sec-Websocket-Version":["13"],"User-Agent":["Cloudflare-Hyperdrive"],"X-Forwarded-For":["2606:4700:3030::"],"X-Forwarded-Proto":["https"]} host=test.myhostname.com ingressRule=0 originService=tcp://postgres:5432 path=/
2024-10-20T19:37:46Z DBG origin connection established connIndex=0 destAddr=tcp://postgres:5432 event=1 ingressRule=0 originService=tcp://postgres:5432
2024-10-20T19:37:46Z DBG proxy stream acknowledged connIndex=0 destAddr=tcp://postgres:5432 event=1 ingressRule=0 originService=tcp://postgres:5432
2024-10-20T19:37:46Z DBG upstream->downstream copy: read tcp 172.18.0.6:44510->172.18.0.5:5432: use of closed network connection connIndex=0 destAddr=tcp://postgres:5432 event=1 ingressRule=0 originService=tcp://postgres:5432
So good news, the worker is connecting to the tunnel, but somehow is not returning what i need. Also it seems to route properly to tcp://postgres:5432
Okay. That's a great start.
Probably from there the steps are to check any logs from your postgres itself to see if it's yelling at all, and then probably you need to poke at your JavaScript for your worker to see if your queries there aren't actually doing what you want them to.
The new workers logs will end up being pretty helpful here, I think
I will check the first one. To check the second one I have already made a NEON db with the same schema and data and binded it to Hyperdrive. It works great. So I think It's not a JS problem i think.
Same script and everything? That's very interesting. Huh.
Yup, I'm just switching the HD from the deployed worker in the Dashboard
I wonder if your postgres just isn't egressing to where the tunnel is listening at. In particular that "postgres:5432" is a bit non-standard. I'd expect your DB to be listening on localhost:5432, or similar. Are you running cloudflared on the same box as your database? It feels like there's a gap between the egress end of the tunnel and the port your database is listening on?
Let me try all that, as is it a Docker Compose it creates a default network, so maybe is what you are telling.. A gap between the services.
I added a service for pgAdmin, so I will try to create db connections in pgAdmin for localhost:5432 and postgres:5432
Ah. That feels right. I recall Dax on his recent Livestream a week or two ago did a bunch of fiddling with a compose setup to get that working.
If it were me, I'd spin up a tunnel locally, cloudflared at both ends, and try to get psql to talk across that setup first, and then go back to incorporating a worker and hyperdrive
I will give a try on this.
localhost:5432 not stablishing connection
postgres:5432 working good
Also if I run docker inspect <container_id> i get that postgres is running locally on 172.18.0.5. So I tested and 172.18.0.5:5432 also stablishing connection.
I meant, instead of the ingress end of your tunnel being a hyperdrive, run cloudflared locally, and send traffic across it that way, to check that the egress end of the tunnel is working as expected
Unless that's what you just did?
Ah nono, what I did is just testing that the adresses I set in the hostnames are the correct ones, just to discard. Now I will try to set what you said of creating a cloudflared-cloudflared
Is this the one I have to follow?
https://developers.cloudflare.com/cloudflare-one/applications/non-http/cloudflared-authentication/arbitrary-tcp/
Cloudflare Docs
Arbitrary TCP | Cloudflare Zero Trust docs
Cloudflare Access provides a mechanism for end users to authenticate with their single sign-on (SSO) provider and connect to resources over arbitrary TCP without being on a virtual private network (VPN).
A bit embarrassing, but I actually really like Ryan's blog for this.
https://ryan-schachte.com/blog/cf_tunnel_tcp/
TCP tunneling with Cloudflare Tunnel
TCP connection forwarding with Cloudflare Tunnel
Either's good though, that's the one I'd use if it were me, yeah.
Ahahah ok thanks. Really apreciate your help. I'm learning a lot of things this days...
I set up a new VPS with just posgres on localhost:5432. Created a tunnel and configured an Hyperdrive config. Then hinded the hyoerdrive to the same worker and now it stablishes connection.
So now I'm 100% sure the problem is in the docker networking configuration. There is a problem with the services comunicating to each other.
I'm gonna ask for help in some docker compose forums hoping someone can help me, as this is not at all an Hyperdrive problem.
If I find a solution I will post here the steps I needed to follow... Maybe someone finds them util in a future
Makes sense. Thanks for following up here, much appreciated. Good luck!
Marc & I had the chance to speak, pretty sure it's SSL missing, @marcbejar will keep us updated with this!
Definitely this was the "issue". I have setted up a cert via openssl and initialized docker with the ssl=on command and int worked!
Next step is to figure out how to connect to the db from the orm i use to run migrations and stuff like this. As @thomasgauvin said first move is to follow the arbitrary TCP guide. Thanks a lot for the support!
Awesome! Happy to hear that's the fix @marcbejar, keep us updated!
Yup! Already managed to connect to the db via WARP. I just had to connect the device my PC to cloudlfare One, and add a policy to the tunnel to allow that device (I used the email method).
The only thing is that in the tutorial it says you can connect the db via:
But the only way I got it working is by setting 127.0.0.1 instead of localhost. Once this done I can use localhost:9210 to acces the db with drizzle, pgAdmin, dbAber, etc... Finally all setted up! :dogekek: