Problems with server on fly.io - unexpectedly without any updates or redeployments
Hello, I'v had some problems with my rooms booking wasp app, running on fly.io. It has been running perfectly for four months (bookd.xyz / foretagscenter-client.fly.dev), but yesterday I noticed that it had stopped working, and I am puzzled by the cause of it.
Problem:
- It stopped working out of thin air. No redeployments or anything.
- The client does not get any response from the server.
- Server logs look like attached - (GET 401 /auth/me)
- No error logs from postgres database - no strange behaviour when reviewing the server metrics (I have very few users)
Tried solutions:
- Redeploying the app with wasp deploy fly deploy
- Restarting the server with wasp deploy fly cmd --context server scale count 0 - wasp deploy fly cmd --context server scale count 1
- Restarting the postgres db with flyctl postgres restart -a foretagscenter-db
Troubleshooting steps
- Run the app locally works fine (wasp start)
- Run the app but connected to fly.io db through proxy works fine
Other than that I am not certain on what to do next. Any possible help or ideas would be highly appreciated!
Best regards
Frej
13 Replies
@frej1046 , one thing to to explore -> why does https://bookd.xyz/ not work at all? So if I go to https://foretagscenter-client.fly.dev/, I get an app that is erroring out because server is rejecting it due to CORS errors, which actually make perfect sense since your server is probably configured to accept requests only from https://bookd.xyz/ (am I making sense here @Wasp Team ?).
But, your https://bookd.xyz/ doesn't resolve to a page at all, I just get
This site can't be reached by Chrome, with ERR_CONNECTION_CLOSED error. Any idea why is that happening? Sounds like an issue with your domain possibly, since Fly's client URL works?Have you setup a CNAME record? https://fly.io/docs/networking/custom-domain/
Checking your domains CNAME record here doesn't show it https://dnschecker.org/#CNAME/Bookd.xyz
DNS Checker
DNS Checker - DNS Check Propagation Tool
Check DNS Propagation worldwide. DNS Checker provides name server propagation check instantly. Changed nameservers so do a DNS lookup and check if DNS and nameservers have propagated.
The other solution I could propose is that you upgrade your app to the newest Wasp version. I had a similar issue with Railway. They changed something on their end and a recent update we pushed fixed it (cc: @miho ). I just had to:
1. run
curl -sSL https://get.wasp-lang.dev/installer.sh | sh
2. change the wasp version on my app in main.wasp
3. and then make sure everything worked locally
4. and then redeployed with wasp deploy fly deployThis is a good idea also! Although I would keep it as the last option, if we run out of ideas, so we don't introduce another variable into the whole debugging process.
true
Yes, this I could try of course. When setting it up I followed the guide at https://wasp-lang.dev/docs/advanced/deployment/cli#using-a-custom-domain-for-your-app, without adding any CNAME record. It worked until now, but things might change on their side.
I will try to follow the fly.io documentation for this one.
Deploying with the Wasp CLI | Wasp
Wasp CLI can deploy your full-stack application with only a single command.
https://community.fly.io/t/cert-not-being-issued-after-12h/9795/24 This might have something to do with it. I did not have a AAAA record set up (only A record as per the documentation), EDIT: Sorry, I did have the AAAA record, as per the documentation. and fly.io have apparently made some updates here. I will continue with trying to recreate the certificate and redo the DNS settings using A and AAAA posts. (My DNS provider does not allow me to put CNAME under @
Fly.io
cert not being issued after 12h
A fix has been rolled out. Sorry about all of that, there are more rough edges with shared IPs than we initially thought.
issuing a new certificate did the trick!!
1. Removing the old certificate
fly certs remove bookd.xyz -a foretagscenter-client
2. Creating a new certificate fly certs create bookd.xyz -a foretagscenter-client
3. Adding the proper DNS records accoring to logs (A and AAAA)
4. Watch progress fly certs show bookd.xyz -a foretagscenter-client
5. Repeat for www (but with CNAME post on www subdomain)
Thank you all so much for the help, I highly appreciate it!Wohooo @frej1046, you just became a Waspeteer level 1!
Awesome @frej1046 , I am glad we figured it out!
Do you think there is something that needs updating in our docs?
No, not at all. The docs are fine. This seems to have been a fly.io thing, where they struggled with the certificates for shared IPs.
Got it, thanks :)!