Machine doesn't start stays at waiting for machine to be reachable
Hi, I'm having an issue when deploying, my server machine is never starting, it just stays at "waiting for machine to be reachable on 0.0.0.0:3001" and then stops after a few tries. In my local the server works, but on fly it never starts. Logs don't say the cause of the issue and I can't ssh either if the machine doesn't start. At some point I thought this was related to missing some env variables, I added them and did a clean deploy but it still doesn't start.
My dockerfile is the same as the default but I'm using a library that doesn't support Alpine so I added a line at the end:
# Any user-defined Dockerfile contents will be appended below.
# NOTE: Why do we specify alpine version here?
# Because if not, we had situations where it would use the different version
# locally and on Github CI. This way we ensure exact version is used,
# and also have control over updating it (instead of update surprising us).
FROM node:18-bullseye AS node
Fly.toml:
app = 'servername'
primary_region = 'mia'
[build]
[http_service]
internal_port = 3001
force_https = true
auto_stop_machines = 'stop'
auto_start_machines = true
min_machines_running = 1
processes = ['app']
[[vm]]
memory = '1gb'
cpu_kind = 'shared'
cpus = 1
9 Replies
Hi @phlx. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
----------------------
Based on the information provided in your question and the knowledge sources available, it seems you're experiencing a deployment issue with your Wasp application on Fly.io. Let's address this step by step:
1. Machine not starting: The issue you're describing where the machine doesn't start and stays at "waiting for machine to be reachable on 0.0.0.0:3001" could be related to various factors. One community member mentioned a similar issue in a Discord support thread (https://discord.com/channels/686873244791210014/1207206194872057917), suggesting it could be due to a failed database connection or incorrect configuration.
2. Environment variables: You mentioned setting all the required environment variables, which is good. Make sure you've set the following required variables as mentioned in the Wasp documentation (https://wasp-lang.dev/docs/advanced/deployment/manually#environment-variables):
-
b. Database connection: Ensure that your database is properly set up and accessible from your Fly.io deployment. Check the database logs and metrics on Fly.io as mentioned in the support thread.
c. Port configuration: Your
DATABASE_URL
- WASP_WEB_CLIENT_URL
- WASP_SERVER_URL
- JWT_SECRET (if using Wasp Auth)
3. Dockerfile modification: Your modification to use node:18-bullseye instead of Alpine should not cause this specific issue, as long as all the necessary dependencies are available in the Bullseye-based image.
4. Troubleshooting steps:
a. Check server logs: Try to access more detailed logs from Fly.io. As suggested in the Discord thread (https://discord.com/channels/686873244791210014/1207206194872057917), look for any error messages or indications of what might be preventing the server from starting.
b. Database connection: Ensure that your database is properly set up and accessible from your Fly.io deployment. Check the database logs and metrics on Fly.io as mentioned in the support thread.
c. Port configuration: Your
fly.toml file seems correct with the internal_port = 3001. Make sure this matches the port your server is actually listening on.
d. Redeployment: As suggested in another Discord thread (https://discord.com/channels/686873244791210014/1217941277979574315), sometimes redeploying the app can resolve issues. Try running wasp deploy fly deploy again.
5. Additional troubleshooting: If the issue persists, you might want to:
- Check if there are any network-related issues on Fly.io's end.
- Verify that your app's dependencies are all compatible with the Node.js version you're using.
- Consider temporarily removing any custom code or dependencies to see if the base application deploys successfully.
If none of these steps resolve the issue, you might need to reach out to Fly.io support or the Wasp community with more detailed logs and information about your specific application setup.
Remember, deploying to cloud platforms can sometimes be tricky, and issues like this often require a bit of detective work to resolve. Don't hesitate to ask for more help if you need it!
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------
Sorry @phlx, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:
Perhaps try again in 15mins?@kapa.ai Database is looking good, no issues in the logs its running normally, I have the enviroment variables my app uses in flyctl secrets myserver. There isn't any errors in the logs of the server, it just stops after trying to start multiple times. The port its 3001 in fly.toml and also in the fly secrets. Tried wasp deploy fly deploy and still with the issue. I'll try removing one of the libraries that node-alpine doesn't support but this would make my app unusable even if it deploys correctly
Let me check the knowledge base for you again. One moment! :books:
----------------------
Thank you for providing more details about your deployment issue. Based on the information you've shared and the knowledge sources available, let's explore some potential solutions:
1. Port configuration: You've confirmed that the port is set to 3001 in both
fly.toml and fly secrets, which is good. However, make sure that your server application is actually listening on this port. Sometimes, there can be a mismatch between the configured port and the one the application is using.
2. Environment variables: You mentioned setting the environment variables using flyctl secrets. This is the correct approach. Double-check that all required variables are set, including DATABASE_URL, JWT_SECRET, WASP_WEB_CLIENT_URL, and WASP_SERVER_URL as mentioned in the Wasp documentation (https://wasp-lang.dev/docs/advanced/deployment/manually#set-up-a-flyio-app).
3. Deployment process: Since you've tried wasp deploy fly deploy without success, we might need to look into the deployment process itself. As suggested in a GitHub issue (https://github.com/wasp-lang/wasp/issues/1498), there could be issues with how the Wasp CLI handles Fly.io deployments.
4. Rollback and retry: If the deployment keeps failing, you might want to try deleting the existing resources on Fly.io and starting fresh. As mentioned in the GitHub issue, you can use flyctl apps destroy <app-name> to remove the existing app and then try deploying again from scratch.
5. Dockerfile modifications: Your plan to remove the library that's incompatible with node-alpine is a good troubleshooting step, but as you mentioned, it might make your app unusable. Instead, you could try creating a minimal version of your app without the problematic library just to test if it deploys successfully. This could help isolate whether the issue is with the deployment process or with your app's dependencies.
6. Fly.io resource allocation: Check if your app is getting enough resources. As mentioned in your fly.toml, you're using 1GB of memory. Make sure this is sufficient for your application.
7. Database connection: Although you mentioned the database is looking good, it's worth double-checking the connection string and ensuring that the server can actually connect to the database when it starts up.
8. Logs and debugging: Since you mentioned there are no errors in the logs, it might be helpful to add more verbose logging to your server startup process. This could provide more insight into where exactly the server is hanging or failing.
If none of these suggestions help, you might want to consider reaching out to Fly.io support directly or posting your issue in the Wasp community forums with more detailed logs and information about your app's structure. The Wasp Discord channel (https://discord.com/channels/686873244791210014/1201898985300971540) could also be a good place to get more specific help from the community.
Remember, deploying to cloud platforms can sometimes be challenging, and it often requires patience and systematic troubleshooting to resolve issues like this.
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------
Sorry @phlx, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:
Perhaps try again in 15mins?Tried everything but didn't work 😦
Wohooo @phlx, you just became a Waspeteer level 1!
@phlx sorry for slow response we are on our yearly retreat so we are a bit slow!
So you just put that one line in your custom Dockerfile? Problem is, that Dockerfile does nothing then. If you need to change the base image completely, you will want to copy the whole Dockerfile and rewrite it to use your image (ubuntu).
@martinsos Yeah I also tried manual deployment and changing the first line in the generated Dockerfile in .wasp/build to ubuntu but I still had the same issue in deployment 😔
did you try manually restarting your server?
(https://fly.io/docs/apps/restart/)
the good old off and on trick helped me.
In each deployment I had to manually restart a service (for me it was always the database) to get it working, my error logs looked similar
@phlx
"waiting for machine to be reachable on 0.0.0.0:3001" means the PORT secret is maybe not set, is the PORT secret set to 3001?