Deployments (still) stuck in Waiting
This has been going on pretty much all weekend, sadly, despite the several incidents and them being "resolved". Until I recently tried an empty commit to kick the deployment process and try again, I had deployments sitting in "Waiting" for 5h+. The one that was an attempt to kick the process is currently at 25m and counting (despite the github actions flows being done for at least 10 minutes). Is there anything I can do to help this other than wait and hope railway starts being semi-reliable?
22 Replies
Project ID:
41f3e5b0-3233-406e-938c-ed0cbe42616341f3e5b0-3233-406e-938c-ed0cbe426163
abort them
Can't
why?
Triple dots menu on them has no entries
interesting, and what would happen to a new deployment
i've had like 2 succeed over the weekend while others are still sitting waiting
so on one of the services, right now, after the last good deploy, there are 5 waiting, 2 removed, 1 skipped
others are similar
when was the last time you tried kicking off a new deployment
25 minutes ago, with an empty commit
and whats the current state of that
Waiting
Flagging this thread. A team member will be with you shortly.
Hey, I see you have Check Suites enabled. Are they resolving properly?
Yes, all the GitHub actions are completing successfully in approximately 10-15 minutes. So I expect the start to be delayed that long
The github actions are actually doing docker builds and pushing the images to the github registry. The railway builds are building essentially empty dockerfiles that just pull and re-push those images, which is why the check suites are on (so the docker images exist before railway tries to pull them)
Hmm, I suspect there's an issue with the check suites workflow in that case. I can open a ticket for us to look into it this week.
I'd appreciate it. I'm so close to getting an alpha version of this out the door, and having to wait days for a config fix to go live has been pretty frustrating
If it's a blocker, you can technically switch to using GitHub actions to deploy. https://blog.railway.app/p/github-actions
Railway Blog
Using Github Actions with Railway
Github Actions come with a pretty neat set of features to automate your workflows. In this post, we talk about using Github Actions to automate your deployments on Railway.
Are there any downsides of doing it that way? It seems like it is sending a tarball of the repo contents out from the action instead of railway doing a proper git clone. Is that going to cause issues for me with github over actions bandwidth or anything?
I am not sure as I haven't extensively used that flow. Just suggested it as an alternative in case you couldn't wait for the fix.
After some brief testing today, I think there may be a weird condition that happens if I push to master and my prod deployment branch approximately simultaneously. Even though both the master and prod checks all succeed, I suspect that railway doesn't check or see that the master checks are successful. If i push to master and wait until it is green, and then push to prod, the deployments have been working pretty cleanly.
This has reliably worked (push to master, wait, and then push to prod after master is green) for the last couple days, for what it is worth