Applying changes stuck?
This happens only on the staging environment, I changed a variable, and then applied then changes. and it stayed stuck on the "Apply changes..." status.
15 Replies
Project ID:
63f743a2-2373-455a-848d-33602e0c751dhttps://railway.app/project/63f743a2-2373-455a-848d-33602e0c751d/service/21c5ffb6-fa37-48d2-9174-5b33dd492a5f/settings
For debugging:
the request
https://backboard.railway.app/graphql/internal?q=workflowStatus with the data
workflowId: "commitChanges/d7cc47b2-805d-47d7-a917-dcc0fe2513f2/152ec7b2-bf14-40cb-92ce-b6a08ccb6c9a"
I get {data: {workflowStatus: {status: "Running", error: null}}}
This is still not working. I tried to remove the staging environment and I get an error. The Apply changes is blocking everything.
Project ID: 63f743a2-2373-455a-848d-33602e0c751dHey there,
Is the linked project still having and issue? I don't see any changes applying
Yes
Okay those changes are now marked as staged
It might have timed-out, but when I apply them again, it keeps being stuck... an no changes are applied
I have services that crashes on the staging environemnet, and at the moment I cannot delete services. It stays on the "Applying changes" status, and then nothing happens.
Can someone from the team look into this please?
We can definitely look into this. Sorry about the delay here. We will look into it first thing on Monday. In the meantime, it seems like you are deleting a bunch of services. Does deleting smaller batches work (maybe there is a single service that is not deleting propertly)
None of the service can be deleted (while trying individually)
Is your end goal to remove all the services and volumes in the staging environment?
If yes, lmk if I can press the commit changes button on any of the environments? There is definitely something strange going on with one/both of the environments
Ok I think I've resolved the issue in staging for deleting those volumes. Please try again and lmk. The problem was that one of the volume instances got in a bad state on the host
I haven't touched anything in the production environment yet. But lmk whats not working and we can get it in the correct state
I want to restart from scratch for the staging deployment as publishing the changes gets stuck.
Now clearly it's still the case with the volumes: I get the error an unknown error has occurred... I cannot even wipe the volume.
There's clearly a bug somewhere. I'm glad this is affecting only the staging and not the production
The strangest part is the on the observability, I still have log message from services that were deleted:
I identified the volume that is problematic: https://railway.app/project/63f743a2-2373-455a-848d-33602e0c751d/volume/3e52f92e-3f27-4100-8bed-f7839b918d94/metrics
I've resolved this error
you should be able to delete the environment now
Thanks it's fixed