Deployments failing after Plugin Migrations
Need help ASAP. Production services are down after running redis plugin migration. Also no logs shows in deployments
Deploy: c128abaf-453e-470b-806e-f6e7d13e5236
76 Replies
Project ID:
c128abaf-453e-470b-806e-f6e7d13e5236where the migrations successful?
It showed dialog saying everything was successful
this does relate to the same project as your previous help thread does?
Nope different one
Can you check logs for my services that are down?
deployments run without showing any logs
I do not work for Railway so I would not have access to such things
ah my bad
can you try to manually redeploy the effected services?
I thought you are part of the team 😄
I did already multiple times 😦
lots of people do, check my bio 🤣
okay tagging in team now
@Christian - deployments in crashed state with no logs
thanks
We're looking into this now; thank you for escalating
thanks
The issues here are caused by of connectivity errors with GitHub
@Rem Kim could you help us explore a couple of avenues here:
- Do you have check suites enabled for impacted service? https://docs.railway.app/guides/github-autodeploys#check-suites
- Did you rename anything recently? Service, GitHub repo, GitHub org name, etc?
Hey @Rem Kim. Can you answer the above
Also this resolveai-cms and resolveai-node-resque, did the deploys get removed despite these failing builds?
yes
I did move repos to org but I did reconnect them later on
Can you try doing this now please?
We're getting "NotFound" errors from GitHub for these repos
Still seeing source as personal account and not an org.
khm
let me check
Could you share username and org name as well to avoid confusion?
1 min
org name is ResolveaiHQ
So the linked user is
rem4ik4ever.
Could you go here and change this? https://railway.app/project/5bcbe412-75a3-4e83-bd3b-e05ad2c5db06/service/76e50908-ae5c-4c8b-bb8d-a8f6a0d86498/settingsi get an error trying to connect
Error plz
it still tries to
One sec.
connect to private repo
I'll remove old one manually.
Try now for
resolveai-cms?This part.
Once you're good, I can do it for
resolveai-node-resque too.still same error
refreshed still error
refreshed page
Now?
Old one has been removed.
worked now
Your logs are logging now
see build logs too
Awesome, let me do it for the other one too.
Please try for the other resolveai-node-resque
sure thanks
(I'm looking into why a failed build was able to remove your deploy. That absolutely shouldn't happen as is completely unacceptable)
connected deploying too
I'm surprised that happened now after redis plugin update
Surprised what happened now?
The "Not Found" issue?
deployments failed and repo disconnected
When did you rename/reorg the thing?
because when I moved repos I checked what my services were pointing to
that was at the beginning of this week I believe
@fp Do we redeploy if there's a source change? We do right?
I ask, because your last deploy on these two services was 9 days ago @Rem Kim
So, I'm wondering if somehow that config change got dropped for these 2 services
(Either as in forgot on your side or not committed on ours. That, I do not know)
We do. I'd imagine these were due to the plugin migration though?
Whatcha mean
Plugin migration redeploys affected services.
But it redeploys using the standard deploy flow
And that should have not removed stuff when it hits "Failed"
Thank you gents. Everything is up now
I hope my postgres migration is not gona break everything again
I don't understand this either btw. On our end, it was pointing to the
user/repo combo.The non-org one right?
yes.
So, somehow it didn't get committed. We can look into that
@Rem Kim when did you change the repo source? Any chance you forgot? 😛
I will sit here and watch as you do it 
(asking for a sanity check before I go digging in)
I migrated to org this monday.
Then I went to Railway and checked where repos point and it was showing ResolveaiHQ
Very weird. We will absolutely have a look
Interesting. It was already showing the org without you needing to change the source?
yeah
Will dig in.
starting postgres migration 🤞
Watching!
(BTW, the migration will remove all deploys so we don't write at all to your DB while we're migrating it)
So, mystery solved there
But, bad timing on that bug I'm affraid 😦
We'll have a look into it
ahh makes sense
Running downstairs to grab water but have migration open on my
not sure if it succeeded I'm back to migrate dialog
Yup it's going
See it's redeploying all your services
resolveai-node-resque active again
ok so just wrong dialog popped up
Well, it'll let you re-run it multiple times
in case it fails
i think all looks good.
Looks good to me
Awesome thanks for looking over!
Nice, apologies for the issue. We've already lined up tickets for fixes. also added $100 in credits to your team so your next couple of bills are on us. 😄
also added $100 in credits to your team so your next couple of bills are on us. 😄wow thanks 😄