Railway incident ?? Ghost deployment without our interaction
This is incredibly serious. We just found out that railway created a deployment out of nothing out of thin air, taking our site down, using a very old random commit.
13 Replies
Project ID:
N/Acc @unstoppablerizz
Is it possible we got hacked, or is there something else happening here?
120b5ec5-59d8-4087-84ae-4e0b3d934aa7
ruling out that we got hacked, it looks like railway just triggered a ghost commit from may 2023
since to log into railway you'd have to use magic, and that didn't happen
so this is very messed up???
in order for railway to deploy your commits they listen to webhooks that github fires, it's most likely that this was github's doing
let me check if vercel triggered
this would quickly clarify it
vercel?
if it's a github issue, both railway and vercel should be affected
since they also use the same webhooks
and we deploy on every commit for backend/frontend
not necessarily, vercel could have some additional logic in there to prevent such things
in this case, it looks like only railway got triggered, so it narrows down to you
Any way we can have the team look at the requests? Also wondering if there's an attack vector somewhere
@Brody My naive thinking at this point is that there was like a queued deployment request somehow that was stuck in a "processing" state in the db, some eng in Railway ran a script that had nothing to do with it and it got triggered
possible, could be a dozen causes, regardless of cause, this definitely shouldn't have happened. will flag team
quick question, your service is back up now right?
Thread has been flagged to Railway team by @Brody.
Yeah! @Brody
we just pushed an empty commit
Can you link the deployment?