Lost data from east region to west region. Wondering if there's any way to rollback
With the big issue that's happened over the last hour, I tried to move regions from East to West. While this worked for all the other services, our mongoDB env was completely dropped. Is there any way to recover that data?
Looking specifically for staff from the Railway team to see if there's anything on your side to help us with this.
Solution:Jump to solution
You should see a new Mongo service named "MongoDB-Restored"
https://railway.app/project/5fd7c146-2bc8-4c17-ad18-4d0b59c44fa0/service/439f896d-fefd-4946-830d-0f258383cf90
Can you check if the data is accurate?...
71 Replies
Project ID:
5fd7c146-2bc8-4c17-ad18-4d0b59c44fa05fd7c146-2bc8-4c17-ad18-4d0b59c44fa0
!t
New reply sent from Help Station thread:
This thread has been escalated to the Railway team.You're seeing this because this thread has been automatically linked to the Help Station thread.
Hoping that y'all have access to some other MongoDB interface from the other region.
My guess is that you have migration scripts when moving from one region to another, but since the whole thing was down, it failed so it gave me a blank database
is this effecting a production application with a userbase?
Yes
I had to migrate data from our staging environment to our production environment, as the production database was completely wiped
We're in an OK spot now, but we lost a lot of data
@Ray - production database wiped during region migration
For context, we host websites for Funeral Homes. So people's obituary pages were down for a good hour
Thankfully our obits are stored in a separate datastore so none of that actual data was lost
https://railway.app/project/5fd7c146-2bc8-4c17-ad18-4d0b59c44fa0/service/ae145910-e686-4262-85da-9ee64354c6c7
Is this the service?
Railway
Railway
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
Yes that's the service
I'll defer to you guys if there is a backup... or if our data is still technically available in the east region. Open to any solution as long as we can get that data back
To confirm, was this in the production environment and a migration from US East -> US West?
Is the volume name the same before and after you performed the migration?
Correct, this was in the production environment. I migrated from US east to US west (as a possible solution to US east being down)
I did not change the volume name at all (I don't think I can since there is no volume attached)
there is no volume attached...?
Replicas
The number of instances that will be run for each deployment of this service.
Replicas
1
Replication is only available for services that do not have a volume attached.
Unless I am misunderstanding the question... it doesn't look like we configured a volume
screenshot of the database service please
is the MongoDB Settings page sufficient?
no
Thanks for the info - going to investigate this now.
(No worries on the volume name, I only asked to make sure I'm looking at the correct thing)
thank you both
@Brody do you still want that screenshot?
there is indeed a volume on the mongo service
yeah that makes sense. i just don't see anywhere to name it
I know these things take time but just getting a sense for timing—is this something I should expect a response about tonight or should I expect something more realistically tomorrow
(I also need to know so I can have our team re-create any data that was lost)
@zackword sorry, was attempting to see if I can restore your data
Do you have anything in the database linked above currently?
Can you also give me some table names so I can confirm?
We have our staging data in the DB now, I migrated it just to restore customers
But I would prefer the last production from us-east if you do have that
Some table names- lol I think we everything is under the “test” database, and the tables would look like users, venues, pages
Gotcha. Is it alright if I created a new database in your project so I can restore your previous data from US East into it?
Absolutely
Solution
You should see a new Mongo service named "MongoDB-Restored"
https://railway.app/project/5fd7c146-2bc8-4c17-ad18-4d0b59c44fa0/service/439f896d-fefd-4946-830d-0f258383cf90
Can you check if the data is accurate?
I just checked the users table and that looks right
Thank you so much
Guessing the best way to switch is to redeploy our app with the new mongodb service
Is that what you’d recommend?
Yep, redeploy with MONGO_URI set to the new one.
Got it. Thank you so much
We’re going to flip the switch after doing another quick test
Super sorry about this, it should never happen. I've sent some credits your way (average of your past 3 month's bill) that will be used to offset your next bill
Please let me know if you see anything off 🙏
Thank you for the credits
I need to run but this is super helpful
Nearly had a heart attack
Me too 😅
Sorry to bug you guys—it looks like I can't actually deploy since I don't have access to thew new MongoDB env?
could you explain what you mean? ray mentioned he created a new database in your project
See bottom right
I tried to point the existing website to the new MongoDB uri, and that's the error I get
that means you dont have permissions for the source repo of that service, nothing to do with mongo
please ask the owner to grant you access
Got it, so you're saying it's a github permission issue then?
(We are a startup of 3 people, and I'm the only one with access to this repo)
Uhh that's weird
yes ive ran into this issue myself when working on someone else's service, they needed to grant me the correct permissions on the repo and then it worked perfectly
An old employee may have connected the repo in the first place?
try to disconnect deploy and then reconnect?
yep, let's see
thanks
Do you have access to the Github repository?
Yep
Looks like you don't have a Github linked to your Railway account?
(I just disconnected it)
oh thats not what i meant
https://railway.app/account what do you see in here under "Account Integrations"?
oh, on my personal account?
ya
When I try to connect Github
This is also weird because I've definitely deployed in the last little while
What's your Github username?
@zackmoy
Also, it looks like now I don't have access to my Afterword corporate account
Sorry for the extra trouble, but like what lol
I think you have two different accounts :thinkies:
I connected my organization
The account inside your team is zack@<yourdomain.com> -- try logging into that via email?
wow
I'm really showing my best colors over here
I relogged in and it's back. How did that happen though? I didn't login with github, I just went to connect it
Seem like it took now. Thank you guys again
Just waiting for the deploy
Nice
you just clicked "Connect" under your personal account and it created a new account for you? 👀
yep this is a thing, if the github email isnt the same as the email on the railway account
well that definitely shouldn't be a thing :kekw:
Okay, we're all back up and running
For a brief moment it looked like the site was down again but that may have been intermittent
Thank you again. I'm sure you guys are bombarded with other issues after the downtime so I appreciate all your help
If you enable this + have healthchecks, you'll have near zero-downtime deploys 🙂
Currently, traffic switchover to your latest deploy is the slowest part of the process due to slow internal routing table updates. With the new edge proxy enabled, this becomes near-instant
Always happy to help, and sorry for the shock earlier!
Thanks for the heads up on this
Also noticed the runtime for v2 on deployment, Will these these out our staging env
not sure what you meant due to the typo, could you try correcting the question please?