R
Railway8mo ago
Froge

❗Urgent - Deleted fork environment, all database volumes removed from production 🙃

alert were in the process of preparing to test an isolated environment, which was forked. We shut it down after it didn't work as expected, and it appears all our volumes from our databases in our development and production environments were removed... As it's a new environment backups were not enabled, we were actually in the process of setting them up, however we do have a backup from a few days ago. Thee was some important time series information in there, is there a way we can restore that database as we didn't explicitly delete it, so the volume (should, I hope) still exist in your backend and just need re-attaching.
36 Replies
Percy
Percy8mo ago
Project ID: 80d54f52-7f6a-415c-b857-cb6aef417c81
Froge
Froge8mo ago
80d54f52-7f6a-415c-b857-cb6aef417c81
Duchess
Duchess8mo ago
Thread has been flagged to Railway team by @Brody.
Froge
Froge8mo ago
<a:party_railway:812486353853218816> pepelovehearts
Brody
Brody8mo ago
its still Sunday night for a good chunk of the team, please be patient
Froge
Froge8mo ago
No worries, we have created a new database, and restored the backup, we won't be touching the old ones for now. Happy to be patient.
Brody
Brody8mo ago
in the mean time can you provide more clarification? the volumes where deleted off of the service? leaving a database service without a volume?
Froge
Froge8mo ago
Correct!
Brody
Brody8mo ago
mind sending some screenshots
Froge
Froge8mo ago
- All our deploys disappeared, and turned off - Database volumes deleted Steps to cause the issue: - Fork development environment (not production) - We reconfigured some services (so we wouldn't get duplicates on 2 of our listening services) - Then decided we didn't need the fork, removed it - Our monitoring system detected everything was offline - Checked Railway, all deployments gone on everything (even the items we didn't touch in the fork) Steps taken since - Created a new DB, restored backup from 2 days ago (we're still missing 2 days worth of time series data) - Pointed APIs at the new DB
Froge
Froge8mo ago
No description
Brody
Brody8mo ago
that definitely goes against what the docs says
No description
Froge
Froge8mo ago
That's what I thought! 🙃
Brody
Brody8mo ago
can you show me how you made the fork of the dev env?
Froge
Froge8mo ago
Sure It was 100% a fork - I saw the arrow icon below the dev environment, pointing to the fork environment in the environments section in app - When we made a change it said "merge changes" at the bottom
Want results from more Discord servers?
Add your server