❗Urgent - Deleted fork environment, all database volumes removed from production 🙃
were in the process of preparing to test an isolated environment, which was forked.
We shut it down after it didn't work as expected, and it appears all our volumes from our databases in our development and production environments were removed...
As it's a new environment backups were not enabled, we were actually in the process of setting them up, however we do have a backup from a few days ago.
Thee was some important time series information in there, is there a way we can restore that database as we didn't explicitly delete it, so the volume (should, I hope) still exist in your backend and just need re-attaching.36 Replies
Project ID:
80d54f52-7f6a-415c-b857-cb6aef417c8180d54f52-7f6a-415c-b857-cb6aef417c81
Thread has been flagged to Railway team by @Brody.
<a:party_railway:812486353853218816> 
its still Sunday night for a good chunk of the team, please be patient
No worries, we have created a new database, and restored the backup, we won't be touching the old ones for now.
Happy to be patient.
in the mean time can you provide more clarification? the volumes where deleted off of the service? leaving a database service without a volume?
Correct!
mind sending some screenshots
- All our deploys disappeared, and turned off
- Database volumes deleted
Steps to cause the issue:
- Fork development environment (not production)
- We reconfigured some services (so we wouldn't get duplicates on 2 of our listening services)
- Then decided we didn't need the fork, removed it
- Our monitoring system detected everything was offline
- Checked Railway, all deployments gone on everything (even the items we didn't touch in the fork)
Steps taken since
- Created a new DB, restored backup from 2 days ago (we're still missing 2 days worth of time series data)
- Pointed APIs at the new DB
that definitely goes against what the docs says
That's what I thought! 🙃
can you show me how you made the fork of the dev env?
Sure
It was 100% a fork
- I saw the arrow icon below the dev environment, pointing to the fork environment in the environments section in app
- When we made a change it said "merge changes" at the bottom
I have notified the team
I mean I have directly notified the team via chat and the flag
Thank you ser 🙏
Just as an FYI, my team is in Australia, it has just gone midnight, it may be 6-7 hours before we can respond from now.
But would love an update in the morning.
Would also love to discuss enterprise plan too!
Hey yeah, just checking in?
theres a good chunk of the team that are still off today
Right, can we get someone to look?
I'm sure a quick check of whether the volume still exists should be relatively quick and simple?
That's a good place to start IMO
this would be likey something jr would deal with, but as far as i know its a holiday for him today
There's only a single staff member who can check? <:pepe_okbruh:679051001189105702>
What does that look like on enterprise?
enterprise offers custom support, so if you wanted i dont see why you couldn’t get on call support if you wanted
Noted
But that would assume JR isn't the only person who can check something like this, right?
right, there would be a chain of whos on call
Okay
So we're waiting another day for someone to look at it?
It deleted production volumes when you deleted the forked env?
Yep, and all the deployments if the environments stopped
core and game services, right?
and game-2 is something you spun up after?Yep, the volumes gone
The other services just lost their deployments in production and development
Yeah we had a 2 day old backup, which we restored to.
We didn't want to add a new volume just in case
This is 100% on us 😦
Volumes are shared across forks right now; we have a new feature in preview that fixes this (staged changes)
What can I do to make this right for you? We can attempt to restore your volumes and the data will be >24hr old depending on when we snapshotted the host
Yeah okay
Unsure why they delete though?
Can you get a timestamp of that snapshot?
Looking into it now. It likely will be just 1d fresher than your backup, tho 😕
Unsure why they delete though?When you delete a volume in a forked environment, it deletes it in all environments because forks are not 100% isolated copies in the current flow Staged Changes solves that by isolating them within the environment they're in. You can enable that at https://railway.app/account/feature-flags We really should've put some giant warning banner on this current flow, sorry about that I'm gonna credit you 3x your last bill for the inconvenience caused, and will look into the timestamp of our backups for you
Thank you, we greatly appreciate it!
We are a software agency, and have some clients with some decent apps, and we love Railway for not having to think about infra (and that's coming from an ex-infra guy).
Would love to discuss enterprise options, we're building everything new in Railway and will be scaling these services!
Send you the credits!
Would love to discuss enterprise options, we're building everything new in Railway and will be scaling these services!Can you see the #direct-support channel?
Yep! Got it 🔥