Froge
Explore posts from servers❗Urgent - Deleted fork environment, all database volumes removed from production 🙃
Thank you, we greatly appreciate it!
We are a software agency, and have some clients with some decent apps, and we love Railway for not having to think about infra (and that's coming from an ex-infra guy).
Would love to discuss enterprise options, we're building everything new in Railway and will be scaling these services!
46 replies
❗Urgent - Deleted fork environment, all database volumes removed from production 🙃
Right, can we get someone to look?
I'm sure a quick check of whether the volume still exists should be relatively quick and simple?
That's a good place to start IMO
46 replies
❗Urgent - Deleted fork environment, all database volumes removed from production 🙃
Just as an FYI, my team is in Australia, it has just gone midnight, it may be 6-7 hours before we can respond from now.
But would love an update in the morning.
Would also love to discuss enterprise plan too!
46 replies
❗Urgent - Deleted fork environment, all database volumes removed from production 🙃
It was 100% a fork
- I saw the arrow icon below the dev environment, pointing to the fork environment in the environments section in app
- When we made a change it said "merge changes" at the bottom
46 replies
❗Urgent - Deleted fork environment, all database volumes removed from production 🙃
- All our deploys disappeared, and turned off
- Database volumes deleted
Steps to cause the issue:
- Fork development environment (not production)
- We reconfigured some services (so we wouldn't get duplicates on 2 of our listening services)
- Then decided we didn't need the fork, removed it
- Our monitoring system detected everything was offline
- Checked Railway, all deployments gone on everything (even the items we didn't touch in the fork)
Steps taken since
- Created a new DB, restored backup from 2 days ago (we're still missing 2 days worth of time series data)
- Pointed APIs at the new DB
46 replies