🚫 MongoDB doesnt work after disk grow
Hello, i did disk increase, and after that my databaze is not working.
Solution:Jump to solution
had a look at their project, they deployed a new mongo database and swapped the volumes around.
they would need to undo everything they did to get back to a working state.
they would also need to know the password of the original database because they deleted the original service....
34 Replies
Project ID:
ee8056af-fbe1-471a-82a0-db5d10cdaf8fee8056af-fbe1-471a-82a0-db5d10cdaf8f
Have you redeployed the database?
Yes, redeploy. restart
redeploy restart?
Redeploy / Restart
both
Oh so you redeployed and restarted the database?
I tried both yes
Only a redeploy is necassery, it should've worked after redeploying 🤔
Can you connect to the database via a db client like MongoDB Compass?
Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/vol_fzhp6wu9vmqj11ae
Initialized wire specification
Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize.
Successfully registered PrimaryOnlyService
Successfully registered PrimaryOnlyService
Multi threading initialized
Starting TenantMigrationAccessBlockerRegistry
MongoDB starting
Memory available to mongo process is less than total system memory
Build Info
Operating System
Options set by command line
Storage engine to use detected by data files
Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
Opening WiredTiger
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger opened
WiredTiger recoveryTimestamp
/sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never' in this binary version
vm.max_map_count is too low
Updated wire specification
current featureCompatibilityVersion value
Clearing temp directory
Initializing cluster server parameters from disk
Flow Control is enabled on this deployment
Initializing full-time diagnostic data capture
Setting new configuration state
Timestamp monitor starting
Listening on
Listening on
Listening on
Waiting for connections
mongod startup complete
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
WiredTiger message
Log
No i cant. Even envs doesnt work
my servers are down
Its not starting post deploy
That UI is a bit buggy, so can't really rely on that
I dont know, but it doesnt work
Worth asking, did you change anything other than growing the volume? Like you didn't swap volumes or anything, right?
Yes and no. I did that AFTER that, vecause i was trying to somhow export that db from server and maybe import it somehow back. But like i said, AFTER i tried everything, so after redeploy, restart, and more
Are you using the original volume again? Also did you change environment variables?
It's very odd to me that your database is refusing to connect 🤔
My guess is something changed on the database service that broke it
yes its same volume, because its 'production" volume. No i did not change enviroments. I just changed size, and after deploy it was loading and loading and its still loading
Then I'm out of ideas
It woukd be good if someone from team answered me 🙂
You'd need to be on pro-plan to be eligible for team support
I'm fairly confident this isn't a platform error so I can't really involve the team here and the team doesn't work over the weekends anyway
Although if we're lucky then brody will see this and help anyway
Sorry that I can't help further, I have a presentation in 7 hours and it's 1 am so I don't have time to look deeper into this
Yeah, its not problem to buy pro plan. but here is the thing. I switched to railyway from google cloud because it seems like good choice, but i asked gpt about that log, and he told its something with replication. And because i dont have access to anything, i cant fix it. So now there is question, when i have problem with something like this, and i cant do anything with it, or noone is going to help me, its worth it to buy pro plan ?
I think ChatGPT is wrong, your logs seem completely fine to me 🤔
Hobby plan only offers community support (which is what this is), if you buy pro plan then you'll be guaranteed a response from the team
Then when log looks nice, there is problem somewher on railway side. or something, becuase thing i did is railway thing
Okay, at least i have backups not from today but yeah, so i am going to create new mongo instance
with new volume
If it was a railway problem then I'd have seen this happen to other users, but i've never seen anyone's database break after growing their volume
that can definitally work
Can i somehow export from railway ?
What do you mean by that? Do you mean export your MongoDB data from the volume?
yeah
Usually what I do is I connect to the database from a db client (Compass in this case) and download the database from there
I'm unsure whether it's possible to extract it directly from the volume, I've never done that before
@Brody ^
Solution
had a look at their project, they deployed a new mongo database and swapped the volumes around.
they would need to undo everything they did to get back to a working state.
they would also need to know the password of the original database because they deleted the original service.
Yeah, but all that happene after problem with strorage increase, because i panicked and was trying to fix it, but nvm, i used backup and created new mongo. Thanks
next time you are having issues and you don't know what to do, please reach out to the community and someone will be able to guide you through what and what not to do.
for future reference, if you did not have a backup and you lost the data there would not be anything railway could do for you unfortunately, as we can only restore data if the platform was at fault.