29 Replies
Project ID:
6a6c4123-f4c9-4c41-ba5f-c3115ddc1753Those line repeat over and over and then mysql is paused by the crash watchdog.
I'm not seeing any errors in those logs, maybe it's something like you've ran out of disk space?
Nope - it's just Railway is having issues I think
full logs please https://bookmarklets.up.railway.app/log-downloader/
I'm not seeing repeating anything?
That is current after I set crash watchdog to stay up until 1000 crashes.
That is the one that was repeating
Like you say - no errors - mostly warnings - its a railway thing not an mysql thing
Have seen a number of issues today - dropped table (which could have been caused by this weird constant reset thing) - billing on the wrong day - now this - think there is something not quite right
well it looks like from your most recent logs that the database is up and operational
Which is good?
But what would be better would be identifying the cause and preventing it from getting into that state again?
I mean the whole thing was redeployed a few hours before because of the sudden billing causing everything to be paused - which is another thing that needs sorted - but not by community forums and that is being handled via email to billing.
If the logs can't tell us why the mysql was stuck in a loop rebooting then this could be a bit of an issue, no?
I also note someone else reporting similar with postgres mongo
you have to keep in mind these databases are completely unmanaged, the only logs railway has for these databases are the logs you see in the deployment logs
They are dockerised - there are no logs as to why the docker container is restarting unexpectedly?
Like I say - this seems to be at the infrastructure level rather than application - so there is nothing further I can add - but I would like to know what happened and if there are any steps to prevent it in the future.
if mysql decided to exit without printing any log as to why, there's really not much to go off of unfortunately
It comes back to my previous issues with binary logs (which the team have changed the mysql docker to avoid for everyone else) - the mysql docker is not something we have any control over (quite rightly) but it does mean the lack of ability to diagnose and fix issues with it causes things like this current issue to be very frustrating.
However, the DB does seem to be up, I can't add anything further as to how the plugin is setup and managed so will just have to "live with it" and move on.
Although it's great that I've been logged out and can't get back into Railway
you have more control than you probably think, you could have disabled binary logging the same way they did
Adds some weight to the "railway is having issues" thing
I already had - it was to do with the way disks were connected to the container
which I have no control over
that's the backboard, a completely separate system compared to the system that runs the containers
I'm sorry, but no it didn't
disabling binary logging was as simple as setting a command line flag in the start command, they just made it the default by updating the template
from mysql's docs
Because their disks were not purging the logs because of the way the disks were connected. ↻
Anyway, the mystery db issue will remain a mystery and maybe the unrelated issue about being able to login will also resolve itself, and I can get resolution on the billing issues also, all unrelated.
that's just not correct
it had absolutely nothing to do with the disk/volume itself, it was a mysql setting that was at fault
Its correct that non-railway deployments of the "standard mysql docker" or standard installs of mysql didn't have the binary log build-up, but only railway did.
a "disk" is not responsible for clearing any data stored on it, that's up to whatever is storing the data
railway updated the service's start command to include one of these flags, I assure you, that's all that was done, and all that could be done
The evidence speaks for itself - besides - being locked out of railway is far more of an issue than a docker plugin issue that was solved months weeks ago
I know, I'm just trying to set facts straight
These were the facts.
Multiple other locations with the exact code (and in fact synced data) didn't do it - only at Railway.
railway deploys the standard image, there's is nothing special or magical going on, it's most literally just a simple docker run command when it comes down to it