Postgres DB Crashed Due to Storage
Hello!
I'm using a js package called ponder (https://ponder.sh/) to index the ethereum blockchain. My deployment crashed from what looks like a lack of storage. When I look at the memory chart of my postgres database on railway though, I had around 3.5GB of memory remaining when it stopped.
Is there another reason why this could have broken? I'm not sure what final storage size will end up being, but I expect it should fall under 8GB. Any help is appreciated!
Project ID:
72e571de-3aaf-4521-bfcb-d36891096f9cPonder – A backend framework for crypto apps
Ponder is an open-source framework for crypto apps focused on developer experience and performance.
26 Replies
Project ID:
72e571de-3aaf-4521-bfcb-d36891096f9cYou would be looking at your memory, postgres does not store persistent data in ram.
Hobby users get 5gb disks and that would dictate how much you are able to store in the database, you can check how much disk space postgres has used by clicking the volume that is attached to the bottom of the database service within your project canvas.
If you need more than 5gb you would need to upgrade to Pro for 50gb volumes, then you can move the project over to your Pro workspace and grow the volume to 50gb from within the volumes settings.
Understood - looks like it failed at around 4.5GB, so that would make sense
You would still be looking at the memory, postgres does not store it's persistent data in memory.
In my message above I included instructions on how you would go about viewing the disk usage.
thanks for pointing that out again
I see it's tapped out - is there anything I need to do after upgrading to pro to increase the storage limit, or will it increase automatically?
I also included instructions for that in my initial message.
🤦♂️ sorry. Thanks for everything
No worries, let me know if you run into any difficulties.
I have a follow up question: I moved the project over and grew the storage, but the db deployment failed again because it says 'invalid length of startup packet'
I'm not very experienced with deploying services like this, so apologies for any dumb questions - I'm assuming there is some corrupted data because the db was interrupted last time.
Does this mean the data in the db is corrupted, or just some files that are needed for startup/other tasks?
If the db data is corrupted, is there a way I can easily roll back? No worries if not, I can run the indexing script again if needed
Can you access the database still? just wanna make sure even though I've seen these errors on my own database and have not faced any issues.
no, the deployment failed during build
First off can you make sure you can connect to the database with a database client like dbgate?
And second, please send your build logs using this -
https://bookmarklets.up.railway.app/log-downloader/
Sure - I think I'm unable to connect with dbgate. I get: "Error: Connection terminated unexpectedly"
What is the status of your postgres deployment?
failed
Build logs of the most recent deployment of the postgres deployment that failed during the build
Did you set a healthcheck on the postgres service?
I did. /health
I did because the indexing package said to add a healthcheck path for their deployment. I assumed it wouldn't hurt. I can remove it for the db though
You would have wanted to add that healthcheck to the indexing service, not postgres.
okay, understood. I set it for both. I'll remove it for postgres and try again
Sounds good
confirming the build was successful!
Awesome