Database Backup Guide for V2 database?
Howdy,
We recently migrated from a v1 database to a v2 instance.
The new v2 instance is this project: https://railway.app/project/3644e3a1-b748-46b4-b196-2a94c381d1f0/service/bf6524db-d1f2-41ee-953b-55cba9112421
The backup cron for this service is:
https://railway.app/project/ddbced4c-9cfd-4736-9c8b-bfc892c86873/service/fe3ec062-651b-42b8-be7c-6f0981cbd146/variables
Unfortunately when I actually check the bucket, these backups dont have any data?
I tried loading some of these and they don't actually include any data from my production environment....
Just wondering if the process has changed for v2 backups? Or if I am doing something wrong
Railway
404 - Page not found
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
Railway
404 - Page not found
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
52 Replies
Project ID:
3644e3a1-b748-46b4-b196-2a94c381d1f0,bf6524db-d1f2-41ee-953b-55cba9112421,ddbced4c-9cfd-4736-9c8b-bfc892c86873,fe3ec062-651b-42b8-be7c-6f0981cbd146can you provide your deployment logs for the backup service https://bookmarklets.up.railway.app/log-downloader/
yes
maybe that warning?
š¤
you're running an old version of the backup script, do you still have the upstream repo linked to your service?
Yes let me try updating that and triggering a backup. That will probably resolve the issue but ill come back if it doesn't. Thank you brody! šāāļø š«¶
classic PEBKAC issue probably š
there was an update?
Updated the backup script yes
redeployed the service, waiting for the cron to trigger now
you can set a run on start variable
ohh let me do that good idea
RUN_ON_STARTUPyup just set and re triggering
so I set this variable in the config of the backup cron
then I pushed a readme change to our fork of the job to trigger a full redeploy
and it doesn't look like it ran?
what does it look like then?
This is all the deploy logs say now
ill try manually running it on my machine
looks like its running an old version, the new version would print the version of both node and pg_dump
Yup it wasn't running with the code from the PR i just merged. Even after running post PR merge the deploy logs are just completely empty.
I'm honestly just gonna delete the service and try reinstantiating from scratch š¤·āāļø
Thank you for your help!
Okay I actually got some info!
redeploying the service from scratch got some good logs
gonna look into this
šÆāāļø
those are good lol?
error while running backup?
im probably dumb haha
last i checked errors arent good š¤£
I was confused from your comment š¤£
but at least it didnt silently make a 20 byte gzip file in your bucket
No not good but at least having error messages is progress
haha love those 20 bytes 
what is the image source of your database?
postgres!
https://github.com/railwayapp-templates/timescale-postgis-ssl/pkgs/container/timescale-postgis-ssl
GitHub
Build software better, together
GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 420 million projects.
what tag?
ghcr.io/railwayapp-templates/timescale-postgis-ssl:pg15-ts2.12
will do some testing and get back to you
Should I try downgrading that maybe? This fk constraint doesn't actually look like its one of the tables in my schema š¤
Thank you
its just a warning, but the warning is treated as an error, i introduced this bug myself
lol! š
on one hand the backup will now tell you if there was an error and not upload an empty dump.
on the other hand the backup will now yell you if there was a warning and not upload anything.
interesting
are you going to leave this behavior as is? I can try adjusting the script and opening a PR later today maybe?
i already fixed this in a local branch, just not super happy with how its done
since pg_dump prints both errors and warnings to stderr theres no super good way to differentiate between a warning and error
wtf pg dump š
a command line utility to dump a postgres database
no I know I just meant
oh lol i see what you mean
that behavior is kinda wack!
sounds like a pain in the ass to deal with
my current error checking code just checks for anything in stderr and throws an error
previous code didnt check stderr at all, this is how those empty gzip files where able to make it into s3
in my local branch, i check if stderr has text and the dump is empty and then throw an error, but if stderr has text and the dump isnt empty i print the text in stderr and an extra message that it was likely a warning and to make sure the backup worked
If possible can I have that current intermediary version of your code that you aren't happy with?
Just so that I can get this job up and running for my team while you come up with a clean sexy sollution that you are happy with š¤ ?
no promises it will work, wrote it a while ago and i dont know if i tested it with the postgres image you mentioned above
https://github.com/brody192/postgres-s3-backups/tree/fix-foreign-key-constraints
Thank you testing now š¹
i too will be testing, hopefully i can come to a good solution
it indeed works as intended
yup forgot to come back and let you know that the changes you sent me worked!!!
Thank you
šāāļø
no problem!