Migrating Postgres to v2 databases has gone horribly wrong. Help
I should mention that I'd already migrated another project with near enough exactly the same stack, with no issues.
Started migration and it got stuck at "Deploy V2 Database".
After ~10 minutes I decided to have a look at the new database - there were no logs.
I manually attempted to redeploy the database (a few times) and it logged some errors about the database already existing.
One of the redeploys appeared to be successful, but still showed error messages.
I checked the migration status and it was still stuck at "Deploy V2 Database".
I decided to delete the new database and create another myself - it successfully deployed within seconds.
My migration status is still stuck at "Deploy V2 Database" (~30 minutes later).
What can I do now?
48 Replies
Project ID:
a8d6a2f4-5aaf-462c-ad92-3e4d9b4c6399Project ID:
a8d6a2f4-5aaf-462c-ad92-3e4d9b4c6399
Also, I should apologize for kinda fanning the flames on this one. 😬
Maybe if I'd waited a bit longer it would have succeeded. 🤷migration hiccup, can you show me a screenshot of the project?
What exactly would you like to see?
a screenshot of your browser's viewport
I think the first of the above 2 covers that?
indeed, and just so i know how urgent this is, is your app offline?
Nope. Still functioning.
go ahead and delete the new postgres database, and that stray volume
To clarify, all of these?
Postgres
pgdata
Postgres data
correct
Okay, they're gone. Just the Postgres Legacy and my main server process remaining.
deploy a new postgres database
Done. Still stuck at "Deploy V2 Database".
we can ignore that, we're doing this manually
😐
Scary stuff. 😂
it's easy, trust me
is your app using reference variables?
As in the database ones?
Should they have curly braces in them or something?
They appear to all be hard coded... but they have the pg logo next to them.
look in the raw editor
they should look like this example https://docs.railway.app/guides/variables#referencing-another-services-variable
If the references are the ${{}}, then yes.,
perfect
I'm gonna finish my dinner then we can get into this!
No worries. It may be best we continue this much later today as it's 1am here, and I've gotta be up kinda early. 😬
alright no worries ping me when you are ready to begin
I should have known better than to deploy at midnight. 😂
Catch you later. 👋
see ya
@Brody you free to walk me through this now? 😁
Actually, "Migration Errored".
"Deploy V2 Database - Failed to migrate"
Worth me just restarting it?
it could be, but first let's see a screenshot of your project to catch me up
Just these and my main server process below.
I assume if I were to restart the migraiton I should delete the new PG + pgdata?
that's correct, but you would also want to rename the legacy database back to just
Postgres
and I'm forgetting if I've ask you, but you are using variable references right?Yup.
yep then do the delete and rename stuff and see if it gives you an option to run the migration again
Will do. 😁
<a:party_parrot:812481236114079784>
progress
Would you mind explaining the process you would have taken if the migration was still stuck on "Deploy V2 Database"?
It'd be useful for my knowledge, and help anyone else that comes across this issue. 😊
Migration has succeded. Looks like all my data's in there, and the app is functioning. 😁
Just an overview would be fine.
this is the process we would have ran though, just manually.
that migration process just deploys this template to migrate your data
https://railway.app/template/postgres-plugin-migration
Oh, nice. Just install the migration service, define 2 variables, stop my server, deploy the migration service?
yep, then swap the namespace of the reference variables over, and you're done
As an alternative, what downsides would creating a database dump and then populating a fresh database with it have?
More downtime?
it depends, because if you don't take your service offline you could end up with missing data in your new database, but if you aren't doing any writes then you could do a backup and restore with zero downtime without taking the service down
It wouldn't have been the end of the world for this app, as it's a bespoke built thing that only gets used every few months. 😂
Thank you for your help.
Think this is my first experience with Railway support via Discord, and I'm liking it. 😁
I'm happy to help!
Will the legacy postgres clean themselves up after the 31st?
they will be shut down on the 31st and it will offer you one last chance to do a migration before they completely remove them on the 12th of February
Alright. May as well delete them to save on some fees anyway. 😂
yeah and to get rid of that migration message