Railway•3mo ago

Django migration hanging

I landed some changes which included a Dango migration, these normally run flawlessly, but today it just hangs, never finishes. I tried redeploying, also tried running the migration manually from the CLI and still hangs. The migration ran locally without issue. The dependency listed in the migration does exsit. Any advice on how to troubleshoot this? Not seeing any errors either. TIA

18 Replies

Percy•3mo ago

Project ID: 70cfde23-f2f2-4d58-8987-2a19bea968ed

Joshie•3mo ago

(project id so this doesn't close?)

Marj•3mo ago

70cfde23-f2f2-4d58-8987-2a19bea968ed

Joshie•3mo ago

Is it possible that the migration is just something that takes a long time on live data? The local db is just smaller and therfor fast to migrate? I guess to that end, do you mind sending the migration you are trying to run?

Marj•3mo ago

It is adding a new table with no data, it should not take more than a couple of seconds. My app is pretty new, so db is not very big This is the code, let me know if I should send it as a file? Generated by Django 4.2.4 on 2024-07-26 10:58 from django.db import migrations, models import django.db.models.deletion class Migration(migrations.Migration): dependencies = [ ('api', '0087_modify_planlimits_data'), ] operations = [ migrations.CreateModel( name='StripeReLinkActivity', fields=[ ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), ('created_at', models.DateTimeField(auto_now_add=True)), ('linked_to_stripe', models.BooleanField(default=False)), ('stripe_email', models.EmailField(blank=True, max_length=100)), ('stripe_customer_id', models.CharField(blank=True, max_length=100)), ('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, to='api.userprofile')), ], ), ]

Joshie•3mo ago

To send code, you can put it in a code block: ```py CODE HERE ``` That would look like

import django
print("hi")

import django
print("hi")

(I escaped the character for you. But it is the `. The character on the ~ key.

Marj•3mo ago

# Generated by Django 4.2.4 on 2024-07-26 10:58

from django.db import migrations, models
import django.db.models.deletion


class Migration(migrations.Migration):

    dependencies = [
        ('api', '0087_modify_planlimits_data'),
    ]

    operations = [
        migrations.CreateModel(
            name='StripeReLinkActivity',
            fields=[
                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                ('created_at', models.DateTimeField(auto_now_add=True)),
                ('linked_to_stripe', models.BooleanField(default=False)),
                ('stripe_email', models.EmailField(blank=True, max_length=100)),
                ('stripe_customer_id', models.CharField(blank=True, max_length=100)),
                ('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, to='api.userprofile')),
            ],
        ),
    ]

# Generated by Django 4.2.4 on 2024-07-26 10:58

from django.db import migrations, models
import django.db.models.deletion


class Migration(migrations.Migration):

    dependencies = [
        ('api', '0087_modify_planlimits_data'),
    ]

    operations = [
        migrations.CreateModel(
            name='StripeReLinkActivity',
            fields=[
                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                ('created_at', models.DateTimeField(auto_now_add=True)),
                ('linked_to_stripe', models.BooleanField(default=False)),
                ('stripe_email', models.EmailField(blank=True, max_length=100)),
                ('stripe_customer_id', models.CharField(blank=True, max_length=100)),
                ('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, to='api.userprofile')),
            ],
        ),
    ]

Thanks 🙂

Joshie•3mo ago

Ok, so the next thing I wonder, is if you have any open transactions during the migration. If you have running transactions, the migration won't happen. So,,, you would have to kill the web interface to the db and then let the migrations go through. Otherwise, yea, it hangs. But if you have a small server,,, I wonder if this is actually the issue :shrug: How are you running the migration?

Marj•3mo ago

The migration first ran as part of a deployment that was kicked off my merging to master on Github. When that didn't work, I used the CLI to connect to the container and kick it off. It is still running...

Operations to perform:
  Apply all migrations: admin, api, auth, authtoken, contenttypes, sessions
Running migrations:
  Applying api.0088_striperelinkactivity...

Operations to perform:
  Apply all migrations: admin, api, auth, authtoken, contenttypes, sessions
Running migrations:
  Applying api.0088_striperelinkactivity...

Let me try restarting the front end Actually, the backend is showing as up and running, but the front end is getting a 503 error, service unavailable

Joshie•3mo ago

So did the migration go through?

Marj•3mo ago

No, and I rebooted the railway container but it seems to try to apply the migration and just hangs, so app is not working at all Should I reboot the postgres db?

Joshie•3mo ago

Do you not see any other logs of what the error is? If you are able to reboot the db, you can try that. But I would first look at what the logs say about the errors

Marj•3mo ago

Restarted the postgres DB, that caused the dango container to crash, after it restarted, it responds, but getting these errors on every request:

Error getting new users over time: connection to server at "roundhouse.proxy.rlwy.net" (35.212.138.205), port 11654 failed: server closed the connection unexpectedly

This probably means the server terminated abnormally

before or while processing the request.

Error getting new users over time: connection to server at "roundhouse.proxy.rlwy.net" (35.212.138.205), port 11654 failed: server closed the connection unexpectedly

This probably means the server terminated abnormally

before or while processing the request.

On the bright side, the migration worked and my new table was added

Joshie•3mo ago

Ok. Well we found the cause of the issue then. So, DBs don't let you do migrations if there are open connections. I am sure there are some enterprise way of getting around that. But :shrug: Anyways, railway had an open connection through that roundhouse endpoint. And so, you couldn't migrate. By reseting the db, you forced the connection to close. There are other ways you can force all connections to be closed. Not the point. I just wonder why this proxy is hanging like that? Are you using TCP? (I assume since you got this error)

Marj•3mo ago

Backend is 2 containers, Django and Postgres, frontend is hosted on Netlify So it is still down, should I reboot Django again? Oh wait, it is back up now

Joshie•3mo ago

👍. I wonder. Maybe netlify was just taking a second?

Marj•3mo ago

Ya maybe that's what was going on. Well phew! Thanks for your help! Will try to see if I can check for open connections before migrating or maybe upgrade if that would solve the problem.

Joshie•3mo ago

I am going to ping someone who might have a better idea about this than me. @Brody (tldr solved for now, but something is still up)

Gaming

Programming

Django migration hanging