Twenty•7mo ago

Update from 0.3.2

Hello, I have a docker install in version 0.3.2. What are the right steps to update the system to the last version?

54 Replies

Hi @iero! Hope you are doing well For now, we recommend migrating step by step: - 0.3.2 > 0.3.3 > 0.4.0 > 0.10.0 However, I can assist you and go directly to 0.10.0 if you like You'll experience some downtime though First, update your containers to use v0.10.0

greg [iero]OP•7mo ago

Yes it will be good to go directly to 0.10.0 I will backup first

charles•7mo ago

ok!

greg [iero]OP•7mo ago

Could you confirm that I'm OK with this dump? Nothing more to backup?

pg_dump -U twenty -h localhost -p 5432 -Fc default > db_twenty.dump

pg_dump -U twenty -h localhost -p 5432 -Fc default > db_twenty.dump

charles•7mo ago

I'm doing: pg_dump postgres://user:password@domain:5432/default > 04-22-2024.sql so it looks good, not sure about -Fc options and what they do

greg [iero]OP•7mo ago

Ok done both to be sure:

iero@endor ~/Twenty/backup % ls -hl                                                               total 2272
-rw-r--r--  1 iero  staff   337K Apr 24 14:15 db_240424.dump
-rw-r--r--  1 iero  staff   749K Apr 24 14:16 db_240424.sql

iero@endor ~/Twenty/backup % ls -hl                                                               total 2272
-rw-r--r--  1 iero  staff   337K Apr 24 14:15 db_240424.dump
-rw-r--r--  1 iero  staff   749K Apr 24 14:16 db_240424.sql

I remove containers and update to last version ?

charles•7mo ago

keep your database instance, are you using a persistent storage? you should only update the container images but keep the data (even if you have made a dump)

greg [iero]OP•7mo ago

I have the three containers:

iero@endor ~/Twenty/backup % docker-compose ps                                                                                                                                                                           [14:32:11]
WARN[0000] /Users/iero/Twenty/compose.yaml: `version` is obsolete 
NAME                    IMAGE                              COMMAND                  SERVICE        CREATED       STATUS        PORTS
twenty-twenty-back-1    twentycrm/twenty-server:latest     "docker-entrypoint.s…"   twenty-back    6 weeks ago   Up 41 hours   0.0.0.0:3000->3000/tcp
twenty-twenty-db-1      twentycrm/twenty-postgres:latest   "/opt/bitnami/script…"   twenty-db      6 weeks ago   Up 41 hours   0.0.0.0:5432->5432/tcp
twenty-twenty-front-1   twentycrm/twenty-front:latest      "docker-entrypoint.s…"   twenty-front   4 weeks ago   Up 41 hours   0.0.0.0:3001->3000/tcp

iero@endor ~/Twenty/backup % docker-compose ps                                                                                                                                                                           [14:32:11]
WARN[0000] /Users/iero/Twenty/compose.yaml: `version` is obsolete 
NAME                    IMAGE                              COMMAND                  SERVICE        CREATED       STATUS        PORTS
twenty-twenty-back-1    twentycrm/twenty-server:latest     "docker-entrypoint.s…"   twenty-back    6 weeks ago   Up 41 hours   0.0.0.0:3000->3000/tcp
twenty-twenty-db-1      twentycrm/twenty-postgres:latest   "/opt/bitnami/script…"   twenty-db      6 weeks ago   Up 41 hours   0.0.0.0:5432->5432/tcp
twenty-twenty-front-1   twentycrm/twenty-front:latest      "docker-entrypoint.s…"   twenty-front   4 weeks ago   Up 41 hours   0.0.0.0:3001->3000/tcp

The database is in a docker container I might test in a test environment to be sure 🙂

charles•7mo ago

could you share your docker-compose.yaml (or compose.yaml)

greg [iero]OP•7mo ago

sure:

version: "3.9"
services:

  twenty-front:
    image: twentycrm/twenty-front:${TAG}
    ports:
      - 3001:3000
    environment:
      - SIGN_IN_PREFILLED=${SIGN_IN_PREFILLED}
      - REACT_APP_SERVER_BASE_URL=${LOCAL_SERVER_URL}
      - REACT_APP_SERVER_AUTH_URL=${LOCAL_SERVER_URL}/auth
      - REACT_APP_SERVER_FILES_URL=${LOCAL_SERVER_URL}/files
    depends_on:
      - twenty-back

  twenty-back:
    image: twentycrm/twenty-server:${TAG}
    ports:
      - 3000:3000
    environment:
      - SIGN_IN_PREFILLED=${SIGN_IN_PREFILLED}
      - PG_DATABASE_URL=${PG_DATABASE_URL}
      - FRONT_BASE_URL=${FRONT_BASE_URL}
      - PORT=3000
      - STORAGE_TYPE=local
      - STORAGE_LOCAL_PATH=.local-storage
      - ACCESS_TOKEN_SECRET=${ACCESS_TOKEN_SECRET}
      - LOGIN_TOKEN_SECRET=${LOGIN_TOKEN_SECRET}
      - REFRESH_TOKEN_SECRET=${REFRESH_TOKEN_SECRET}
    depends_on:
      - twenty-db

  twenty-db:
    image: twentycrm/twenty-postgres:${TAG}
    volumes:
      - twenty-db-data:/bitnami/postgresql
    ports:
      - 5432:5432
    environment:
      - POSTGRESQL_USER=${POSTGRES_USER}
      - POSTGRESQL_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRESQL_DB=default

volumes:
  twenty-db-data:

version: "3.9"
services:

  twenty-front:
    image: twentycrm/twenty-front:${TAG}
    ports:
      - 3001:3000
    environment:
      - SIGN_IN_PREFILLED=${SIGN_IN_PREFILLED}
      - REACT_APP_SERVER_BASE_URL=${LOCAL_SERVER_URL}
      - REACT_APP_SERVER_AUTH_URL=${LOCAL_SERVER_URL}/auth
      - REACT_APP_SERVER_FILES_URL=${LOCAL_SERVER_URL}/files
    depends_on:
      - twenty-back

  twenty-back:
    image: twentycrm/twenty-server:${TAG}
    ports:
      - 3000:3000
    environment:
      - SIGN_IN_PREFILLED=${SIGN_IN_PREFILLED}
      - PG_DATABASE_URL=${PG_DATABASE_URL}
      - FRONT_BASE_URL=${FRONT_BASE_URL}
      - PORT=3000
      - STORAGE_TYPE=local
      - STORAGE_LOCAL_PATH=.local-storage
      - ACCESS_TOKEN_SECRET=${ACCESS_TOKEN_SECRET}
      - LOGIN_TOKEN_SECRET=${LOGIN_TOKEN_SECRET}
      - REFRESH_TOKEN_SECRET=${REFRESH_TOKEN_SECRET}
    depends_on:
      - twenty-db

  twenty-db:
    image: twentycrm/twenty-postgres:${TAG}
    volumes:
      - twenty-db-data:/bitnami/postgresql
    ports:
      - 5432:5432
    environment:
      - POSTGRESQL_USER=${POSTGRES_USER}
      - POSTGRESQL_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRESQL_DB=default

volumes:
  twenty-db-data:

charles•7mo ago

ok so you have a persistent volume which is good make sure your database backup contains your data by browsing it quickly and you should be good to go to update your TAG to v0.10

greg [iero]OP•7mo ago

Yes, I looked for Companies, Projects and other objects and they are in in the dumps

charles•7mo ago

ok! let's upgrade and fix the migration issues

greg [iero]OP•7mo ago

I keep the DB container and I upgrade others or do I update them all?

charles•7mo ago

Keep the DB container : - it's fine as we haven't change anything there since 0.3.2. However it's good to see how to upgrade it for the future Upgrade the DB container : - should be fine but I have never tried it with a self host version so I'm not sure

greg [iero]OP•7mo ago

OK I updated my docker-compose.yml with those parameters:

name: twenty

services:
  server:
    image: twentycrm/twenty:${TAG}
    volumes:
      - server-local-data:/app/${STORAGE_LOCAL_PATH:-.local-storage}
    ports:
      - "3000:3000"
    environment:
      PORT: 3000
      PG_DATABASE_URL: postgres://twenty:twenty@${PG_DATABASE_HOST}/default
      SERVER_URL: ${SERVER_URL}
      FRONT_BASE_URL: ${FRONT_BASE_URL:-$SERVER_URL}

      ENABLE_DB_MIGRATIONS: true

      SIGN_IN_PREFILLED: ${SIGN_IN_PREFILLED}
      STORAGE_TYPE: ${STORAGE_TYPE}
        #STORAGE_S3_REGION: ${STORAGE_S3_REGION}
        #STORAGE_S3_NAME: ${STORAGE_S3_NAME}
        #STORAGE_S3_ENDPOINT: ${STORAGE_S3_ENDPOINT}
      ACCESS_TOKEN_SECRET: ${ACCESS_TOKEN_SECRET}
      LOGIN_TOKEN_SECRET: ${LOGIN_TOKEN_SECRET}
      REFRESH_TOKEN_SECRET: ${REFRESH_TOKEN_SECRET}
      FILE_TOKEN_SECRET: ${FILE_TOKEN_SECRET}
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: curl --fail http://localhost:3000/healthz
      interval: 5s
      timeout: 5s
      retries: 10
    restart: always

  db:
    image: twentycrm/twenty-postgres:${TAG}
    volumes:
      - db-data:/bitnami/postgresql
    hostname: localhost
    ports:
      - "5432:5432"
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_ADMIN_PASSWORD}
    healthcheck:
      test: pg_isready -U twenty -d default
      interval: 5s
      timeout: 5s
      retries: 10
    restart: always

volumes:
  db-data:
  server-local-data:

name: twenty

services:
  server:
    image: twentycrm/twenty:${TAG}
    volumes:
      - server-local-data:/app/${STORAGE_LOCAL_PATH:-.local-storage}
    ports:
      - "3000:3000"
    environment:
      PORT: 3000
      PG_DATABASE_URL: postgres://twenty:twenty@${PG_DATABASE_HOST}/default
      SERVER_URL: ${SERVER_URL}
      FRONT_BASE_URL: ${FRONT_BASE_URL:-$SERVER_URL}

      ENABLE_DB_MIGRATIONS: true

      SIGN_IN_PREFILLED: ${SIGN_IN_PREFILLED}
      STORAGE_TYPE: ${STORAGE_TYPE}
        #STORAGE_S3_REGION: ${STORAGE_S3_REGION}
        #STORAGE_S3_NAME: ${STORAGE_S3_NAME}
        #STORAGE_S3_ENDPOINT: ${STORAGE_S3_ENDPOINT}
      ACCESS_TOKEN_SECRET: ${ACCESS_TOKEN_SECRET}
      LOGIN_TOKEN_SECRET: ${LOGIN_TOKEN_SECRET}
      REFRESH_TOKEN_SECRET: ${REFRESH_TOKEN_SECRET}
      FILE_TOKEN_SECRET: ${FILE_TOKEN_SECRET}
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: curl --fail http://localhost:3000/healthz
      interval: 5s
      timeout: 5s
      retries: 10
    restart: always

  db:
    image: twentycrm/twenty-postgres:${TAG}
    volumes:
      - db-data:/bitnami/postgresql
    hostname: localhost
    ports:
      - "5432:5432"
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_ADMIN_PASSWORD}
    healthcheck:
      test: pg_isready -U twenty -d default
      interval: 5s
      timeout: 5s
      retries: 10
    restart: always

volumes:
  db-data:
  server-local-data:

greg [iero]OP•7mo ago

I updated the TAG variable to 0.10.0 in .env file and run

message.txt

greg [iero]OP•7mo ago

seems to work. But when I log to localhost:3000 I have a blank page

greg [iero]OP•7mo ago

OK I found some errors in db container:

message.txt

charles•7mo ago

your db seems fine 🙂 you could use v0.10 as a tag (you'll get the patch upgrades for free, we are already in v0.10.2) If the frontend is blank, the starting point is chrome network tab!

greg [iero]OP•7mo ago

Sounds not bad

charles•7mo ago

what's the return of the graphql call?

greg [iero]OP•7mo ago

I have

  <body>
    <noscript>You need to enable JavaScript to run this app.</noscript>
    <div id="root"></div>
  </body>

  <body>
    <noscript>You need to enable JavaScript to run this app.</noscript>
    <div id="root"></div>
  </body>

in the source

charles•7mo ago

I suspect it's failing 🙂

greg [iero]OP•7mo ago

graphql:

{
    "errors": [
        {
            "message": "Cannot query field \"environment\" on type \"Sentry\".",
            "locations": [
                {
                    "line": 29,
                    "column": 7
                }
            ]
        },
        {
            "message": "Cannot query field \"release\" on type \"Sentry\".",
            "locations": [
                {
                    "line": 30,
                    "column": 7
                }
            ]
        }
    ]
}

{
    "errors": [
        {
            "message": "Cannot query field \"environment\" on type \"Sentry\".",
            "locations": [
                {
                    "line": 29,
                    "column": 7
                }
            ]
        },
        {
            "message": "Cannot query field \"release\" on type \"Sentry\".",
            "locations": [
                {
                    "line": 30,
                    "column": 7
                }
            ]
        }
    ]
}

charles•7mo ago

This is normal as react code is not present in the source code (we don't support SSR) ok, so your FE is correctly setup and your backend is not what do you have in your server logs?

greg [iero]OP•7mo ago

I just have server and db now, just two containers

greg [iero]OP•7mo ago

here is the complete log:

message.txt

charles•7mo ago

mmh, looks good. what do you see on sgl.local/graphql ? Another question, why have you reset your database? is this intentionnal?

greg [iero]OP•7mo ago

I did not reset the DB, it was done during the update.. ok I found one issue, server name was not good has to be localhost here

charles•7mo ago

you are right. It's a good think you have a backup... @Quentin G FYI, I feel the auto setup of the database should be a parameter Does your FE load?

Freebios•7mo ago

The yarn database:migrate:prod on first boot ?

charles•7mo ago

mmmh sorry! You are right, it's not a reset it's a migrate all good!

Freebios•7mo ago

It's relying on a lock on the fs, which was missing i assume. Auto init would be interesting to keep to ease new installs, but I could look for signs of already initialised database

charles•7mo ago

it's all good, I thought it was the reset script but it's only the setup one which is non destructive

Freebios•7mo ago

:aniblobsweat: all good then 😄

greg [iero]OP•7mo ago

I do have to launch "yarn database:migrate:prod" ?

charles•7mo ago

No it has already been done!

greg [iero]OP•7mo ago

OK the front is working now I have to restore the db I think

charles•7mo ago

I expect your workspace schema to be broken

greg [iero]OP•7mo ago

I cannot log with my old account

charles•7mo ago

You don't have your data in your db anymore ? Check your postgres content first? It should still be there But your server should show erroeq

greg [iero]OP•7mo ago

Nope, the DB is empty I will load the backup tomorrow after I just need to launch yarn database:migrate:prod ? I have this error in the server :

2024-04-24 22:28:34 [Nest] 1  - 04/24/2024, 8:28:34 PM   ERROR [UserWorkspaceMiddleware] Error while validating token in middleware.
2024-04-24 22:28:34 [Nest] 1  - 04/24/2024, 8:28:34 PM   ERROR [UserWorkspaceMiddleware] UnauthorizedException: Token invalid.

2024-04-24 22:28:34 [Nest] 1  - 04/24/2024, 8:28:34 PM   ERROR [UserWorkspaceMiddleware] Error while validating token in middleware.
2024-04-24 22:28:34 [Nest] 1  - 04/24/2024, 8:28:34 PM   ERROR [UserWorkspaceMiddleware] UnauthorizedException: Token invalid.

Because the user doesn't exists I imagine?

Freebios•7mo ago

About the missing data, i would say it comes from the volume name mismatch between old and new docker compose It went from

volumes:
  twenty-db-data:

volumes:
  twenty-db-data:

volumes:
  db-data:
  server-local-data:

volumes:
  db-data:
  server-local-data:

Data is probably still one the disk 🙂 If you wish to use the new name, here is a quick way to transfer data i found https://github.com/moby/moby/issues/31154#issuecomment-360531460

greg [iero]OP•7mo ago

OK so I tried to restore database:

docker exec -it twenty-db-1 sh
$ dropdb -U twenty -h localhost -p 5432 default
Password: 
$ createdb  -U twenty -h localhost -p 5432 default
Password: 
$ pg_restore -U twenty -h localhost -p 5432 -C --clean --create --if-exists -d default /db_240424.dump
Password: 
pg_restore: error: could not execute query: ERROR:  cannot drop the currently open database
Command was: DROP DATABASE IF EXISTS "default";
pg_restore: error: could not execute query: ERROR:  database "default" already exists
Command was: CREATE DATABASE "default" WITH TEMPLATE = template0 ENCODING = 'UTF8' LOCALE_PROVIDER = libc LOCALE = 'en_US.UTF-8';


pg_restore: warning: errors ignored on restore: 2
$ exit

docker exec -it twenty-db-1 sh
$ dropdb -U twenty -h localhost -p 5432 default
Password: 
$ createdb  -U twenty -h localhost -p 5432 default
Password: 
$ pg_restore -U twenty -h localhost -p 5432 -C --clean --create --if-exists -d default /db_240424.dump
Password: 
pg_restore: error: could not execute query: ERROR:  cannot drop the currently open database
Command was: DROP DATABASE IF EXISTS "default";
pg_restore: error: could not execute query: ERROR:  database "default" already exists
Command was: CREATE DATABASE "default" WITH TEMPLATE = template0 ENCODING = 'UTF8' LOCALE_PROVIDER = libc LOCALE = 'en_US.UTF-8';


pg_restore: warning: errors ignored on restore: 2
$ exit

greg [iero]OP•7mo ago

I connected to the database and it seems to be there. I started a yarn db migrate but I had some errors

message.txt

greg [iero]OP•7mo ago

When I try to log-in, I have this error

message.txt

greg [iero]OP•7mo ago

@charles Any idea? I can erase and restart if needed

charles•7mo ago

yes we have an issue with migrations since 0.3.2 🙂 You can take a look at the thread here: https://discord.com/channels/1130383047699738754/1232303531256578099/1232596961828405309 migrating to 0.10 is possible but not easy. If you think it's easier to restart from scratch and put back your data it might be a better way. Upcoming migrations will be easier 🙂 If you still want to migrate: regarding your error, there is a missing FK in the database: FK_2ec910029395fa7655621c88908 you need to create it manually before re-reunning migrations again

greg [iero]OP•7mo ago

I have a lot of data to migrate, so I'm not sure I can put back my data manualy.. or I have to code a lot using API I'm a little bit lost. How/Where can I add this FK?

charles•7mo ago

Ok, let's migrate your workspace then! do you have a postgres UI client? like Dbeaver or Postico? We can schedule a 30 min session if needed 🙂

greg [iero]OP•7mo ago

I have "Tableplus" and I just installed Postico I'm available today if you have some time

charles•7mo ago

does 2pm Paris time work for you?

greg [iero]OP•7mo ago

yes good for me!

Gaming

Programming

Update from 0.3.2