Twenty•7d ago

Google Calendar sync stuck

At some point our Google Calendar sync was stuck and I can't find the way how to force the re-sync. This is what I see in the database: for message channel I see syncStatus: [null] syncStage: FULL_MESSAGE_LIST_FETCH_PENDING syncedAt: [null] for calendar channel: syncStatus: [null] syncStage: FULL_CALENDAR_EVENT_LIST_FETCH_PENDING syncedAt: [null]

9 Replies

thomast•7d ago

@Raphaël

charles•7d ago

Hi @Sebastian are you self hosting? If yes could you check that your crons are registered? We don't have a nice UI yet for that so you'll have to check a bit manually https://twenty.com/developers/section/self-hosting/setup

yarn command:prod cron:messaging:messages-import
yarn command:prod cron:messaging:message-list-fetch
yarn command:prod cron:calendar:calendar-event-list-fetch
yarn command:prod cron:calendar:calendar-events-import
yarn command:prod cron:messaging:ongoing-stale
yarn command:prod cron:calendar:ongoing-stale

yarn command:prod cron:messaging:messages-import
yarn command:prod cron:messaging:message-list-fetch
yarn command:prod cron:calendar:calendar-event-list-fetch
yarn command:prod cron:calendar:calendar-events-import
yarn command:prod cron:messaging:ongoing-stale
yarn command:prod cron:calendar:ongoing-stale

` (that's to register the messaging / calendar crons) To check if they are running you can check your worker logs (docker logs if you are on docker)

SebastianOP•7d ago

thanks, commands applied. In worker logs I see only these repeating messages and nothing else:

worker-1  | query: SELECT * FROM current_schema()
worker-1  | query: CREATE EXTENSION IF NOT EXISTS "uuid-ossp"
worker-1  | query: SELECT version();
worker-1  | query: SELECT * FROM "information_schema"."tables" WHERE "table_schema" = 'metadata' AND "table_name" = '_typeorm_migrations'
worker-1  | query: SELECT * FROM "metadata"."_typeorm_migrations" "_typeorm_migrations" ORDER BY "id" DESC
worker-1  | No migrations are pending
worker-1  | query: SELECT * FROM current_schema()
db-1      | 2025-02-14 10:36:26,045 INFO: no action. I am (0a878dffcf46), the leader with the lock
worker-1  | query: CREATE EXTENSION IF NOT EXISTS "uuid-ossp"
worker-1  | query: SELECT version();
worker-1  | query: SELECT * FROM "information_schema"."tables" WHERE "table_schema" = 'core' AND "table_name" = '_typeorm_migrations'
worker-1  | query: SELECT * FROM "core"."_typeorm_migrations" "_typeorm_migrations" ORDER BY "id" DESC
worker-1  | No migrations are pending
worker-1  | Successfuly migrated DB!
worker-1  | touch: /app/docker-data/db_status: No such file or directory
worker-1  | Performed 'create extension "vector (pgvector)"' successfully
worker-1  | Performed 'create schema "public"' successfully
worker-1  | Performed 'create schema "metadata"' successfully
worker-1  | Performed 'create schema "core"' successfully
worker-1  | Performed 'create extension "uuid-ossp"' successfully

worker-1  | query: SELECT * FROM current_schema()
worker-1  | query: CREATE EXTENSION IF NOT EXISTS "uuid-ossp"
worker-1  | query: SELECT version();
worker-1  | query: SELECT * FROM "information_schema"."tables" WHERE "table_schema" = 'metadata' AND "table_name" = '_typeorm_migrations'
worker-1  | query: SELECT * FROM "metadata"."_typeorm_migrations" "_typeorm_migrations" ORDER BY "id" DESC
worker-1  | No migrations are pending
worker-1  | query: SELECT * FROM current_schema()
db-1      | 2025-02-14 10:36:26,045 INFO: no action. I am (0a878dffcf46), the leader with the lock
worker-1  | query: CREATE EXTENSION IF NOT EXISTS "uuid-ossp"
worker-1  | query: SELECT version();
worker-1  | query: SELECT * FROM "information_schema"."tables" WHERE "table_schema" = 'core' AND "table_name" = '_typeorm_migrations'
worker-1  | query: SELECT * FROM "core"."_typeorm_migrations" "_typeorm_migrations" ORDER BY "id" DESC
worker-1  | No migrations are pending
worker-1  | Successfuly migrated DB!
worker-1  | touch: /app/docker-data/db_status: No such file or directory
worker-1  | Performed 'create extension "vector (pgvector)"' successfully
worker-1  | Performed 'create schema "public"' successfully
worker-1  | Performed 'create schema "metadata"' successfully
worker-1  | Performed 'create schema "core"' successfully
worker-1  | Performed 'create extension "uuid-ossp"' successfully

my worker config:

  worker:
    image: twentycrm/twenty:${TAG}
    command: ["yarn", "worker:prod"]
    environment:
      #PG_DATABASE_URL: postgres://twenty:twenty@${PG_DATABASE_HOST}/default
      PG_DATABASE_URL: postgres://${PGUSER_SUPERUSER:-postgres}:${PGPASSWORD_SUPERUSER:-postgres}@${PG_DATABASE_HOST:-db:5432}/default
      SERVER_URL: ${SERVER_URL}
      REDIS_URL: ${REDIS_URL:-redis://localhost:6379}

      FRONT_DOMAIN: ${FRONT_DOMAIN}
      FRONT_PROTOCOL: ${FRONT_PROTOCOL}
      FRONT_PORT: ${FRONT_PORT}

      STORAGE_TYPE: ${STORAGE_TYPE}
      STORAGE_S3_REGION: ${STORAGE_S3_REGION}
      STORAGE_S3_NAME: ${STORAGE_S3_NAME}
      STORAGE_S3_ENDPOINT: ${STORAGE_S3_ENDPOINT}

      APP_SECRET: ${APP_SECRET}

      AUTH_PASSWORD_ENABLED: ${AUTH_PASSWORD_ENABLED}
      AUTH_GOOGLE_ENABLED: ${AUTH_GOOGLE_ENABLED}
      AUTH_GOOGLE_CLIENT_ID: ${AUTH_GOOGLE_CLIENT_ID}
      AUTH_GOOGLE_CLIENT_SECRET: ${AUTH_GOOGLE_CLIENT_SECRET}
      AUTH_GOOGLE_CALLBACK_URL: ${AUTH_GOOGLE_CALLBACK_URL}
      AUTH_GOOGLE_APIS_CALLBACK_URL: ${AUTH_GOOGLE_APIS_CALLBACK_URL}

      MESSAGING_PROVIDER_GMAIL_ENABLED: ${MESSAGING_PROVIDER_GMAIL_ENABLED}
      CALENDAR_PROVIDER_GOOGLE_ENABLED: ${CALENDAR_PROVIDER_GOOGLE_ENABLED}

      EMAIL_DRIVER: ${EMAIL_DRIVER}
      EMAIL_FROM_ADDRESS: ${EMAIL_FROM_ADDRESS}
      EMAIL_FROM_NAME: ${EMAIL_FROM_NAME}
      EMAIL_SYSTEM_ADDRESS: ${EMAIL_SYSTEM_ADDRESS}
      EMAIL_SMTP_HOST: ${EMAIL_SMTP_HOST}
      EMAIL_SMTP_PORT: ${EMAIL_SMTP_PORT}
      EMAIL_SMTP_USER: ${EMAIL_SMTP_USER}
      EMAIL_SMTP_PASSWORD: ${EMAIL_SMTP_PASSWORD}

      ACCESS_TOKEN_EXPIRES_IN: ${ACCESS_TOKEN_EXPIRES_IN}
      LOGIN_TOKEN_EXPIRES_IN: ${LOGIN_TOKEN_EXPIRES_IN}

    depends_on:
      db:
        condition: service_healthy
      server:
        condition: service_healthy
    restart: always

  worker:
    image: twentycrm/twenty:${TAG}
    command: ["yarn", "worker:prod"]
    environment:
      #PG_DATABASE_URL: postgres://twenty:twenty@${PG_DATABASE_HOST}/default
      PG_DATABASE_URL: postgres://${PGUSER_SUPERUSER:-postgres}:${PGPASSWORD_SUPERUSER:-postgres}@${PG_DATABASE_HOST:-db:5432}/default
      SERVER_URL: ${SERVER_URL}
      REDIS_URL: ${REDIS_URL:-redis://localhost:6379}

      FRONT_DOMAIN: ${FRONT_DOMAIN}
      FRONT_PROTOCOL: ${FRONT_PROTOCOL}
      FRONT_PORT: ${FRONT_PORT}

      STORAGE_TYPE: ${STORAGE_TYPE}
      STORAGE_S3_REGION: ${STORAGE_S3_REGION}
      STORAGE_S3_NAME: ${STORAGE_S3_NAME}
      STORAGE_S3_ENDPOINT: ${STORAGE_S3_ENDPOINT}

      APP_SECRET: ${APP_SECRET}

      AUTH_PASSWORD_ENABLED: ${AUTH_PASSWORD_ENABLED}
      AUTH_GOOGLE_ENABLED: ${AUTH_GOOGLE_ENABLED}
      AUTH_GOOGLE_CLIENT_ID: ${AUTH_GOOGLE_CLIENT_ID}
      AUTH_GOOGLE_CLIENT_SECRET: ${AUTH_GOOGLE_CLIENT_SECRET}
      AUTH_GOOGLE_CALLBACK_URL: ${AUTH_GOOGLE_CALLBACK_URL}
      AUTH_GOOGLE_APIS_CALLBACK_URL: ${AUTH_GOOGLE_APIS_CALLBACK_URL}

      MESSAGING_PROVIDER_GMAIL_ENABLED: ${MESSAGING_PROVIDER_GMAIL_ENABLED}
      CALENDAR_PROVIDER_GOOGLE_ENABLED: ${CALENDAR_PROVIDER_GOOGLE_ENABLED}

      EMAIL_DRIVER: ${EMAIL_DRIVER}
      EMAIL_FROM_ADDRESS: ${EMAIL_FROM_ADDRESS}
      EMAIL_FROM_NAME: ${EMAIL_FROM_NAME}
      EMAIL_SYSTEM_ADDRESS: ${EMAIL_SYSTEM_ADDRESS}
      EMAIL_SMTP_HOST: ${EMAIL_SMTP_HOST}
      EMAIL_SMTP_PORT: ${EMAIL_SMTP_PORT}
      EMAIL_SMTP_USER: ${EMAIL_SMTP_USER}
      EMAIL_SMTP_PASSWORD: ${EMAIL_SMTP_PASSWORD}

      ACCESS_TOKEN_EXPIRES_IN: ${ACCESS_TOKEN_EXPIRES_IN}
      LOGIN_TOKEN_EXPIRES_IN: ${LOGIN_TOKEN_EXPIRES_IN}

    depends_on:
      db:
        condition: service_healthy
      server:
        condition: service_healthy
    restart: always

charles•7d ago

your worker does not seem to be running any cron can you check your redis content (using redis Insight for example) and show me what you have inside (i'm interested into bull: keys)

SebastianOP•7d ago

that's part of them:

127.0.0.1:6379> keys bull:*
  1) "bull:entity-events-to-db-queue:151"
  2) "bull:webhook-queue:35"
  3) "bull:entity-events-to-db-queue:97"
  4) "bull:entity-events-to-db-queue:115"
  5) "bull:entity-events-to-db-queue:47"
  6) "bull:webhook-queue:19"
  7) "bull:entity-events-to-db-queue:145"
  8) "bull:webhook-queue:47"
  9) "bull:entity-events-to-db-queue:40"
 10) "bull:entity-events-to-db-queue:meta"
 11) "bull:entity-events-to-db-queue:163"
 12) "bull:entity-events-to-db-queue:100"
 13) "bull:entity-events-to-db-queue:157"
 14) "bull:calendar-queue:1"
 15) "bull:entity-events-to-db-queue:139"
 16) "bull:entity-events-to-db-queue:50"
 17) "bull:entity-events-to-db-queue:85"
 18) "bull:cron-queue:events"
 19) "bull:entity-events-to-db-queue:36"
 20) "bull:cron-queue:repeat:dddf90bd802998940aab1261051870c9:1739529300000"
 21) "bull:entity-events-to-db-queue:144"
 22) "bull:cron-queue:repeat:14166af08923f86d1550d0acfcc34009:1739530800000"
 23) "bull:entity-events-to-db-queue:22"
 24) "bull:entity-events-to-db-queue:55"
 25) "bull:entity-events-to-db-queue:28"
 26) "bull:webhook-queue:41"
 27) "bull:entity-events-to-db-queue:34"
 28) "bull:webhook-queue:24"
 29) "bull:cron-queue:id"
 30) "bull:entity-events-to-db-queue:104"
 31) "bull:entity-events-to-db-queue:142"
 32) "bull:webhook-queue:22"
 33) "bull:entity-events-to-db-queue:159"
 34) "bull:webhook-queue:48"
 35) "bull:entity-events-to-db-queue:166"
 36) "bull:email-queue:meta"
 37) "bull:entity-events-to-db-queue:60"
 38) "bull:entity-events-to-db-queue:128"
 39) "bull:entity-events-to-db-queue:172"
 40) "bull:webhook-queue:18"
 41) "bull:entity-events-to-db-queue:135"
 42) "bull:calendar-queue:7"
 43) "bull:entity-events-to-db-queue:180"
 44) "bull:webhook-queue:71"
 45) "bull:cron-queue:meta"
 46) "bull:entity-events-to-db-queue:165"
 47) "bull:entity-events-to-db-queue:122"
 48) "bull:entity-events-to-db-queue:58"
 49) "bull:entity-events-to-db-queue:102"

127.0.0.1:6379> keys bull:*
  1) "bull:entity-events-to-db-queue:151"
  2) "bull:webhook-queue:35"
  3) "bull:entity-events-to-db-queue:97"
  4) "bull:entity-events-to-db-queue:115"
  5) "bull:entity-events-to-db-queue:47"
  6) "bull:webhook-queue:19"
  7) "bull:entity-events-to-db-queue:145"
  8) "bull:webhook-queue:47"
  9) "bull:entity-events-to-db-queue:40"
 10) "bull:entity-events-to-db-queue:meta"
 11) "bull:entity-events-to-db-queue:163"
 12) "bull:entity-events-to-db-queue:100"
 13) "bull:entity-events-to-db-queue:157"
 14) "bull:calendar-queue:1"
 15) "bull:entity-events-to-db-queue:139"
 16) "bull:entity-events-to-db-queue:50"
 17) "bull:entity-events-to-db-queue:85"
 18) "bull:cron-queue:events"
 19) "bull:entity-events-to-db-queue:36"
 20) "bull:cron-queue:repeat:dddf90bd802998940aab1261051870c9:1739529300000"
 21) "bull:entity-events-to-db-queue:144"
 22) "bull:cron-queue:repeat:14166af08923f86d1550d0acfcc34009:1739530800000"
 23) "bull:entity-events-to-db-queue:22"
 24) "bull:entity-events-to-db-queue:55"
 25) "bull:entity-events-to-db-queue:28"
 26) "bull:webhook-queue:41"
 27) "bull:entity-events-to-db-queue:34"
 28) "bull:webhook-queue:24"
 29) "bull:cron-queue:id"
 30) "bull:entity-events-to-db-queue:104"
 31) "bull:entity-events-to-db-queue:142"
 32) "bull:webhook-queue:22"
 33) "bull:entity-events-to-db-queue:159"
 34) "bull:webhook-queue:48"
 35) "bull:entity-events-to-db-queue:166"
 36) "bull:email-queue:meta"
 37) "bull:entity-events-to-db-queue:60"
 38) "bull:entity-events-to-db-queue:128"
 39) "bull:entity-events-to-db-queue:172"
 40) "bull:webhook-queue:18"
 41) "bull:entity-events-to-db-queue:135"
 42) "bull:calendar-queue:7"
 43) "bull:entity-events-to-db-queue:180"
 44) "bull:webhook-queue:71"
 45) "bull:cron-queue:meta"
 46) "bull:entity-events-to-db-queue:165"
 47) "bull:entity-events-to-db-queue:122"
 48) "bull:entity-events-to-db-queue:58"
 49) "bull:entity-events-to-db-queue:102"

do you need all of them?

charles•7d ago

your crons are not registered

yarn command:prod cron:messaging:messages-import
yarn command:prod cron:messaging:message-list-fetch
yarn command:prod cron:calendar:calendar-event-list-fetch
yarn command:prod cron:calendar:calendar-events-import
yarn command:prod cron:messaging:ongoing-stale
yarn command:prod cron:calendar:ongoing-stale

yarn command:prod cron:messaging:messages-import
yarn command:prod cron:messaging:message-list-fetch
yarn command:prod cron:calendar:calendar-event-list-fetch
yarn command:prod cron:calendar:calendar-events-import
yarn command:prod cron:messaging:ongoing-stale
yarn command:prod cron:calendar:ongoing-stale

SebastianOP•3d ago

yes, I did run all these commands. I just didn't paste you my whole redis output due to Discord limitations. Do you want me to paste all I have in Redis? @charles also - should I run these commands on server or worker? @charles @charles ?

dialnco•2d ago

worker! as per docs https://twenty.com/developers/section/self-hosting/setup#start-the-cron-jobs. You can also add env var

DISABLE_DB_MIGRATIONS=true

DISABLE_DB_MIGRATIONS=true

on your worker container, as migrations runs on the server container

SebastianOP•2d ago

oh, thanks! it worked!

Gaming

Programming

Google Calendar sync stuck

Did you find this page helpful?