Twenty•5mo ago

v0.22.0 Webhooks failing to send

After upgrading to v0.22.0 (and running db migrations) I'm now receiving the following errors in the container logs from Twenty:

[Nest] 1  - 07/17/2024, 11:25:32 PM   ERROR [SyncDriver] No handler found for job: webhook-queue
[Nest] 1  - 07/17/2024, 11:25:32 PM   ERROR [SyncDriver] No handler found for job: entity-events-to-db-queue
[Nest] 1  - 07/17/2024, 11:25:32 PM   ERROR [SyncDriver] No handler found for job: entity-events-to-db-queue

[Nest] 1  - 07/17/2024, 11:25:32 PM   ERROR [SyncDriver] No handler found for job: webhook-queue
[Nest] 1  - 07/17/2024, 11:25:32 PM   ERROR [SyncDriver] No handler found for job: entity-events-to-db-queue
[Nest] 1  - 07/17/2024, 11:25:32 PM   ERROR [SyncDriver] No handler found for job: entity-events-to-db-queue

I do have multiple Webhooks configured but payload is not being sent and doesn't appear to be attempting to make connection to the configured endpoint.

39 Replies

Skyfox675OP•5mo ago

This is my current env configuration if there's something that I missed when upgrading versions:

          env {
            name  = "PORT"
            value = "3000"
          }
          env {
            name  = "FRONT_BASE_URL"
            value = "https://crm.dominus.haus:443"
          }
          env {
            name  = "BACKEND_SERVER_URL"
            value = "https://crm.dominus.haus:443"
          }
          env {
            name  = "PG_DATABASE_URL"
            value = "postgres://<redacted>/default"
          }
          env {
            name  = "ENABLE_DB_MIGRATIONS"
            value = "true"
          }
          env {
            name  = "SIGN_IN_PREFILLED"
            value = "true"
          }
          env {
            name  = "STORAGE_TYPE"
            value = "local"
          }
          env {
            name  = "ACCESS_TOKEN_EXPIRES_IN"
            value = "7d"
          }
          env {
            name  = "LOGIN_TOKEN_EXPIRES_IN"
            value = "1h"
          }
          env {
            name = "ACCESS_TOKEN_SECRET"
            value_from {
              secret_key_ref {
                name = "tokens"
                key  = "accessToken"
              }
            }
          }
          env {
            name = "LOGIN_TOKEN_SECRET"
            value_from {
              secret_key_ref {
                name = "tokens"
                key  = "loginToken"
              }
            }
          }
          env {
            name = "REFRESH_TOKEN_SECRET"
            value_from {
              secret_key_ref {
                name = "tokens"
                key  = "refreshToken"
              }
            }
          }
          env {
            name = "FILE_TOKEN_SECRET"
            value_from {
              secret_key_ref {
                name = "tokens"
                key  = "fileToken"
              }
            }
          }

          env {
            name  = "PORT"
            value = "3000"
          }
          env {
            name  = "FRONT_BASE_URL"
            value = "https://crm.dominus.haus:443"
          }
          env {
            name  = "BACKEND_SERVER_URL"
            value = "https://crm.dominus.haus:443"
          }
          env {
            name  = "PG_DATABASE_URL"
            value = "postgres://<redacted>/default"
          }
          env {
            name  = "ENABLE_DB_MIGRATIONS"
            value = "true"
          }
          env {
            name  = "SIGN_IN_PREFILLED"
            value = "true"
          }
          env {
            name  = "STORAGE_TYPE"
            value = "local"
          }
          env {
            name  = "ACCESS_TOKEN_EXPIRES_IN"
            value = "7d"
          }
          env {
            name  = "LOGIN_TOKEN_EXPIRES_IN"
            value = "1h"
          }
          env {
            name = "ACCESS_TOKEN_SECRET"
            value_from {
              secret_key_ref {
                name = "tokens"
                key  = "accessToken"
              }
            }
          }
          env {
            name = "LOGIN_TOKEN_SECRET"
            value_from {
              secret_key_ref {
                name = "tokens"
                key  = "loginToken"
              }
            }
          }
          env {
            name = "REFRESH_TOKEN_SECRET"
            value_from {
              secret_key_ref {
                name = "tokens"
                key  = "refreshToken"
              }
            }
          }
          env {
            name = "FILE_TOKEN_SECRET"
            value_from {
              secret_key_ref {
                name = "tokens"
                key  = "fileToken"
              }
            }
          }

charles•5mo ago

Hi @Skyfox675, I'm taking a look 🙂 @martmull could you take a look at this one actually?

martmull•5mo ago

@charles @Skyfox675 checking

martmull•5mo ago

same error on local main for me. Investigating

martmull•5mo ago

@Jérémy here is the issue we got. Think it is introduced here https://github.com/twentyhq/twenty/pull/5657. Not all queues are registered by twenty-server, only messaging-queue, cron-queue and calendar-queue. I try to figure out how to registered them all in syncDriver mode

GitHub

Build software better, together

GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 420 million projects.

charles•5mo ago

@martmull let's not use the syncDriver mode, it's too limited we are likely going to deprecate it fully @Skyfox675 could you also change your MESSAGE_QUEUE_TYPE to pg-boss ?

martmull•5mo ago

ok, then @Skyfox675 you just need to add a MESSAGE_QUEUE_TYPE=pg-boss or MESSAGE_QUEUE_TYPE=bull-mq (bull-mq requires REDIS) env variable pg-boss don't need REDIS, both pg-boss and bull-mq need to run an instance of the worker: yarn nx worker twenty-server

Skyfox675OP•5mo ago

Will do! I'll set this shortly 🙂 Does this require an instance of the worker to be always running, or is it a one-time run to migrate essentially? Since I run twenty-server in K3s, is there an env var to set the worker instance to auto-start? Assuming I'm following correctly.. (Planning on using MESSAGE_QUEUE_TYPE=pg-boss)

martmull•5mo ago

always running

Skyfox675OP•5mo ago

Going to check the docs really quick, I thought I remembered seeing an env for this @charles there's an instance of the worker running by default as part of twenty-server already right? Could have sworn I saw env vars regarding worker counts before but not seeing them in .env.examples Thank you both for jumping on this so quickly!! 🔥

Skyfox675OP•5mo ago

@martmull looks like I'm getting the following error when attempting to run yarn nx worker twenty-server in the twenty-server docker container

Skyfox675OP•5mo ago

I've set MESSAGE_QUEUE_TYPE=pg-boss no furthers 🙌 but webhooks don't seem to sent. I think that was what you/Charles found above though

martmull•5mo ago

It should work but it needs the worker to run in order to launch jobs (think you can find them in pgboss.schedule postgresql table)

Skyfox675OP•5mo ago

Currently not seeing any entries in that taable: I made some updates to a company object that should have triggered update.company

Skyfox675OP•5mo ago

Thinking that these might be picked up faster than I can query but figured I would share

charles•5mo ago

webhooks job are not in the schedule table (this is reserved for crons). You should see them in the jobs table though 🙂

martmull•5mo ago

Indeed 😅 Sorry for the wrong info @Skyfox675

putna•5mo ago

is this the same issue? how to solve it. I want to send member invite emails, and see this error

martmull•5mo ago

you need to setup the pg-boss MESSAGE_QUEUE_TYPE env: MESSAGE_QUEUE_TYPE=pg-boss

putna•5mo ago

thanks, will try

Skyfox675OP•5mo ago

@charles do I need to run yarn nx worker twenty-server on the twenty-server container after it's started?

charles•5mo ago

Hi @Skyfox675, sorry for the misleading information above! nx is a tool for developers contributing to Twenty (it's a monorepo tool providing caching to speed up tooling) We should never use it in produciton / self-host as everything is pre-built for you in the docker image in a prod environment, the command are: yarn start:prod and yarn worker:prod but you don't need to run these are these are the entrypoint of server and worker containers 🙂 you just have to stop and start your container again

Skyfox675OP•5mo ago

That's what I thought! Was going to offer to update the entrypoint if it wasn't already there 🙂

Skyfox675OP•5mo ago

FYI I'm still not seeing webhooks firing after adding MESSAGE_QUEUE_TYPE=pg-boss and a container restart/redeploy Here's the Job table with a webhook-queue events showing state of created I haven't preformed anything this morning in the app so expecting these webhooks to be from yesterday

Skyfox675OP•5mo ago

Here's a better query with created/completed timestamps:

charles•5mo ago

I think this is legit do you see CallWebhookJob logs in your worker container logs whenever you make an update on a record?

Skyfox675OP•5mo ago

Checking!

Skyfox675OP•5mo ago

I don't.. Only seeing TwentyORMCoreModule, and WorkspaceQueryRunnerService logs and made multiple changes on a Company (standard) object and a "Plan" (custom) object:

charles•5mo ago

this does not look like worker logs, it looks like server logs! is your entrypoint right on your worker? see ligne 55 here: https://github.com/twentyhq/twenty/blob/main/packages/twenty-docker/docker-compose.yml#L55

Skyfox675OP•5mo ago

So the container is running using the DOCKERFILE defaults, I don't have a cmd or entrypoint override. I run the two following containers: - twentycrm/twenty:v0.22.0 - twentycrm/twenty-postgres:v0.22.0 Is there another container I need to run a worker? or is a worker embedded in the twenty container?

charles•5mo ago

mmmh! you are missing the worker!

charles•5mo ago

Could you mimick https://github.com/twentyhq/twenty/blob/main/packages/twenty-docker/docker-compose.yml ?

GitHub

twenty/packages/twenty-docker/docker-compose.yml at main · twentyhq...

Building a modern alternative to Salesforce, powered by the community. - twentyhq/twenty

charles•5mo ago

the worker is exactly like the server, but the entrypoint is overriden to be worker:prod server: twentycrm/twenty:v0.22.0 worker: twentycrm/twenty:v0.22.0 db: twentycrm/twenty-postgres:v0.22.0 (btw, you can use v0.22.1!)

Skyfox675OP•5mo ago

Where has my brain been these last few week.. will add and update shortly.. Thank you! Perfect, I had a feeling I was missing something! Will pair with @xSwampxFox to update the Terraform and K8s manifests as well

Skyfox675OP•5mo ago

Well would you look at that, follow directions and things just start working lol

Skyfox675OP•5mo ago

That did the trick, thank you so much! I've still got a lot to learn about how the TwentyCRM app works internally but I feel like I've finally understand the overall service architecture 🙂

charles•5mo ago

Wondeful! Enjoy using it!

xSwampxFox•5mo ago

@charles Just finished a pr for this which adds the worker and updates for v20+. I also did some refactoring to clean up the terraform variables. When you have time do you mind approving and let me know if there are any changes you would like. 😊 https://github.com/twentyhq/twenty/pull/6356

GitHub

Lumosviridi v20 kubernetes updates by LumosViridi · Pull Request #6...

Updates for v20+ and misc terraform bug fixes. Also refactored to use terraform variables instead of locals which helps with readability and ease of use for new users. Terraform validation is curre...

charles•4mo ago

ty! Taking a look! @xSwampxFox Thanks a lot, I have left review. I really love it! Closing this as we have the PR to move forward

Gaming

Programming

v0.22.0 Webhooks failing to send