Twenty•4w ago

Webhook Event duplication

Hi team! I'm working on extending TwentyCRM with a Zapier-like automation tool called N8n (N8n.io). I'm seeing some odd behavior with how TwentyCRM is sending Webhooks. I have a custom data model called "Automations" that is effectively an extention of the standard Companies model for things such as client onboarding, system email send status, etc. I'm on v0.23.2, and I have a Webhook filtered to Automations, UPDATE events to be sent. But I'm receiving ~4 duplicate webhook payloads on a single field's value change. I've incorperated some consistancy/idempotency into my worflow to set concurrency=1 and a GET to ensure we're using TwentyCRM's latest value and not the value in the Webhook payload. This has solved the duplication issue on my end, but it is 4x the workflow execution load at the moment to filter out these events. Does anyone know what the cause could be of why so many events are being thrown? Do record view's or GET's via API cause these events to be thrown? Thanks! Additional info: Data model this is being seen on: Custom - Automations Field type: Select - esigContract Allowed values: NOT_SENT SEND QUEUED SENT VIEWED DECLINED SIGNED Example field change: NOT_SENT -> SEND

33 Replies

Skyfox675•4w ago

I guess another question I have is, how many webhooks are sent in parallel? Or does TwentyCRM wait until each webhook receives a response code before moving to the next configured webhook? I've been experimenting with this issue for a while now and I made some improvements to my workflow outside of Twenty so responses to webhooks are sent after the workflow has finished, this slowed down the 4x race conditions quite a bit, but it's still present. I am seeing some off exceptions being thrown on the Twenty Worker right after an event is emitted, 4 consecutive exceptions for message queue(s) and then the webhooks fire.

Exception Captured
  undefined
  [
    ReferenceError: event is not defined
        at CallWebhookJobsJob.handle (/app/packages/twenty-server/dist/src/engine/api/graphql/workspace-query-runner/jobs/call-webhook-jobs.job.js:87:65)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:111:17)
        at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:105:13)
        at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:79:17
        at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/drivers/pg-boss.driver.js:34:13
        at async resolveWithinSeconds (/app/node_modules/pg-boss/src/manager.js:35:14)
        at async /app/node_modules/p-map/index.js:57:22
  ]

Exception Captured
  undefined
  [
    ReferenceError: event is not defined
        at CallWebhookJobsJob.handle (/app/packages/twenty-server/dist/src/engine/api/graphql/workspace-query-runner/jobs/call-webhook-jobs.job.js:87:65)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:111:17)
        at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:105:13)
        at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:79:17
        at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/drivers/pg-boss.driver.js:34:13
        at async resolveWithinSeconds (/app/node_modules/pg-boss/src/manager.js:35:14)
        at async /app/node_modules/p-map/index.js:57:22
  ]

ɃØĦɆᵾS•4w ago

I'm checking the worklow you've sent but what stood out for me it position as negative float, why as it a float in first place when in logs it's a negative integer?

ɃØĦɆᵾS•4w ago

@charles something to check on your side but for Monday or next week in general, why position is presented as a float?

ɃØĦɆᵾS•4w ago

@Skyfox675 I imported your workflow and while I get why Trello nodes is not working, I don't understand why basic nodes like if are also not working

ɃØĦɆᵾS•4w ago

What version of n8n are you using? I'm self-hosting it in Docker, ver. 1.55.3

Skyfox675•4w ago

Not sure why those If nodes are showing as unknown, I'm on v1.56.1 (pre-release)

Skyfox675•4w ago

Here's the first IF node

Skyfox675•4w ago

And the second IF node

Skyfox675•4w ago

And the bottom Swtich node:

Skyfox675•4w ago

Just a note though I'm still running with MESSAGE_QUEUE_TYPE=pg-boss @charles I wonder if I'm still missing tables or another psql objects coming from v0.12 origionally

ɃØĦɆᵾS•4w ago

So that's why, I'm running older version than you

ɃØĦɆᵾS•4w ago

@Skyfox675 someone reported the same issue as you reported here https://github.com/twentyhq/twenty/issues/6735

GitHub

Webhooks are firing 4 times for each event · Issue #6735 · twentyhq...

Bug Description I'm using tarampampam/webhook-tester to test the webhook and I'm seeing 4 posts for each event. They all come a few milliseconds apart and the payload is the same on all of ...

Skyfox675•3w ago

@ɃØĦɆᵾS Thanks for finding that! I've added a note about my environment there as well posting here too: Version: v0.23.2 Environment: Self-Hosted MESSAGE_QUEUE_TYPE=pg_boss Seen on: Custom Data Models, haven't ruled out Standard Models though Webhook Filtering: Yes, filtered to Custom Model name and Update Webhook Responses are sending 200 OK's after external processing has finished on Webhook receiver (typically .5-1 seconds) Hey @charles hope your week is starting off well! Curious, should I try switching over to Redis to see if this issue is still present? Saw your note on @xSwampxFox 's PR - https://github.com/twentyhq/twenty/pull/6356/files#r1692558322 we can get that added and PR'ed as part of the Terraform/Helm/Manifests moving forward

charles•3w ago

I think we should force redis by default on Twenty and queue to be pg_boss so users don't have to worry about it I wanted to do it yesterday and today but I haven't found the time, likely tomorrow. We will need to update: - docker-compose for selfhost - render - terraform stuff indeed - deprecate the sync queue driver anyway (does not work and too painful to maintain

Skyfox675•3w ago

Sounds like a plan! Saw your Redis config here - https://discord.com/channels/1130383047699738754/1276587741144154193/1276602115153858662 I'll get to work on switching over to it. Any issues with data loss or weirdness going from pg-boss to bull-mq Maybe I'm overthinking it

Gaming

Programming

Webhook Event duplication