Twenty•5mo ago

Webhook Event duplication

Hi team! I'm working on extending TwentyCRM with a Zapier-like automation tool called N8n (N8n.io). I'm seeing some odd behavior with how TwentyCRM is sending Webhooks. I have a custom data model called "Automations" that is effectively an extention of the standard Companies model for things such as client onboarding, system email send status, etc. I'm on v0.23.2, and I have a Webhook filtered to Automations, UPDATE events to be sent. But I'm receiving ~4 duplicate webhook payloads on a single field's value change. I've incorperated some consistancy/idempotency into my worflow to set concurrency=1 and a GET to ensure we're using TwentyCRM's latest value and not the value in the Webhook payload. This has solved the duplication issue on my end, but it is 4x the workflow execution load at the moment to filter out these events. Does anyone know what the cause could be of why so many events are being thrown? Do record view's or GET's via API cause these events to be thrown? Thanks! Additional info: Data model this is being seen on: Custom - Automations Field type: Select - esigContract Allowed values: NOT_SENT SEND QUEUED SENT VIEWED DECLINED SIGNED Example field change: NOT_SENT -> SEND

33 Replies

Skyfox675OP•4mo ago

I guess another question I have is, how many webhooks are sent in parallel? Or does TwentyCRM wait until each webhook receives a response code before moving to the next configured webhook? I've been experimenting with this issue for a while now and I made some improvements to my workflow outside of Twenty so responses to webhooks are sent after the workflow has finished, this slowed down the 4x race conditions quite a bit, but it's still present. I am seeing some off exceptions being thrown on the Twenty Worker right after an event is emitted, 4 consecutive exceptions for message queue(s) and then the webhooks fire.

Exception Captured
  undefined
  [
    ReferenceError: event is not defined
        at CallWebhookJobsJob.handle (/app/packages/twenty-server/dist/src/engine/api/graphql/workspace-query-runner/jobs/call-webhook-jobs.job.js:87:65)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:111:17)
        at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:105:13)
        at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:79:17
        at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/drivers/pg-boss.driver.js:34:13
        at async resolveWithinSeconds (/app/node_modules/pg-boss/src/manager.js:35:14)
        at async /app/node_modules/p-map/index.js:57:22
  ]

Exception Captured
  undefined
  [
    ReferenceError: event is not defined
        at CallWebhookJobsJob.handle (/app/packages/twenty-server/dist/src/engine/api/graphql/workspace-query-runner/jobs/call-webhook-jobs.job.js:87:65)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:111:17)
        at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:105:13)
        at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:79:17
        at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/drivers/pg-boss.driver.js:34:13
        at async resolveWithinSeconds (/app/node_modules/pg-boss/src/manager.js:35:14)
        at async /app/node_modules/p-map/index.js:57:22
  ]

ɃØĦɆᵾS•4mo ago

I'm checking the worklow you've sent but what stood out for me it position as negative float, why as it a float in first place when in logs it's a negative integer?

ɃØĦɆᵾS•4mo ago

@charles something to check on your side but for Monday or next week in general, why position is presented as a float?

ɃØĦɆᵾS•4mo ago

@Skyfox675 I imported your workflow and while I get why Trello nodes is not working, I don't understand why basic nodes like if are also not working

ɃØĦɆᵾS•4mo ago

What version of n8n are you using? I'm self-hosting it in Docker, ver. 1.55.3

Skyfox675OP•4mo ago

Not sure why those If nodes are showing as unknown, I'm on v1.56.1 (pre-release)

Skyfox675OP•4mo ago

Here's the first IF node

Skyfox675OP•4mo ago

And the second IF node

Skyfox675OP•4mo ago

And the bottom Swtich node:

Skyfox675OP•4mo ago

Just a note though I'm still running with MESSAGE_QUEUE_TYPE=pg-boss @charles I wonder if I'm still missing tables or another psql objects coming from v0.12 origionally

ɃØĦɆᵾS•4mo ago

So that's why, I'm running older version than you

ɃØĦɆᵾS•4mo ago

@Skyfox675 someone reported the same issue as you reported here https://github.com/twentyhq/twenty/issues/6735

GitHub

Webhooks are firing 4 times for each event · Issue #6735 · twentyhq...

Bug Description I'm using tarampampam/webhook-tester to test the webhook and I'm seeing 4 posts for each event. They all come a few milliseconds apart and the payload is the same on all of ...

Skyfox675OP•4mo ago

@ɃØĦɆᵾS Thanks for finding that! I've added a note about my environment there as well posting here too: Version: v0.23.2 Environment: Self-Hosted MESSAGE_QUEUE_TYPE=pg_boss Seen on: Custom Data Models, haven't ruled out Standard Models though Webhook Filtering: Yes, filtered to Custom Model name and Update Webhook Responses are sending 200 OK's after external processing has finished on Webhook receiver (typically .5-1 seconds) Hey @charles hope your week is starting off well! Curious, should I try switching over to Redis to see if this issue is still present? Saw your note on @xSwampxFox 's PR - https://github.com/twentyhq/twenty/pull/6356/files#r1692558322 we can get that added and PR'ed as part of the Terraform/Helm/Manifests moving forward

charles•4mo ago

I think we should force redis by default on Twenty and queue to be pg_boss so users don't have to worry about it I wanted to do it yesterday and today but I haven't found the time, likely tomorrow. We will need to update: - docker-compose for selfhost - render - terraform stuff indeed - deprecate the sync queue driver anyway (does not work and too painful to maintain

Skyfox675OP•4mo ago

Sounds like a plan! Saw your Redis config here - https://discord.com/channels/1130383047699738754/1276587741144154193/1276602115153858662 I'll get to work on switching over to it. Any issues with data loss or weirdness going from pg-boss to bull-mq Maybe I'm overthinking it

ɃØĦɆᵾS•4mo ago

Maybe way bigger RAM usage when spamming, after all Redis is in-memory database (don't do it)

Skyfox675OP•4mo ago

Great point, thank you for calling that out, what kind of usage are you seeing at the moment? (I know it's probably subjective to Twenty's record count, etc)

ɃØĦɆᵾS•4mo ago

None as I can't log in 🙂

Skyfox675OP•4mo ago

If you wanna DM I can try to help

ɃØĦɆᵾS•4mo ago

Aside this, mainly for testing if it works correctly and such Thanks, but I'm writing what's wrong in #💻︱development, I think that some merges causes this issue but I can't pinpoint which one

Skyfox675OP•4mo ago

Up and running with Redis, holy crap that is fast... 😅 might need to double check my event handling to make sure it doesn't cause a race condition again lol Ok so looks like the 4x events are still being emitted, but the exception ordering is different with bull-mq opposed to pg-boss, looks like 1 exception followed by all defined Webhook targets, then the process loops 3 more times:

charles•4mo ago

I'll try to reproduce and troubleshoot this today

Skyfox675OP•4mo ago

Really appreciate your time, I'll be by my computer pretty much all day, if I can provide any details/tables let me know

charles•4mo ago

@Skyfox675 I'm back from PTO and happy to investigate this together

ɃØĦɆᵾS•4mo ago

@charles fix for this issue is already merged FYI

charles•4mo ago

Oooh ok! Thank you @ɃØĦɆᵾS, I feel you are everywhere haha

ɃØĦɆᵾS•4mo ago

https://github.com/twentyhq/twenty/issues/6735

GitHub

Webhooks are firing 4 times for each event · Issue #6735 · twentyhq...

Bug Description I'm using tarampampam/webhook-tester to test the webhook and I'm seeing 4 posts for each event. They all come a few milliseconds apart and the payload is the same on all of ...

ɃØĦɆᵾS•4mo ago

About PR, I'm not sure which one is responsible for this

charles•4mo ago

what do you mean responsible for this?

ɃØĦɆᵾS•4mo ago

Responsible in a sense PR which fixed this issue, but no PR was linked

charles•4mo ago

I think we have refactored webhooks and it also fixes the issue, the issue was closed by the author

ɃØĦɆᵾS•4mo ago

Ah okay, that explains why there's no linked PR 👍

charles•4mo ago

yep!

Gaming

Programming

Webhook Event duplication