Webhook Event duplication

Hi team! I'm working on extending TwentyCRM with a Zapier-like automation tool called N8n (N8n.io). I'm seeing some odd behavior with how TwentyCRM is sending Webhooks. I have a custom data model called "Automations" that is effectively an extention of the standard Companies model for things such as client onboarding, system email send status, etc. I'm on v0.23.2, and I have a Webhook filtered to Automations, UPDATE events to be sent. But I'm receiving ~4 duplicate webhook payloads on a single field's value change. I've incorperated some consistancy/idempotency into my worflow to set concurrency=1 and a GET to ensure we're using TwentyCRM's latest value and not the value in the Webhook payload. This has solved the duplication issue on my end, but it is 4x the workflow execution load at the moment to filter out these events. Does anyone know what the cause could be of why so many events are being thrown? Do record view's or GET's via API cause these events to be thrown? Thanks! Additional info: Data model this is being seen on: Custom - Automations Field type: Select - esigContract Allowed values: NOT_SENT SEND QUEUED SENT VIEWED DECLINED SIGNED Example field change: NOT_SENT -> SEND
33 Replies
Skyfox675
Skyfox675OP3mo ago
I guess another question I have is, how many webhooks are sent in parallel? Or does TwentyCRM wait until each webhook receives a response code before moving to the next configured webhook? I've been experimenting with this issue for a while now and I made some improvements to my workflow outside of Twenty so responses to webhooks are sent after the workflow has finished, this slowed down the 4x race conditions quite a bit, but it's still present. I am seeing some off exceptions being thrown on the Twenty Worker right after an event is emitted, 4 consecutive exceptions for message queue(s) and then the webhooks fire.
Exception Captured
undefined
[
ReferenceError: event is not defined
at CallWebhookJobsJob.handle (/app/packages/twenty-server/dist/src/engine/api/graphql/workspace-query-runner/jobs/call-webhook-jobs.job.js:87:65)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:111:17)
at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:105:13)
at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:79:17
at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/drivers/pg-boss.driver.js:34:13
at async resolveWithinSeconds (/app/node_modules/pg-boss/src/manager.js:35:14)
at async /app/node_modules/p-map/index.js:57:22
]
Exception Captured
undefined
[
ReferenceError: event is not defined
at CallWebhookJobsJob.handle (/app/packages/twenty-server/dist/src/engine/api/graphql/workspace-query-runner/jobs/call-webhook-jobs.job.js:87:65)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:111:17)
at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:105:13)
at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:79:17
at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/drivers/pg-boss.driver.js:34:13
at async resolveWithinSeconds (/app/node_modules/pg-boss/src/manager.js:35:14)
at async /app/node_modules/p-map/index.js:57:22
]
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
I'm checking the worklow you've sent but what stood out for me it position as negative float, why as it a float in first place when in logs it's a negative integer?
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
@charles something to check on your side but for Monday or next week in general, why position is presented as a float?
No description
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
@Skyfox675 I imported your workflow and while I get why Trello nodes is not working, I don't understand why basic nodes like if are also not working
No description
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
What version of n8n are you using? I'm self-hosting it in Docker, ver. 1.55.3
Skyfox675
Skyfox675OP3mo ago
Not sure why those If nodes are showing as unknown, I'm on v1.56.1 (pre-release)
No description
Skyfox675
Skyfox675OP3mo ago
Here's the first IF node
No description
Skyfox675
Skyfox675OP3mo ago
And the second IF node
No description
Skyfox675
Skyfox675OP3mo ago
And the bottom Swtich node:
No description
Skyfox675
Skyfox675OP3mo ago
Just a note though I'm still running with MESSAGE_QUEUE_TYPE=pg-boss @charles I wonder if I'm still missing tables or another psql objects coming from v0.12 origionally
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
So that's why, I'm running older version than you
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
@Skyfox675 someone reported the same issue as you reported here https://github.com/twentyhq/twenty/issues/6735
GitHub
Webhooks are firing 4 times for each event · Issue #6735 · twentyhq...
Bug Description I'm using tarampampam/webhook-tester to test the webhook and I'm seeing 4 posts for each event. They all come a few milliseconds apart and the payload is the same on all of ...
Skyfox675
Skyfox675OP3mo ago
@ɃØĦɆᵾS Thanks for finding that! I've added a note about my environment there as well posting here too: Version: v0.23.2 Environment: Self-Hosted MESSAGE_QUEUE_TYPE=pg_boss Seen on: Custom Data Models, haven't ruled out Standard Models though Webhook Filtering: Yes, filtered to Custom Model name and Update Webhook Responses are sending 200 OK's after external processing has finished on Webhook receiver (typically .5-1 seconds) Hey @charles hope your week is starting off well! Curious, should I try switching over to Redis to see if this issue is still present? Saw your note on @xSwampxFox 's PR - https://github.com/twentyhq/twenty/pull/6356/files#r1692558322 we can get that added and PR'ed as part of the Terraform/Helm/Manifests moving forward
charles
charles3mo ago
I think we should force redis by default on Twenty and queue to be pg_boss so users don't have to worry about it I wanted to do it yesterday and today but I haven't found the time, likely tomorrow. We will need to update: - docker-compose for selfhost - render - terraform stuff indeed - deprecate the sync queue driver anyway (does not work and too painful to maintain
Skyfox675
Skyfox675OP3mo ago
Sounds like a plan! Saw your Redis config here - https://discord.com/channels/1130383047699738754/1276587741144154193/1276602115153858662 I'll get to work on switching over to it. Any issues with data loss or weirdness going from pg-boss to bull-mq Maybe I'm overthinking it
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
Maybe way bigger RAM usage when spamming, after all Redis is in-memory database (don't do it)
Skyfox675
Skyfox675OP3mo ago
Great point, thank you for calling that out, what kind of usage are you seeing at the moment? (I know it's probably subjective to Twenty's record count, etc)
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
None as I can't log in 🙂
Skyfox675
Skyfox675OP3mo ago
If you wanna DM I can try to help
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
Aside this, mainly for testing if it works correctly and such Thanks, but I'm writing what's wrong in #💻︱development, I think that some merges causes this issue but I can't pinpoint which one
Skyfox675
Skyfox675OP3mo ago
Up and running with Redis, holy crap that is fast... 😅 might need to double check my event handling to make sure it doesn't cause a race condition again lol Ok so looks like the 4x events are still being emitted, but the exception ordering is different with bull-mq opposed to pg-boss, looks like 1 exception followed by all defined Webhook targets, then the process loops 3 more times:
No description
charles
charles3mo ago
I'll try to reproduce and troubleshoot this today
Skyfox675
Skyfox675OP3mo ago
Really appreciate your time, I'll be by my computer pretty much all day, if I can provide any details/tables let me know
charles
charles3mo ago
@Skyfox675 I'm back from PTO and happy to investigate this together
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
@charles fix for this issue is already merged FYI
charles
charles3mo ago
Oooh ok! Thank you @ɃØĦɆᵾS, I feel you are everywhere haha
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
GitHub
Webhooks are firing 4 times for each event · Issue #6735 · twentyhq...
Bug Description I'm using tarampampam/webhook-tester to test the webhook and I'm seeing 4 posts for each event. They all come a few milliseconds apart and the payload is the same on all of ...
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
About PR, I'm not sure which one is responsible for this
charles
charles3mo ago
what do you mean responsible for this?
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
Responsible in a sense PR which fixed this issue, but no PR was linked
charles
charles3mo ago
I think we have refactored webhooks and it also fixes the issue, the issue was closed by the author
ɃØĦɆᵾS
ɃØĦɆᵾS3mo ago
Ah okay, that explains why there's no linked PR 👍
charles
charles3mo ago
yep!
Want results from more Discord servers?
Add your server