Webhook Event duplication
Hi team!
I'm working on extending TwentyCRM with a Zapier-like automation tool called N8n (N8n.io). I'm seeing some odd behavior with how TwentyCRM is sending Webhooks.
I have a custom data model called "Automations" that is effectively an extention of the standard Companies model for things such as client onboarding, system email send status, etc.
I'm on v0.23.2, and I have a Webhook filtered to Automations, UPDATE events to be sent. But I'm receiving ~4 duplicate webhook payloads on a single field's value change.
I've incorperated some consistancy/idempotency into my worflow to set concurrency=1 and a GET to ensure we're using TwentyCRM's latest value and not the value in the Webhook payload. This has solved the duplication issue on my end, but it is 4x the workflow execution load at the moment to filter out these events.
Does anyone know what the cause could be of why so many events are being thrown?
Do record view's or GET's via API cause these events to be thrown?
Thanks!
Additional info:
Data model this is being seen on: Custom -
Automations
Field type: Select - esigContract
Allowed values: NOT_SENT
SEND
QUEUED
SENT
VIEWED
DECLINED
SIGNED
Example field change: NOT_SENT
-> SEND
33 Replies
I guess another question I have is, how many webhooks are sent in parallel? Or does TwentyCRM wait until each webhook receives a response code before moving to the next configured webhook?
I've been experimenting with this issue for a while now and I made some improvements to my workflow outside of Twenty so responses to webhooks are sent after the workflow has finished, this slowed down the 4x race conditions quite a bit, but it's still present.
I am seeing some off exceptions being thrown on the Twenty Worker right after an event is emitted, 4 consecutive exceptions for message queue(s) and then the webhooks fire.
I'm checking the worklow you've sent but what stood out for me it position as negative float, why as it a float in first place when in logs it's a negative integer?
@charles something to check on your side but for Monday or next week in general, why position is presented as a float?
@Skyfox675 I imported your workflow and while I get why Trello nodes is not working, I don't understand why basic nodes like if are also not working
What version of n8n are you using? I'm self-hosting it in Docker, ver. 1.55.3
Not sure why those If nodes are showing as unknown, I'm on v1.56.1 (pre-release)
Here's the first IF node
And the second IF node
And the bottom Swtich node:
Just a note though I'm still running with
MESSAGE_QUEUE_TYPE=pg-boss
@charles I wonder if I'm still missing tables or another psql objects coming from v0.12
origionallySo that's why, I'm running older version than you
@Skyfox675 someone reported the same issue as you reported here https://github.com/twentyhq/twenty/issues/6735
GitHub
Webhooks are firing 4 times for each event · Issue #6735 · twentyhq...
Bug Description I'm using tarampampam/webhook-tester to test the webhook and I'm seeing 4 posts for each event. They all come a few milliseconds apart and the payload is the same on all of ...
@ɃØĦɆᵾS Thanks for finding that! I've added a note about my environment there as well posting here too:
Version:
v0.23.2
Environment: Self-Hosted
MESSAGE_QUEUE_TYPE=pg_boss
Seen on: Custom Data Models, haven't ruled out Standard Models though
Webhook Filtering: Yes, filtered to Custom Model name and Update
Webhook Responses are sending 200 OK's after external processing has finished on Webhook receiver (typically .5-1 seconds)
Hey @charles hope your week is starting off well!
Curious, should I try switching over to Redis to see if this issue is still present?
Saw your note on @xSwampxFox 's PR - https://github.com/twentyhq/twenty/pull/6356/files#r1692558322 we can get that added and PR'ed as part of the Terraform/Helm/Manifests moving forwardI think we should force redis by default on Twenty and queue to be pg_boss so users don't have to worry about it
I wanted to do it yesterday and today but I haven't found the time, likely tomorrow. We will need to update:
- docker-compose for selfhost
- render
- terraform stuff indeed
- deprecate the sync queue driver anyway (does not work and too painful to maintain
Sounds like a plan!
Saw your Redis config here - https://discord.com/channels/1130383047699738754/1276587741144154193/1276602115153858662
I'll get to work on switching over to it. Any issues with data loss or weirdness going from
pg-boss
to bull-mq
Maybe I'm overthinking itMaybe way bigger RAM usage when spamming, after all Redis is in-memory database (don't do it)
Great point, thank you for calling that out, what kind of usage are you seeing at the moment? (I know it's probably subjective to Twenty's record count, etc)
None as I can't log in 🙂
If you wanna DM I can try to help
Aside this, mainly for testing if it works correctly and such
Thanks, but I'm writing what's wrong in #💻︱development, I think that some merges causes this issue but I can't pinpoint which one
Up and running with Redis, holy crap that is fast... 😅 might need to double check my event handling to make sure it doesn't cause a race condition again lol
Ok so looks like the 4x events are still being emitted, but the exception ordering is different with
bull-mq
opposed to pg-boss
, looks like 1 exception followed by all defined Webhook targets, then the process loops 3 more times:I'll try to reproduce and troubleshoot this today
Really appreciate your time, I'll be by my computer pretty much all day, if I can provide any details/tables let me know
@Skyfox675 I'm back from PTO and happy to investigate this together
@charles fix for this issue is already merged FYI
Oooh ok! Thank you @ɃØĦɆᵾS, I feel you are everywhere haha
GitHub
Webhooks are firing 4 times for each event · Issue #6735 · twentyhq...
Bug Description I'm using tarampampam/webhook-tester to test the webhook and I'm seeing 4 posts for each event. They all come a few milliseconds apart and the payload is the same on all of ...
About PR, I'm not sure which one is responsible for this
what do you mean responsible for this?
Responsible in a sense PR which fixed this issue, but no PR was linked
I think we have refactored webhooks and it also fixes the issue, the issue was closed by the author
Ah okay, that explains why there's no linked PR 👍
yep!