When Bulk Triggering events with a new subscriber, the subscriber is being created multiple times
Hi there! So our current flow has us generating a series of reminders to send at scheduled times. As a result, we might end up 3 events scheduled at 3, 5, 10 days out. We trigger these events in a single call to the bulk trigger endpoint. It seems that when we do this with a new subscriber in the payload, the subscriber is getting created multiple times.
Checking this against a locally run version of Novu, I was able to get the timestamps in MongoDb of some example subscribers getting created as a part of a bulk request:
createdAt
2024-02-14T22:25:52.290+00:00
createdAt
2024-02-14T22:25:52.298+00:00
Have you all seen anything like this? Is this something we are doing wrong on our end? Thank you in advance!
7 Replies
Hey @Shaun 👋 thanks for reporting that, @Pawan Jain, @Moderator have you seen that?? Cc @Dima Grossman
I would suspect concurrency issues.
In MongoDB schema for subscriber I don't see any unique indexes for subsriberId.
Per the Novu docs the subscriberId should be a unique identifier. I think making use of unique indexes for subscriberId would make sense here 👍
One small concern is it does seem like unique indexes aren't supported if the dataset already violates the uniqueness: https://www.mongodb.com/docs/manual/core/index-unique/#restrictions
@Pawan Jain @Engineering , would love to get your thoughts here
@Shaun, it does seem to be related to a race condition our team discovered + discussed just a few hours ago! It seems to be related to multiple concurrent calls with subscriber creation inline in the trigger call- in creating a subscriber during the trigger event. Which we support, but thats where the race condition originate. We do have an upsert behavior, and plan to keep that. we already have some ideas to fix it
Will keep you updated on that
Thank you for all your feedback! We really do appreciate it
Awesome, thank you so much for looking into it! Appreciate it 🙏