Error in topics endpoint
Hello, i have this error in production: PlanExecutor error during aggregation :: caused by :: Total size of documents in topicsubscribers matching pipeline's $lookup stage exceeds 104857600 bytes
I fetch topics through this point: https://docs.novu.co/api-reference/topics/filter-topics
Can you explain what is this ?
Novu
Filter topics - Novu
Returns a list of topics that can be paginated using the
page query parameter and filtered by the topic key with the key query parameter8 Replies
seems like you have lot of topics 👀
no, i have only 1 topic, but i have too many subscribers
I counted,at this time I have more than 2 million subscribers in my topic
Hi @Евгений Толченников , thanks for sharing this with us. Right now we are not limiting the amount of subscribers a topic has so you have hit an infrastructure limitation. When we implemented the Topics API we already had in our minds this situation would happen but we didn't have any metric that could allow us to think how to handle topic subscribers limit.
We appreciate your feedback as it will help us to tackle that issue. So right now we are sorry but you will have to manage your subscribers in batches in different topics to overcome this limitation. 🙏
Thank for your answer @Pablo Fernández
Can you tell me optimal maximum size for 1 topic ?
We are not sure about what would the sweet post but would you mind to help us testing this 2 limits?
- A topic with 100,000 subscribers
- A different topic with 500,000 subscribers
That would help us a lot to decide how to tackle this. Sorry for using you as guinea pig. 🙏
ok, i try to test this cases
@Евгений Толченников, you just advanced to level 4!
i tested this endpoint point on new version 0.21, same error (