Novu•3y ago

Error in topics endpoint

Hello, i have this error in production: PlanExecutor error during aggregation :: caused by :: Total size of documents in topicsubscribers matching pipeline's $lookup stage exceeds 104857600 bytes

I fetch topics through this point: https://docs.novu.co/api-reference/topics/filter-topics

Can you explain what is this ?

Novu

Filter topics - Novu

Returns a list of topics that can be paginated using the

page

page

query parameter and filtered by the topic key with the

key

key

query parameter

Pawan Jain•11/7/23, 1:59 PM

seems like you have lot of topics

PidginEnemyOP•11/7/23, 2:20 PM

no, i have only 1 topic, but i have too many subscribers

PidginEnemyOP•11/7/23, 8:14 PM

I counted,at this time I have more than 2 million subscribers in my topic

--•11/8/23, 3:07 PM

Hi @Евгений Толченников , thanks for sharing this with us. Right now we are not limiting the amount of subscribers a topic has so you have hit an infrastructure limitation. When we implemented the Topics API we already had in our minds this situation would happen but we didn't have any metric that could allow us to think how to handle topic subscribers limit.
We appreciate your feedback as it will help us to tackle that issue. So right now we are sorry but you will have to manage your subscribers in batches in different topics to overcome this limitation.

PidginEnemyOP•11/8/23, 3:26 PM

Thank for your answer @Pablo Fernández
Can you tell me optimal maximum size for 1 topic ?

PPidginEnemy Thank for your answer @Pablo Fernández Can you tell me optimal maximum size for...

--•11/8/23, 3:29 PM

We are not sure about what would the sweet post but would you mind to help us testing this 2 limits?

A topic with 100,000 subscribers
A different topic with 500,000 subscribers
That would help us a lot to decide how to tackle this. Sorry for using you as guinea pig.

PidginEnemyOP•11/8/23, 3:32 PM

ok, i try to test this cases

Novu_Bot•11/8/23, 3:32 PM

@Евгений Толченников, you just advanced to level 4!

PidginEnemyOP•11/10/23, 4:53 PM

i tested this endpoint point on new version 0.21, same error (