DI
discord.js - Imagine ❄•16mo ago
War

Cache growing too huge too fast for interacting with members in a role

I have a bot that sends money to a specified role at a certain time of the day. I'd like to send money to all the individuals inside a role. Currently I'm using the following fetches per guild: 
guild = await client.guilds.fetch(guildId);
...
if (guild.memberCount !== guild.members.cache.size) {
    await guild.members.fetch();
}
guild = await client.guilds.fetch(guildId);
...
if (guild.memberCount !== guild.members.cache.size) {
    await guild.members.fetch();
}
This unfortunatly loads all the thousands of users into the bot, making it crash. Is there a more efficient way to get the members of a role? I only need the userIDs of people inside a role without caching all the members on a guild. Thank you for the help!
9 Replies
d.js toolkit
d.js toolkit•16mo ago
- What's your exact discord.js npm list discord.js and node node -v version? - Not a discord.js issue? Check out #other-js-ts. - Consider reading #how-to-get-help to improve your question! - Explain what exactly your issue is. - Post the full error stack trace, not just the top part! - Show your code! - Issue solved? Press the button!
Squid
Squid•16mo ago
No, you are currently doing the most efficient route Role#members is just a getter that returns which cached members have a role, so fetching is needed regardless
loads all the thousands of users into the bot, making it crash
That is not normal, though. Make sure you have the GuildMembers intent if you don't already
War
WarOP•16mo ago
I do have the intent, still don't know why the heap size jumps from 100mb to 500mb when the cron for it is executed. It states that its unable to garbage collect. What is a normal instance size for a bot with 800 servers? is 500mb max RAM too little, should I upgrade my cloud server?
Squid
Squid•16mo ago
I'm not going to pretend to be an expert about this, but yesterday someone was being berated for only having 1gb ram for a bot that's in more than just one server 500mb is definitely on the low-low side, even compared to most servers' free tiers, so I would absolutely suggest upgrading ASAP
War
WarOP•16mo ago
Yeah, but this bot doesn't store any messages, I think the AWS instance is 1GB, but the bot maxes out at 500mb. Doesn't fetching the users of a server store them all into the cache? I see a cache = false option but thats only for single member fetches I figured out another solution 
makeCache: Options.cacheWithLimits({
        GuildMemberManager: {
            maxSize: 30,
            keepOverLimit: member => member.id === client.user!.id,
        },
    }),
makeCache: Options.cacheWithLimits({
        GuildMemberManager: {
            maxSize: 30,
            keepOverLimit: member => member.id === client.user!.id,
        },
    }),
That makes it only save 30 to cache, so I have to use stack cache members = guild.members.fetch() to temporarily store 30+ No the cache was growing too big tho I think Yeah, that's true It's supposed to send out the payment in 25 mins, I'll see if it crashes If so, I'll implement your solution on top thank you! Didn't work I didn't try your solution, my solution was wrong, seems like its one server that has a massive amount of members I just upgraded the RAM to 2 GB from 1, maybe it will help But since it's costly I'll implement your solution and try that I don't have messages intent Only guild and guild member intents Gotcha Wonder why discord makes it so hard just to get members that have a particular role like I just want a list of IDs, not their entire life history Can discord.js not optimize this by filtering on the backend? well by backend I mean as the members come in Yeah but thats only for the cached members right? Yeah, makes sense What's the typical RAM requirement for 1k server bot, or does it vary significantly? Makes sense I see, makes sense Welp, time to ban this feature for servers over 1k people or implement your solution that works too actually it won't because I'l still have to fetch from that large server and it's literally crashing the entire bot before it even completes that transaction I guess that would only be a one-time cost Woah Maybe it's not the members then I have the stack trace 
<--- Last few GCs --->

[2500656:0x5118880]   847896 ms: Scavenge (reduce) 474.0 (483.4) -> 473.4 (483.7) MB, 48.0 / 0.0 ms  (average mu = 0.186, current mu = 0.139) allocation failure


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb09980 node::Abort() [node /home/ec2-user/currency-bot/dist/index.js]
 2: 0xa1c235 node::FatalError(char const*, char const*) [node /home/ec2-user/currency-bot/dist/index.js]
 3: 0xcf784e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node /home/ec2-user/currency-bot/dist/index.js]
 4: 0xcf7bc7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node /home/ec2-user/currency-bot/dist/index.js]
 5: 0xeaf465  [node /home/ec2-user/currency-bot/dist/index.js]
 6: 0xeaff46  [node /home/ec2-user/currency-bot/dist/index.js]
 7: 0xebe46e  [node /home/ec2-user/currency-bot/dist/index.js]
 8: 0xebeeb0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node /home/ec2-user/currency-bot/dist/index.js]
 9: 0xec1e2e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node /home/ec2-user/currency-bot/dist/index.js]
10: 0xe830a2 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node /home/ec2-user/currency-bot/dist/index.js]
11: 0xe7b6b4 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node /home/ec2-user/currency-bot/dist/index.js]
12: 0xe7d3c0 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [node 
<--- Last few GCs --->

[2500656:0x5118880]   847896 ms: Scavenge (reduce) 474.0 (483.4) -> 473.4 (483.7) MB, 48.0 / 0.0 ms  (average mu = 0.186, current mu = 0.139) allocation failure


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb09980 node::Abort() [node /home/ec2-user/currency-bot/dist/index.js]
 2: 0xa1c235 node::FatalError(char const*, char const*) [node /home/ec2-user/currency-bot/dist/index.js]
 3: 0xcf784e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node /home/ec2-user/currency-bot/dist/index.js]
 4: 0xcf7bc7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node /home/ec2-user/currency-bot/dist/index.js]
 5: 0xeaf465  [node /home/ec2-user/currency-bot/dist/index.js]
 6: 0xeaff46  [node /home/ec2-user/currency-bot/dist/index.js]
 7: 0xebe46e  [node /home/ec2-user/currency-bot/dist/index.js]
 8: 0xebeeb0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node /home/ec2-user/currency-bot/dist/index.js]
 9: 0xec1e2e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node /home/ec2-user/currency-bot/dist/index.js]
10: 0xe830a2 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node /home/ec2-user/currency-bot/dist/index.js]
11: 0xe7b6b4 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node /home/ec2-user/currency-bot/dist/index.js]
12: 0xe7d3c0 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [node 
/home/ec2-user/currency-bot/dist/index.js]
13: 0x1258d45 v8::internal::IncrementalStringBuilder::Extend() [node /home/ec2-user/currency-bot/dist/index.js]
14: 0xfa7980 v8::internal::JsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>) [node /home/ec2-user/currency-bot/dist/index.js]
15: 0xfa975e v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/ec2-user/currency-bot/dist/index.js]
16: 0xfaa7f5 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/ec2-user/currency-bot/dist/index.js]
17: 0xfabb4c v8::internal::JsonStringifier::SerializeJSReceiverSlow(v8::internal::Handle<v8::internal::JSReceiver>) [node /home/ec2-user/currency-bot/dist/index.js]
18: 0xfacc5d v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/ec2-user/currency-bot/dist/index.js]
19: 0xfaed8f v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [node /home/ec2-user/currency-bot/dist/index.js]
20: 0xd79a77 v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [node /home/ec2-user/currency-bot/dist/index.js]
21: 0x15f0bf9  [node /home/ec2-user/currency-bot/dist/index.js]
/home/ec2-user/currency-bot/dist/index.js]
13: 0x1258d45 v8::internal::IncrementalStringBuilder::Extend() [node /home/ec2-user/currency-bot/dist/index.js]
14: 0xfa7980 v8::internal::JsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>) [node /home/ec2-user/currency-bot/dist/index.js]
15: 0xfa975e v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/ec2-user/currency-bot/dist/index.js]
16: 0xfaa7f5 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/ec2-user/currency-bot/dist/index.js]
17: 0xfabb4c v8::internal::JsonStringifier::SerializeJSReceiverSlow(v8::internal::Handle<v8::internal::JSReceiver>) [node /home/ec2-user/currency-bot/dist/index.js]
18: 0xfacc5d v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/ec2-user/currency-bot/dist/index.js]
19: 0xfaed8f v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [node /home/ec2-user/currency-bot/dist/index.js]
20: 0xd79a77 v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [node /home/ec2-user/currency-bot/dist/index.js]
21: 0x15f0bf9  [node /home/ec2-user/currency-bot/dist/index.js]
I see wonder what could be causing it then unless it's a server with like 100k people Wait, JsonStringifier? huh 8k+ (that I know of) Must be that one, actually Wish I could only temporarily allocate more ram Wait, how does the swap work again? Maybe its the disk size? it's only 8gb hmm okay huh? Wait I'm doing members = guild.members.fetch() So the bot sends money hourly and daily, this only happens when the daily action triggers at 7PM cst when I restart the bot wouldn't that be empty? wait nvm memberCount, yea lemme try this 
client.once('ready', async () => {
    console.log('Bot is up!');
    console.log('here');
    console.log('member number', client.guilds.cache.map(g => g.memberCount).reduce((a, b) => a + b, 0));
client.once('ready', async () => {
    console.log('Bot is up!');
    console.log('here');
    console.log('member number', client.guilds.cache.map(g => g.memberCount).reduce((a, b) => a + b, 0));
running above code 75,157 The stack trace gives no clues lemme try and AI that
War
WarOP•16mo ago
will this be beneficial: https://www.npmjs.com/package/heapdump
npm
heapdump
Make a dump of the V8 heap for later inspection.. Latest version: 0.3.15, last published: 4 years ago. Start using heapdump in your project by running npm i heapdump. There are 180 other projects in the npm registry using heapdump.
War
WarOP•16mo ago
I'm looking I'm using JSON.parse I'll paste it here 
async function handleGuildTasks(taskEntries: TaskEntry[], client: Client): Promise<void> {
    const guildId = taskEntries[0].ID.split('_')[0];

    let guild;
    try {
        guild = await client.guilds.fetch(guildId);
    }
    catch (err) {
        return;
    }
    if (!guild) {
        return;
    }

    let members;
    if (guild.memberCount !== guild.members.cache.size) {
        console.log('Fetching members from API');
        members = await guild.members.fetch();
    }
    else {
        members = guild.members.cache;
    }

    for (const taskEntry of taskEntries) {
        await handleTaskEntry(taskEntry, client, guild, members);
    }
}
async function handleGuildTasks(taskEntries: TaskEntry[], client: Client): Promise<void> {
    const guildId = taskEntries[0].ID.split('_')[0];

    let guild;
    try {
        guild = await client.guilds.fetch(guildId);
    }
    catch (err) {
        return;
    }
    if (!guild) {
        return;
    }

    let members;
    if (guild.memberCount !== guild.members.cache.size) {
        console.log('Fetching members from API');
        members = await guild.members.fetch();
    }
    else {
        members = guild.members.cache;
    }

    for (const taskEntry of taskEntries) {
        await handleTaskEntry(taskEntry, client, guild, members);
    }
}
async function handleTaskEntry(taskEntry: TaskEntry, client: Client, guild: Guild, members: Collection<Snowflake, GuildMember>): Promise<void> {
    const [guildId, ticker, roleId] = taskEntry.ID.split('_');
    const task : TaskData = JSON.parse(taskEntry.data);

    const role = client.guilds.cache.get(guild!.id)?.roles.cache.get(roleId);
    if (!role) {
        return;
    }

    // filter out members that dont have role.id
    members = members.filter(member => member.roles.cache.has(role!.id));

    // console.log(guildId, task['userId'], ticker, task['amount'], role!.members.size);
    const balance = await currHelpers.openCurrAccount(guildId, task['userId'], ticker);

    // Check if ticker exists
    if (balance === false) {
        // Delete the entry if the ticker doesn't exist anymore
        await autoSends.delete(taskEntry.ID);
        return;
    }

    // Check if the user has enough money
    if (balance < task.amount * members.size) {
        // Mark as inactive if insufficient funds
        task.status = 'inactive';
        await autoSends.set(taskEntry.ID, task);
        return;
    }

    task.status = 'active';
    await autoSends.set(taskEntry.ID, task);

    // fetch user
    const theUser = await client.users.fetch(task['userId']);

    // subtract money from user
    await currHelpers.subtractCurrBalance(guildId, theUser, task['amount'] * role!.members.size, '/role auto_send', ticker);
    for (const member of members.values()) {
        currHelpers.addCurrBalance(guildId, member.user, task['amount'], '/role auto_send', ticker);
    }
}
async function handleTaskEntry(taskEntry: TaskEntry, client: Client, guild: Guild, members: Collection<Snowflake, GuildMember>): Promise<void> {
    const [guildId, ticker, roleId] = taskEntry.ID.split('_');
    const task : TaskData = JSON.parse(taskEntry.data);

    const role = client.guilds.cache.get(guild!.id)?.roles.cache.get(roleId);
    if (!role) {
        return;
    }

    // filter out members that dont have role.id
    members = members.filter(member => member.roles.cache.has(role!.id));

    // console.log(guildId, task['userId'], ticker, task['amount'], role!.members.size);
    const balance = await currHelpers.openCurrAccount(guildId, task['userId'], ticker);

    // Check if ticker exists
    if (balance === false) {
        // Delete the entry if the ticker doesn't exist anymore
        await autoSends.delete(taskEntry.ID);
        return;
    }

    // Check if the user has enough money
    if (balance < task.amount * members.size) {
        // Mark as inactive if insufficient funds
        task.status = 'inactive';
        await autoSends.set(taskEntry.ID, task);
        return;
    }

    task.status = 'active';
    await autoSends.set(taskEntry.ID, task);

    // fetch user
    const theUser = await client.users.fetch(task['userId']);

    // subtract money from user
    await currHelpers.subtractCurrBalance(guildId, theUser, task['amount'] * role!.members.size, '/role auto_send', ticker);
    for (const member of members.values()) {
        currHelpers.addCurrBalance(guildId, member.user, task['amount'], '/role auto_send', ticker);
    }
}
I deleted the first function because it's not relevant subtracts and adds money + logs where is what stored? the code? or the database type? I'm using quick.db 
// Opens a currency account for users if they don't have one
// and returns account value
export async function openCurrAccount(interaction: CommandInteraction | string, userID: string, currTicker: string) : Promise<number | false> {
    let guildID: string;
    if (typeof interaction === 'string') {
        guildID = interaction;
    }
    else {
        guildID = await getGuildID(interaction);
    }
    // Check if the currency actually exists
    if (await guildCurr.has(`${guildID}.${currTicker}`) == false) {
        return false;
    }
    const account = await guildCurr.get(`${guildID}.${currTicker}.users.${userID}.balance`);

    if (account == undefined) {
        await guildCurr.set(`${guildID}.${currTicker}.users.${userID}.balance`, 0);
    }
    // Check if user balance < 0
    if (await guildCurr.get(`${guildID}.${currTicker}.users.${userID}.balance`) < 0) {
        await guildCurr.set(`${guildID}.${currTicker}.users.${userID}.balance`, 0);
    }
    return await guildCurr.get(`${guildID}.${currTicker}.users.${userID}.balance`);
}
// Opens a currency account for users if they don't have one
// and returns account value
export async function openCurrAccount(interaction: CommandInteraction | string, userID: string, currTicker: string) : Promise<number | false> {
    let guildID: string;
    if (typeof interaction === 'string') {
        guildID = interaction;
    }
    else {
        guildID = await getGuildID(interaction);
    }
    // Check if the currency actually exists
    if (await guildCurr.has(`${guildID}.${currTicker}`) == false) {
        return false;
    }
    const account = await guildCurr.get(`${guildID}.${currTicker}.users.${userID}.balance`);

    if (account == undefined) {
        await guildCurr.set(`${guildID}.${currTicker}.users.${userID}.balance`, 0);
    }
    // Check if user balance < 0
    if (await guildCurr.get(`${guildID}.${currTicker}.users.${userID}.balance`) < 0) {
        await guildCurr.set(`${guildID}.${currTicker}.users.${userID}.balance`, 0);
    }
    return await guildCurr.get(`${guildID}.${currTicker}.users.${userID}.balance`);
}
Don't think that's the probem tho It only happens with one server The bot jumps from 128mb to 540mb while fetching Thank you for the help anyways, I've doubled the RAM, lets see if it happens again If it doesn't I'm banning the feature for servers over 1000 and downgrading I'll try this I think I know why, they have 70+ roles for many of the users on that server I increased it RAM up to 2GB, it worked the bot jumped from 151mb to 1GB when it was fetching that one server, then immediatly back to 162mb Yes, I do So if I hadn't had that 30 member thing, it would have been in there forever? So this is what users not having a lifetime means? cache lifetimes read something about that in the wiki If the type of cache has a lifetime associated with it, such as invites, messages, or threads, then you can set the lifetime option to sweep items older than specified. Otherwise, you can set the filter option for any type of cache, which will select the items to sweep.
War
WarOP•16mo ago
No description
War
WarOP•16mo ago
Hmm, so the cache just simply grows and doesn't replace players who haven't interacted with the bot in a while with newer players? Gotcha Gotcha, that makes sense. Thank you so much for your help! I really appreciate it! I like your solution for the database storage of all users with a role, but at this stage of the bot I'm unwilling to do that. I'll simply ban big servers from using this feature until increased revenue justifies a special bot for them or an increase in RAM. Hopefully this will help a lot of people who have a similar issue. Wish there was a way to only do segemented fetching with the API, but probably never going to happen Damn wtf, that's an insane idea The problem is removal, if someone is removed from a role what if the bot misses it? is there documentation on this? Gotcha, don't know what those are but will read into them 👍

Did you find this page helpful?