discord.js - Imagine ❄•16mo ago

Cache Sweeper

Hello, I wanted to know why, even with these cache sweepers, my ram usage continues to increase as the bot's uptime increases. (5 process with 10 shards of ~1000 guilds and 59 shards in total, 58.7k guilds)

function filter() {
    return false;
}

const client = new Discord.Client({
    shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
    shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
    intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
    partials: [Discord.Partials.Channel, Discord.Partials.Message],
    sweepers: {
        ...Discord.Options.DefaultSweeperSettings,
        messages: { interval: 3600, lifetime: 1800 }, // 30 minutes
        bans: { interval: 3600, filter: () => filter },
        threads: { interval: 3600, lifetime: 1800 }, // 30 minutes
        threadMembers: { interval: 3600, filter: () => filter },
        invites: { interval: 3600, lifetime: 1800 }, // 30 minutes
        presences: { interval: 3600, filter: () => filter },
        reactions: { interval: 3600, filter: () => filter },
        stickers: { interval: 3600, filter: () => filter },
        users: { interval: 3600, filter: () => filter }
    }
});

function filter() {
    return false;
}

const client = new Discord.Client({
    shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
    shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
    intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
    partials: [Discord.Partials.Channel, Discord.Partials.Message],
    sweepers: {
        ...Discord.Options.DefaultSweeperSettings,
        messages: { interval: 3600, lifetime: 1800 }, // 30 minutes
        bans: { interval: 3600, filter: () => filter },
        threads: { interval: 3600, lifetime: 1800 }, // 30 minutes
        threadMembers: { interval: 3600, filter: () => filter },
        invites: { interval: 3600, lifetime: 1800 }, // 30 minutes
        presences: { interval: 3600, filter: () => filter },
        reactions: { interval: 3600, filter: () => filter },
        stickers: { interval: 3600, filter: () => filter },
        users: { interval: 3600, filter: () => filter }
    }
});

34 Replies

d.js toolkit•16mo ago

- What's your exact discord.js npm list discord.js and node node -v version? - Not a discord.js issue? Check out #other-js-ts. - Consider reading #how-to-get-help to improve your question! - Explain what exactly your issue is. - Post the full error stack trace, not just the top part! - Show your code! - Issue solved? Press the button!

SayrixOP•16mo ago

[email protected] v17.9.0 Or he should return true to be removed from the cache ?

Digital•16mo ago

it increases because you dont have lifetime for presences, reactions, and so on.. those pile up quickly if you have their intents why are you returing false on filter? if you dont want to store them, just modify the makeCache object

SayrixOP•16mo ago

I want to store them but only for 1 hour

Digital•16mo ago

you cant do that since there is no that lifetime option

SayrixOP•16mo ago

??? the lifetime option is only for invites messages and threads

Digital•16mo ago

SayrixOP•16mo ago

Are you blind ?

SayrixOP•16mo ago

Digital•16mo ago

you dont get me, your usage goes up because of those specific caches (users, presences especially, ....), and since there is no option to put lifetime on it, you cant prevent ram from increasing if you dont fully disable those caches

SayrixOP•16mo ago

But if i clear it every hour it shouldn't increase no ?

Digital•16mo ago

you can only clear it manually then it shouldnt increase

SayrixOP•16mo ago

Even with the sweeper ?

Digital•16mo ago

i literally just said that you cant with sweeper because half of them dont support lifetime

SayrixOP•16mo ago

But on the doc i can use all of them I will try it this day, I’ll let you know and check the ram usage after some days I tried this:

function users(a) {
    if (a.id === client.user.id) return false;
    console.log("Clearing user", a.id, "from cache");
    return true;
}

// eslint-disable-next-line no-unused-vars
function filter(a, b, c) {
    // console.log(a, 1, b, 2, c, 3, "filter");
    return true;
}

const interval = 10;
const client = new Discord.Client({
    shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
    shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
    intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
    partials: [Discord.Partials.Channel, Discord.Partials.Message],
    sweepers: {
        ...Discord.Options.DefaultSweeperSettings,
        bans: { interval: interval, filter: filter },
        messages: { interval: interval, lifetime: 1800 },
        threads: { interval: interval, lifetime: 1800 }, // 30 minutes
        threadMembers: { interval: interval, filter: filter },
        invites: { interval: interval, lifetime: 1800 }, // 30 minutes
        presences: { interval: interval, filter: filter },
        reactions: { interval: interval, filter: filter },
        stickers: { interval: interval, filter: filter },
        autoModerationRules: { interval: interval, filter: filter },
        users: {
            interval: interval,
            filter: () => users,
        },
        applicationCommands: { interval: interval, filter: filter },
        emojis: { interval: interval, filter: filter },
    },
    makeCache: Discord.Options.cacheWithLimits({
        ApplicationCommandManager: 5, // guild.commands
        BaseGuildEmojiManager: 5, // guild.emojis
        GuildBanManager: 0, // guild.bans
        GuildInviteManager: 0, // guild.invites
        GuildManager: Infinity, // client.guilds
        GuildMemberManager: {
            maxSize: 200,
            keepOverLimit: member => member.id === client.user.id,
        },
        GuildStickerManager: 0, // guild.stickers
        GuildScheduledEventManager: 0, // guild.scheduledEvents
        MessageManager: 5, // channel.messages
        PresenceManager: 0, // guild.presences
        ReactionManager: 0, // message.reactions
        ReactionUserManager: 0, // reaction.users
        StageInstanceManager: 500, // guild.stageInstances
        ThreadManager: 0, // channel.threads
        ThreadMemberManager: 0, // threadchannel.members
    }),
});

function users(a) {
    if (a.id === client.user.id) return false;
    console.log("Clearing user", a.id, "from cache");
    return true;
}

// eslint-disable-next-line no-unused-vars
function filter(a, b, c) {
    // console.log(a, 1, b, 2, c, 3, "filter");
    return true;
}

const interval = 10;
const client = new Discord.Client({
    shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
    shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
    intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
    partials: [Discord.Partials.Channel, Discord.Partials.Message],
    sweepers: {
        ...Discord.Options.DefaultSweeperSettings,
        bans: { interval: interval, filter: filter },
        messages: { interval: interval, lifetime: 1800 },
        threads: { interval: interval, lifetime: 1800 }, // 30 minutes
        threadMembers: { interval: interval, filter: filter },
        invites: { interval: interval, lifetime: 1800 }, // 30 minutes
        presences: { interval: interval, filter: filter },
        reactions: { interval: interval, filter: filter },
        stickers: { interval: interval, filter: filter },
        autoModerationRules: { interval: interval, filter: filter },
        users: {
            interval: interval,
            filter: () => users,
        },
        applicationCommands: { interval: interval, filter: filter },
        emojis: { interval: interval, filter: filter },
    },
    makeCache: Discord.Options.cacheWithLimits({
        ApplicationCommandManager: 5, // guild.commands
        BaseGuildEmojiManager: 5, // guild.emojis
        GuildBanManager: 0, // guild.bans
        GuildInviteManager: 0, // guild.invites
        GuildManager: Infinity, // client.guilds
        GuildMemberManager: {
            maxSize: 200,
            keepOverLimit: member => member.id === client.user.id,
        },
        GuildStickerManager: 0, // guild.stickers
        GuildScheduledEventManager: 0, // guild.scheduledEvents
        MessageManager: 5, // channel.messages
        PresenceManager: 0, // guild.presences
        ReactionManager: 0, // message.reactions
        ReactionUserManager: 0, // reaction.users
        StageInstanceManager: 500, // guild.stageInstances
        ThreadManager: 0, // channel.threads
        ThreadMemberManager: 0, // threadchannel.members
    }),
});

But i have this error:

Uncaught exception: TypeError [SweepFilterReturn]: The return value of the sweepFilter function was not false or a Function

@qjuh

🌺🇫🇷 Shigu :3 🌺•16mo ago

On which line?

SayrixOP•16mo ago

at Timeout._onTimeout (C:\Users\Sayrixx\Desktop\ChilledBot\node_modules\discord.js\src\util\Sweepers.js:461:48) So all sweepers where i use the function filter

Pulse•16mo ago

i think that advice was wrong, filter is supposed to be a function that returns a function (or null)

export interface SweepOptions<K, V> {
  interval: number;
  filter: GlobalSweepFilter<K, V>;
}

export interface SweepOptions<K, V> {
  interval: number;
  filter: GlobalSweepFilter<K, V>;
}

export type GlobalSweepFilter<K, V> = () => ((value: V, key: K, collection: Collection<K, V>) => boolean) | null; from https://old.discordjs.dev/#/docs/discord.js/14.13.0/typedef/SweepOptions

filter: The function used to determine the function passed to the sweep method

as for the original question, you probably wanted to return true in your filter

SayrixOP•16mo ago

ok. I will try to return true

Pulse•16mo ago

from https://discord.js.org/docs/packages/collection/1.5.3/Collection:Class#sweep

Removes items that satisfy the provided filter function.

SayrixOP•16mo ago

Here is my code that i will push into production:

function users(a) {
    if (a.id === client.user.id) return false;
    console.log("Clearing user", a.id, "from cache");
    return true;
}

// eslint-disable-next-line no-unused-vars
function filter(a, b, c) {
    // console.log(a, 1, b, 2, c, 3, "filter");
    return true;
}

const interval = 10;
const client = new Discord.Client({
    shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
    shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
    intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
    partials: [Discord.Partials.Channel, Discord.Partials.Message],
    sweepers: {
        ...Discord.Options.DefaultSweeperSettings,
        bans: { interval: interval, filter: () => filter },
        messages: { interval: interval, lifetime: 1800 },
        threads: { interval: interval, lifetime: 1800 }, // 30 minutes
        threadMembers: { interval: interval, filter: () => filter },
        invites: { interval: interval, lifetime: 1800 }, // 30 minutes
        presences: { interval: interval, filter: () => filter },
        reactions: { interval: interval, filter: () => filter },
        stickers: { interval: interval, filter: () => filter },
        autoModerationRules: { interval: interval, filter: () => filter },
        users: { interval: interval, filter: () => users },
        applicationCommands: { interval: interval, filter: () => filter },
        emojis: { interval: interval, filter: () => filter },
    },
    makeCache: Discord.Options.cacheWithLimits({
        ApplicationCommandManager: 5, // guild.commands
        BaseGuildEmojiManager: 5, // guild.emojis
        GuildBanManager: 0, // guild.bans
        GuildInviteManager: 0, // guild.invites
        GuildManager: Infinity, // client.guilds
        GuildMemberManager: {
            maxSize: 200,
            keepOverLimit: member => member.id === client.user.id,
        },
        GuildStickerManager: 0, // guild.stickers
        GuildScheduledEventManager: 0, // guild.scheduledEvents
        MessageManager: 5, // channel.messages
        PresenceManager: 0, // guild.presences
        ReactionManager: 0, // message.reactions
        ReactionUserManager: 0, // reaction.users
        ThreadManager: 0, // channel.threads
        ThreadMemberManager: 0, // threadchannel.members
    }),
});

function users(a) {
    if (a.id === client.user.id) return false;
    console.log("Clearing user", a.id, "from cache");
    return true;
}

// eslint-disable-next-line no-unused-vars
function filter(a, b, c) {
    // console.log(a, 1, b, 2, c, 3, "filter");
    return true;
}

const interval = 10;
const client = new Discord.Client({
    shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
    shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
    intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
    partials: [Discord.Partials.Channel, Discord.Partials.Message],
    sweepers: {
        ...Discord.Options.DefaultSweeperSettings,
        bans: { interval: interval, filter: () => filter },
        messages: { interval: interval, lifetime: 1800 },
        threads: { interval: interval, lifetime: 1800 }, // 30 minutes
        threadMembers: { interval: interval, filter: () => filter },
        invites: { interval: interval, lifetime: 1800 }, // 30 minutes
        presences: { interval: interval, filter: () => filter },
        reactions: { interval: interval, filter: () => filter },
        stickers: { interval: interval, filter: () => filter },
        autoModerationRules: { interval: interval, filter: () => filter },
        users: { interval: interval, filter: () => users },
        applicationCommands: { interval: interval, filter: () => filter },
        emojis: { interval: interval, filter: () => filter },
    },
    makeCache: Discord.Options.cacheWithLimits({
        ApplicationCommandManager: 5, // guild.commands
        BaseGuildEmojiManager: 5, // guild.emojis
        GuildBanManager: 0, // guild.bans
        GuildInviteManager: 0, // guild.invites
        GuildManager: Infinity, // client.guilds
        GuildMemberManager: {
            maxSize: 200,
            keepOverLimit: member => member.id === client.user.id,
        },
        GuildStickerManager: 0, // guild.stickers
        GuildScheduledEventManager: 0, // guild.scheduledEvents
        MessageManager: 5, // channel.messages
        PresenceManager: 0, // guild.presences
        ReactionManager: 0, // message.reactions
        ReactionUserManager: 0, // reaction.users
        ThreadManager: 0, // channel.threads
        ThreadMemberManager: 0, // threadchannel.members
    }),
});

(i will just change the interval)

LeMorrow•16mo ago

@sayrix how did it go? Did true work better?

SayrixOP•16mo ago

I forgot to disable something so i will re try now, i'll let you know friday

LeMorrow•16mo ago

perfect, thank you I've also had a bunch of issues similar to this and struggled with my sweepers

SayrixOP•16mo ago

function users(a) {
    if (a.id === client.user.id) return false;
    console.log("Clearing user", a.id, "from cache");
    return true;
}

// eslint-disable-next-line no-unused-vars
function filter(a, b, c) {
    // console.log(a, 1, b, 2, c, 3, "filter");
    return true;
}

const interval = 15 * 60; // 15 minutes
const client = new Discord.Client({
    shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
    shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
    intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
    partials: [Discord.Partials.Channel, Discord.Partials.Message],
    sweepers: {
        ...Discord.Options.DefaultSweeperSettings,
        bans: { interval: interval, filter: () => filter },
        messages: { interval: interval, lifetime: 1800 },
        threads: { interval: interval, lifetime: 1800 }, // 30 minutes
        threadMembers: { interval: interval, filter: () => filter },
        invites: { interval: interval, lifetime: 1800 }, // 30 minutes
        presences: { interval: interval, filter: () => filter },
        reactions: { interval: interval, filter: () => filter },
        stickers: { interval: interval, filter: () => filter },
        autoModerationRules: { interval: interval, filter: () => filter },
        users: { interval: interval, filter: () => users },
        applicationCommands: { interval: interval, filter: () => filter },
        emojis: { interval: interval, filter: () => filter },
    },
    makeCache: Discord.Options.cacheWithLimits({
        ApplicationCommandManager: 5, // guild.commands
        BaseGuildEmojiManager: 5, // guild.emojis
        GuildBanManager: 0, // guild.bans
        GuildInviteManager: 0, // guild.invites
        GuildStickerManager: 0, // guild.stickers
        GuildScheduledEventManager: 0, // guild.scheduledEvents
        MessageManager: 5, // channel.messages
        PresenceManager: 0, // guild.presences
        ReactionManager: 0, // message.reactions
        ReactionUserManager: 0, // reaction.users
        ThreadManager: 0, // channel.threads
        ThreadMemberManager: 0, // threadchannel.members
    }),
});

function users(a) {
    if (a.id === client.user.id) return false;
    console.log("Clearing user", a.id, "from cache");
    return true;
}

// eslint-disable-next-line no-unused-vars
function filter(a, b, c) {
    // console.log(a, 1, b, 2, c, 3, "filter");
    return true;
}

const interval = 15 * 60; // 15 minutes
const client = new Discord.Client({
    shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
    shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
    intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
    partials: [Discord.Partials.Channel, Discord.Partials.Message],
    sweepers: {
        ...Discord.Options.DefaultSweeperSettings,
        bans: { interval: interval, filter: () => filter },
        messages: { interval: interval, lifetime: 1800 },
        threads: { interval: interval, lifetime: 1800 }, // 30 minutes
        threadMembers: { interval: interval, filter: () => filter },
        invites: { interval: interval, lifetime: 1800 }, // 30 minutes
        presences: { interval: interval, filter: () => filter },
        reactions: { interval: interval, filter: () => filter },
        stickers: { interval: interval, filter: () => filter },
        autoModerationRules: { interval: interval, filter: () => filter },
        users: { interval: interval, filter: () => users },
        applicationCommands: { interval: interval, filter: () => filter },
        emojis: { interval: interval, filter: () => filter },
    },
    makeCache: Discord.Options.cacheWithLimits({
        ApplicationCommandManager: 5, // guild.commands
        BaseGuildEmojiManager: 5, // guild.emojis
        GuildBanManager: 0, // guild.bans
        GuildInviteManager: 0, // guild.invites
        GuildStickerManager: 0, // guild.stickers
        GuildScheduledEventManager: 0, // guild.scheduledEvents
        MessageManager: 5, // channel.messages
        PresenceManager: 0, // guild.presences
        ReactionManager: 0, // message.reactions
        ReactionUserManager: 0, // reaction.users
        ThreadManager: 0, // channel.threads
        ThreadMemberManager: 0, // threadchannel.members
    }),
});

btw for my sharding manager i use "discord-hybrid-sharding" @lemorrow When i look the graph thats something like ~100MiB every 3 hours

LeMorrow•16mo ago

Right, so no big improvement with that I'll let you know if I can find any improvements with my experiments

mahakadema•16mo ago

Something that would definitely help in diagnosing where this RAM usage originates from (because i see nothing here indicating that caches are what drive up this ram usage) would be to diagnose whether the sweepers run as expected. this event listener:

client.on(Events.CacheSweep, (msg) => console.log("SWEEPER RUN", msg))

client.on(Events.CacheSweep, (msg) => console.log("SWEEPER RUN", msg))

would allow you to diagnose whether the numbers approximately match up. If they do, ur RAM leakage is likely caused by other stuff

SayrixOP•16mo ago

If you also have a big bot like mine idk if you also use discord-hybrid-sharding for your sharding manager because maybe that it's caused by my sharding manager 🤷‍♂️ I will try to add this code in production to see I think i will install a profiler to see what's wrong

BradyDaLlama .*˖•14mo ago

did you ever figure this out? I have a pretty similar issue but it's a smaller bot. I have pretty aggressive makeCache settings (i think) and some sweepers but the ram still seems to go up and then crash/ i reset it before it does

BradyDaLlama .*˖•14mo ago

the top command shows super low memory usage, but the free -g/-m shows a higher amount is being used, closer to what my VPS monitoring reports.

Gaming

Programming

Cache Sweeper