Cache Sweeper

Hello, I wanted to know why, even with these cache sweepers, my ram usage continues to increase as the bot's uptime increases. (5 process with 10 shards of ~1000 guilds and 59 shards in total, 58.7k guilds)
function filter() {
return false;
}

const client = new Discord.Client({
shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
partials: [Discord.Partials.Channel, Discord.Partials.Message],
sweepers: {
...Discord.Options.DefaultSweeperSettings,
messages: { interval: 3600, lifetime: 1800 }, // 30 minutes
bans: { interval: 3600, filter: () => filter },
threads: { interval: 3600, lifetime: 1800 }, // 30 minutes
threadMembers: { interval: 3600, filter: () => filter },
invites: { interval: 3600, lifetime: 1800 }, // 30 minutes
presences: { interval: 3600, filter: () => filter },
reactions: { interval: 3600, filter: () => filter },
stickers: { interval: 3600, filter: () => filter },
users: { interval: 3600, filter: () => filter }
}
});
function filter() {
return false;
}

const client = new Discord.Client({
shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
partials: [Discord.Partials.Channel, Discord.Partials.Message],
sweepers: {
...Discord.Options.DefaultSweeperSettings,
messages: { interval: 3600, lifetime: 1800 }, // 30 minutes
bans: { interval: 3600, filter: () => filter },
threads: { interval: 3600, lifetime: 1800 }, // 30 minutes
threadMembers: { interval: 3600, filter: () => filter },
invites: { interval: 3600, lifetime: 1800 }, // 30 minutes
presences: { interval: 3600, filter: () => filter },
reactions: { interval: 3600, filter: () => filter },
stickers: { interval: 3600, filter: () => filter },
users: { interval: 3600, filter: () => filter }
}
});
No description
34 Replies
d.js toolkit
d.js toolkitβ€’14mo ago
- What's your exact discord.js npm list discord.js and node node -v version? - Not a discord.js issue? Check out #other-js-ts. - Consider reading #how-to-get-help to improve your question! - Explain what exactly your issue is. - Post the full error stack trace, not just the top part! - Show your code! - Issue solved? Press the button!
Sayrix
SayrixOPβ€’14mo ago
[email protected] v17.9.0 Or he should return true to be removed from the cache ?
Digital
Digitalβ€’14mo ago
it increases because you dont have lifetime for presences, reactions, and so on.. those pile up quickly if you have their intents why are you returing false on filter? if you dont want to store them, just modify the makeCache object
Sayrix
SayrixOPβ€’14mo ago
No description
Sayrix
SayrixOPβ€’14mo ago
I want to store them but only for 1 hour
Digital
Digitalβ€’14mo ago
you cant do that since there is no that lifetime option
Sayrix
SayrixOPβ€’14mo ago
??? the lifetime option is only for invites messages and threads
Digital
Digitalβ€’14mo ago
^
Sayrix
SayrixOPβ€’14mo ago
Are you blind ?
Sayrix
SayrixOPβ€’14mo ago
No description
Sayrix
SayrixOPβ€’14mo ago
No description
Digital
Digitalβ€’14mo ago
you dont get me, your usage goes up because of those specific caches (users, presences especially, ....), and since there is no option to put lifetime on it, you cant prevent ram from increasing if you dont fully disable those caches
Sayrix
SayrixOPβ€’14mo ago
But if i clear it every hour it shouldn't increase no ?
Digital
Digitalβ€’14mo ago
you can only clear it manually then it shouldnt increase
Sayrix
SayrixOPβ€’14mo ago
Even with the sweeper ?
Digital
Digitalβ€’14mo ago
i literally just said that you cant with sweeper because half of them dont support lifetime
Sayrix
SayrixOPβ€’14mo ago
No description
Sayrix
SayrixOPβ€’14mo ago
But on the doc i can use all of them I will try it this day, I’ll let you know and check the ram usage after some days I tried this:
function users(a) {
if (a.id === client.user.id) return false;
console.log("Clearing user", a.id, "from cache");
return true;
}

// eslint-disable-next-line no-unused-vars
function filter(a, b, c) {
// console.log(a, 1, b, 2, c, 3, "filter");
return true;
}

const interval = 10;
const client = new Discord.Client({
shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
partials: [Discord.Partials.Channel, Discord.Partials.Message],
sweepers: {
...Discord.Options.DefaultSweeperSettings,
bans: { interval: interval, filter: filter },
messages: { interval: interval, lifetime: 1800 },
threads: { interval: interval, lifetime: 1800 }, // 30 minutes
threadMembers: { interval: interval, filter: filter },
invites: { interval: interval, lifetime: 1800 }, // 30 minutes
presences: { interval: interval, filter: filter },
reactions: { interval: interval, filter: filter },
stickers: { interval: interval, filter: filter },
autoModerationRules: { interval: interval, filter: filter },
users: {
interval: interval,
filter: () => users,
},
applicationCommands: { interval: interval, filter: filter },
emojis: { interval: interval, filter: filter },
},
makeCache: Discord.Options.cacheWithLimits({
ApplicationCommandManager: 5, // guild.commands
BaseGuildEmojiManager: 5, // guild.emojis
GuildBanManager: 0, // guild.bans
GuildInviteManager: 0, // guild.invites
GuildManager: Infinity, // client.guilds
GuildMemberManager: {
maxSize: 200,
keepOverLimit: member => member.id === client.user.id,
},
GuildStickerManager: 0, // guild.stickers
GuildScheduledEventManager: 0, // guild.scheduledEvents
MessageManager: 5, // channel.messages
PresenceManager: 0, // guild.presences
ReactionManager: 0, // message.reactions
ReactionUserManager: 0, // reaction.users
StageInstanceManager: 500, // guild.stageInstances
ThreadManager: 0, // channel.threads
ThreadMemberManager: 0, // threadchannel.members
}),
});
function users(a) {
if (a.id === client.user.id) return false;
console.log("Clearing user", a.id, "from cache");
return true;
}

// eslint-disable-next-line no-unused-vars
function filter(a, b, c) {
// console.log(a, 1, b, 2, c, 3, "filter");
return true;
}

const interval = 10;
const client = new Discord.Client({
shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
partials: [Discord.Partials.Channel, Discord.Partials.Message],
sweepers: {
...Discord.Options.DefaultSweeperSettings,
bans: { interval: interval, filter: filter },
messages: { interval: interval, lifetime: 1800 },
threads: { interval: interval, lifetime: 1800 }, // 30 minutes
threadMembers: { interval: interval, filter: filter },
invites: { interval: interval, lifetime: 1800 }, // 30 minutes
presences: { interval: interval, filter: filter },
reactions: { interval: interval, filter: filter },
stickers: { interval: interval, filter: filter },
autoModerationRules: { interval: interval, filter: filter },
users: {
interval: interval,
filter: () => users,
},
applicationCommands: { interval: interval, filter: filter },
emojis: { interval: interval, filter: filter },
},
makeCache: Discord.Options.cacheWithLimits({
ApplicationCommandManager: 5, // guild.commands
BaseGuildEmojiManager: 5, // guild.emojis
GuildBanManager: 0, // guild.bans
GuildInviteManager: 0, // guild.invites
GuildManager: Infinity, // client.guilds
GuildMemberManager: {
maxSize: 200,
keepOverLimit: member => member.id === client.user.id,
},
GuildStickerManager: 0, // guild.stickers
GuildScheduledEventManager: 0, // guild.scheduledEvents
MessageManager: 5, // channel.messages
PresenceManager: 0, // guild.presences
ReactionManager: 0, // message.reactions
ReactionUserManager: 0, // reaction.users
StageInstanceManager: 500, // guild.stageInstances
ThreadManager: 0, // channel.threads
ThreadMemberManager: 0, // threadchannel.members
}),
});
But i have this error: Uncaught exception: TypeError [SweepFilterReturn]: The return value of the sweepFilter function was not false or a Function @qjuh
πŸŒΊπŸ‡«πŸ‡· Shigu :3 🌺
On which line?
Sayrix
SayrixOPβ€’14mo ago
at Timeout._onTimeout (C:\Users\Sayrixx\Desktop\ChilledBot\node_modules\discord.js\src\util\Sweepers.js:461:48) So all sweepers where i use the function filter
Pulse
Pulseβ€’14mo ago
i think that advice was wrong, filter is supposed to be a function that returns a function (or null)
export interface SweepOptions<K, V> {
interval: number;
filter: GlobalSweepFilter<K, V>;
}
export interface SweepOptions<K, V> {
interval: number;
filter: GlobalSweepFilter<K, V>;
}
export type GlobalSweepFilter<K, V> = () => ((value: V, key: K, collection: Collection<K, V>) => boolean) | null; from https://old.discordjs.dev/#/docs/discord.js/14.13.0/typedef/SweepOptions
filter: The function used to determine the function passed to the sweep method
as for the original question, you probably wanted to return true in your filter
Sayrix
SayrixOPβ€’14mo ago
ok. I will try to return true
Pulse
Pulseβ€’14mo ago
from https://discord.js.org/docs/packages/collection/1.5.3/Collection:Class#sweep
Removes items that satisfy the provided filter function.
Sayrix
SayrixOPβ€’14mo ago
Here is my code that i will push into production:
function users(a) {
if (a.id === client.user.id) return false;
console.log("Clearing user", a.id, "from cache");
return true;
}

// eslint-disable-next-line no-unused-vars
function filter(a, b, c) {
// console.log(a, 1, b, 2, c, 3, "filter");
return true;
}

const interval = 10;
const client = new Discord.Client({
shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
partials: [Discord.Partials.Channel, Discord.Partials.Message],
sweepers: {
...Discord.Options.DefaultSweeperSettings,
bans: { interval: interval, filter: () => filter },
messages: { interval: interval, lifetime: 1800 },
threads: { interval: interval, lifetime: 1800 }, // 30 minutes
threadMembers: { interval: interval, filter: () => filter },
invites: { interval: interval, lifetime: 1800 }, // 30 minutes
presences: { interval: interval, filter: () => filter },
reactions: { interval: interval, filter: () => filter },
stickers: { interval: interval, filter: () => filter },
autoModerationRules: { interval: interval, filter: () => filter },
users: { interval: interval, filter: () => users },
applicationCommands: { interval: interval, filter: () => filter },
emojis: { interval: interval, filter: () => filter },
},
makeCache: Discord.Options.cacheWithLimits({
ApplicationCommandManager: 5, // guild.commands
BaseGuildEmojiManager: 5, // guild.emojis
GuildBanManager: 0, // guild.bans
GuildInviteManager: 0, // guild.invites
GuildManager: Infinity, // client.guilds
GuildMemberManager: {
maxSize: 200,
keepOverLimit: member => member.id === client.user.id,
},
GuildStickerManager: 0, // guild.stickers
GuildScheduledEventManager: 0, // guild.scheduledEvents
MessageManager: 5, // channel.messages
PresenceManager: 0, // guild.presences
ReactionManager: 0, // message.reactions
ReactionUserManager: 0, // reaction.users
ThreadManager: 0, // channel.threads
ThreadMemberManager: 0, // threadchannel.members
}),
});
function users(a) {
if (a.id === client.user.id) return false;
console.log("Clearing user", a.id, "from cache");
return true;
}

// eslint-disable-next-line no-unused-vars
function filter(a, b, c) {
// console.log(a, 1, b, 2, c, 3, "filter");
return true;
}

const interval = 10;
const client = new Discord.Client({
shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
partials: [Discord.Partials.Channel, Discord.Partials.Message],
sweepers: {
...Discord.Options.DefaultSweeperSettings,
bans: { interval: interval, filter: () => filter },
messages: { interval: interval, lifetime: 1800 },
threads: { interval: interval, lifetime: 1800 }, // 30 minutes
threadMembers: { interval: interval, filter: () => filter },
invites: { interval: interval, lifetime: 1800 }, // 30 minutes
presences: { interval: interval, filter: () => filter },
reactions: { interval: interval, filter: () => filter },
stickers: { interval: interval, filter: () => filter },
autoModerationRules: { interval: interval, filter: () => filter },
users: { interval: interval, filter: () => users },
applicationCommands: { interval: interval, filter: () => filter },
emojis: { interval: interval, filter: () => filter },
},
makeCache: Discord.Options.cacheWithLimits({
ApplicationCommandManager: 5, // guild.commands
BaseGuildEmojiManager: 5, // guild.emojis
GuildBanManager: 0, // guild.bans
GuildInviteManager: 0, // guild.invites
GuildManager: Infinity, // client.guilds
GuildMemberManager: {
maxSize: 200,
keepOverLimit: member => member.id === client.user.id,
},
GuildStickerManager: 0, // guild.stickers
GuildScheduledEventManager: 0, // guild.scheduledEvents
MessageManager: 5, // channel.messages
PresenceManager: 0, // guild.presences
ReactionManager: 0, // message.reactions
ReactionUserManager: 0, // reaction.users
ThreadManager: 0, // channel.threads
ThreadMemberManager: 0, // threadchannel.members
}),
});
(i will just change the interval)
LeMorrow
LeMorrowβ€’14mo ago
@sayrix how did it go? Did true work better?
Sayrix
SayrixOPβ€’14mo ago
I forgot to disable something so i will re try now, i'll let you know friday
LeMorrow
LeMorrowβ€’14mo ago
perfect, thank you I've also had a bunch of issues similar to this and struggled with my sweepers
Sayrix
SayrixOPβ€’14mo ago
No description
Sayrix
SayrixOPβ€’14mo ago
function users(a) {
if (a.id === client.user.id) return false;
console.log("Clearing user", a.id, "from cache");
return true;
}

// eslint-disable-next-line no-unused-vars
function filter(a, b, c) {
// console.log(a, 1, b, 2, c, 3, "filter");
return true;
}

const interval = 15 * 60; // 15 minutes
const client = new Discord.Client({
shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
partials: [Discord.Partials.Channel, Discord.Partials.Message],
sweepers: {
...Discord.Options.DefaultSweeperSettings,
bans: { interval: interval, filter: () => filter },
messages: { interval: interval, lifetime: 1800 },
threads: { interval: interval, lifetime: 1800 }, // 30 minutes
threadMembers: { interval: interval, filter: () => filter },
invites: { interval: interval, lifetime: 1800 }, // 30 minutes
presences: { interval: interval, filter: () => filter },
reactions: { interval: interval, filter: () => filter },
stickers: { interval: interval, filter: () => filter },
autoModerationRules: { interval: interval, filter: () => filter },
users: { interval: interval, filter: () => users },
applicationCommands: { interval: interval, filter: () => filter },
emojis: { interval: interval, filter: () => filter },
},
makeCache: Discord.Options.cacheWithLimits({
ApplicationCommandManager: 5, // guild.commands
BaseGuildEmojiManager: 5, // guild.emojis
GuildBanManager: 0, // guild.bans
GuildInviteManager: 0, // guild.invites
GuildStickerManager: 0, // guild.stickers
GuildScheduledEventManager: 0, // guild.scheduledEvents
MessageManager: 5, // channel.messages
PresenceManager: 0, // guild.presences
ReactionManager: 0, // message.reactions
ReactionUserManager: 0, // reaction.users
ThreadManager: 0, // channel.threads
ThreadMemberManager: 0, // threadchannel.members
}),
});
function users(a) {
if (a.id === client.user.id) return false;
console.log("Clearing user", a.id, "from cache");
return true;
}

// eslint-disable-next-line no-unused-vars
function filter(a, b, c) {
// console.log(a, 1, b, 2, c, 3, "filter");
return true;
}

const interval = 15 * 60; // 15 minutes
const client = new Discord.Client({
shards: sharding.getInfo().SHARD_LIST, // An array of shards that will get spawned
shardCount: sharding.getInfo().TOTAL_SHARDS, // Total number of shards
intents: ["Guilds", "GuildVoiceStates", "GuildMessages"],
partials: [Discord.Partials.Channel, Discord.Partials.Message],
sweepers: {
...Discord.Options.DefaultSweeperSettings,
bans: { interval: interval, filter: () => filter },
messages: { interval: interval, lifetime: 1800 },
threads: { interval: interval, lifetime: 1800 }, // 30 minutes
threadMembers: { interval: interval, filter: () => filter },
invites: { interval: interval, lifetime: 1800 }, // 30 minutes
presences: { interval: interval, filter: () => filter },
reactions: { interval: interval, filter: () => filter },
stickers: { interval: interval, filter: () => filter },
autoModerationRules: { interval: interval, filter: () => filter },
users: { interval: interval, filter: () => users },
applicationCommands: { interval: interval, filter: () => filter },
emojis: { interval: interval, filter: () => filter },
},
makeCache: Discord.Options.cacheWithLimits({
ApplicationCommandManager: 5, // guild.commands
BaseGuildEmojiManager: 5, // guild.emojis
GuildBanManager: 0, // guild.bans
GuildInviteManager: 0, // guild.invites
GuildStickerManager: 0, // guild.stickers
GuildScheduledEventManager: 0, // guild.scheduledEvents
MessageManager: 5, // channel.messages
PresenceManager: 0, // guild.presences
ReactionManager: 0, // message.reactions
ReactionUserManager: 0, // reaction.users
ThreadManager: 0, // channel.threads
ThreadMemberManager: 0, // threadchannel.members
}),
});
btw for my sharding manager i use "discord-hybrid-sharding" @lemorrow When i look the graph thats something like ~100MiB every 3 hours
LeMorrow
LeMorrowβ€’14mo ago
Right, so no big improvement with that I'll let you know if I can find any improvements with my experiments
mahakadema
mahakademaβ€’14mo ago
Something that would definitely help in diagnosing where this RAM usage originates from (because i see nothing here indicating that caches are what drive up this ram usage) would be to diagnose whether the sweepers run as expected. this event listener:
client.on(Events.CacheSweep, (msg) => console.log("SWEEPER RUN", msg))
client.on(Events.CacheSweep, (msg) => console.log("SWEEPER RUN", msg))
would allow you to diagnose whether the numbers approximately match up. If they do, ur RAM leakage is likely caused by other stuff
Sayrix
SayrixOPβ€’14mo ago
If you also have a big bot like mine idk if you also use discord-hybrid-sharding for your sharding manager because maybe that it's caused by my sharding manager πŸ€·β€β™‚οΈ I will try to add this code in production to see I think i will install a profiler to see what's wrong
BradyDaLlama .*Λ–
BradyDaLlama .*Λ–β€’12mo ago
did you ever figure this out? I have a pretty similar issue but it's a smaller bot. I have pretty aggressive makeCache settings (i think) and some sweepers but the ram still seems to go up and then crash/ i reset it before it does
No description
BradyDaLlama .*Λ–
BradyDaLlama .*Λ–β€’12mo ago
the top command shows super low memory usage, but the free -g/-m shows a higher amount is being used, closer to what my VPS monitoring reports.
Want results from more Discord servers?
Add your server