tRPC, Prisma, Discord Bot and tracking mass messages
Currently rebuilding building an app which is using NextJS (on Vercel), tRPC, Prisma, PlanetScale with a Discord bot all housed in a nice little Turborepo.
Now, you can create posts from the web app that get posted to fairly large number of Discord Servers (~700, and growing).
When the post gets approved by an admin (in the web app), it updates the post to approved through tRPC, tRPC calls the Discord bots little Express API to tell it to get the approved post and start sending the messages.
Then the messages are being sent out to the Discord servers via the bot, which will create an entry in the DB to track the status.
This is where I'm unsure, should the bot be creating those entries directly with Prisma initialized on the bot itself, or be going through tRPC on Next/Vercel.
I suppose if I'm batch creating all those entries from the Discord Bot, and then making a single call back to tRPC, but again it'll likely be ~700 entries to create and I'm unsure on the efficiencies here, as if there's 10 posts, that's 7K entries just to track which messages have been sent...
Or maybe there is a better way to track which messages have been sent?
44 Replies
Do you need to store the transaction "receipts" long term? You could use a message queue system instead maybe.
Do they need to be sent as bot messages or can they be sent as webhooks? When someone makes a post does it get posted in all servers or just a subset?
Instead of storing this data in planetscale also, Redis / Upstash might be a better fit
Agreed - redis/upstash is my go-to for this kind of thing. Then have other processes consume out of there on a schedule or event.
I feel like what you want is webhooks not a discord bot sending messages for this https://discord.com/developers/docs/resources/webhook
Discord Developer Portal
Discord Developer Portal — API Docs for Bots and Developers
Integrate your service with Discord — whether it's a bot or a game or whatever your wildest imagination can come up with.
That’d scale significantly better since I don’t think you’d get rate limited
I was thinking that.
The bot can be used to manage the webhooks in that case.
Say we do that, the logic is moved from the Discord bot to tRPC hosted on Next/Vercel.
They don't need to be long-term receipts, we do need to perform some analytics. Don't really need to store every message receipt.
The "trigger" event is that a posting is approved by an admin in the web app.
Then we work out which channels (or webhooks) need to receive the messages.
Ideally we get all the channels/webhooks, then send to a queue system, however I'm not really sure how to manage that, and how that'd play with upstash/redis.
Hmm, doing a bit of research, and thinking something like BullMQ, and Upstash.
So post gets approved, creates a job in the queue to register all of the messages that need to be sent about the approved post, registering each message in the queue.
Then each message get's sent out from the queue.
Can just be a fairly lightweight node server, I'm not sure if you'd be able to do a message queue serverless
Doing a bit more research now
Maybe QStash with Vercel serverless functions world work? Rather than an entire service for sending/receiving requests? Maaaybe? I don't know if sending 1000 requests to Vercel serverless functions to send a post to each Discord server is efficient?
Reading through this I feel like you may be overcomplicating it, just to break down the problem statement again:
You need the same post to be sent in 700+ Discord servers once approved by an admin via a vercel dashboard
The way these posts are made, whether it be via webhook, discord bot posting, does not matter
Any reason you can't just use an announcement channel and crosspost the message letting Discord do all of the work? https://old.discordjs.dev/#/docs/discord.js/main/class/Message?scrollTo=crosspost
Discord.js
Discord.js is a powerful node.js module that allows you to interact with the Discord API very easily. It takes a much more object-oriented approach than most other JS Discord libraries, making your bot's code significantly tidier and easier to comprehend.
Yes, that's correct, it's that simple.
It's pretty old school is my understanding, we already have ~700 servers with the bot installed, so the bot needs to do the work as we can't tell the servers to do any additional work or we'll likely lose a bunch.
(The bot can create and manage webhooks if we use them), as the people in the Discord server will be managing the subscription to the approved posts.
So the post will have categories such as:
- Marketing
- Design
The user will run
/watch-category the bot will register that server and channel to receive approved posts of that category.
But that's easy enough, no issues there. (That's also why we can't do announcement channels).
It's about when the post comes out, how to manage the distribution of the messages into the specific channels.I suggest asking around in the discord dev server, they probably have a lot more experience than most when it comes to running this kind of bot
From a non technical perspective, it may be worth making the switch now to using Discords' official system as once your bot reaches a certain size they would probably want you to use the official methods
Along with that, you can programmatically subscribe to announcements channels same as you would webhooks, so it wouldn't require any extra work from your subscribers https://discord.js.org/#/docs/discord.js/main/class/NewsChannel?scrollTo=addFollower
So all you'd need to do is make an announcement channel for each topic, the only constraint then would be you can only follow a max of 9 channels per channel
discord.js
discord.js
discord.js is a powerful Node.js module that allows you to interact with the Discord API very easily. It takes a much more object-oriented approach than most other JS Discord libraries, making your bot's code significantly tidier and easier to comprehend.
Hmm, interesting.
I might do some research on that, and come back to you.
Forgetting the Discord component just for the moment, is there a way we can translate a single job being approved, then get all of the webhook/or similar and creating the message queue (of 700-100)
I'm not sure if that latter part of that is possible on Vercel, as that's a large DB query and then posting that off message queue system may exceed timeouts, etc.
At that point, Vercel is not the right fit for what you're trying to do
It's similar to where if you have some sort of AI model, you need to run that outside of Vercel and have the functions call into it, or just have the functions hit it directly
If I were you I would host this on my Discord bot since I already have the resources allocated for that, as for the message queue part, something like https://upstash.com/blog/qstash-announcement would probably work incase the bot goes offline in the middle of sending the announcements
QStash: Messaging for the Serverless
Designed for the serverless with per-request pricing and Redis®/Kafka® API.
Yeah, I think we'll do that, the bot can handle creating the queue, and then the queue can call Vercel serverless functions to send out the messages to the servers.
tbh going from the bot -> vercel may be overkill
instead of writing this to a queue, maybe just store in redis something like
post-id-channel-id-server-id: true / false, with a time to live of 7 days and dispatch the events from the discord bot, updating those as you go
you've already done all the work on the bot to get that data there, no point spinning up extra vercel charges even though they are pretty cheap