Best way to handle fetching large amounts of emails
So I have a T3 app that is essentially a project manager for a construction/remodel company. Every time the company is starting a project on a client's house, they create a new "project' in our app. One of the features for the scope is to have an "email" tab where we use the gmail API to fetch every conversation an employee of the company has had with the client, and display them in the app.
The gmail API for orgs does have an endpoint to fetch all emails built in, but based on the fact that my fetch for single users emails takes like 3-5 seconds ATM, I'm guessing that doing a fetch on all this data at runtime is not going to to be viable.
What approach would you guys recommend? One approach one of my coworkers floated was to essentially have a separate server that's running 24/7, and does this fetch call once every minute. We then have a DB that essentially just has one table, customers, and each customer has an array of messageIds. All we do is add in add in the ID of any message that has happened since the last fetch to the array. Then on the main app, we fetch the list of messageIds from the DB for the customer, and use that pre-compiled list of messages to fetch each email needed.
Another similar approach would be that whenever an "admin" logs in on the app, and opens a project, we do a fetch for their conversations with the given client, then push that to the main DB we already have. Then we can fetch everyone's previous convos. Downside to this approach is that the emails would only be updated when one of the admin logs in and views the projects, so recent emails could take a while to come, users don't have a way of knowing if the list of emails their looking at is stale or not.
Curious to hear thoughts about better approaches as I'm not really a big fan of either of these!
2 Replies
I would disregard the 2nd approach, because having it trigger only for admins is a much worse than it taking 3-5s, or any amount of time actually 😂
The one with the 24/7 server is better IMO, but will maybe bump up your costs too much (or maybe not).
I would maybe consider this approach:
Have a Redis DB (or KeyDB or w/e since the Redis drama) and just when you would fetch the emails, save it into Redis so next time you have something to show initially when the user loads it up.
However, you would still need to fetch at runtime whenever someone opens that page. Still I think 3-5s showing with and indicator for "loading new messages" doesn't sound that bad... and they can still look at old emails. However, if the time is going to get much longer then this option is probably not viable. But I guess it all depends on "How important is it that your users always have up to date emails in real time"?
Also, I just tried this with my API, why does it take you like 5s for the requests?
@JulieCezar Oh just looked into redis, this is actually super OP for this use case, good call. So easy to use as well wtf.
I think the current plan is to essentially just have this as a Cron job that runs every minute and pushes into the Redis DB