System architecture for processing data from multiple 3rd party APIs
Hello, I need some advice on the best practices and general direction I should consider for handling data for my application.
The current data flow is outlined in the attached image. Issues I have with it: 1. NextJS/Vercel Serverless functions can only run for so long. Thus, I return
The current data flow is outlined in the attached image. Issues I have with it: 1. NextJS/Vercel Serverless functions can only run for so long. Thus, I return
res.status(200).json(message) before the processes I want to run are actually finished.
2. NextJS Api routes should generally only be used by the frontend, not for querying from outside of the system. Separations of concerns suggests to me that I should have another service for handling my data.
3. No queue, so if two http requests towards the same endpoint happens at the same time I'm not even sure what happens, or how I could monitor that my data, and especially my articles, are being inserted correctly.
Possible solutions:
- Building a service to handle the data using:
- AWS SQS (1 million requests for free)
- Google Cloud Computing
- Using Quirrel
- Queue system
- Cron jobs
- Self-hosted (Not sure about cost)
Concerns:
- I would like to share types between all systems. Especially those that are generated by Prisma, my zod schemas and possibly my tRPC client. Is this feasible with separated services?
- Am I thinking about this the wrong way? Am I missing something?
4 Replies
before the processes I want to run are actually finished.Does the function still keep on going once you return immediately? Or are they still stopped at the 10 second timeout?
share types between all systems.Monorepo or if separate repos use a shared package
>Does the function still keep on going once you return immediately? Or are they still stopped at the 10 second timeout?
I'm not actually sure. In my local environment it keeps running, for sure, but in my vercel dev environment it seems that it stops, or I'm unable to see more console.logs.
Monorepo or if separate repos use a shared packageI am in fact in a monorepo, using create-t3-turbo, but I'm unsure whether it's a good idea to include other services in it, that are not more tightly connected to my NextJS application.
Google and Facebook use monorepos for everything. I’m sure you will be okay having some of your services in the same repo. Just make sure to tweak the deployment settings so you don’t redeploy when there’s a change to an unrelated package
Good point, thank you!
Do you have any input on how you would design the structure for this sort of data handling?