Using Upstash
Hey guys,
I was wondering if i could get some insight into the best way to design the following system. I have a rough idea on how to design it but not 100% sure.
So I have IoT devices in the field, and only activate when a certain sensor is enabled (which should send a notification to the user with the data).
Once activated they will start sending data to my api every Xms for Y seconds. (just say data every 500ms, for 10 seconds)
This data is sent over HTTP to my nextjs api with data. Sometimes it might spam the api endpoint if the frequency is quite low. To account for this I dont want to send a user a notification on their phone every time this happens.
Currently what i am doing, is inserting all this data into a
MessageBus table. which is exactly what it says, a message bus. I've realised this is probably in efficient, and going to move to redis streams on upstash.
Now this is where im a little unsure on what to do next.
So when data comes in from a device, push it to the redis stream. Cool that works, but then how should i know when to send a notification to the user? I would need some form of server that reads the redis stream, and calls the appropriate actions, but this isnt possible to do in nextjs...32 Replies
As it's http based
Use some form of rate limiting
Loss of data? Possibly
But won't spam that much
yea ofc
not really sure how to handle the stream though
because its all serverless
If you really want
You can use planet scale http driver
And just send directly to the database
And check on next with a cron
i need it real time tho
i want to be abel to send a notification as soon as data starts coming in (obviously only 1 notif)
im not really sure if i need redis at this point
You can check upstash Kafka then
It's based on streams and almost realtime
http request with data comes in
put data in database
send qstash message to request notification if its new data
would that be better instead of kafka or redis?
same issue tho - still need something that subs to the kafka stream
If it's based on realtime
Serverless is hard
yea...
hmmm
i might need a server then hey
maybe qstash publish a message to endpoint?
Idk if it's possible
You can push messages directly to redis
And cron it to analyze and flush
But it's back to issue 1
im not 100% familiar with qstash, but isnt the point of qstash to forward messages?
so like when i get data on my api
publish it on qstash.
oh but ye same issue
Yeah
I guess i could just not use redis, kafka or qstash
it would look like
IoT --> send data to API --> insert data into planetscale db --> call a function to send a notification to user (this then checks the db when the last message was, it if was more than 5mins, send a notification)
i believe that would still be real time
Sounds nice and easy to maintain tbh
yea
means i dont have to worry about redis etc.
Yes
And planetscale does have the perf for that
and in my notification function i can rate limit notifications by doing a simple check.
Yep yep
now i just need to figure out where i should upload my files too
S3
Don't handle file uploads on lambdas
the IoT sends the image to my nextjs api.
then i can just pipe the image to S3?
Get a signed url, and send directly from the device
š ok
And send just the link to store somewhere
any providers like S3?
so send the url from the device to my api, which adds it to the table?
You have other options from providers like azure and such
But S3 is easier overall
fair
Less of the worse
Create url
Send to S3 from device
Store url on database
š
assuming its an easy api to use, just would need an api key or something?
TomDoesTech
YouTube
Use Presigned PUT URLs to Easily Upload Files to AWS S3
Theo's Tweet: https://twitter.com/t3dotgg/status/1564019039877271552
Repository: https://github.com/TomDoesTech/pre-signed-put-url
0:00 Intro
0:43 Data flow
1:45 Bootstrap application
2:33 S3 bucket setup
5:07 Get PUT URL handler
10:36 Form
15:24 Does it work?
š Follow me here:
Discord: https://discord.gg/4ae2Esm6P7
Twitter: https://twitter.c...
cheers
big fan of cloudflare r2, and really cloudflare in general