What Services Should I use?

Hello everyone. I'm looking to build an application that can achieve the following: - Receive event data from Shopify's web pixel API (https://shopify.dev/docs/api/web-pixels-api) - Send the event data to storage (ideally to BigQuery but open to something else) I'm unsure what all the tools I would need for this. Should I use a worker to create an endpoint for ingesting the events? Then connect the worker to BigQuery? BigQuery only allows 1500 "load jobs" per day so I couldn't insert events 1 by 1. How could I batch and load say every minute? Or should the worker be bound to D1? R2? (not sure what the major differences are there). Thank you to anyone with thoughts and ideas 🙂
6 Replies
MK
MK•13mo ago
D1 is sqlite based database, R2 is S3 supported storage (basically for your files). So if you wanna run SQL commands on your data you better go with D1. Or you can use Workers-KV (key value store) if you need to access them fast. Question is how much data you're gonna push to your desired infrastructure?
Stockton
StocktonOP•13mo ago
It's going to be a lot. Think all the events that happen on an ecommerce store. Page View, view item, add to cart, purchase, etc.
MK
MK•13mo ago
Then you need to use another cloud database because cloudflare has some limitations on D1 (https://developers.cloudflare.com/d1/platform/limits/) And since it's still in beta...
Stockton
StocktonOP•13mo ago
Right, so I was thinking BigQuery. Any thoughts on that? And batching for inserts every minute?
MK
MK•13mo ago
I've never used or dig into it sorry. But you can check aws db products. also I didn't put them side by side.
Stockton
StocktonOP•13mo ago
Thank you so much for your inputs MK! Really appreciate you jumping in. Wondering if anyone else has some thoughts to? Looks like the new queue service would work nicely. Shopify events -> Worker -> Queue -> BigQuery
Want results from more Discord servers?
Add your server