What Services Should I use?
Hello everyone. I'm looking to build an application that can achieve the following:
- Receive event data from Shopify's web pixel API (https://shopify.dev/docs/api/web-pixels-api)
- Send the event data to storage (ideally to BigQuery but open to something else)
I'm unsure what all the tools I would need for this.
Should I use a worker to create an endpoint for ingesting the events?
Then connect the worker to BigQuery? BigQuery only allows 1500 "load jobs" per day so I couldn't insert events 1 by 1. How could I batch and load say every minute?
Or should the worker be bound to D1? R2? (not sure what the major differences are there).
Thank you to anyone with thoughts and ideas 🙂
6 Replies
D1 is sqlite based database, R2 is S3 supported storage (basically for your files). So if you wanna run SQL commands on your data you better go with D1. Or you can use Workers-KV (key value store) if you need to access them fast. Question is how much data you're gonna push to your desired infrastructure?
It's going to be a lot. Think all the events that happen on an ecommerce store. Page View, view item, add to cart, purchase, etc.
Then you need to use another cloud database because cloudflare has some limitations on D1 (https://developers.cloudflare.com/d1/platform/limits/) And since it's still in beta...
Right, so I was thinking BigQuery. Any thoughts on that? And batching for inserts every minute?
I've never used or dig into it sorry. But you can check aws db products. also I didn't put them side by side.
Thank you so much for your inputs MK! Really appreciate you jumping in.
Wondering if anyone else has some thoughts to?
Looks like the new queue service would work nicely.
Shopify events -> Worker -> Queue -> BigQuery