❔ DB and web API data potential desync- how to resolve?
I have an app that will cross compare DB contract data to usage data fetched from a web API.
The API returns details for users like "id", "name", "email" and so forth, which we also store on our server/DB ecause we need to tally before billing. But the API side can be modified by our client at any time. My current solution is that I will always treat the API as the source of ground truth.
So if any of the user Id on the API changes and causes a mismatch, there would need to be a manual/automatic adjustment on the database side for the client records.
My question is, is this an appropriate solution?
1. If so, how can this be done with the least amount of human intervention?
2. If not, what is an appropriate solution?
From what I can see, the web API does not provide any endpoints to listen for changes to the user's id.
4 Replies
I think the only way this really makes sense is if the API DOES provide an endpoint to listen for changes
that's exactly what @MODiX does, for example
we keep a couple different tables of user data in our own database, which is just cloned data from Discord
and we listen to the appropriate gateway events from Discord, to know when to make updates
also on startup, IIRC, when Discord sends us a full list of users, anyway
that lets us do various info queries exclusively from our database, instead of having to always call out to Discord
you're basically proposing the same thing, but you need to decide a strategy for when to update your cache
if you can't do it immediately, with notifications from the API, then you'll have to be okay with stale data, to some degree
and you need to decide what degree
maybe you actively track the staleness of each user record, by it's LastUpdated timestamp, and refresh from the API after X amount of time has passed
maybe you just do a bulk refresh of all users nightly
or weekly
or whatever
but if you don't actively have an issue with bandwidth or latency for making API calls, I would argue you've got a solution looking for a problem
to me this seems kinda wrong
how do you expect to keep track of this without daily headaches?
why are you keeping client's data and why does matter for billing?
why didn't you agree on a basic structure/fields beforehand?
and in the end, what kind of service are you offering that is this messy
i mean why is not client's data a blackbox
TBH not a single one of those points seems strange to me
Was this issue resolved? If so, run
/close - otherwise I will mark this as stale and this post will be archived until there is new activity.