Dealing with concurrent updates
Does anyone have suggestions on resources, concepts and topics to look into on handling concurrent updates? Like I have two different systems possibly updating the same record/object, how should I be handling concurrent updates to prevent duplicates and other issues. The two different systems have their own db but eventually the shared data has to be written in a middleware postgres db that has to handle the concurrent scenarios.
53 Replies
ā This post has been reserved for your question.
Hey @Kale Vivi! Please useTIP: Narrow down your issue to simple and precise questions to maximize the chance that others will reply in here./closeor theClose Postbutton above when your problem is solved. Please remember to follow the help guidelines. This post will be automatically marked as dormant after 300 minutes of inactivity.
try database locking
optimistic and pessimistic depending on the use case
it depends on what you are dealing with
if it's about concurrency within your application, you can use language mechanisms: https://discord.com/channels/648956210850299986/1027179147086282752
If you it's about synchronizing access to a DB from multiple applications, you would want to use the mechanisms provided by the DB (e.g. locking or transactions)
How does the db or language know the order of the inserts/updates/changes? Is it based on ids or timestamps or..? I'll start reading up on the concepts above. thanks!
If you are finished with your post, please close it.
If you are not, please ignore this message.
Note that you will not be able to send further messages here after this post have been closed but you will be able to create new posts.
if it gets a request for doing an operation and it can execute the operation, it does that
š¤ Post marked as dormant
This post has been inactive for over 300 minutes, thus, it has been archived.
If your question was not answered yet, feel free to re-open this post or create a new one.
In case your post is not getting any attention, you can try to use /help ping.
Warning: abusing this will result in moderative actions taken against you.Hey, I guess I have questions on handling something like this:
I have an user facing application A that is a SaaS (full stack web app) that can make an update to the same record in another user facing application B (full stack). Both have their own data stores but I have to sync updates to both data stores using a middleware layer that also has it's own db. Using locks and transactions should handle majority of the cases?
I mean also there is latency between systems, so thinking to keep track using a timestamp for each system and compare timestamps?
maybe consider adding an API in application B for application A to update stuff?
though you could also let both access the common DB
What do you mean by both access the common DB? I mean both systems will update the common db via apis/streaming etc
Eventually, we would like to retire Application B, hence why better not to add api and other items in B
Just need to solve this issue temporarily until App A is fully good and well
If you are finished with your post, please close it.
If you are not, please ignore this message.
Note that you will not be able to send further messages here after this post have been closed but you will be able to create new posts.
I'm thinking at this point I will need to compare timestamps but not sure how much more overhead this introduces
oh so A is replacing B
eventually but need to keep both live for some time
so thats' where the data sync is needed
until then
So both have the same data but in a different format and both do both reading and writing?
yeah exactly
Is it maybe possible that you replace the writing functionality first?
so that application B becomes read-only and all write access happens with application A?
Unfortunately no
both need to be live until the full cutover
Do you need more or less realtime updates?
ideally realtime updates between systems if possible
but thinking 5 to 10 min delay is fine too
I think the best option is the following:
- Add an API to application A that allows updating and reading stuff
- change application B to use that API for all its operations instead of the DB
so you let application A be the DB of application B
or you use the same DB for both applications
i.e. have more or less the same schema (you might be able to use views or stored procedure for some things)
App A is vendor locked, we can't make direct changes to it. They have their own apis we can refer to however to get data in and out of the system
alternatively (if there are not that many different types of write applications), you can do the following:
- Add an API to application A that allows application B to modify A's DB
- Add an API to application B that allows application A to modify B's DB
App B is retiring, so we don't want to add more functionality to App B. We are building out the middleware however that sits between A and B
that can handle keeping everything in sync
Is there an option on getting notified via an API for every change being made in application B?
i.e. can application A subscribe to all changes in application B?
We can subscribe to App A not app B. The are webhooks functionality in App A
So you have absolutely no option of getting informed about changes being made in app B?
One thought is setting up a kafka producer in app B
to get the updates in real time
and we can consume the updates in the middleware
adn then eventually post that to app A
you need something that gets all updates of app B and sends that to app A, yes
I mean let's say I have something liek that
How do I handle simul. updates?
I have to compare timestamps right
when you make an operation in app A, you use application B's API to perform the same change
and you can maybe do something like rolling back the transaction in A if that one fails
Do you have access to app B's database?
Do you have an option to block changes in app B (e.g. via your middleware)?
Can you maybe simulate application B's database using application A's database?
What do you mean by simulate app B and mean by block changes? Middleware also has it's own db
You have a middleware between app B and the database - Are you controlling that middleware/can you change that?
Also is it an option for app A to just use the DB of app B instead of its own?
App A + DB <> Middlware + DB <> App B + DB
It's more like this right now
No App A can't use app B
db
Does app B communicate with the DB directly?
or is there something in between you could intercept?
š¤ Post marked as dormant
This post has been inactive for over 300 minutes, thus, it has been archived.
If your question was not answered yet, feel free to re-open this post or create a new one.
In case your post is not getting any attention, you can try to use /help ping.
Warning: abusing this will result in moderative actions taken against you.couldnt you use some sort of pessimistic locking or optimistic in both apps
wouldnt that just solve all the problems
as long as u can edit both apps
they can only modify one of the apps - they said that before
and both apps have different databases
o mamma mia
hol up
theres middleware plus db
so just apply the locking on the middleware
and then once that operation is done you can let the other db know
to do stuff
hmmmmmmmmmmmmmmmmmmmmmm
what about using something to make a queue
kafka sounds like a great answer to this
have 1 channel for both apps to do stuff
and 1 same channel for both apps to receive instructions
I'm not sure about what happens with the middleware of app B (the one they cannot modify)
that's what I was trying to find out
But they cannot change (much) about app B
then u cant do anything with app b db really
this is why I asked about it
unless u connect the db from app b to app a
but there are some possibilities - but these depend on the answers to these things
yes your sudoness
ur new title
š
there's also some APIs of app B but there are limitations on it
yea kinda weird architecture ngl
well at least it's better than the last one and I'll probably not need to automod it
it's migrating from B (foreign vendor) to A
what was bad about the last one smh
it was the merge of two words
skibidi and majesty
the first one
oh yea
that makes sense
Hey, I'll diagram some things and get some feedback soon just to clarify any doubts here
š¤ Post marked as dormant
This post has been inactive for over 300 minutes, thus, it has been archived.
If your question was not answered yet, feel free to re-open this post or create a new one.
In case your post is not getting any attention, you can try to use /help ping.
Warning: abusing this will result in moderative actions taken against you.