Kysely•16mo ago

Transaction response time

Hey everyone, in my code I have a createUser(), createProfile(), createBusiness(), and createAddress() for this example. Currently, I have creaeUser() execute on it's own and then the other 3 free functions execute as one transaction. Which gives me the benefit of having a fast response time for when a user creates an account. (the loading screen is quick) but I don't know if that benefit outweighs the risk of having user records in the database without a profile, business, and/or address records - *if for some reason the transaction fails but the createUser() executed fine. I really like the fast signup user experience provided by just awaiting createUser() without having to wait for a whole transaction to complete. Is there any way to handle this scenario other than optimizing the functions within the transaction as much as possible?

21 Replies

Igal•16mo ago

Hey 👋 You could create user immediately and offload the rest of the data + user id to a queue with a retry mechanism and dead letter queue. thus the other entities will be created eventually and asynchronously.

NazCodelandOP•16mo ago

async createUser(data) {
  const userData = coerceInputData(data, 'emailVerified');

  get(isCurrentSignUpABusinessStore) ? (userData.role = 'business') : (userData.role = 'client');

  // create 'profile','address' and 'business' if it's a   business signup
  const result = db.transaction().execute(async (trx) => {
  // create user
  const query = db.insertInto('User').values(userData);
  // return user
  const result = supportsReturning ? await query.returningAll().executeTakeFirstOrThrow() : await query.executeTakeFirstOrThrow().then(async () => {
    return await db.selectFrom('User').selectAll().where('email', '=', `${userData.email}`).executeTakeFirstOrThrow();
});

  const userId = result.id;
  const profile = await createProfile(
{ 
  ...get(currentSigningUpBusinessStore).profile,
  userId: userId
},
  trx
);

let business;
if (get(isCurrentSignUpABusinessStore)) {
  business = await createBusiness(
    {
      ...get(currentSigningUpBusinessStore).business,          userId: userId
    },
    trx
  );
}

const address = await createAddress(                  {
        ...get(currentSigningUpBusinessStore).address,
        ...(get(isCurrentSignUpABusinessStore) ? {   businessId: business?.id 
      } : { profileId: profile.id })
    },
    trx
  );
  return result;
});

return coerceReturnData(await result, 'emailVerified');
},

async createUser(data) {
  const userData = coerceInputData(data, 'emailVerified');

  get(isCurrentSignUpABusinessStore) ? (userData.role = 'business') : (userData.role = 'client');

  // create 'profile','address' and 'business' if it's a   business signup
  const result = db.transaction().execute(async (trx) => {
  // create user
  const query = db.insertInto('User').values(userData);
  // return user
  const result = supportsReturning ? await query.returningAll().executeTakeFirstOrThrow() : await query.executeTakeFirstOrThrow().then(async () => {
    return await db.selectFrom('User').selectAll().where('email', '=', `${userData.email}`).executeTakeFirstOrThrow();
});

  const userId = result.id;
  const profile = await createProfile(
{ 
  ...get(currentSigningUpBusinessStore).profile,
  userId: userId
},
  trx
);

let business;
if (get(isCurrentSignUpABusinessStore)) {
  business = await createBusiness(
    {
      ...get(currentSigningUpBusinessStore).business,          userId: userId
    },
    trx
  );
}

const address = await createAddress(                  {
        ...get(currentSigningUpBusinessStore).address,
        ...(get(isCurrentSignUpABusinessStore) ? {   businessId: business?.id 
      } : { profileId: profile.id })
    },
    trx
  );
  return result;
});

return coerceReturnData(await result, 'emailVerified');
},

Hey! Just for context, I have createUser(), createProfile(), createBusiness() and createAddress() so, now I need to learn about how to make a queue with a retry mechanism and dead letter queue, but before I do that I have a question about "... thus the other entities will be created eventually and asynchronously." If the response is returned immediately after createUser() has been created, so the front-end UI proceeds as it's awaiting the return of createUser() and then say later, in the transaction, something goes wrong, how will that work. Does the user get signed out? Does the record created by createUser() get deleted?

Igal•16mo ago

and then say later, in the transaction, something goes wrong, how will that work.

you either poll for the eventually consistent data OR its errors, or use web sockets (preferably a managed service for that) that inform the FE things succeeded/failed.

Does the user get signed out? Does the record created by createUser() get deleted?

depends on product requirements and UX you're going for.

NazCodelandOP•16mo ago

What do you mean by "Poll"? as in choose?

Igal•16mo ago

https://en.wikipedia.org/wiki/Polling_(computer_science)

Polling (computer science)

Polling, or interrogation, refers to actively sampling the status of an external device by a client program as a synchronous activity. Polling is most often used in terms of input/output (I/O), and is also referred to as polled I/O or software-driven I/O. A good example of hardware implementation is a watchdog timer.

Igal•16mo ago

polling is when you keep requesting for the data or its status in intervals from the FE

NazCodelandOP•16mo ago

I see, I heard of it but I thought ppl were saying "pulling" lol hmm, in the above code if I moved

...
 const userData = coerceInputData(data, 'emailVerified');

  get(isCurrentSignUpABusinessStore) ? (userData.role = 'business') : (userData.role = 'client');

  // create 'profile','address' and 'business' if it's a   business signup
  const result = db.transaction().execute(async (trx) => {
  // create user
  const query = db.insertInto('User').values(userData);
  // return user
  const result = supportsReturning ? await query.returningAll().executeTakeFirstOrThrow() : await query.executeTakeFirstOrThrow().then(async () => {
    return await db.selectFrom('User').selectAll().where('email', '=', `${userData.email}`).executeTakeFirstOrThrow();
});
...

...
 const userData = coerceInputData(data, 'emailVerified');

  get(isCurrentSignUpABusinessStore) ? (userData.role = 'business') : (userData.role = 'client');

  // create 'profile','address' and 'business' if it's a   business signup
  const result = db.transaction().execute(async (trx) => {
  // create user
  const query = db.insertInto('User').values(userData);
  // return user
  const result = supportsReturning ? await query.returningAll().executeTakeFirstOrThrow() : await query.executeTakeFirstOrThrow().then(async () => {
    return await db.selectFrom('User').selectAll().where('email', '=', `${userData.email}`).executeTakeFirstOrThrow();
});
...

out of the transaction, that would leave createProfile, createBusiness and createAddress in the one transaction. I need to keep track of the status of this transaction I can keep track of it by polling it or use web sockets to do it Just wanted to make sure I'm following but if that is correct it's on me to implement error handling to notify myself and maybe the user along with keeping a record of which user (userId) the transaction failed for

Igal•16mo ago

you're too focused on the low-level details

handler(req, res) {
  const Aresult = await A(req.body);

  await enqueue({ body: req.body, Aresult });

  res.status(201).json(Aresult);
}

handler(req, res) {
  const Aresult = await A(req.body);

  await enqueue({ body: req.body, Aresult });

  res.status(201).json(Aresult);
}

enqueue would just enqueue the message in a queue (SQS, RabbitMQ, etc.). another place in your codebase will consume the queue and do the transaction.

NazCodelandOP•16mo ago

Okay, I follow so far, I am assuming these SQS, RabbitMW platforms /API offer error handling etc or somewhere inbetween I need to add that incase something in the queue goes wrong

Igal•16mo ago

yeah messages can be "dead-lettered" when they fail too much (a configurable threshold) or they expire (TTL) or the queue is full

NazCodelandOP•16mo ago

okay, I am assuming messages are the individual queued items in the queue awaiting exeuction

Igal•16mo ago

yeah the structure of a message body is totally your decision

NazCodelandOP•16mo ago

I haven't looked into SQS, RabbitMW etc but you are saying that I can just pass it my transaction as a queue and somewhere else in the code I will be using SQS/RabbitMQ to process those queues very nice if I am following you correctly so far, that brings me to my other question I asked you in the other channel I guess I'll close that channel and continue here, unless you think it's a good idea to continue there?

Igal•16mo ago

let's continue here

NazCodelandOP•16mo ago

You said "... monitored (+ alerts), you might have queues in between with retry mechanism + dead letter queue. dead letter queues could be consumed too for error handling ..." and "The system could be event-sourced, so you always have the event persisted and could replay it from the event stream. " or just returning a response the moment as error occurs is also fine depending on the use case we already talked about queues with retry mechanisms, you said "messages" can be dead lettered - I am assuming this is how you determine when to stop retrying ( a threshold that's configurable) "The system could be event-sourced, so you always have the event persisted and could replay it from the event stream. " - that's new does event-sourced mean that any function call, api call or get/mutations are recorded?

Igal•16mo ago

in event-sourcing, events are the source of truth of the system you can "play" them in order and calculate a state.

NazCodelandOP•16mo ago

event-sourcing sounds like a applicaiton level design choice Unless, I am understanding it wrong. Every application has events, so when a user signs up, that signup event is the source of truth and say createUser() succeds based on that event, but the transaction fails. In a "event-sourcing" setup you would be able to have access to that event and re-submit that event if you need to if that's wrong, then I'll have to read up on it some more to understand

Igal•16mo ago

read about it, it's a cool pattern not a silver bullet, not suitable for many cases.

NazCodelandOP•16mo ago

ya looking into it right, so far sounds cool Hey Igal, I ended up looking into Event Sourcing and Messaging Queues and the categories they are part of. I think I have better understanding of what solution would work for my case. I wanna share and ask you something. So, I learned that with Event Sourcing I'll get Atomicity out of the box. The above createUser() and it's subfunctions would be one operation - saving the state as an event into the Event Store. Compared to what it is currently - 3 (if business 4) operations of type CRUD with the database This would give me the solution that I wanted; not having to manually go in and create/edit a record in the database incase the transaction within createUser() failed The overall objective was to give a fast response so the UI would be snappy - I am unable to find out if storing an event equivalent to the createUser() function above, is faster than doing the 3-4 operations

Igal•16mo ago

one of the most important things is, does it make sense? - domain-wise. do the pros outweigh the cons

NazCodelandOP•16mo ago

not entirely, after some reading, I realize that majority of applications can be designed to use either one. In my case, I probably would gain some benefits for the domain but I don't think those benefits out weight the complexity, what I think outweighs the complexity is the addition of the few benefits I think the domain will get + the vast amount of data that I could in the future use to figure out a lot of metrics because I would be able to create event stores and put in any type of user behavior event into it and set up a service to make sense of that data my options as I see them are, I use my current implementation and just pass the transaction to a message queue, handle errors from the dead lettered queues and manually deal with those dead lettered queues after their retry limit or TTL expires. This is the easiest implementation. Vs. Event sourcing, way more complexity but many benefits

Gaming

Programming

Transaction response time