Cloudflare Developers•3y ago

Transactional outbox

I wrote another library in the "Make building reliable DOs easier" series.. this time it is to help create DO that use the Transactional Outbox pattern: https://github.com/evanderkoogh/do-transactional-outbox. Would love any feedback about how easy it is to setup and if you come across any edge-cases I haven't thought of..

5 Replies

Cole•3y ago

Thank you so much for sharing this pattern. I'm not entirely sure why I'd use it, but I haven't heard about it before, and it's really nice to see DO code examples. Btw, I opened a PR to fix something I caught in an initial scan https://github.com/evanderkoogh/do-transactional-outbox/pull/1/files

GitHub

Fix function bind in proxyDO by colelawrence · Pull Request #1 · ev...

Cole•3y ago

I'm stuck re-reading the full example in the README.md, but I'm just not seeing the bigger picture. Is there any chance you might add comments to the example which can point out why using .put in this context is interesting?

import { OutboxManager, TOB_DurableObject, withTransactionalOutbox } from 'do-transactional-outbox'

export interface Env {
  TEST_DO_TOB: DurableObjectNamespace
  OUTBOX_MANAGER: OutboxManager
}

class DO implements TOB_DurableObject {
  private storage: DurableObjectStorage
  constructor(state: DurableObjectState, protected readonly env: Env) {
    this.storage = state.storage
  }
  // What actually causes this method to be invoked?
  async sendMessages(messages: { id: string; msg: any }[]): Promise<void> {
    // What kind of business logic is this?
    // Or is this wiring for TO pattern?
    Object.entries(messages).forEach(async ([key, msg]) => {
      await this.storage.put(`__msg::${key}`, msg)
    })
  }
  
  async fetch(request: Request): Promise<Response> {
    // Retrying is something talked about in the TO pattern, does that mean that the response can fail, and this still gets retried?
    const storage = this.env.OUTBOX_MANAGER
    // Is this interesting?
    // Is it interesting if this failed?
    await storage.put(`one`, 'first', 'first')
    const puts = {
      two: 'second',
      three: 'third',
    }
    // Is "multiples" part of DurableObjects base API, or is it special transaction?
    await storage.put(puts, { type: 'multiples', blah: true })
    return new Response('Ok', { status: 200 })
  }
}

const exportedDO = withTransactionalOutbox(TestDO)

export { exportedDO }

import { OutboxManager, TOB_DurableObject, withTransactionalOutbox } from 'do-transactional-outbox'

export interface Env {
  TEST_DO_TOB: DurableObjectNamespace
  OUTBOX_MANAGER: OutboxManager
}

class DO implements TOB_DurableObject {
  private storage: DurableObjectStorage
  constructor(state: DurableObjectState, protected readonly env: Env) {
    this.storage = state.storage
  }
  // What actually causes this method to be invoked?
  async sendMessages(messages: { id: string; msg: any }[]): Promise<void> {
    // What kind of business logic is this?
    // Or is this wiring for TO pattern?
    Object.entries(messages).forEach(async ([key, msg]) => {
      await this.storage.put(`__msg::${key}`, msg)
    })
  }
  
  async fetch(request: Request): Promise<Response> {
    // Retrying is something talked about in the TO pattern, does that mean that the response can fail, and this still gets retried?
    const storage = this.env.OUTBOX_MANAGER
    // Is this interesting?
    // Is it interesting if this failed?
    await storage.put(`one`, 'first', 'first')
    const puts = {
      two: 'second',
      three: 'third',
    }
    // Is "multiples" part of DurableObjects base API, or is it special transaction?
    await storage.put(puts, { type: 'multiples', blah: true })
    return new Response('Ok', { status: 200 })
  }
}

const exportedDO = withTransactionalOutbox(TestDO)

export { exportedDO }

ErwinOP•3y ago

Ahh.. those are super good comments! Thanks for that So basically the challenge you sometimes have is that you want to save something to a database and tell someone that you updates something. So what you want is that either both the save and the sending happen, or neither happen. With this library, this is what happens. (Assuming you are able to send messages at all). It handles things like saving both state and messages and automatically retries sending the messages.

Cole•3y ago

Would that be applied for something like an operational transformation oriented collab architecture? Then, making sure the actual messages are idempotent for the receiver (as I understand the referenced write up)

ErwinOP•3y ago

I am not entirely sure what you mean with an operational transformation oriented collab architecture, but yes, you want your messages to be idempotent for the receiver, because there is always the option you will get a message multiple times (at least once guarantee)

Gaming

Programming

Transactional outbox