Cloudflare Developers•8mo ago

RPC Error handling and retries

The new RPC stuff is really great. One question I have is does the RPC calls handle retries? In my current RPC library I retry on these errors which are not uncommon:

const retryOn = new RegExp([
  'Network connection lost',
  'Cannot resolve Durable Object due to transient issue on remote node',
  'Durable Object reset because its code was updated',
  'The Durable Object\'s code has been updated',
].join('|'));

const retryOn = new RegExp([
  'Network connection lost',
  'Cannot resolve Durable Object due to transient issue on remote node',
  'Durable Object reset because its code was updated',
  'The Durable Object\'s code has been updated',
].join('|'));

(from https://github.com/dabblewriter/durable-apis/blob/main/src/durable-apis.ts#L12-L17) Usually 1 retry is all that is needed, though I do use exponential backoff to continue trying for up to several seconds. This is built-in to my RPC lib. It would be great if it were built-in to your implementation. I don't want to wrap every RPC call with a try/catch to retry on regular network/lifecycle errors. Thanks!

6 Replies

DaniFoldi•8mo ago

cc @kenton I couldn't find most of the error messages from DOs in workerd, and same for jsrpc, sadly also not documented. Is there perhaps a list of these retryable error messages you could give us? :MeowHeartCloudflare:

kenton•8mo ago

At present there is no automatic retry. Note that error.isDurableObjectReset should be true for the latter two errors on your list. Meanwhile, error.remote should be true if the error was thrown by application code on the remote end (i.e. not by system code in between). We have some plans to add more information along these lines to errors to help with this. automatic retry wouldn't be safe in general unless either (a) we know that the message was never delivered to application code on the remote end, or (b) somehow the application tells us that the request is idempotent. For (a) we try our best to retry automatically (though we're not as good at this as we should be today). We don't have a mechanism for (b) at present.

DaniFoldi•8mo ago

Oh interesting, I haven't seen these properties in any docs, I assume these aren't new?

kenton•8mo ago

oh sorry, it's actually durableObjectReset, not isDurableObjectReset but indeed it seems missing from docs, that's weird

Jacob WrightOP•8mo ago

Thank you @kenton. So error.remote will be true if the error was from my code, otherwise the error was from something on the platform?

kenton•8mo ago

Right. .remote is documented here FWIW: https://developers.cloudflare.com/durable-objects/reference/error-handling/

Cloudflare Docs

Error handling · Cloudflare Durable Objects docs

Review how to handle errors (exceptions) generated by both your own Durable Object code as well as exceptions thrown by Durable Objects’ …

Gaming

Programming

RPC Error handling and retries