How to know if it's worth it to implement a circuit breaker ?
I have an microservice that have two background services. The first one consumes internal events from kafka, transform them to integration events, store them in the db. The second one is querying that db and sending http requests with those integration events to an external service.
There is already a retry pattern implemented. But I am wondering, Is it worth it in this case to implement the circuit breaker ? What if The external service is not reliable at all ?
17 Replies
What we do is we use Polly for automatic retry policies. If Polly fails (I.E. no connection at all) we serialize our requests, store them on the machine, and then retry them later when something else going on is successful and the connection is restored
that'sa retry pattern
what's the relation with circuit breakers here ?
Polly can implement a circuit breaker as well, FWIW.
True Polly has that. I was just sharing with ya what we do since you asked if its worth it. I guess it comes down to if you think allowing the app to constantly retry something that is likely to fail exposes you to any sort of DoS attacks or unexpected behaviors
For us, we are internal tooling, so we just hold onto requests that failed Polly's retry policy and then execute them when another request going out finally succeeds
What do you mean by "holding" requests ?
We serialize them (in order) and store them on the machine until one of our requests succeeds. Once one succeeds we have a service that transform them back into requests and send them out. This approach may not apply to you though if your subsequent requests rely on the request before it in which case circuit breaking may be worth it for you
In terms of worth I think the biggest things (for us anyways) were:
• Can it be abused and create a DoS attack intentionally or unintentionally
• Can it cause unexpected errors or data corruption
• Does order matter? And if it does can we somehow guarantee order with/without it
I am not sure I understand why a circuit breaker can be useful with ordering
If you're not sure if you need a circuit breaker you probably don't
yeah, that's what I am saying
but for the sake of learning I want to know about about situations where it's necessary/good to have it
If you have situations where you can detect that something is almost certainly going to fail (the breaker is tripped) you can save doing work unnecessarily or avoiding other potentially undesirable issues - for instance avoiding flooding your logs with millions of errors.
Whether or not you need such a thing is really going to depend on your specific circumstances
okay thanks ! I
@Mango (original) why did you decide to store them instead of just using Polly ?
We use Polly. We are using a fallback policy that has 5 retries
If all 5 retries fails the fallback is storing them until a connection is restored
Okay I see ! When you have a lot of events to send and also 5 retries each, doesn't that put a burden on memory ?
I mean, before acheiving the stade where you store them in db or something, before that everything is in memeory right ?
Yes every request will be in memory until its handled in some fashion
If thats a concern you can do a stress test benchmark, but its a non-issue with our setup. Some megabytes isn't gonna hurt anything for us
@Mango (original) How many requests/second do you have approximately ?
Really depends on what our users do. I don't expect more than 10 requests in a minute, it'd be really difficult. But if the service goes down for a hour that can be 600 stored requests. But once we store them they are living on the system not memory
we don't focus on tiny optimizations in early phases, we make something work then refine it. Works pretty well
okay thanks !