C#•2y ago

❔ HttpClient does not respect CancellationToken

App & Deployment Details Applications X and Y are DotnetCore 6.0 applications. X runs on K8s and Y on VMs on the cloud. Context of the problem I have two applications, X and Y. Y is a dumb app and it's job is to fetch data from various datasources. Y has certain REST styled endpoints. X has a logic to call the said endpoints of application Y. All calls are async and the use case needs the Http Post call to adhere to the cancellationToken. Problem The cancellationToken is set in this fashion

using var cts = CancellationTokenSource.CreateLinkedTokenSource(token);
  cts.CancelAfter(certainValue);

using var cts = CancellationTokenSource.CreateLinkedTokenSource(token);
  cts.CancelAfter(certainValue);

and used in the following manner

response = await _client.SendAsync(request, cancellationToken);

response = await _client.SendAsync(request, cancellationToken);

we also have logging on top of the httpClient call and it looks like so

var stopWatch = Stopwatch.StartNew();
try
{
  response = await PostAsync(request, cancellationToken);
}
catch
{
  ElapsedTime = stopWatch.ElapsedInMilliseconds();
}

var stopWatch = Stopwatch.StartNew();
try
{
  response = await PostAsync(request, cancellationToken);
}
catch
{
  ElapsedTime = stopWatch.ElapsedInMilliseconds();
}

Observations 1. The CancellationToken value is always set to 20s. From the ElapsedTime that is logged, we observed that the call does not timeout ~20s but in 1/10th of cases is >20s and has no pattern 2. The 20s timeout was changed to 30s and similar behaviour was observed. 3. Found this SO article describing the same problem Solutions Tried 1. Used the Polly.Timout but still observing the same issue

Stack Overflow

HttpClient cancellation doesn't kill underlying TCP call

I'm trying to set a default timeout for my HttpClient calls to 5 seconds. I've done this via CancellationTokenSource. Here's the pertinent bit of code: var cancellationToken = new

GitHub

Timeout

Polly is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and ...

15 Replies

MetalPuppyTheThirdOP•2y ago

Additional Details 1. HttpClient is initialised as a singleton 2. HttpClient.Timeout is set to 10mins 3. The Polly.Timeout() solution was implemented using the Optimistic Strategy as recommended in the readme More details on what is desired state - Ideally : If app X can tell Y to stop processing because X does not care anymore Not Ideal but can work : X can chop off the call to Y and Y can keep processing the request

Mayor McCheese•2y ago

Nothing comes to mind, you can inspect the source and see how the cancellation token is respected in httpclient.

MetalPuppyTheThirdOP•2y ago

thanks for the tip! from the SO, IMO this looks like a fairly common problem and yet there's no recommended solutions for this

DaVinki•2y ago

This might seem like a shot in the dark but may be worth trying

DaVinki•2y ago

Have you tried not creating a linked source and just passing the original token source instead? The documentation for it really only says it’s an observer of the passed token https://learn.microsoft.com/en-us/dotnet/api/system.threading.cancellationtokensource.createlinkedtokensource?view=net-7.0#system-threading-cancellationtokensource-createlinkedtokensource(system-threading-cancellationtoken)

CancellationTokenSource.CreateLinkedTokenSource Method (System.Thre...

Creates a CancellationTokenSource that will be in the canceled state when any of the source tokens are in the canceled state.

MetalPuppyTheThirdOP•2y ago

hrm, but the linkedsource cancellationRequested is tied to the one passed to it

DaVinki•2y ago

Just wrote a small app to test it and yeah, any cancellations to the created linked token are not forwarded to the original token used Unless it's not your problem, I'm not sure. I don't know if cts is the same as cancellationToken in the SendAsync method or PostAsync

MetalPuppyTheThirdOP•2y ago

yeah but I think that would not solve my problem. The token passed to the HttpClient is X. X must cancel after Y timespan. but it is not. Would like to point that the linkedTokenSource is needed because this call has to be capped to a certain value. My use case is the original token cannot be passed to this HttpClient call as the call has a different "max run time" setup in the config. the whole end to end process has allowed runtime of X timespan. This HttpClient call that I describe has allowed runtime of Y timespan. X > Y always

I don't know if cts is the same as cancellationToken in the SendAsync method or PostAsync

It is the same

DaVinki•2y ago

X and Y here are original and linked token?

MetalPuppyTheThirdOP•2y ago

apologies, let me elaborate in a simple way. My use case is : call n services in X time. X is the original cancellationToken. One of these n services is the one that I have described above. This service has to have it's own cancellationToken which must be < original cancellationToken so as to give other services their own time to run. The tokens that is passed to these services are created by doing a createLinkedTokenSource(originalToken) and cancelAfter(time_for_this_particular_service)

DaVinki•2y ago

And X should be canceled if Y times out?

MetalPuppyTheThirdOP•2y ago

no vice versa is true though The client requests to give a car in the response. There are 3 services, one to create the engine, one to create the body, and one to do the electricals. Client gives a 20s timeout to the app. The services, by historical data and agreement, are decided that they will complete their task in 5s each. Now, one of the service, a shitty service - electricals in this instance, takes 8s or 10s to do it's thing. Don't care why, but it takes that much time. Now I've passed a 5s timeout to that service -- which is not being respected. I create that 5s timeout by using linkedTokenSource. What is happening currently is, the runtime( whole app ) is >20s since that shitty service is not respecting the token. When it takes > cancellationToken, eventually it fails with a TaskCancelledException, post that, the application observes the original token is also marked cancelled so it too times out.

Unknown User•2y ago

Message Not Public

Mayor McCheese•2y ago

Much what tebeclone said it's a bit nebulous as to "when" or "if" in some cases you're waiting for code to get to a point where it says "oh I'm cancelled, okay" That's why I suggested you look at how it's respected.

Accord•2y ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

Gaming

Programming

❔ HttpClient does not respect CancellationToken