C#•17mo ago

✅ ✅ C# HttpClient returning 400, but other tools return 200

I am trying to make a simple web request to a URL using .NET/C# HttpClient. I am getting a 400 error "Bad Request". Whereas everything else such as Python Requests, Postman, and the Web browser dev tools all return a 200 Success. The URL is http://ericsmithrocks.com. This is a random URL I found that exhibits this behavior. Other URLs seem to work as expected. I have experimented with replacing "http" with "https" and adding an HttpClientHandler with different settings, with no success. I am thinking it may be something to do with my headers or an http permanent redirect issue. Any help would be greatly appreciated. I have attached pictures of all my code / tests. Thanks.

ERIC SMITH

11 Replies

dotnet123123OP•17mo ago

 c#
var client = new HttpClient();
var request = await client.GetAsync("http://www.ericsmithrocks.com");
var code = request.StatusCode;

 c#
var client = new HttpClient();
var request = await client.GetAsync("http://www.ericsmithrocks.com");
var code = request.StatusCode;

 python
import requests
r = requests.get("http://www.ericsmithrocks.com")
r.status_code

 python
import requests
r = requests.get("http://www.ericsmithrocks.com")
r.status_code

nukleer bomb•17mo ago

very strange litteraly the same code WorksOnMyMachine

version of .NET you are using?

dotnet123123OP•17mo ago

.net 6

canton7•17mo ago

For whatever reason, that website wants you to have a User-Agent header:

client.DefaultRequestHeaders.UserAgent.Add(new("Test", "1.2.3"));

client.DefaultRequestHeaders.UserAgent.Add(new("Test", "1.2.3"));

HimmDawg•17mo ago

That's the deal with a couple of websites. From what I heard, back in the day websites claimed that you'd require a specific browser. idk if that was a necessacity because of technical reasons or simply marketing shrug

people went ahead and spoofed their user agent

dotnet123123OP•17mo ago

thank you @canton7 , you're a lifesaver. Can I ask how you figured that out? Did you just happen to know about the weird history that HImmDawg mentioned above?

canton7•17mo ago

I've seen exactly this problem before, so it's the first thing I tried https://webaim.org/blog/user-agent-string-history/ is worth a read But I don't think it's directly related -- it's just that some servers want to see a user agent, for whatever reason

dotnet123123OP•17mo ago

lol nice. I had been quite stumped for awhile. Don't know if I ever would have solved that naturally on my own. Didn't find anything googling online

canton7•17mo ago

https://stackoverflow.com/questions/24274789/is-lack-of-user-agent-in-http-request-valid The thing to do when you have a working case and a non-working case is to find the difference between them In this case, you need to work out the difference. You could do that by sniffing the request that each makes (fiddler used to be the go-to tool before they went commercial, not sure what the replacement is), going all the way and using wireshark, or making requests to a server which you can control, e.g. requestbin Eventually you get it down so that the only difference between the working request and the non-working one is the user-agent, and adding in the user-agent fixes it. Now you know what the problem is

dotnet123123OP•17mo ago

Thanks again. I will read those 2 links So, this fixed my issue, but I'm wondering why the .NET HttpClient is seemingly defaulting to an empty agent string in the header, when from what I can tell, most other mainstream libraries/languages put a default value in.

Accord•17mo ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

Gaming

Programming

✅ ✅ C# HttpClient returning 400, but other tools return 200