Cloudflare Developers•3mo ago

Is Cloudflare blocking my site?

If I use curl, with its default headers, I get a response saying that my site is unavailable (with a 404 response code). If I use curl, but change its headers to ones that match my normal browser, I get the response I expect.

$> # the following command returns 404 and references cloudflare's IP addresses in the response headers
$> curl -v https://beakpointinsights.com
$> # the following command works correctly and doesn't refer to cloudflare in the response headers
$> curl -v https://beakpointinsights.com/ -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0"

$> # the following command returns 404 and references cloudflare's IP addresses in the response headers
$> curl -v https://beakpointinsights.com
$> # the following command works correctly and doesn't refer to cloudflare in the response headers
$> curl -v https://beakpointinsights.com/ -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0"

I am on the free Cloudflare plan. My DNS settings are shown in the attachment. I have no custom WAF settings. Bot Fight Mode and Block AI Scrapers and Crawlers are both disabled.

17 Replies

Idle•3mo ago

seems to be a user issue, are you relying on curl resolving your hostname in any way?

1.1.1.1•3mo ago

DNS over Discord: A records

beakpointinsights.com A @1.1.1.1 +noall +answer

NAME                  | TTL  | DATA         
----------------------+------+--------------
beakpointinsights.com | 288s | 199.60.103.31

NAME                  | TTL  | DATA         
----------------------+------+--------------
beakpointinsights.com | 288s | 199.60.103.31

diggy diggy hole

Idle•3mo ago

if you disable proxying cloudflare doesn't do any blocking of sorts unless you are using the malware/family dns

AlanOP•3mo ago

I'm not doing anything special with curl, as far as I know. I started looking into this because semrush was telling me that they are unable to access my site.

Idle•3mo ago

did you have proxying enabled for your site before?

AlanOP•3mo ago

I've had that turned off forever, except for a few minutes today where I turned it on to try to debug the issue.

Idle•3mo ago

unfortunately, this does exactly what it says, if someone is unable to resolve your hostname the issue lies somewhere else. cloudflare status doesn't seem to be reporting any dns issues, but perhaps someone else knows something we don't :shrugpepe:

AlanOP•3mo ago

i'm not certain that it's a name resolution issue, since the behavior differs depending on what the user-agent header is in the request.

Idle•3mo ago

okay, i did some more testing. if i omit my user agent then i also get the 404 error when i try visiting your site, however the 404 response is coming from your server

Idle•3mo ago

so this is an issue on your end

AlanOP•3mo ago

okay perfect. thanks for helping me narrow it down. this is what confused me and pointed me in the wrong direction:

* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 404
< date: Wed, 28 Aug 2024 20:42:27 GMT
< content-type: text/html
< strict-transport-security: max-age=3628800; includeSubDomains; preload
< content-security-policy: upgrade-insecure-requests
< x-hs-reason: 404 predicted at edge
< set-cookie: __cf_bm=rLMjXOIqL6WeCRQNCucfwg8qIWU3KuAVzM__dWrdu9Y-1724877747-1.0.1.1-Y_L_kGZA5Yov_7.fpvuHqCpzNuQ65ixKtOCqnjpO4fjrgHDvYUg3kSjHPqDkMJmg2PUH4SNCpr2kNXHbLUD2pw; path=/; expires=Wed, 28-Aug-24 21:12:27 GMT; domain=.beakpointinsights.com; HttpOnly; Secure; SameSite=None
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=YtYjVpqHsFFzCftwOWBmETFCPpstzU0y9wOMDB1688j1gv94pgagUh69bWP%2FNkhIhCG2SjVIEJ4Wh6H%2BG3weponmcdmSGyY8Q1CMLXm3WdH%2FVgwaEgXXGTKjvrjFFn2uQmLj%2FOFd%2BQ%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}
< set-cookie: __cfruid=009dd54e61dd282dda679c0626fb8c1fde213d5a-1724877747; path=/; domain=.beakpointinsights.com; HttpOnly; Secure; SameSite=None
< server: cloudflare
< cf-ray: 8ba720c0dd374529-ATL
< alt-svc: h3=":443"; ma=86400

* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 404
< date: Wed, 28 Aug 2024 20:42:27 GMT
< content-type: text/html
< strict-transport-security: max-age=3628800; includeSubDomains; preload
< content-security-policy: upgrade-insecure-requests
< x-hs-reason: 404 predicted at edge
< set-cookie: __cf_bm=rLMjXOIqL6WeCRQNCucfwg8qIWU3KuAVzM__dWrdu9Y-1724877747-1.0.1.1-Y_L_kGZA5Yov_7.fpvuHqCpzNuQ65ixKtOCqnjpO4fjrgHDvYUg3kSjHPqDkMJmg2PUH4SNCpr2kNXHbLUD2pw; path=/; expires=Wed, 28-Aug-24 21:12:27 GMT; domain=.beakpointinsights.com; HttpOnly; Secure; SameSite=None
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=YtYjVpqHsFFzCftwOWBmETFCPpstzU0y9wOMDB1688j1gv94pgagUh69bWP%2FNkhIhCG2SjVIEJ4Wh6H%2BG3weponmcdmSGyY8Q1CMLXm3WdH%2FVgwaEgXXGTKjvrjFFn2uQmLj%2FOFd%2BQ%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}
< set-cookie: __cfruid=009dd54e61dd282dda679c0626fb8c1fde213d5a-1724877747; path=/; domain=.beakpointinsights.com; HttpOnly; Secure; SameSite=None
< server: cloudflare
< cf-ray: 8ba720c0dd374529-ATL
< alt-svc: h3=":443"; ma=86400

Idle•3mo ago

you can see that there is no Cf-Cache-Status header, meaning that the response is coming from your server

AlanOP•3mo ago

You have helped me out so much today. I am very, very grateful for your time.

Idle•3mo ago

no problem! if you want to replicate the 404 error in your browser, using firefox you can visit about:config and create a string value with the name general.useragent.override and just leave it empty. then you can visit your site and using ctrl + shift + i check the network tab for any errors. if you then click on the file that is served with the 404 response, and open the collapsed get request information, you can see the IP that the response is coming from. maybe this can help with any future errors too :shrugpepe:

AlanOP•3mo ago

i think it will! have a great rest of your day.

Gaming

Programming

Is Cloudflare blocking my site?