Cloudflare Developers•3mo ago

Authenticated Origin Pull Not working

I followed this article: https://developers.cloudflare.com/ssl/origin-configuration/authenticated-origin-pull/set-up/zone-level/

    ssl_certificate /etc/ssl/nginx/cert.pem;
    ssl_certificate_key /etc/nginx/clf/key.pem;
    ssl_verify_client on;
   ssl_client_certificate /etc/nginx/clf/cloudflare.crt;

    ssl_certificate /etc/ssl/nginx/cert.pem;
    ssl_certificate_key /etc/nginx/clf/key.pem;
    ssl_verify_client on;
   ssl_client_certificate /etc/nginx/clf/cloudflare.crt;

Spend entire day to figure this out only getting

400 Bad Request
No required SSL certificate was sent
nginx

400 Bad Request
No required SSL certificate was sent
nginx

Cloudflare Docs

Zone-level | Cloudflare SSL/TLS docs

When you enable Authenticated Origin Pulls for a zone, all proxied traffic to your zone is authenticated at the origin web server.

31 Replies

PriyashOP•3mo ago

https://interlacekit.com/ The strange part is the authenticated pull works fine on my other domain which do not even show origin certificates in dashboard but has authenticated pull enabled If I disable the ssl_verify_client on; everything works fine

Chaika•3mo ago

It "works fine" for the same reason why you can enable AOP (Auth. Origin Pulls) without breaking anything, just enabling it in Cloudflare doesn't do anything other then start serving it, and then your origin has to be specifically enabled tjo start verifying it What do you mean "origin certs"? Origin Certs are completely seperate from AoP, did you download the certificate on that guide and which setup did you use?

PriyashOP•3mo ago

Thanks for quick revert. Can you help understand how it works. My current understanding is:

ssl_certificate /etc/ssl/nginx/cert.pem; // im putting here origin ca certificate  - https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/
ssl_certificate_key /etc/nginx/clf/key.pem; // and the rsa key used for creating CSR

ssl_verify_client on;

ssl_client_certificate /etc/nginx/clf/cloudflare.crt;  // Cloudflare CA for auth pull verification as specified : https://developers.cloudflare.com/ssl/origin-configuration/authenticated-origin-pull/set-up/zone-level/#1-upload-certificate-to-origin

ssl_certificate /etc/ssl/nginx/cert.pem; // im putting here origin ca certificate  - https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/
ssl_certificate_key /etc/nginx/clf/key.pem; // and the rsa key used for creating CSR

ssl_verify_client on;

ssl_client_certificate /etc/nginx/clf/cloudflare.crt;  // Cloudflare CA for auth pull verification as specified : https://developers.cloudflare.com/ssl/origin-configuration/authenticated-origin-pull/set-up/zone-level/#1-upload-certificate-to-origin

Chaika•3mo ago

You're just using the shared Auth. Origin Pull cert and enabling in the dash, you're not uploading your own/used api to enable?

PriyashOP•3mo ago

I first tried using my cert CSR and KEY via terraform to issue Origin CA Certificate. But currently trying to do it manually using Cloudflare's managed CSR and KEY to issue it Also for Origin Pull i did upload the the CA cert from CLF to my server. Also enabled the tls client auth via API.

{
    "success": true,
    "errors": [],
    "messages": [],
    "result": {
        "enabled": true
    }
}

{
    "success": true,
    "errors": [],
    "messages": [],
    "result": {
        "enabled": true
    }
}

Chaika•3mo ago

The origin CA cert stuff is unlikely to be your issue unless your SSL/TLS Mode is "Flexible" or "Off" Should be Full (Strict)

PriyashOP•3mo ago

The SSL mode is strict. Again if i disable the verifiy client on nginx it works fine

Chaika•3mo ago

You can enable nginx custom log format to log the mTLS/Client cert as well, something like:

        log_format combined_sslclient '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$ssl_client_s_dn($ssl_client_serial)" $ssl_client_fingerprint ';

        access_log /var/log/nginx/access.log combined_sslclient;

        log_format combined_sslclient '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$ssl_client_s_dn($ssl_client_serial)" $ssl_client_fingerprint ';

        access_log /var/log/nginx/access.log combined_sslclient;

so you don't need to break your site in order to see if CF is serving the cert or not

PriyashOP•3mo ago

okay let me try this.

Chaika•3mo ago

btw you'd want ssl_verify_client optional; so the cert gets requested and verified if it does exist

PriyashOP•3mo ago

172.69.87.85 - - [25/Aug/2024:19:02:02 +0000] "GET / HTTP/1.1" 400 632 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "-(-)" - 
172.69.87.20 - - [25/Aug/2024:19:02:03 +0000] "GET /favicon.ico HTTP/1.1" 400 632 "https://interlacekit.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "-(-)" - 
172.69.94.123 - - [25/Aug/2024:19:02:42 +0000] "GET / HTTP/1.1" 400 632 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "-(-)" - 
172.69.95.39 - - [25/Aug/2024:19:03:54 +0000] "GET / HTTP/1.1" 200 9367 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "-(-)" -

172.69.87.85 - - [25/Aug/2024:19:02:02 +0000] "GET / HTTP/1.1" 400 632 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "-(-)" - 
172.69.87.20 - - [25/Aug/2024:19:02:03 +0000] "GET /favicon.ico HTTP/1.1" 400 632 "https://interlacekit.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "-(-)" - 
172.69.94.123 - - [25/Aug/2024:19:02:42 +0000] "GET / HTTP/1.1" 400 632 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "-(-)" - 
172.69.95.39 - - [25/Aug/2024:19:03:54 +0000] "GET / HTTP/1.1" 200 9367 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "-(-)" -

not getting anything

Chaika•3mo ago

would make sense why you got that error in the dash you see AoP enabled for that specific zone? You've never used the per-hostname api/config?

PriyashOP•3mo ago

PriyashOP•3mo ago

Oh.. should I be setting up per host instead of zone level?

Chaika•3mo ago

no but it's an option which would override this should be settings/tls_client_auth not origin_tls_client_auth, curious

PriyashOP•3mo ago

is it possible i might have override some zone level settings with terraform settings overrides? let me try this one

Chaika•3mo ago

in a few different ways. configuration rules can override ssl mode for example

Chaika•3mo ago

/zones/<zoneId>/settings/tls_client_auth is the normal universal one /zones/{zone_id}/origin_tls_client_auth/ is the sub-path all the per hostname settings are under. You could try seeing if you have overriden it for your apex https://developers.cloudflare.com/api/operations/per-hostname-authenticated-origin-pull-get-the-hostname-status-for-client-authentication and if you have any custom ones https://developers.cloudflare.com/api/operations/per-hostname-authenticated-origin-pull-list-certificates

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

PriyashOP•3mo ago

Noted Correct me If i'm wrong. Regardless of ssl certificates configuration on my server I should see Origin Pull logs in nginx logs. Or do they matter only for CA cert?

Chaika•3mo ago

I don't understand the question "regardless of ssl cert configuration"? AoP is just mTLS so as the bare min your origin web server needs to be requesting the client cert I was specifically talking about ssl mode because Off/Flexible is HTTP (not https) and Flexible is the semi-default, and of course without https not going to get a client cert

PriyashOP•3mo ago

Got it. So if strict mode is working. then there some miss configuration at Zone level? The same verification CA cert is working with other domain.

Chaika•3mo ago

it's working with another domain in the sense that the "other domains" web server is verifying the mtls cert (ssl_verify_client) and you can see it in logs?

PriyashOP•3mo ago

yes its working with other domain

Chaika•3mo ago

cool, then you have a working setup to compare to. Yea I'd check out the config fully, could try running the url through the Trace tool account level and see what you match

PriyashOP•3mo ago

Im trying over again manually to replicate the config No its not working. Something is wrong with zone settings. Is there any way to reset the zone

Chaika•3mo ago

no way to reset without removing and waiting 7 days. Did you try tracing like I said above, and the get hostname config/certs for aop endpoints?

PriyashOP•3mo ago

trying trace. don't want to setup per host but will give it try as well

172.68.22.12 - - [25/Aug/2024:21:27:44 +0000] "GET /privacy-policy HTTP/1.1" 200 16289 "-" "Cloudflare Trace" "-(-)" - 
162.158.41.251 - - [25/Aug/2024:21:30:21 +0000] "GET /privacy-policy HTTP/1.1" 200 16290 "-" "Cloudflare Trace" "-(-)" -

172.68.22.12 - - [25/Aug/2024:21:27:44 +0000] "GET /privacy-policy HTTP/1.1" 200 16289 "-" "Cloudflare Trace" "-(-)" - 
162.158.41.251 - - [25/Aug/2024:21:30:21 +0000] "GET /privacy-policy HTTP/1.1" 200 16290 "-" "Cloudflare Trace" "-(-)" -

Chaika•3mo ago

I mean look at the overall flow, did it hit any config rules overriding ssl/any specific rules of interest?

PriyashOP•3mo ago

no there's no rule hit in trace

PriyashOP•3mo ago

After configuring per host the aop requests are working.

172.69.179.80 - - [26/Aug/2024:10:32:27 +0000] "GET /favicon.ico HTTP/1.1" 400 610 "https://interlacekit.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "CN=CloudFlare Origin Certificate,OU=CloudFlare Origin CA,O=CloudFlare\x5C, Inc.(0779EA24E40DF4E05099ADCBE7CF2B887BB4A324)" bd0ae53fc0f1cbf37ac5b7ec3c999b1755bda305 
172.69.179.55 - - [26/Aug/2024:10:32:30 +0000] "GET / HTTP/1.1" 400 610 "https://interlacekit.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "CN=CloudFlare Origin Certificate,OU=CloudFlare Origin CA,O=CloudFlare\x5C, Inc.(0779EA24E40DF4E05099ADCBE7CF2B887BB4A324)" bd0ae53fc0f1cbf37ac5b7ec3c999b1755bda305 
172.69.179.79 - - [26/Aug/2024:10:32:30 +0000] "GET /favicon.ico HTTP/1.1" 400 610 "https://interlacekit.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "CN=CloudFlare Origin Certificate,OU=CloudFlare Origin CA,O=CloudFlare\x5C, Inc.(0779EA24E40DF4E05099ADCBE7CF2B887BB4A324)" bd0ae53fc0f1cbf37ac5b7ec3c999b1755bda305

172.69.179.80 - - [26/Aug/2024:10:32:27 +0000] "GET /favicon.ico HTTP/1.1" 400 610 "https://interlacekit.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "CN=CloudFlare Origin Certificate,OU=CloudFlare Origin CA,O=CloudFlare\x5C, Inc.(0779EA24E40DF4E05099ADCBE7CF2B887BB4A324)" bd0ae53fc0f1cbf37ac5b7ec3c999b1755bda305 
172.69.179.55 - - [26/Aug/2024:10:32:30 +0000] "GET / HTTP/1.1" 400 610 "https://interlacekit.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "CN=CloudFlare Origin Certificate,OU=CloudFlare Origin CA,O=CloudFlare\x5C, Inc.(0779EA24E40DF4E05099ADCBE7CF2B887BB4A324)" bd0ae53fc0f1cbf37ac5b7ec3c999b1755bda305 
172.69.179.79 - - [26/Aug/2024:10:32:30 +0000] "GET /favicon.ico HTTP/1.1" 400 610 "https://interlacekit.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "CN=CloudFlare Origin Certificate,OU=CloudFlare Origin CA,O=CloudFlare\x5C, Inc.(0779EA24E40DF4E05099ADCBE7CF2B887BB4A324)" bd0ae53fc0f1cbf37ac5b7ec3c999b1755bda305

Now getting 400 Bad Request The SSL certificate error nginx Wooho.. Go it working.. Thanks @Chaika for the help.. 🙌

Gaming

Programming

Authenticated Origin Pull Not working