Custom Hostname DCV Delegation does not work (Pending Validation TXT) for domain with DNSSEC
is CF for SaaS DCV delegation for Custom Hostnames, not possible if the domain is not a zone in Cloudflare and has DNSSEC enabled (doesn't seem to have any misconfiguration)?
- TLD is
.us,
- DNSSEC is enabled but no issues as far as I can tell that could cause anything to be unreachable
- Registrar is Pairnic,
- CA that Cloudflare picked seems to be Google.
- Hostname did pre-validate and is active, but Certificate is stuck in Pending Validation (TXT).
- No CAAs in the domain/TLD
It's odd as the _acme-challenge.preview.exampledomain.us CNAME resolves correctly, which means that _acme-challenge-preview.exampledomain.us is issuing the correct TXT that should activate it.
This happens for the 3 hostnames we're attempting to add (preview. which does not exist at the moment, www. which does point to their current/previous site, and root which also does and redirects to www.)
Already opened a support ticket (Ticket 3290689) but I'm a bit fearful that if we don't solve it soon we'll end up getting rate-limited / temporarily banned (?) by the CA soon 😬
The automatic notifications have been stuck in DCV has failed (which they also tend to happen on successful validations but then work) - this time they've just been repeating that error for ~2 hours now instead of working after 2-3 minutes.
We've already activated tens of custom hostnames without any hiccups with this kind of setup (e.g. prevalidating both the cert with Delegated DCV and the hostname with the prevalidation TXT and only then moving the target CNAME)
Thank you!
EDIT: solved - pairdomains support was excellent and was able to discover something very-specific to their DNS system which they've been able to diagnose.
31 Replies
Hm it does seem to be related to DNSSEC. The odd thing is that...
-
dig _acme-challenge.preview.exampledomain.us TXT +dnssec @1.1.1.1 always works
- dig _acme-challenge.preview.exampledomain.us TXT +dnssec @8.8.8.8 intermittently works with the same response:
but sometimes dig _acme-challenge.preview.exampledomain.us TXT +dnssec @8.8.8.8 actually fails with SERVFAIL.
I'm definitely not familiar with DNSSEC enough from seeing DNSviz to understand whether...
1. DNSSEC simply must be disabled when using CF for SaaS DCV delegation OR...
2. DNSSEC fails for Google DNS sometimes because there's 1 DS record that use digest algorithm 1 (SHA-1) or some other misconfiguration and fixing it would work?DNSSEC simply must be disabled when using CF for SaaS DCV delegation OR...No, DNSSEC is completely fine
DNSSEC fails for Google DNS sometimes because there's 1 DS record that use digest algorithm 1 (SHA-1) or some other misconfiguration and fixing it would work?Yea, it looks like they have two DS records at their Registrar
Hey Chaika! thanks for your response. That makes sense, and Cloudflare's 1.1.1.1 just ignores the digest alg=1 one whilst Google might sometime use it and fail from there?
If their DNSSEC wasn't broken, even though our Cloudflare Zone where we have Custom Hostnames enabled does not currently use DNSSEC, this setup should still work and be OK?
also - thanks for your help on this discord in the past, our team learned a ton on how everything worked for Custom Hostnames just by seeing your responses to other people's questions
your zone doesn't matter at all, the cname is to dcv.cloudflare.com which is DNSSEC. but yea, you can point a zone with dnssec via cname to a zone without it, that's no issue. DNSSEC is enforced by a hierachy, a CNAME redoes the entire hierarchy (ex: root is trusted by known certs, com is trusted by root, website.com is trusted by com, or website.com doesn't have dnssec so com serves a trusted nsec record signing that there's no dnssec configured)
That's what dnsviz is trying to show you, there's two complete trees for the different domains
excellent - that makes a lot of sense now, thank you for confirming it! I'm still somewhat confused as to why Google DNS would pick the SHA-1 if in theory the error message says everyone would just ignore it.
but I'll see if the registrar can fix their DS records or alternatively temporarily disable DNSSEC
It failing on Google's DNS and the CA being GTS are related is a safe bet
but yea I'm not sure why CF/Google DNS would have different behavior on that. Sometimes Extended DNS messages can help, but in this case all it's giving is
EDE: 10 (RRSIGs Missing): (For ./soa)
it does only say "might" be ignored for the other type
yeah and when it works I do consistently see the
`_acme-challenge.preview.exampledomain.us. 1579 IN RRSIG CNAME 13 4
so maybe it does just depend on what DS algo appears first, Cloudflare ignores SHA-1 but Google doesn't, if it follows that chain it stops at exampledomain.us
although that's also obscure to me - from the screenshot I'd assume the intermittent failures would affect for example the A reocrd on www.exampledomain.us - but it does not, I can only replicate when going through the Delegated DCV CNAME
yea that error message is just weird too "for ./soa", which zone's Soa! lol
The Key tag on both the DS records is right, so I guess it could find the DNSKEY but then gets an RRSIG back with the wrong algo and dies
yeah - let's see if their registrar support is helpful / customer's willing to temporarily disable DNSSEC* (as I understand renewal happens via HTTP DCV anyways, I'm just wanting to avoid any downtime on the switch over). Just when you think you're comfortable with DNS, a rogue DNSSEC intedeterministic issue pops up 😂
many, many thanks Chaika! really appreciate your time and support here!
Of course! Just in case it's worth mentioning: You don't want the customer to deletethe DNSKEY record, just the DS records at the registrar (or at least the bad one). Removing the DS Records at Registrar will make the registry/tld respond with NSEC, saying there's no dnssec configured. It's fine to keep the DNSKEY records being served from auth. dns. However if you deleted the DNSKEY records/disable dnssec in auth dns without removing the DS records, you'd cause resolution to fail/all responses to be seen as bogus
* * * (as I understand renewal happens via HTTP DCV anyways, I'm just wanting to avoid any downtime on the switch overIf you pick TXT it'll continue to renew as TXT. That's the point/the good thing about the DCV Delegation, since you have them create that CNAME CF can keep creating new TXTs to validate certs renewal. It's more reliable then http too, as no firewalls/security stuff can get in the way of it (although afaik CF has made that harder to happen these days)
haha sorry yeah I re-read my message whilst you were typing and realized that literally that option would be the absolute worst possible thing to do 😂 . Cheers for mentioning it, hope I didn't freak you out there
noted on the renewal - so we'll have to figure this one out, but I guess we'll all learn something out of it!
The DS record for SHA-1 has been removed a few hours ago and seems to be correctly updated everywhere. Sadly I can still replicate the issue where Google DNS fails sometimes for
dig _acme-challenge.preview.exampledomain.us TXT +dnssec @8.8.8.8 (already flushed the cache for Google DNS in https://developers.google.com/speed/public-dns/cache and can also be seen using dns.google) https://dns.google/resolve?name=_acme-challenge.preview.exampledomain.us&type=TXT&do=true
DNSviz initially seems to say everything's OK. Although we've run it with Advanced Options: Extra Types TXT and:
_acme-challenge.preview.exampledomain.us/A has errors; select the "Denial of existence" DNSSEC option to see them. appears as an error (nothing really appears in the graph)
https://dnsviz.net/d/_acme-challenge.preview.exampledomain.us/ZmMsrA/dnssec/?rr=all&a=all&ds=all&ta=.&tk=
(EDIT: original domain was replaced)
Sadly, when enabling that option the error dissappears instead of showing more information 🤔 - still not sure I understand why this appears when adding extra types TXT, as CF doesn't serve any A record there through the CNAMEthat's really curious, can you reproduce the same failure with any other custom hostnames you've delegated dcv for?
none - but this is the first one that uses DNSSEC outside of Cloudflare unfortunately
It's also quite interesting to me that both hierarchies ( _acme-challenge.preview.exampledomain.us CNAME as well as preview.exampledomain.us.6f0d68e920b655b1.dcv.cloudflare.com TXT) seems to individually have no DNSSEC issues for Google - but the full chain does intermittently
need more info from google on the failure or someone who knows more about dnssec probably
another potential solution though: do you have enterprise/the ability to pick a diff CA?
or maybe even if you haven't done any setup, recreate until you get a different CA then GTS/get LE
not an enterprise zone sadly! it did cross my mind to re-add them and see if I got letsencrypt assigned 😂
yea I just don't know who's side this is on. It could be a mix even,of google doing something weird and the other side not supporting it. Your customer has dnssec through some weird dns provider too, could be an edge case in their setup
reason #159854 to not like/use dnssec
I'm right there with you! Always kind of avoided DNSSEC from the war stories – so far the registrar is being really responsive and helpful. I'll see if the Cloudflare Support ticket gets assigned to someone that has any further things worth trying
Another interesting bit: all our hostnames picked GTS (~80 created in the last 3 weeks), I thought it was a more uniform split
could be split by zone
would match up with the behavior I've seen w/ Pages Custom domains in the past
or maybe just more complex then a random in some other way
Ah that makes sense! I'll start playing with HTTP validation as a plan B too, but it's the kind of thing that you don't want to run into a blocking issue whilst the service is down
something curious I've noticed with their dns provider is that when you ask for the SOA of
_acme-challenge.preview.exampledomain.us., it responds with the CNAME
;_acme-challenge.preview.exampledomain.us. IN SOA ;; ANSWER SECTION: _acme-challenge.preview.exampledomain.us. 3499 IN CNAME preview.exampledomain.us.6f0d68e920b655b1.dcv.cloudflare.com. ;; AUTHORITY SECTION: dcv.cloudflare.com. 1699 IN SOA graham.ns.cloudflare.com. dns.cloudflare.com. 2343262667 10000 2400 604800 1800which is not the behavior of a cname on Cloudflare
;; QUESTION SECTION: ;test-simple-proxied.tylerobrien.dev. IN SOA ;; AUTHORITY SECTION: tylerobrien.dev. 1639 IN SOA pineapple.tylerobrien.dev. dns.cloudflare.com. 2343261772 10000 2400 604800 1800
but isn't that the reason why CNAMEs dont exist at @ in the spec? I'll try a few examples, it's a good shout
that the soa record needs to exist? That was my understanding yea
idk what is the right behavior or not, but a curious difference
yeah exactly - also not sure but it's something worth playing with. Thanks for helping debug this Chaika! I'll make sure to post here when ever we get to the bottom of it
please do! It'd be curious what it ends up being
the nsec/denial of existence thing you pointed out is curious too, CF does respond with an nsec on getting the soa of
preview.intentionallyblank.us.6f0d68e920b655b1.dcv.cloudflare.com, because the soa is on dcv.cloudflare.com, and it does respond with that soa with the rrsigs for it in the same response
wish google would just tell you where the rrsigs issue is. It's not even possible to test the specific edge case I pointed out above with dnsviz as far as I know because it just follows the CNAME/impossible to tell it to query the soa record on _acme-challenge specificallyI poked around a bit more and it looks like CF only does this when the cname is to the same zone, it still upsets dnsviz though lol
that makes sense, I was seeing and in general it seems that a CNAME takes over the SOA for that hostname - but you're right that the extended DNS error code hints at there being a problem there
Hi! We ended up validating using HTTP validation instead to work around, but pairdomains/pairnic support was excellent and found out the issue was something unique to their DNS system implementation. Thank you, Chaika, for your time and support! We've definitely learned a few things throughout this one.
The root cause of the issue (I've edited my messages to exampledomain.us now):
When our custom DNS system was created, the name servers were set to give a default answer to any A record query where we don't have a specific zone file for the domain. So if a domain has DNS set up, we give the normal answer for it. If the domain doesn't have DNS set up, we answer with the IP address of a placeholder server (216.92.3.120). [...] What happens is the query to _acme-challenge.preview.exampledomain.us reaches us and we respond with an answer that it is a CNAME to preview.exampledomain.us.6f0d68e920b655b1.dcv.cloudflare.com. If the query doesn't use DNSSEC, the CNAME is followed and the query is successful. If the query uses DNSSEC and it checks for an A record, the CNAME is followed and the A record (or lack of record) is given and the query is successful. If the query uses DNSSEC and it checks for any other record type, the querying server can make a followup query asking for the A record of the target. If there is an A record for the target, the A record is given and the query is successful. This problem happens because the querying server cant find an A record for preview.exampledomain.us.6f0d68e920b655b1.dcv.cloudflare.com, so they query us and ask for the IP address. We respond with 216.92.3.120. We give an IP but Cloudflare correctly says there is no A record. The disagreement causes the querying server to respond that the DNSSEC check failed.
🤔 That's interesting, I blame DNSSEC still.
That explanation is a bit curious though because it was Google that had issues querying, not Cloudflare, and Google specifically said on the SOA, although if it was issues with any other query type then A, then makes some sense
well eitherway happy to hear you got to the bottom of it/found a way around it, I edited the original domain name out of my responses as well
(you also missed editing the domain out of the answer section of this message, https://discord.com/channels/595317990191398933/1248323742938173512/1248369165614448730, if you were trying to avoid it being indexed)
Discord
Discord - Group Chat That’s All Fun & Games
Discord is great for playing games and chilling with friends, or even building a worldwide community. Customize your own space to talk, play, and hang out.