Custom Hostname DCV Delegation does not work (Pending Validation TXT) for domain with DNSSEC
is CF for SaaS DCV delegation for Custom Hostnames, not possible if the domain is not a zone in Cloudflare and has DNSSEC enabled (doesn't seem to have any misconfiguration)?
- TLD is
.us,
- DNSSEC is enabled but no issues as far as I can tell that could cause anything to be unreachable
- Registrar is Pairnic,
- CA that Cloudflare picked seems to be Google.
- Hostname did pre-validate and is active, but Certificate is stuck in Pending Validation (TXT).
- No CAAs in the domain/TLD
It's odd as the _acme-challenge.preview.exampledomain.us CNAME resolves correctly, which means that _acme-challenge-preview.exampledomain.us is issuing the correct TXT that should activate it.
This happens for the 3 hostnames we're attempting to add (preview. which does not exist at the moment, www. which does point to their current/previous site, and root which also does and redirects to www.)
Already opened a support ticket (Ticket 3290689) but I'm a bit fearful that if we don't solve it soon we'll end up getting rate-limited / temporarily banned (?) by the CA soon 😬
The automatic notifications have been stuck in DCV has failed (which they also tend to happen on successful validations but then work) - this time they've just been repeating that error for ~2 hours now instead of working after 2-3 minutes.
We've already activated tens of custom hostnames without any hiccups with this kind of setup (e.g. prevalidating both the cert with Delegated DCV and the hostname with the prevalidation TXT and only then moving the target CNAME)
Thank you!
EDIT: solved - pairdomains support was excellent and was able to discover something very-specific to their DNS system which they've been able to diagnose.
31 Replies
Hm it does seem to be related to DNSSEC. The odd thing is that...
-
dig _acme-challenge.preview.exampledomain.us TXT +dnssec @1.1.1.1 always works
- dig _acme-challenge.preview.exampledomain.us TXT +dnssec @8.8.8.8 intermittently works with the same response:
but sometimes dig _acme-challenge.preview.exampledomain.us TXT +dnssec @8.8.8.8 actually fails with SERVFAIL.
I'm definitely not familiar with DNSSEC enough from seeing DNSviz to understand whether...
1. DNSSEC simply must be disabled when using CF for SaaS DCV delegation OR...
2. DNSSEC fails for Google DNS sometimes because there's 1 DS record that use digest algorithm 1 (SHA-1) or some other misconfiguration and fixing it would work?DNSSEC simply must be disabled when using CF for SaaS DCV delegation OR...No, DNSSEC is completely fine
DNSSEC fails for Google DNS sometimes because there's 1 DS record that use digest algorithm 1 (SHA-1) or some other misconfiguration and fixing it would work?Yea, it looks like they have two DS records at their Registrar
Hey Chaika! thanks for your response. That makes sense, and Cloudflare's 1.1.1.1 just ignores the digest alg=1 one whilst Google might sometime use it and fail from there?
If their DNSSEC wasn't broken, even though our Cloudflare Zone where we have Custom Hostnames enabled does not currently use DNSSEC, this setup should still work and be OK?
also - thanks for your help on this discord in the past, our team learned a ton on how everything worked for Custom Hostnames just by seeing your responses to other people's questions
your zone doesn't matter at all, the cname is to dcv.cloudflare.com which is DNSSEC. but yea, you can point a zone with dnssec via cname to a zone without it, that's no issue. DNSSEC is enforced by a hierachy, a CNAME redoes the entire hierarchy (ex: root is trusted by known certs, com is trusted by root, website.com is trusted by com, or website.com doesn't have dnssec so com serves a trusted nsec record signing that there's no dnssec configured)
That's what dnsviz is trying to show you, there's two complete trees for the different domains
excellent - that makes a lot of sense now, thank you for confirming it! I'm still somewhat confused as to why Google DNS would pick the SHA-1 if in theory the error message says everyone would just ignore it.
but I'll see if the registrar can fix their DS records or alternatively temporarily disable DNSSEC
It failing on Google's DNS and the CA being GTS are related is a safe bet
but yea I'm not sure why CF/Google DNS would have different behavior on that. Sometimes Extended DNS messages can help, but in this case all it's giving is
EDE: 10 (RRSIGs Missing): (For ./soa)
it does only say "might" be ignored for the other type
yeah and when it works I do consistently see the
`_acme-challenge.preview.exampledomain.us. 1579 IN RRSIG CNAME 13 4
so maybe it does just depend on what DS algo appears first, Cloudflare ignores SHA-1 but Google doesn't, if it follows that chain it stops at exampledomain.us
although that's also obscure to me - from the screenshot I'd assume the intermittent failures would affect for example the A reocrd on www.exampledomain.us - but it does not, I can only replicate when going through the Delegated DCV CNAME
yea that error message is just weird too "for ./soa", which zone's Soa! lol
The Key tag on both the DS records is right, so I guess it could find the DNSKEY but then gets an RRSIG back with the wrong algo and dies
yeah - let's see if their registrar support is helpful / customer's willing to temporarily disable DNSSEC* (as I understand renewal happens via HTTP DCV anyways, I'm just wanting to avoid any downtime on the switch over). Just when you think you're comfortable with DNS, a rogue DNSSEC intedeterministic issue pops up 😂
many, many thanks Chaika! really appreciate your time and support here!
Of course! Just in case it's worth mentioning: You don't want the customer to deletethe DNSKEY record, just the DS records at the registrar (or at least the bad one). Removing the DS Records at Registrar will make the registry/tld respond with NSEC, saying there's no dnssec configured. It's fine to keep the DNSKEY records being served from auth. dns. However if you deleted the DNSKEY records/disable dnssec in auth dns without removing the DS records, you'd cause resolution to fail/all responses to be seen as bogus
* * * (as I understand renewal happens via HTTP DCV anyways, I'm just wanting to avoid any downtime on the switch overIf you pick TXT it'll continue to renew as TXT. That's the point/the good thing about the DCV Delegation, since you have them create that CNAME CF can keep creating new TXTs to validate certs renewal. It's more reliable then http too, as no firewalls/security stuff can get in the way of it (although afaik CF has made that harder to happen these days)
haha sorry yeah I re-read my message whilst you were typing and realized that literally that option would be the absolute worst possible thing to do 😂 . Cheers for mentioning it, hope I didn't freak you out there
noted on the renewal - so we'll have to figure this one out, but I guess we'll all learn something out of it!
The DS record for SHA-1 has been removed a few hours ago and seems to be correctly updated everywhere. Sadly I can still replicate the issue where Google DNS fails sometimes for
dig _acme-challenge.preview.exampledomain.us TXT +dnssec @8.8.8.8 (already flushed the cache for Google DNS in https://developers.google.com/speed/public-dns/cache and can also be seen using dns.google) https://dns.google/resolve?name=_acme-challenge.preview.exampledomain.us&type=TXT&do=true
DNSviz initially seems to say everything's OK. Although we've run it with Advanced Options: Extra Types TXT and:
_acme-challenge.preview.exampledomain.us/A has errors; select the "Denial of existence" DNSSEC option to see them. appears as an error (nothing really appears in the graph)
https://dnsviz.net/d/_acme-challenge.preview.exampledomain.us/ZmMsrA/dnssec/?rr=all&a=all&ds=all&ta=.&tk=
(EDIT: original domain was replaced)
Sadly, when enabling that option the error dissappears instead of showing more information 🤔 - still not sure I understand why this appears when adding extra types TXT, as CF doesn't serve any A record there through the CNAMEthat's really curious, can you reproduce the same failure with any other custom hostnames you've delegated dcv for?
none - but this is the first one that uses DNSSEC outside of Cloudflare unfortunately
It's also quite interesting to me that both hierarchies ( _acme-challenge.preview.exampledomain.us CNAME as well as preview.exampledomain.us.6f0d68e920b655b1.dcv.cloudflare.com TXT) seems to individually have no DNSSEC issues for Google - but the full chain does intermittently
need more info from google on the failure or someone who knows more about dnssec probably