Cloudflare Developers•5d ago

Unexpected 404 errors - Possible O2O issue

Starting last week, we started observing occasional 404 errors for requests that definitely should not have resulted in 404s. We're quite sure the 404s do not originate from our app: the failed requests do not appear in our application logs, and the actual response payload (a simple "404 page not found" plain text string) is not what our app returns for expected 404 errors. We've reached out to our hosting provider, Render.com. Render.com also uses Cloudflare, and their support mentioned that this could be a possible O2O issue. We tentatively disabled Cloudflare proxying for our app's hostname and the issue went away. However this is not a viable long term solution for us (we want to be able to set our own WAF rules, etc.). Render.com is adament the problem is not on their end. I've opened a ticket with Cloudflare last week (request #01312293) but haven't heard anything back yet. Additionally, Cloudflare's support ticket system seems to be having issues and I can't check the ticket's status. I'd really appreciate it if a Cloudflare support agent could get in touch with me and help investigate this issue. Thanks! 🙏

22 Replies

Chaika•5d ago

When you get 404s, did you see cf-cache-status HIT on them? Is the record targeting Render a CNAME on apex or subdomain, or an A/AAAA record (which don't O2O)?

OlivierOP•5d ago

The record is a CNAME on a subdomain. (We've observed the issue on multiple subdomains and domains.) All the 404s where we've managed to capture headers have 'cf-cache-status': 'DYNAMIC' .

Chaika•5d ago

The record is a CNAME on a subdomain. (

So it could be O2O if Proxied as well

All the 404s where we've managed to capture headers have 'cf-cache-status': 'DYNAMIC' .

so it's not cache. That'd be an easy one. Lots of other things run in the middle though. Account level you have a "trace" tool, magic link: https://dash.cloudflare.com/?to=/:account/trace/search. I believe only works if it's actually proxied though. But I'd jump to it being a worker route or a snippet intercepting potentially. Not too many products can return 404s in a non-determistic way though

OlivierOP•5d ago

I can provide some ray IDs if that's helpful?

Chaika•5d ago

Not helpful to me, not a Cloudflare employee. Realistically not helpful to Cloudflare employees either due to sampling and such Worth noting w/ O2O, CF-Cache-Status is of the last zone executing. That is, with O2O enabled you see cache status of your own zone, with disabled you see of render's zone (if it's going through the cache layer)

OlivierOP•5d ago

Gotcha, thanks. I'll check out the trace tool.

Chaika•5d ago

If you post on the community forums: https://community.cloudflare.com/ including ticket number can throw the ticket in escalation queue from there if you want to go that way, as long as you're biz or pro plan on a domain in that acct. (can't escalate non-dev platform issues directly in Discord, and community is just easier since it's directly tied to account). I don't hold out too much hope for them to respond with anything too helpful or different though unless it was actively and reproducibility occurring.

OlivierOP•5d ago

I managed to trigger the issue with the trace tool

OlivierOP•5d ago

👍 will post on the community forums. Thanks @Chaika, appreciate the help.

Chaika•5d ago

I would add as a side note tho: If you don't have a use case for O2O it's going to hurt more then help with confusing configurations, and Render is likely Enterprise, so Enterprise routing, which you lose with your own proxy hmm on a url that shouldn't 404?

OlivierOP•5d ago

Yeah. I had to try a few times to trigger the 404.

Chaika•5d ago

I assume you re-enabled proxy to test? Can you share url? Curious to see if any of the response headers give any more detail

OlivierOP•5d ago

This was on a non-production URL where proxying is still enabled. I'd rather not share the URL in a public space but I can share the response headers for a sample 404:

    'cf-cache-status': 'DYNAMIC',
    'cf-ray': '8f39f65b1e8fba36-SEA',
    connection: 'keep-alive',
    'content-length': '19',
    'content-type': 'text/plain; charset=utf-8',
    date: 'Tue, 17 Dec 2024 21:21:21 GMT',
    nel: '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}',
    'report-to': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v4?s=MaDL%2F9iuIyj0INKF3liSVTfy1y0ArgOTi%2FfvcMOPzG%2BP3E1sROy2sqQDlzM2SgaKmDDBW3PIi19Ar8TbhHagZYREMKcOLXi8DYhIFdFTu2FSE%2Fe0Y2eDHJfH%2BzuFmqhp7AX0OOImq5Ga%2Fw%3D%3D"}],"group":"cf-nel","max_age":604800}',
    server: 'cloudflare',
    'server-timing': 'cfL4;desc="?proto=TCP&rtt=7553&min_rtt=7550&rtt_var=2837&sent=5&recv=5&lost=0&retrans=0&sent_bytes=2849&recv_bytes=798&delivery_rate=557697&cwnd=128&unsent_bytes=0&cid=02423b005368cedf&ts=85&x=0"',
    vary: 'Accept-Encoding',
    'x-content-type-options': 'nosniff'

    'cf-cache-status': 'DYNAMIC',
    'cf-ray': '8f39f65b1e8fba36-SEA',
    connection: 'keep-alive',
    'content-length': '19',
    'content-type': 'text/plain; charset=utf-8',
    date: 'Tue, 17 Dec 2024 21:21:21 GMT',
    nel: '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}',
    'report-to': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v4?s=MaDL%2F9iuIyj0INKF3liSVTfy1y0ArgOTi%2FfvcMOPzG%2BP3E1sROy2sqQDlzM2SgaKmDDBW3PIi19Ar8TbhHagZYREMKcOLXi8DYhIFdFTu2FSE%2Fe0Y2eDHJfH%2BzuFmqhp7AX0OOImq5Ga%2Fw%3D%3D"}],"group":"cf-nel","max_age":604800}',
    server: 'cloudflare',
    'server-timing': 'cfL4;desc="?proto=TCP&rtt=7553&min_rtt=7550&rtt_var=2837&sent=5&recv=5&lost=0&retrans=0&sent_bytes=2849&recv_bytes=798&delivery_rate=557697&cwnd=128&unsent_bytes=0&cid=02423b005368cedf&ts=85&x=0"',
    vary: 'Accept-Encoding',
    'x-content-type-options': 'nosniff'

Chaika•5d ago

If that's a 404, CF is saying it contacts your origin (CF-Cache-Status existing and being Dynamic proves that) and no worker matches in trace so nothing possibly modifying it, so that would indictate your origin 404ing

OlivierOP•5d ago

I agree, but there's nothing in our app logs. So I figure it has to be Render's infrastructure, after Cloudflare but before our app server.

Chaika•5d ago

(Cloudflare would also be part of that)

OlivierOP•5d ago

but Render also inserts some custom headers (x-render-routing, maybe others) which are not seen here, so they're saying it's not them

Chaika•5d ago

only other thing that comes to mind is snippets, under Rules -> snippets of your website. They don't appear in trace currently They could modify a response from an origin to end up like that, would be weird tho

OlivierOP•5d ago

we're not using snippets

Chaika•5d ago

then idk and not much more to go off of, link community post when you make it and I can throw it in escalation queue, could try telling Render as well that it seems to be hitting them fine/cf-cache-status dynamic and nothing in trace

OlivierOP•5d ago

👍 thanks. I'll post on community forums https://community.cloudflare.com/t/unexpected-random-404-errors/748989 I've also asked Render to check their own Cloudflare setup, still waiting for them to get back to me about that.

Chaika•5d ago

You got a reply from cf? anything helpful? must have been funny timing, I didn't have a chance to escalate that and doesn't look like Laudian did either

Gaming

Programming

Unexpected 404 errors - Possible O2O issue