Cloudflare websites extremely slow
Hey, I'll try to provide as much information as I can, and everything censored can be uncensored on request.
Since the past 4+ weeks, I get daily issues with many websites becoming unusably slow. What I experience is that the website either loads forever, or it loads within a few MINUTES. For example, a website with a few images on it, takes 2 minutes to load a 20 kb thumbnail! Or I once tried to watch a 3MB video on Discord and it took 5 minutes to buffer! (See image 1 and 2). It feels like using dial up internet on the modern web.
The strange thing is that all speedtests I try return that I get my full 250 Mbit/s connection without any issues while this happens. Eventually I realised that the one thing the slow websites all have in common is that they all use Cloudflare.
So I went to https://speed.cloudflare.com/ and I noticed that whenever the websites become unusable, speed.cloudflare.com also gives me abyssmal results. Single-digit download speed instead of the 250 Mbit that I get on every other speedtest. And I get up to 50% packet loss, if the packet loss test even works at all (most of the time it just failed without a result). Ping and jitter are also high, I once got a jitter result of 400ms. I repeated this test at various times, and it's only the cloudflare speedtest that is having terrible results, all other speedtests achieve the max speed 24/7.
I tried an MTR to various websites that were having issues, and I noticed that they all routed me through
CLOUDFLARE.edge6.Dusseldorf1.Level3.net which is giving me a 50% packet loss in the MTR.
The slow websites usually don't occur in the morning, they start in the afternoon and evening, so it's probably related to load.
My neighbor (in the same building) is having the same problem, using the same ISP.
I contacted my ISP and they said my connections is great and there's nothing they can do about it (even though they're our Tier 1 network).
How can I proceed?
64 Replies
Your ISP is DTAG/Telekom?
It's Deutsche Telekom
There's known and well reported issues with Telekom and CF. As far as I know, something along the lines of Telekom wants CF to pay for peering with them/their network usage, and since they aren't, they're sending routes over congested links on purpose and/or rerouting stuff to East Coast.
There's a bunch of posts on the forums:
https://community.cloudflare.com/t/connection-to-cf-via-german-telekom-is-very-slow/656849/2
https://community.cloudflare.com/t/latency-to-proxied-dns-entries-high-since-2024-01-30t0715cet/608356/13?u=chaika
https://community.cloudflare.com/t/telekom-cloudflare-routing-problem-only-in-munich/659752/12
and apparently some on telekom's forums too, from what I've read.
Not that it matters too much in this case but packet loss on a single hop that does not go on further/to the end isn't anything of concern btw. You def have issues though, but not necessarily on that specific hop
If you're interested in learning more about how to debug things with traceroute, https://archive.nanog.org/sites/default/files/10_Roisman_Traceroute.pdf is a good read.
Nothing you can do in this case though, other then using a VPN to a nearby location might help, as it'd go over a different route and from the DC have different routes to CF.
Thanks for the fast response, you're very dedicated!
So it seems both sides are battling it out and I'm being used as leverage while only suffering as a result.
Yeah I assumed the tracert does individual pings because the ones after it didn't get any packet loss, but I'll check that resource out too, thanks
So I guess I can only complain to Telekom that they should stop being greedy and fix their routing, and in the meantime pay for a VPN (I assume that breaks my port forwarding?)
And another question, since Deutsche Telekom is Germany's Tier 1 ISP network, doesn't that mean that all German ISPs use this same infrastructure and would have the same problem, or is that not how it works?
So it seems both sides are battling it out and I'm being used as leverage while only suffering as a result.Pretty much. I looked over those threads and it seems some people reported some improvements recently but still def some issues
Yeah I assumed the tracert does individual pings because the ones after it didn't get any packet loss, but I'll check that resource out too, thanksMore then that, to routers there's a big difference between to (like pings) and through (like actual traffic
So I guess I can only complain to Telekom that they should stop being greedy and fix their routing, and in the meantime pay for a VPN (I assume that breaks my port forwarding?)If you have port forwarding in your network to other devices should be fine, if it's going to that specific device you're using with a vpn yea it'd break that. Some VPNs offer portforwarding stuff though It may be helpful to explain that "Tier 1" is essentially a made up and arbitrary term. The idea is just that all Tier 1s don't pay for peering from each other, but the real world is more messy. Additionally they're not "Germany's network" or anything like that. Not like all the other providers or networks in Germany are forced to use them. They are huge and own some other ISPs, but there are other Tier 1s in Germany and other routes https://en.wikipedia.org/wiki/Tier_1_network
Ah I see, I thought it was a hierarchy where all Tier 2 to Tier 2 connections most likely go through the Tier 1 also
it depends. They are called Tier 1 for an example, they're huge, own lots of physical infrastructure, etc, but there are other players. For the big networks specifically like cloudflare/facebook/google, ISPs usually aim to peer with them directly. If you found a VPN in Germany for example, they probably peer directly with Cloudflare and would avoid that issue
Peering with content networks like Cloudflare/Google/etc is settlement free/they don't charge you anything, only thing it costs you (as an ISP) is the price of the interconnect/facility itself. Whereas an ISP would have to otherwise pay for Transit/another ISP to send their traffic through to reach the rest of the internet, and that's usually on a static speed commitment (1 gb peak for xxx per month, etc). Makes sense for them to use less Transit which they pay lots for, and more peering
Internet is messy, routing is often about what is cheapest/business decisions rather then just the fastest/most capacity route
Okay, thanks!
Just one more question, the Telekom support guy kept trying to tell me for 2 hours that "No we can't change (our internal) routing. It's outside of our power. We only provide the connection. We're not able to choose which path your traffic takes through our network. So we can't help you with this problem. Also we have no department that does routing, because we can't change the routing."
... he's completely wrong about that, right?
Yep, when you're sending traffic you can pick exactly which route you want it to go over
Cloudflare, as the receiving side of the connection can merely make routes available to their specific peers/connections, and some Tier 1 provide BGP Communities, sort of like metadata that you can use to hint "depref this route in Asia". Telekom, as the sending side, controls exactly how/where traffic exits their network at/how it travels thru. Both side control who they peer with, but Cloudflare's all settlement free and even has an automated peering portal, they would just have to have a common facility or run a private interconnect, so certainly isn't on them that telekom isn't peering with them lol
Yeahh, I agree with Cloudflare then
And I felt like I was losing my sanity talking to that Telekom "support", I asked them multiple times to connect me with someone else who's more technically inclined and he just refused because "he learned IT" and "apparently I'm too dumb to understand what he's trying to explain to me". But I got a complaint form at the end, so I'm going to write in there now and most likely that'll get me contacted a second time and then I'll talk to someone else with this new information you provided
I'll leave you to it then, thanks for your time and have a nice day!
You're welcome to try and I imagine more people complaining helps but I kind of doubt change will come instant. I know some other people on the community forums have mentioned reaching out to Telekom as well without much luck, and apparently on telekom's german forums online there's been a bunch of talk about it
Same 4 me
Cant use Discord in the evening! Just with VPN
I'm also affected for many weeks now.
I've opened support cases at Telekom Germany, we wrote tons of messages in their support forums.
(It's a User to User forum in it's first place, but there are also workers from telekom around)
Noone seems to man up and talk to each other.
I assume that the traffic between telekom and cloudflare is huge (since cloudflare covers up a lot of the internet these days) so I don't understand why they don't agree on free peering when noone wanna pay each other.
It is very annoying that this dispute is being fought on the backs of the end customers.
Hi, I have the same issue.
Hello, using Hungarian Telekom (AS5483), free tier CF sites are routed to the USA with this speed, since January 30th…
Packet loss is stable 20% nonstop
Pro tiers are routed to Germany on IPv4 and Vienna on IPv6, (even though CF is available in Hungary), and they suffer from 30-40% packet loss at the evening
This is 1.1.1.1
yea, other reports like that, read above for more info
https://discord.com/channels/595317990191398933/1253001766497292420/1253002490308329672
This is what Telekom says about it…
This suggests they won’t really do anything about it and let millions of users suffer from this
that's just how it works with ISPs. CF cannot do anything but say "please peer with us for free" and make routes available to them, they have complete control of the traffic within their network, how it egresses
The funny thing is, Telekom owns Dataplex (a datacenter in Hungary), and Cloudflare is in Dataplex, on the same switch as Telekom
Why would they still use DTAG’s peering?
My understanding is that they want CF to pay for peering/every bit of traffic used (As well as some other providers) and CF is refusing to do so, at least for their standard free network
so in response they're purposefully (or at least, purposefully not fixing) routing traffic far away (used to be via East Coast)/through congested links
both sides have been pretty silent on it though, probably because it's a legal/business dispute. I'm no Cf Employee either, so :shrug:
all I can say is, all CDNs like CF offer settlement free peering everywhere and even have an automated peering portal, so shouldn't be on them
They are killing their international lines with this tactic… Anything that goes through twelve99 is extremely slow…
guess they don't really have a reason to care as long as some companies pay up, it's beneficial to them.
Since it's an issue between Cloudflare and Telekom - Did Cloudflare try to contact Telekom about this in the last few weeks? (Since the issues got more present during the last months)
According to Telekom it's Cloudflare's fault. (Which I'm sure is wrong, still I wanted to ask my question)
Hello, Slovak Telekom user here, I've reported the issue to my ISP and they seem to be aware of it on a deeper technical level, but I doubt they could do much about what their sister company in Germany is doing
Also got here from the Deutsche Telekom forum post
it's been a while and lots of old posts but I don't believe CF has ever said much other then writing this blog post and linking it: https://blog.cloudflare.com/eu-network-usage-fees, after the start of those issues:
In Europe, transit providers play a critical role because many of the largest incumbent telcos won’t do settlement-free direct peering connections. Cloudflare would of course be happy to directly connect with EU telcos because we have an open peering policy. As we’ll show, the performance and reliability improvement for their subscribers and our customers’ content and services would significantly improve. And if the telcos offered us transit – the ability to send traffic to their network and onwards to the Internet – at market rates, we would consider use of that service as part of competitive supplier selection.probably because it's a business dispute, not much has been shared with the public. This is the Developers Discord and it doesn't really fit the params to be escalated from here, but they did try from the community forums side and nothing was said back iirc
I've seen a couple communiry forums posts and pretty much all of them had a reply stating the dispute with Deutsche Telekom
@Chaika it's mindblowing, that Telekom refuses the offered free direct Peering, and instead investing/paying for oder CDNsbetween cloudflare and Telekom. (e.g. more hops, I guess should get what I mean)
So in the end - the have to pay more than they would have to if it's done correctly... weird..
"Tier 1" Networks such as DTAG are called Tier 1 because they do not pay for transit at all, not inbetween them either, at least in theory, all the major T1s have settlement peering to each other
I thought the issue was that they wanted more money for peering than other ISPs
between CF and Telekom? Yea, as described above:
In Europe, transit providers play a critical role because many of the largest incumbent telcos won’t do settlement-free direct peering connections. Cloudflare would of course be happy to directly connect with EU telcos because we have an open peering policy. As we’ll show, the performance and reliability improvement for their subscribers and our customers’ content and services would significantly improve. And if the telcos offered us transit – the ability to send traffic to their network and onwards to the Internet – at market rates, we would consider use of that service as part of competitive supplier selection.It's not just "more" money either, ISPs in North America and other places do free/settlement free peering
It's crazy.. it's like someone's offering you to get something for free, and you refuse. lol
more like someone's offering you something free but you think you could get a whole lot of money from them if you go about it another way
Yeah
I've posted that on the Telekom forums
But is there any chance and way we can get in touch with cloudflare to get this solved?
no because its not a cloudflare problem? xd
Of course it's also a Cloudflare problem, even Telekom itself says so:
„Unfortunately, we had to realize that despite increased traffic with peering, these go through a path that is not at all aligned for it.
In addition, we would like to officially announce that without the cooperation with Cloudflare, we would unfortunately not be able to solve some branches“
even Telekom is saying this? no, only Telekom is saying this.
it might help to actually read the thread before asking questions
From what I've gathered, Cloudflare would be happy to pay to get a better connection with Telekom, if they actually provided such service at market competitive cost
Hey I'm back, after like 10 calls and emails between me and multiple Telekom people who don't know what they're doing...
Apparently once it got elevated to their actual technicians, they said they tried some things and asked me if it got better, but it didn't improve.
Now their technicians think it's cloudflare's fault because they saw the packet loss in the trace. (Should I even bother trying to explain that to them?)
And now they're asking me to also provide a trace in the opposite direction that goes from Cloudflare to me? Is it even possible to request that?
I have already received a response from our technical department. They checked and found no error in the Telekom network, but in the Cloudflare network. Please provide us with a back-trace (preferably including a "to"-trace). The "to"-trace did not show an error at Telekom, but rather packet loss at Cloudflare. We also say that you should contact Cloudflare, and they say that you should contact Telekom. That is what my colleagues in technology told me. I hope that I was able to help you with my answer. Best wishes and have a nice day
Enterprise have a trace api they can use from a specific colo
https://developers.cloudflare.com/api/operations/diagnostics-traceroute?schema_url=https%3A%2F%2Fraw.githubusercontent.com%2Fcloudflare%2Fapi-schemas%2Fmain%2Fopenapi.yaml
If you remind me later and say which colo you are hitting (look at /cdn-cgi/trace) and a test IP, I can run one and see
Cloudflare API Documentation
Interact with Cloudflare's products and services via the Cloudflare API
not that I think they'll do anything with it but better then doing nothing
Slovak Telekom on the other hand hasn't given me much information, they just said that multiple people have reported the issue and they're working on fixing it but can't give a timeline. They seem to be admitting that the issue is on Telekom's side, as they gave me the option to get free mobile internet until they fix the issue
Sure, that would be very helpful
Do you have time right now? Since the problem is occuring right now
in a few mins if you can dm me/send me a test ip and colo to test from
okay sure, thanks! I'll send it now and you can do it when you have time
Keep us updated please! :)
just curious, would mind to test one for me as well?
Sure
Just got an invoice from my ISP (Slovak Telekom) and as they consider this to be an issue on their side, they contacted me and told me they'll be lowering the amount for the duration of the fault. I was not expecting to get a 75% off once I actually got the invoice.
Meanwhile I'm guessing Deutsche Telekom who is actually at fault is still denying everything
Though that does leave me to wonder if everyone got a discount, though most likely only people who reported the issue. But seeing how competent the customer support here seems to be, I'm sure they'll just have a sticky note somewhere saying "if customer states 'internet slow sometimes' just group under this fault"
@araghon007 lucky you! :MeowHeartCloudflare: :party_blob:
For me they refused a discount, and also said they're unable to fix it, so I cancelled my contract and next month I'm switching to Vodafone instead
Which subsidiary are you with?
I'm thinking about switching to o2 (germany), since vodafone often use telecom backbones. (talking about using telecom's routings) (ofc the "last mile" to my house will be the same with every provider)
I was with Deutsche Telekom (Germany)
So they still don't take any blame. Very sad to see
Just as an update, I have since then switched my ISP because of this, from Deutsche Telekom to Vodafone, and the problem completely dissappeared. That was after I was seemingly escalated to the highest tier of Telekom's customer support where a guy was personally overseeing my case for 2 weeks and I was talking to the engineers there (who were not customer service trained, they're the actual tech people there), and still in the end they said they couldn't fix it. So if someone else in Germany is having this problem with Telekom, I can only recommend switching to another ISP.
How does it compare to Telekom so far? In terms of price per performance and availability
That's surreal and outright malicious, I can't believe a legitimate company would stoop so low
And here I thought US cable ISPs were bad enough lol