Traffic from Barcelona routed through Singapore datacenter - Extremely high latency
Hi, Cloudflare is routing traffic for my website through the Singapore (SIN) datacenter, even though the detected location is in Spain (ES). This is significantly increasing latency and impacting performance for users in the region, making the website almost unusable.
The /cdn-cgi/trace output shows:
colo=SIN | sliver=none | http=http/3 | loc=ES | tls=TLSv1.3 | sni=encrypted | warp=off | gateway=off | rbi=off
I’ve checked cloudflarestatus.com, and it shows no issues with the nearest datacenters (Barcelona, Madrid, Paris).
Is anyone else experiencing this issue? Any insights or suggestions would be greatly appreciated.
Thank you!
34 Replies
Just to narrow down your issue, are you routed correctly when you visit an Enterprise website such as https://cloudflare.com/cdn-cgi/trace
Thank you so much for answering so quickly 🥰 When I visit that website I'm routed through the CDG dataserver (Paris)
No worries, happy to help 🙂
Being routed to CDG confirms that it's not just an ISP peering issue on your end. This website is very useful in illustrating what is likely going on here:
https://cloudflare-test.judge.sh/
Depending on your plan and your actual location, you might notice weird routing issues like this because prefixes that are assigned to lower plans can get withdrawn from certain PoPs so Cloudflare can manage traffic when necessary and prioritize higher plans.
If you take a look on that website, I'm assuming you'll see yourself being routed to
SIN on the free plans and as you get closer to Enterprise, it'll get closer to you.
Another thing to note is the LOC=ES doesn't mean anything and will always be the same no matter which PoP you connect to.
That just refers to the geolocation data that is associated with the IP you connected with (based on Maxmind).Oh that website I see the "FREE" column websites being routed to CDG and MAD, but when entering my website it shows SIN 🥲
Oh, that is certainly strange then!
It's usually always ISP peering or traffic management going on!
So your website is the only website being routed through Singapore on that entire list?
Would you mind sharing the URL? I am also in Europe so I can do a traceroute on my end and see if I can figure it out
I see one PRO website routed through SIN, all others are CDG, MAD, MXP and MRS
Sure! It's ferroviaria.cat
Interesting, it does load perfectly for me (at my nearest DC)
🥲
One thing I would do to try and narrow this down further is to use Cloudflare WARP and then see how you're routed to your own website.
WARP should connect to your local DC as opposed to
SIN and then avoids your ISPs routing table, so if your website then loads from MAD / CDG - we know it's an ISP issue.
However, I would be surprised if any of your users are also experiencing issues, because I've connected to your site through a Madrid VPN and it loads instantly.I've set the 1.1.1.1 DNS on my router and on my adapter settings and it still shows SIN and almost same ping
I mean the actual WARP VPN, not just DNS
If you get the app it'll re-route all your internet traffic through Cloudflare
Sorry! Okay, I'm installing it 🙂
Wow with WARP VPN installed on my phone it shows i'm being routed to MAD with a reasonable ping (150 / 88)
It's weird because I'm getting SIN on my laptop and my phone and both networks are different carriers
With WARP all websites listed are routed to MAD
Sweet!
That is good news, because your website is being served through Madrid and other local DCs, so you can reasonably say that most of your users should be fine.
You can do a MTR/traceroute from your laptop/phone to give you more visibility on what happens when it's routed through your ISP, but it's them taking an inefficient route to Cloudflare for certain prefixes.
fwiw that website is sadly pretty outdated now, I think a lot of them are way off, espec the free ones. Free usually have 2 IPs (104 and 172) and alt ips like 188.x in specific regions. firing.it, for example looks Enterprise now, 2 IPs same subnet
Oooh interesting good to know thanks for the update 🙏
I made https://debug.chaika.me/?findColo=true and those plans are right but lack the same sample size. IPv4 vs V6 routing can be interesting tho, otherwise just comparing known free vs cloudflare.com does most of it.
Okay! I've done a traceroute and it shows lots of timeouts:
1 * * * Timeout
2 4 ms 5 ms 4 ms 10.x.x.x
3 14 ms 14 ms 14 ms 10.x.x.x
4 14 ms 13 ms 13 ms 10.x.x.x
5 14 ms 14 ms 14 ms 10.x.x.x
6 13 ms 14 ms 13 ms 10.x.x.x
7 13 ms 13 ms 13 ms 10.x.x.x
8 * * * Timeout
9 * * * Timeout
10 * * * Timeout
11 14 ms 14 ms 15 ms 188.114.108.9
12 14 ms 14 ms 14 ms 104.21.12.150
I don't know if it's an ISP issue, I've tested through different devices on all carriers and all of them are incredibly slow. Even asked friends and tell me they have issues
Timeouts are normal throughout a trace, ICMP is low priority or blocked in some cases and that's perfectly fine
you traced
ferroviaria.cat?Yes! That's the traceroute to
ferroviaria.catah, looks like same IP so I assume so. It's only 14ms at the end? If you run mtr (if on linux) or just ping for ~30s to
ferroviaria.cat, what do you see?pinging for 30 seconds looks fine, no packets lost and Minimum = 14ms, Maximum = 22ms, Average = 14ms
did it go over IPv4 or IPv6?
how can I know? 🫠 haha
well the final IP is 104.21.12.150, not IPv6
well the address would either be IPv4 (4 numbers seperated by dots) or IPv6 (many more numbers and letters seperated by semicolons)
it doesn't matter too much on second thought, Happy Eyeballs should land on v4 if it was routing local and v6 was routing sin
If ping/icmp is consistently low latency but http requests/traffic to the same site is higher latency and showing SIN/far away, it's CF rerouting the traffic internally due to resource availability and such. Cloudflare has internal tools such as Plurimog (nice blog on it - https://blog.cloudflare.com/meet-traffic-manager/) to fling traffic out to other locations, a certain % or specific free/pro/biz IP ranges. Not much you can do about it other then upgrade plans (with no guarantee of one plan fixing it, other then custom ent stuff), and not your ISP's peering/etc
on the website you made I see only SIN at "local-pro", all others are CDG, MAD and even BCN. local-free-v4 and local-free are both listed as CDG. but when entering ferroviaria.cat/cdn-cgi/trace it shows SIN without WARP and MAD with WARP
so maybe I reached a certain amount of traffic and Cloudflare is penalizing me?
typically I've seen it as they shift a % of traffic (for all non-ent plans) and not all of it, which makes the plans look a bit weird
not something that would happen without you getting emailed, and also wouldn't make sense, traffic is cheap in europe and super expensive in asia pacific
now it has changed to SIN at local-free-v4 and local-free 😮
yup sounds like https://discord.com/channels/595317990191398933/1328416682288349278/1328434531396489349
The super far locations bit is curious but we've been through it before, for example Sao Paulo / GRU was flinging non-ent traffic to IAD / Us East for a while rather then any other nearby location, like the ~20 other Cf locs in South America, CF Employees confirmed it to be intentional. I think it's because it's picking the lowest utilization location globally and not caring about latency, and it flinging traffic like that is more meant to be a last ditch measure but that's just my guess
regardless not a whole lot that can be changed from your end, other then hoping they normalize / scale up the location soon
I think there's some event in Spain right now right? Cloudflare's Radar shows traffic spiked a lot over baseline earlier, possibly connected if you speculate a bit
yes, yesterday there was a very important football match in the afternoon. maybe Cloudflare is temporarily routing some traffic due to those spikes?
now a visitor is complaining on my Twitter saying she can't load the website 😦
that's what I was thinking
😮 ! It looks like my website is now served by the MAD datacenter!
my website and local-free-v4: MAD, local-free: CDG. now everything loads fast as always, I guess there has been an intentional 24-hour routing to Singapore for some users until now?
cloudflarestatus.com is showing a scheduled maintenance currently in progress at the Singapore datacenter lol
Singapore is a point of presence made up of ~15 seperate cloudflare colocations, safe to say at least at a few different data centers, a beast of a PoP lol
well it seems that my website is running smoothly again for everyone 🥰 I've been struggling all day to figure out what happened, thank you so much for your help and explainations @It's Always DNS @Chaika!! I really appreciate your support 💜