Did workers get slower?
I've had a worker running for years just to test the speed of Cloudflare services. Today, I tested the new KV speed improvements, and they showed significant progress—down to 3-5ms from around 25ms+ after a cold start. However, the worker seems to have slowed down overall. I haven't changed the code in years and have tested it multiple times. While 135ms for a simple response (waiting for the server) isn't too bad, it feels slower than before. I recall response times being around 50-60ms previously. Have there been any recent changes that could explain this?
30 Replies
Is it consistently slower like ~130ms instead of 50-60ms? If so probably your routing changed. You can go to the path
/cdn-cgi/trace on your worker and look at the colo= line for the airport code of the cf location you're hittingHmm, yeah, you're right. I’m not really sure what or where the EWR Data Center is. A simple ping to the server takes 105ms, and I don’t understand why my worker isn’t selecting the nearest Data Center in Frankfurt (17ms), like it usually does.
Edit: Living in Germany (NRW) btw
. I’m not really sure what or where the EWR Data Center is.New York City, all airport codes are googable
A simple ping to the server takes 105ms, and I don’t understand why my worker isn’t selecting the nearest Data Center in Frankfurt (17ms), like it usually does.It's DTAG something like DTAG is a big ISP in Germany, they want Cloudflare to pay them for peering/transit/per link, and since they aren't they're routing traffic far away/on cheaper routes. iirc Cloudflare was paying that dtag tax/better local routing on Pro/Biz/Ent . you could see for yourself on https://cloudflare.manfredi.io/en/tools/datacenters/ and https://debug.chaika.me/?findColo=true and see where you get routed depending on plan
Ohh dang that's some useful inside Information thank you very much. And very nice links. So Cloudflare Workers Paid isn't that fast anymore without a Business Plan, thats sad :c
Pro doesn't get you routed locally in those tests?
I don't have Pro. I only have the 5€ workers plan.
no I mean on the links I sent, they've got Pro sites in there as an example. You specifically mentioned Business so I was confused as to why Biz plan over Pro
Yeah Pro already uses a better Datacenter down to 35ms, but still not the best ^^
The best one actually seems to be HAM and that's down to 15ms (Enterprise Plan and Durable Object only)
So actually I just tested the Durable Object now and because of this change it's super slow ...
The Durable Object still seems to be hosted in HAM, but the request is routed over New York. Now a simple Durable Object Request takes 300ms after cold start!!! RIP
Edit: but maybe it is because it's an old DO pre changes
Edit: but maybe it is because it's an old DO pre changesThey don't move so probably that. What do you see on https://debug.chaika.me/?findColo=true for Durable Object? That's a new DO created on an ent website routing. DOs should just be created off the colo you hit, not your geoip info or anything
It also says HAM, which would be a major issue in the Architecture. Because 300ms for DO is inacceptable.
Would be good for a Websocket Connection, but for a simple Worker to DO it's not good
I'm not a big fan of this changes. I really liked the performance of the Cloudflare network and I don't need anything of the Business Plan for Workers Development
It also says HAM, which would be a major issue in the Architecture. Because 300ms for DO is inacceptable.opps my example isn't good for that -- HAM is right because Ent is connecting locally what do you get from https://durable-object-spawner.chaika.win/ with free plan routing?
IAD
EWR
so going to worker New York (EWR) connecting a DO in IAD (Ashburn), not bad, probably ~20ms or so between worker and do
Yeah but before the changes it was actually in the same Datacenter ^^
it's all capacity based
https://where.durableobjects.live/colo/EWR there's a 60% chance of getting a ewr DO spawning a durable object in ewr
Where Durable Objects Live
Newark, New Jersey, United States (colo)
Tracking where Durable Objects are created, wherever you are in the world.
and a 40% of iad do from ewr
Ahh okay ty, but I'm still thinking why whould they change it for all Worker Products and not only for there free CDN
Worker products just sit behind the cdn
It's still like get 70ms more delay or pay more money for features you don't neeed
But of course only for germany
only for dtag
there was a blog from Cloudflare -- I can't find it offhand now, but basically saying CF is willing to pay market prices for peering with dtag (even though peering should be settlement-free) but not absurd pricing like they want
And it's actually 90-110ms more delay. I misscalculated
Ahh ok, yeah that sucks :c greedy germans
I found this post: https://www.reddit.com/r/de_EDV/comments/1em7mie/telekom_peering_zu_cloudflare_die_bei_mir_ist/
And it only seems to happen if you are in the Deutsche Telekom Network and I'm in it :c
It appears to be so expensive because Telekom is one of the largest (if not the largest) internet providers here, yet they have one of the weakest connections to major data centers like FRA, with a capacity of only about 110 Gbit/s, while others have 20 times that bandwidth.
a lot of ISPs do peering settlement free
if you're big enough Cloudflare/Google/etc will do private interconnects with you
Telekom doesn't do settlement free peering because they pick not to and would rather try to charge people for them
Telekom charges for almost everything; they're really greedy and expensive.
@Chaika one last question: Does workers still select the best Data Center for your location? Because it seems like they had something like better locations for better customers in mind.
: Does workers still select the best Data Center for your location? Because it seems like they had something like better locations for better customers in mind.What do you mean by that? Workers fundamentally live behind the CDN, so all the CDN routing stuff effects them. CDN is Anycast, so Anycast routing. CF has Smart Placement for running your worker in a specific location, but that's still all behind the Anycast CDN, so let's say you have a REST API your Worker calls and it lives in FRA, and you turn on Smart Placement. You'd go all the way to EWR then CF would tunnel traffic internally to run the worke rin fra
I meant you're not in the Telekom network, so you don't have this problem like I do and still enjoy the sub 30ms delay?
Ahh ok and if I remember correctly business user actually have some kind of better routing in the cdn right? So it would still be better with a business plan
yup
it's not guaranteed in any way to not change but all plans have better routing then free by some degree. Some of it is traffic manager (an internal cloudflare tools) removes plans in order of free, pro, biz when capacity is at limits for a location, and some is probably more complex business decisions
Well, this thread was informative. No wonder my workers run in a DC that pings 130ms when the closest one is just 5ms. Even the WARP unlimited VPN service terminates there
Found this check: https://cloudflare-test.judge.sh/