Can you all DM me your Workers account
Can you all DM me your Workers account id and KV namespace id
26 Replies
Thanks all for sharing the details
@danielsundq @ivogt @Anthony @berkinovish we're trying to pinpoint this, how are you measuring the latency? Do you have timestamps in your worker code, are you measuring the aggregate responses of your Workers, etc?
Also, if you can share screenshots that might help us triangulate the issue, that'd be appreciated
We use KV for our session storage. We write to KV on sign-in, but the entry wasn’t yet available on subsequent requests, causing users to be logged out immediately. Once we identified KV as the culprit, we moved away from it for our sessions and didn’t investigate further.
Unknown User•2w ago
Message Not Public
Sign In & Join Server To View
In our case we have some rest api endpoints that handles things related to user sessions. The sessions is stored in KV. So we basically have endpoints like updateSession and getSession. We have pinpointed that after calling updateSession, the changes is not manifested in the response of getSession until around 40 seconds after updateSession is called.
Very similar for us. Login and updating the session is not always updated. Occasunally things work but most of the time the session value is stale for few seconds. Often a refresh still shows old data. This used to work for a lot of time and started being a problem since the other day.
Also, I did double check that we are awaiting the kv update.
Here is an example scenario:
- (1)POST to login
- (2) if above is successfull we revalidate the data on the page (Remix into play here) which refetches the data (caching is also ruled-out) - this call is seeing the old data of the session.
Of course initially we thought its something with our implementation and thoroughly checked if its not the implementation and there are no race conditions. But as it appears sequential calls see stale (likey kv cached) value.
Is it possible that KV became more spec compliant( around the 60sec cache) and the previous behavior was not suppose to work in this way?
I don't really have a way of measuring this. (But if you have a suggestion I could implement it).
For me there's not just latency, but most writes don't ever actually update.
Also just wondering, why are there no updates on the status page?
We are urgently moving sessions to D1 and as of now things seems much better
This is currently the running theory for this impact, @berkinovish @danielsundq @bsam75 @ivogt @Anthony what is the throughput for your individual KV pairs?
Very small number of keys but often updates. During testing it's like 1 or 2 active sessions.
Often* I mean a login click and that's it.
We just tested this with D1 storing sessions and everything looks fine so far. Might stay on this solution
Unknown User•2w ago
Message Not Public
Sign In & Join Server To View
Got it, so we're talking <10 RPS correct?
Unknown User•2w ago
Message Not Public
Sign In & Join Server To View
This is aligned with the eventually consistent model of KV and we recommend using durable object in addition to achieve the consistency behavior you're looking for https://developers.cloudflare.com/kv/concepts/how-kv-works/#consistency
Cloudflare Docs
How KV works | Cloudflare Workers KV
KV is a global, low-latency, key-value data store. It stores data in a small number of centralized data centers, then caches that data in Cloudflare’s data centers after access.
@bsam75 please DM me your KV namespace id
Our throughput is also pretty low, definitely under 10RPS
We’ve never had issues with KV for sessions until two days ago—it’s been running smoothly for us over the past couple of months.
If using KV for sessions (which will be read immediately after writing for subsequent requests) is not recommended, you might want to reach out to the Remix team to update their docs: https://remix.run/docs/en/main/utils/sessions#createworkerskvsessionstorage-cloudflare-workers
Thanks @berkinovish for the details
Unknown User•2w ago
Message Not Public
Sign In & Join Server To View
@bsam75 @berkinovish @ivogt @Anthony @danielsundq Can you folks try again? We tweaked the eventual consistency parameters to help out some of your use cases
Works great!
Unknown User•2w ago
Message Not Public
Sign In & Join Server To View
Thanks @thomasgauvin ! Just to understand, is this a global change, or did you tweak it only for the KVs we mentioned via DM?
Across KV. In general, we don't recommend relying on the per PoP consistency, because it is possible that your subsequent requests are routed to other regions (due to how routing to workers regions is non deterministic) and won't necessarily read your updated values. But until we have a solution that allows you to work around this, we've adjusted to minimize the impact. This is something we're planning in the next couple of months and will be communicated.
Unknown User•2w ago
Message Not Public
Sign In & Join Server To View
Alright, clear. Thanks!
Do you recommend to migrate to something like durable objects in that time or will there be a solution that works with KV?
Works great here also, thanks!