HTTP 429 Too Many Requests
My application makes some calls to the Kinde API from our backend. On Tuesday at 1:24pm (UTC) we saw a spike of HTTP 429 responses from the Kinde API. Can you help me understand the rate limits which apply to use of the Kinde API?
22 Replies
Hey @Martin,
To answer your question, are you able to share what requests you made to the Kinde API?
We were making calls to the
/api/v1/organizations/org_cfd97636f0a7/users API for all of our different organisations.Hey @Martin,
THanks for getting back to me on the API call you were making. My team are looking into this and will get back to you.
Hey @Martin,
A 429 response is returned when you are sending too many requests that exceeds the rate limit.
Responses include a header
RateLimit-Reset with the number of seconds until the rate limit reset.
Whilst I cannot provide specific information on our rate limits, if you let me know the maximum amount of requests you expect to send per minute, I can let you know if that exceeds our rate limit.
Are you calling the API endpoint directly or via a method in one of the Kinde SDKs?
The back-end Kinde SDKs automatically handle waiting/retrying Kinde Management API calls in these instances.Our REST API has an endpoint which lists all the users across all of our Kinde organisations. We currently have ~28 Kinde organisations in Prod, so I'd expect our REST API to attempt to make ~28 Kinde API calls in quick succession. This is in an internal tool so we get far fewer than 1 request a second to this REST API endpoint, and hence I'd expect us to usually be making zero Kinde API calls, and occassionally be making a spike of <100 Kinde API calls very rapidly.
We're calling the API directly through the Python requests library with the following retry strategy:
DEFAULT_RETRY_STRATEGY = Retry(
total=3,
backoff_factor=1,
status_forcelist=[429, 500, 502, 503, 504],
allowed_methods=["HEAD", "GET", "OPTIONS"],
)
This will mean that in the event of a 429 response we'll wait 1 second, then 2 seconds, then 4 seconds, then give up.
I'm basically very surprised that an occassional spike of <100 Kinde API requests is enough to trigger HTTP 429 responses over multiple seconds, to the extent that our retries weren't enough to get a successful response. Can you help me understand this? Have you got data in your metrics showing that we're actually making far more calls than this?
maybe they're counting the retried requests against your quota (the one's that are getting 429'd)?
Hey @Martin,
Thanks for elaborating on the logic of your application and how often you expect to make Kinde Management API calls.
I have passed all of this information to my team.
We'll get back to you soon with guidance on making calls to the Kinde Management API whilst not being restricted by our rate limits.
Hey @Martin,
Our rate limits consider 1 minute timeframes. So if you get rate limited it could be for up to 1 minute.
So waiting a few seconds may not work. As mentioned before, we have a header in the response
RateLimit-Reset that returns the number of seconds until the limit resets.
@Martin regarding the "occassional spike of <100 Kinde API requests", do you know roughly the maximum amount of API requests you expect per minute?
This will help me guide you on how to best approach calling our APIs with the volume/frequency of API requests you expect.How are you computing your rate limiting? I would have hoped for a token bucket (https://en.wikipedia.org/wiki/Token_bucket) to allow for a defined spike of requests, followed by a smooth rate of allowed incoming requests. I'm surprised that you've got an implementation which can result in the API rejecting requests for up to a minute at a time.
Do you have logs/metrics which enable you to see the per-client rate of incoming requests which Kinde actually experienced during this period of throttling on Tuesday 12th Dec at 1:24pm UTC?
On our side I can see that we had 10 invocations of the Lambda which was calling Kinde within a single minute.
I've just noticed that we actually make 2x Kinde API calls each time we list users for an organisation because the first response returns the list of users and a
next_token, and then the next response returns no more users and a null next_token. This looks like a bug to me as I wouldn't expect you to return a next_token when there's no more data to fetch. This bug doubles our number of Kinde API requests to this endpoint.
This means that each of these Lambda invocations fetches the users for ~28 orgs, and each of these operations takes 2 requests, which means 56 Kinde API requests from each Lambda. There were 10 invocations so that's 560 Kinde API requests in a single minute, which is ~9/second.Token bucket
The token bucket is an algorithm used in packet-switched and telecommunications networks. It can be used to check that data transmissions, in the form of packets, conform to defined limits on bandwidth and burstiness (a measure of the unevenness or variations in the traffic flow). It can also be used as a scheduling algorithm to determine the ti...
Hi @Martin,
Thanks again for elaborating on your logic behind the API calls you are making.
I can understand your frustration on how you wouldn't expect to get rate limited on such a small amount of API calls.
I too agree you should not get rate limited for this instance.
How are you computing your rate limiting?Unfortunately, I cannot provide anymore details on how we exactly compute our rate limits.
Do you have logs/metrics which enable you to see the per-client rate of incoming requests which Kinde actually experienced during this period of throttling on Tuesday 12th Dec at 1:24pm UTC?We do have access to logs and can see the behaviour of the API calls you made during this time.
I've just noticed that we actually make 2x Kinde API calls each time we list users for an organisation because the first response returns the list of users and a next_token, and then the next response returns no more users and a null next_token. This looks like a bug to me as I wouldn't expect you to return a next_token when there's no more data to fetch. This bug doubles our number of Kinde API requests to this endpoint.I have recreated this issue, I will bring it up with my teammate first thing on Monday. I, along with a few other teammates, are working during the holiday period but it is limited staff. I will raise this first thing on Monday to my teammate but I cannot guarantee any action is taken until all my teammates are back on the 2nd of January 2024. Is this a blocker for you right now until then? In the meantime, if you are trying to get a list of all users and the organizations they are assigned to, I would recommend calling the List Users API call and including
expand=organizations to get the organizations each user is assigned to. This would make a significant less amount of API calls. Would this achieve what you are after?
I apologise for the inconvenience here. We do not want to inhibit the experience of your application on Kinde. I also understand how this must be very frustrating to hit such rate limits at this stage when you expect to have more API calls as you grow.
I, along with my team, will work with you to alleviate this pain so you don't have to think twice again about hitting rate limits.
It might be worth jumping on a call early Jan 2024 to discuss this further with my team.
For now, I will pass all this information along to my team next week.
Please let me know if this is blocking you right now and can't wait until 2 Jan 2024.Thanks for the detailed response. None of this is blocking me at the moment. I'm happy to wait for a further update from you sometime in the next couple of weeks.
Thanks for the tip about using the List Users API call. I'll bear that in mind as an option which I can switch to if I actually hit any more throttling. For now, it's slightly more convenient from a code point of view to get users for each organisation.
Also, happy new year!
Hey @Martin,
Happy new year to you too!
Please do reach out if this issue becomes more pressing/more of a blocker.
One of my teammates has started looking into your throttling issue and when the
next_token call returns no users.
I will reach out to you when I have any updates.We saw another spike of throttling today at 11:38. Are we sharing a throttle bucket with other Kinde clients? There wasn't much traffic from our service at this time, and I've just tested with slightly more traffic and wasn't throttled.
Hey @Martin,
I am getting my team to look into this instance of throttling. Are you able to confirm the date/time (UTC) that you experienced the spike in throttling?
Hey @Martin,
We have just deployed an update that optimises our rate limiting. The previous requests rates you sent should not be throttled.
Please let me know if you come across any instances of rate limiting again.
That's great news, thanks!
In case you still need it, the issue yesterday was at 11:38 UTC on Jan 4th.
No worries.
Thanks for passing on the date and time of when you experienced throttling. This matches the time we could see throttling on your business. I can confirm the optimised rate limiting we introduced would not have caused throttling in this instance.
Can you also confirm you are using all your own credentials for social (e.g. Google) and/or enterprise (e.g. Azure AD) authentication connections?
If you are using the default Kinde credentials for authentication methods, you will also be throttled.
We're using our own credentials as we need to do that to get our custom domain to be shown on the login screen
Noted.
Let us know if you experience anymore issues @Martin
We just saw a spike of throttling at 15:02 UTC. Can you see if we're sending too much traffic?
@Oli - Kinde ^^^
@Martin are you still experiencing throttling spikes?
We just saw the one spike recently and it hasn't re-occurred.
@Andre @ Kinde - we just saw another spice at 12:33 UTC today. Can you give me any more guidance about what rate of API calls is acceptable?
Hey @Martin,
My team is looking into this. I will get back to you
Hey @Martin,
We can see you are making a lot of request to Kinde.
We can see at one point you made over 400 requests/second, in that time there were only 29 distinct requests repeated 10-18 times.
We can help you reduce these duplicate requests if you can explain more about your logic and why all these requests were made.
Thanks, this is really helpful to hear. I'll look into it further on our side. It sounds like something may be genuinely misconfigured.
Let us know what you find. We are here to help.