JWT with long-lived Refresh Tokens
I've been struggling to understand the purpose of short-lived access tokens and long-lived refresh tokens for a while now. I haven't been able to find many in-depth writeups about why the pattern is recommended and less so about how to use it in a meaningful way. I understood the gist of it - you'd want short-lived access tokens so if it's compromised, the bad actor only has a limited timeframe to cause damage. What I didn't understand is how that would prevent the same issue from happening with a long-lived refresh token.
I did some digging and this is what I concluded: https://medium.com/@getintheshell/finally-understanding-the-benefits-of-a-long-lived-refresh-token-bf021176a9d1
Basically:
- Short-lived access token to make auth stateless on the server. No need to query a DB for every API call.
- Long-lived refresh tokens to send out new access tokens.
- Blacklist for refresh tokens. If a token is compromised, we can add the refresh token to the blacklist to prevent it from generating any new access tokens. This will require a query to the blacklist every time the refresh API is invoked. This is fine because querying once for refresh is faster than querying for every API call for auth.
I thought I finally understood but then someone left a comment on my medium post that completely stumped me and now I'm back to square one.
Their question:
I am using google oauth and it provides me with refresh and access tokens. I just revoked access of the credentials directly from my google account which should have made the refresh token invalid. Now, the access token is immediately not able to access google apis. Do you have anything to add on this front? According to this article, I should still be able to access the google APIs for 4 hours(life of google access tokens) without any issues.If Google stores the access tokens in the blacklist, why not just use a long-lived access token? This seems very much like a stateful approach, no? I guess they could be using the access tokens for the claims so they don't have to query the DB for the user's access permissions, but at that point, why not just make the access token long-lived and remove the refresh token entirely? If a long-lived access token is compromised, why not just blacklist the access token? Seems like it would have the same end result.
Medium
Finally Understanding the Benefits of a Long-lived Refresh Token
This article assumes you understand what access tokens and refresh tokens are and how they are meant to work.
Solution:Jump to solution
if u have an access token valid for 5 mins, and a refresh token valid for 365 days
blacklisting the refresh token, u gotta store the blacklisted value for an entire year.. by which time a lotta blacklists could've happened, so storing it in an in memory database isnt really possible
On the flip side, blacklisting the access token is only necessary for 5 mins, which IS possible in an in memory database.. AND u get the added benefit of instant token revokability..
A Redis lookup isnt really all that time consuming when u really consider things
...
47 Replies
It's not the same result because eliminating refresh tokens for a long-lived access token undermines the efficiency and scalability of the entire pattern.
Think about millions or billions of users and devices, you shouldn't eliminate the refresh token because doing so would make access tokens effectively stateful.
If a long-lived access token is compromised, you must explicitly track and revoke it server-side, whereas with the short-lived access token and refresh token pattern, the compromised access token would naturally expire, limiting the damage without requiring server-side intervention.
But that seems to be how Google does it. See the quoted question I posted - if you revoke the token, the access token stops working too, which implies that theyβre adding the refresh token and access token to the blacklist.
So if they add the access token to the blacklist, we bypass waiting for it to expire, but this seems like it defeats the purpose of having a stateless token.
I don't know if Google really does that. Usually, you just blacklist the refresh token so that new access token can't be generated. But assuming they do, it doesn't defeat the purpose. It's just a security trade off. The access token is still stateless most of the time.
As I understand, the purpose of refresh and access tokens are mainly security, not to lessen the compute time.
You have an access token that is:
* giving you access to stuff
* can be used in parallel
* you send it in almost every request
* BUT it is short-leaved
Refresh tokens are:
* far more dangerous
* they give you longer access to basically everything (via acquiring the access tokens)
* SO usually there are additional strategies to secure them, that would not work with access tokens
For example, best practice is one-time-use refresh tokens.
When you use a refresh token to get an access token, you actually get both (new refresh token too). And the old one is no longer valid.
So even if you leaked it - it would be dangerous for the time of life of access token.
This strategy can't be useful with access tokens, since they are usually sent in parallel.
Solution
if u have an access token valid for 5 mins, and a refresh token valid for 365 days
blacklisting the refresh token, u gotta store the blacklisted value for an entire year.. by which time a lotta blacklists could've happened, so storing it in an in memory database isnt really possible
On the flip side, blacklisting the access token is only necessary for 5 mins, which IS possible in an in memory database.. AND u get the added benefit of instant token revokability..
A Redis lookup isnt really all that time consuming when u really consider things
SO doing both.. the refresh token in a proper database, whereas the access token in an in-memory database works well
Now this is just an educated guess which sorta takes into account ur specific guess.. how they REALLY go is hidden for security reasons.. but they could be using smth like this in combination with "security contexts" that they keep track of to pull this off
Oh, I haven't even considered using an in-memory blacklist for access tokens. That makes a lot of sense to me. Another bandaid on top to increase security for a slight performance hit.
i really reccommend you to learn about jwt auth not from blog posts or first results that came out after searching about it. They are made by people who usually just repeat what other people say 1:1. OWSAP is a great source to learn more about jwt tokens.
So if I understand this correctly, the access token is what you send over the wire for your requests, and the refresh token you only send when you need a new access token.
My question is doesn't the one-time use part run into the same problem of needing to store something related to the refresh token? (Maybe a hash or some such.) Or are the requests for that so infrequent that it technically doesn't matter?
"Need" is a strong word. The security have some trade-offs. It could be UX (you must relogin often or use 2fa), or some compute time or both.
There is no "need" in that. You go as far as you deem neccessary for your project.
But the question was - why not use just one long-live token?
And the answer is - it is less secure, since you can't have different policies (like one-time use and short-lived).
The process of black-listing is a separate topic - and do you want to store blacklist access tokens and check the list every request - is similarly - how much security do you want. Google opts to revoke all access immediately.
I see, makes sense!
Ya all make sense, perfect. BUT, If an actor is that proficient that he can get/steal your access token (that is short lived), how hard would it be for him to steal the refresh token?
I think Google's access token lifespan is 1h, so a person just have to sniff your network for 1.01h at max to get your refresh token also, then your'e f*ed π₯².
Only if it's being sent in plaintext or otherwise unencrypted
That's what the blacklist is for. If they steal your refresh token, you add it to the blacklist and it can no longer generate access tokens.
not really blacklisting is not for stolen tokens, more like 1) log-out of access token 2) banning user.
stolen refresh token can be easily mitigated by updating database with a new refresh token
Why wouldn't you blacklist stolen tokens? In terms of the refresh token, it'd have the exact same result as logging out and banning. In all three cases, adding it to the blacklist would prevent it from making new access tokens.
assuming that you implemented short, long live tokens, it is practically impossible to cause any real harm to user or to have any time to receive a report and add to deny list access token.
Oh, are you talking about access tokens? I was responding to Ahmed's question about refresh tokens.
In the context of access tokens:
it is practically impossible to cause any real harm to userI don't know if I quite agree with this. Even if the attacker has 5 minutes, if they come in with a plan of which APIs to invoke, it could cause serious harm. For example, let's say they invoke an API to delete the user's account. That can be done pretty easily within the lifespan of the access token. That's a pretty simple example, but I'm sure there are far more possibilities that a well thought-out attack could produce in the lifespan of an access token.
have any time to receive a report and add to deny list access tokenThis is where a revoke API can come in handy. If you know that your tokens have been stolen, or if the system is able to automatically detect malicious actors, triggering a revoke API could automatically place the refresh and access tokens into blacklists. Of course, this means that you're going to need to keep a reference of all the non-expired access tokens in order to know which access tokens to place into the blacklist, adding more state - but this state wouldn't be used for auth, just for revocation purposes, so it should be fine.
i was talking about both. Access token and refresh token.
you do not allow a user to delete account without additional step of security, at minimum you require a password.
again you don`t need to put refresh token in blacklist, you simply update database with a new refresh token
that is why i recommended to learn about security not from blog posts or first search results, they will always say 1:1 what other say. I was a victim once of this myself.
Why would you store all refresh tokens in a database? Wouldn't they also be JWT, and therefore self-contained? Storing a blacklist would be much more efficient from a scale POV, no? Regardless, it should have the same end result.
you do not allow a user to delete account without additional step of security, at minimum you require a password.Sure, this was just a general example. There are other ways that an attacker can cause damage in a short timeframe. Do you have any specific recommendations for resources?
you are talking about leakage of refresh token right? The only usage of refresh token is to hit refresh tokens endpoint. In that endpoint you check if refresh toekn are matched with refresh which is signed with a user. If refresh token was leaked, you can update users refresh token basically making it useless to generate new tokens.
But this can also be accomplished with a blacklist, which will hold less data than an exhaustive list of all refresh tokens
no, because refresh token are valid for days/month
you will save in blacklist refresh token until it will expire
imagine if users refresh token will be leaked 3 times in the row, are you gonna add all 3 refresh tokens in black list
From my understanding, yes, since it's a self-contained JWT. If it was a session token, then you wouldn't have to.
I guess a refresh token being a session token could be more effective, though
why would you want to save all 3 refresh tokens in black list when you can just update refresh toekn that is related to uesr once
black list will use more and more memory
Hmmm yeah, I see where you're coming from
So more of a whitelist than a blacklist for refresh tokens
yes you can say that
Actually, no. I still think the blacklist makes more sense here.
Here's an example. You have 1,000 users. 500 of these users sign in with only one device and need only 1 refresh token. The other 500 users sign in with two devices and need 2 refresh tokens.
With a whitelist, you're storing 1,500 tokens.
With a blacklist, you're storing 0 tokens.
Let's say 250 of these tokens leak and the system automatically revokes these refresh tokens.
The whitelist will now store 1,250 tokens.
The blacklist will store 250 tokens.
As the number of users increase, and the number of logins are invoked, the amount of tokens you need to store in the whitelist also increases. This makes it much more stateful on the server-side, which goes against the nature of JWT.
A blacklist could theoretically outgrow the whitelist if someone's able to leak a massive amount of tokens. In the average case, though, the whitelist will be larger.
1)
with black list you have to look up in black list AND in user/token table with each request. With whitelist you have to look up at only database once.
2)
now lets say this scenario occurs:
you are storing 1500 tokens
250 of these tokens leak and the system automatically revokes these refresh tokens.
The blacklist will store 250 tokens.
the question, what will happen if user will generate new refresh token?
do you add old refresh token in black list?
if no, it IS a security concern
if yes, it becomes statefull auth indeed
memory black list will growth really fast
Why not just create new refresh tokens for those users that get leaked? Why store the leaked one
If the only thing those refresh tokens are used for is to get more access tokens, why not on revoke, just roll a new one?
yes, the "white list"
In a blacklist system, for those users who have their refresh tokens leaked, would you update their "good" refresh token or leave it as the leaked one and just say "hey i see your refresh token is valid but it's in the blacklist"?
i am sorry i didnt quite understand the question, english is not my first language
with black list you have to look up in black list AND in user/token table with each requestFrom what I understand, you don't do this. JWTs are self-contained. A user/token table would be a session-based approach, not JWT.
the question, what will happen if user will generate new refresh token? do you add old refresh token in black list?You do not add the old refresh tokens to the black list because this implies that they can only sign in with one device at a time. You'd only add them to the blacklist when the revoke API is invoked (like during logout).
you have to look up in the db when refreshing tokens, or you wont get updated context (updated roles and etc) in jwt token.
i didnt understand your last point a little
you have to look up in the db when refreshing tokens, or you wont get updated context (updated roles and etc) in jwt token.Oooooh this is interesting. You're right.
Plus you have to know when those refresh tokens expire, and you can't trust the client on that
The expiration of the tokens is self-contained inside of the token itself, so that shouldn't be an issue
The client will know not to send an expired refresh token?
I guess you could present the login screen to the user if there is no refresh token, but afaik the token doesn't self-destruct when it expires
You're right that you have to check the user table for every refresh but you'd do the same thing *regardless * if you're using a whitelist or blacklist.
In a whitelist:
1. User sends in refresh token to the refresh API
2. Server checks to see if the refresh token is in the whitelist
3. If it is, it pulls data from the user table to update claims
In a blacklist:
1. User sends in refresh token to the refresh API
2. Server checks to see if the refresh token is not in the blacklist
3. If it is not, it pulls data from the user table to update claims
The only difference is that we're looking for a match in the whitelist and looking for a miss in the blacklist.
In this case, the whitelist will still contain more data with the same number of calls to the DB. Since the user can have multiple refresh tokens (logged in to multiple devices), you'll need to store your whitelist separate from the user table.
If the refresh token expires, it's up to the client to remove it from its local state, which it should be able to do because the expiration is stated in the token. If it tries to send an expired refresh token to the server, the server will send back an error.
Using a refresh token and an access token provides a good mix of security, revocability and performance without adding significant amounts of complexity. Think of the typical lifecycle:
1. User logs in - is granted a refresh token.
2. User calls some APIs, does some things - they would use a short lived access token.
3. Someone hacks both the access token and refresh token
Revocation:
the access token could be allowed to expire, or a small kv-db or cache could be used to revoke it. Problem solved for the access token, but not yet the refresh token.
Since you are rarely touching the refresh token, it can be less optimized - keep a token id in the JWT refresh token and mark it revoked at the database. It shouldn't have a huge performance penalty because the refresh tokens are rarely used.
The best part is since access tokens are very unlikely to be revoked, the amount of storage required to store if an access token is revoked is negligible.
Performance:
There will always be fewer access tokens than refresh tokens. They expire fast, so do not need to be persisted long term. It's really easy to optimize them to use redis or another lightning fast store.
Obviously, if somoene gets a hold of an access token or refresh token, it takes moments to destroy somoene's account. Thus for destructive actions, it's anyways a good idea to re-authenticate the user.
Handling expired refresh tokens depends on your business rules. For some use cases (such as banks, for example), you may not allow the refresh token to be renewed. But in other cases, the refresh tokens can be renewed.
Ok, but would a user need more then 1 refresh token per device? Also why do you want to keep track of all the previouse ones?
It gets leaked then you replace it, old ones are by definition blocked cuz there is a new one in their place
So there is no need to keep the old ones to block them cuz they don't have access anyway
but would a user need more then 1 refresh token per deviceNo, a user won't need more than one refresh token per device but each user can have more than one device.
Also why do you want to keep track of all the previouse ones?You don't. Only the ones that are passed to a revoke API (e.g., logging out or the system automatically detecting fraudulent behaviours).
So there is no need to keep the old ones to block them cuz they don't have access anywayI'm arguing for storing less data. A blacklist will only store *revoked * unexpired refresh tokens. A white list will store all unrevoked unexpired refresh tokens. There will (most likely) be more unrevoked tokens than revoked tokens.
How would you reauthenticate a user if i may ask?
If both tokens are compromised
I was also wondering wouldnt using csrf protect against XSS?
user enters their username / password again
xss is just one possible vulnerability
Oh so you suggest keeping an extra id in the jwt refresh token and storing in the DB
For reference in case both tokens get hacked?
possibly. you'd just need enough information to invalidate the JWT.
i wonder if this is a standard
because im new to backend, and Im following OWASP guidelines but they suggest using a whitelist
but i havent find any info on how to approach security if both tokens get hacked
my guess is just then to send some email like how google does when another device has access to your account
and simply reset their password etc
but even if the hacker gets access, using CSRF should help already and using content policy to prevent this
and getting access seems pretty difficult, someone would have to do it improperly for it to happen