Google Calendar Sync Fails (GH Issue 8743)
https://github.com/twentyhq/twenty/issues/8743
Basically it's a scenario where the google integration works fine (as I can authenticate via google) and the auth screens present fine, but when google sends you back to the twenty server, the server flips out and crashes.
GitHub
When google calendar / auth is enabled, the authentication, authori...
Bug Description Google authentication for SSO and calendar sync is enabled on the server. The google client-id and client-secret are set, and google integration works. SSO login for users is functi...
37 Replies
@Raphaël Heya, figured I'd tag you here as mentioned
So the initial problem I think I've solved. I had the
AUTH_GOOGLE variables defined in the server, but not the worker. Defined them and re-deployed and the worker doesn't flip out now. Sync still shows failed, but not a lot of details as to why. Is there any information / documentation about how that whole process functions? When I try to reconnect I get:
Just get an immediate
Sync Failed
Running any of:
Just dumps a lot of Nest output, but no errors. So I'm kind of in the dark as to what's happening. Is there a way to turn up logging on the nest logger?
Normally we log errors inside the worker
Could you send me the calendarChannel syncStatus and the messageChannel syncStatus and also their syncStage ?
Do you see any events inside the db?
There's no errors in the worker, but the state is always failed. Not even pending. Have to check in the db. Where in the schema should I look?
Inside your workspace schema you will find calendarChannel and messageChannel
For message status it's "ONGOING" but for calendar status it's "FAILED_INSUFFICIENT_PERMISSIONS". Are there specifications for the OIDC requirements for calendar syncing anywhere? There may be something I missed in defining the google entry? That doesn't make a lot of sense since the consent screen explicitly lists the app asking for all of your calendars
Well after a few minutes they're both
FAILED_INSUFFICIENT_PERMISSIONS
There's also no errors in the worker that I can see
They are exiting overly fast (messages specifically)
I think I may have found it
Well I thought. I was looking at the HAR of traffic to and from google to verify the scopes / claims requested since invalid_grant is one of the triggers I could find in the source code to set that state.
I added
https://www.googleapis.com/auth/calendar.events
https://www.googleapis.com/auth/profile.emails.read
To the sensitive scopes for the OIDC client, but no change afterwards. Was hoping that was it
It may be a timing concern. I may try again later to see if the changes affect anything. Honestly if failures like that occur, some additional logging would be beneficial
Even with debug mode on the worker and all logging levels enabled, I don't get any information. It just fails
If it provides anything to assist, the worker configuration (redacted) is below:
Did you enable people api ?
On the google cloud console
We will need to improve this part indeed, we also have to update the documentation for self hosting for email an calendar connection. Sorry if it's a bit confusing!
Yeah google people API is enabled
I just silently get INSUFFICIENT PERMISSIONS for both calendar / gmail sync.
So any additional details from the worker would be highly beneficial, and some clear guidance on requirements from google and its client.
I may try to create a PR if I can figure out javascript to enable additional logging at the closures where the message gets processed
@charles
I think the most efficient would be to schedule a call with @shairozan and to compare the setup with the one we have in production
We can also add a bit more log in this section before putting the channel in INSUFFICIENT_PERMISSIONS
Last question @shairozan could you double check the scopes attached to your individual email acconts
Here is an example (sorry it's french)
C'est bien. Yeah I can double check and provide the full details for the scopes / claims. I went through the source code to find the scope details for allocation.
May be some differences. I notice that yours lists:
* Profile Data
* Reading your Emails
* Sending Emails
* Managing Calendar
For mine it's viewing email and settings but not processing / sending. Looks to be the only real difference I can see.
It is worth pointing out that readonly for gmail is a restricted scope. That may have some affect in workspaces accounts for private / internal clients. That's something someone more familiar with google's odd permissions structure would have to comment on
Here are the scopes enabled for the OIDC client
These were all derived from the OIDC authorization request. Capturing the traffic, the scopes requested were:
I don't think not having email sending is a problem based on the scopes here from an OIDC perspective. It may be that in the ingest process it expects email to be setup for outbound communications, and without that goes into insufficient permissions?
@Raphaël @martmull I don't see the scope to SEND emails ?
Is it potentially that the permissions expects the scope to send emails, but because I haven't configured anything for sending that it isn't getting expressed in OAuth Authorization request?
I'm going to add that to the client and see if it's getting requested at the least
Nah. The requested scopes in the authorization from twenty still don't have email send. OIDC client allows it as a claim, just need to know what toggles
isGmailSendEmailScopeEnabled Maybe that I'm missing a configuration item?
Looks like it's handled in the core schema in postgres based on my provider. I found the features table and am pretty sure the key is IS_GMAIL_SEND_EMAIL_SCOPE_ENABLED, but the workspaceID value expects a UUIDI'm also interested in this.
I haven't dug into it as of lately as it's not a critical thing for us at the moment, BUT the sync isn't as stable as I had hoped it would be.
Ah figured that part out. It's also in the core space
Yeah you have to enable the feature in the core schema
Now let's see if it flips out after auth
Nah immediately goes into "Sync lost with mailbox". I'll give it some time as I have to get to a meeting
Ok, so mine is sort of the opposite. Calendar works flawlessly, and the email sync doesn't complete. I'm going to go back and dig into it all this week. I hope it all works out. I'll follow along.
Neither even starts for me. Both channels fall into
INSUFFICIENT_PERMISSIONS so I haven't even started yet
I'm at least getting closer to the ref from the above
Is what we're seeing in the auth request after enabling the featureI forgot to mention that I'm available for a call to discuss about it, but my times are kind of crappy. You could book some time here
I don't see the AUTH_GOOGLE_APIS_CALLBACK_URL=http://<REDACTED>/auth/google-apis/get-access-token in your env vars
It's set
Also if the callback URL wasn't set, we would have gotten to a network traffic issue. The attempt to come back from google after the authorization would have pointed at the default and failed. Wouldn't have gotten to the point of ingesting a token
So after adding the Gmail / Calendar APIs sync started, but it fails periodically with this in the worker logs
I can always create a separate issue in GH if we want to vs continuing to drag this one on
It seems like it's something dealing with bigquery
At least that's where I see it come up a lot. I don't see any references from the codebase to bigquery. It may be a scenario where its' a specific error that's not catalogued into a try / catch and is falling into a default throw within the message import
It may be because I'm in a workspace with a private (internal) OIDC client. https://developers.google.com/identity/protocols/oauth2/production-readiness/google-workspace. We may be able to trust the app there. may have to investigate with my own personal workspace account
May be the case, but I'm not sure what qualifies as "high risk" from their perspective. If we're accessing attachments, it may be considered high risk
Ok looks like you can specify in API Controls under settings to trust internal apps. That may get it to work. I'd have to work with our google admins to see if that resolves it, but I'm betting that's probably where it's flipping out
I think this may solve it indeed
Is your google app in test mode or is it published ?
It's not published anywhere but in the workspace
To make it public means going through google verification, which takes money
If it does, I'll make a PR for the documentation to explain requirements for an internal (workspace specific) OIDC client
It doesn't cost anything to go through google verification but it takes a little bit of time, but you don't have to publish it to make it work. It's just that users that aren't test users will receive a warning while trying to connect to the app.
Our use case is purely internal so an internal OIDC client inside of the workspaces makes more sense. The users see it like a typical google auth interaction and we don't have to worry about it ever being seen outside of the workspace. Thanks for the info though since I've traditionally shied away from public google OIDC integrations