Understanding the Worker Process?
There have been mentions of the worker process and how this is needed to process tasks I presume in Redis or the alternative postgres extension.
How can this Worker process be setup and what environment variables are required for it? - There have been mentions of an alternative init command to start the worker node but what environment variables does it require?
Attempting to use a docker-compose file
When attempting to use the Gmail integration - it is stuck on pending which I guess is that a worker is not processing the output?
4 Replies
Hi @A, the documentation is inexistant on workers, I agree
Here is a thread on how to set it up:
https://discord.com/channels/1130383047699738754/1236757275880722573/1237383594520084530
We will add this to the doc soon, we are still 100% sure about the command API
@charles Hi again haha!
Regarding this then:
env variable: MESSAGE_QUEUE_TYPE=pg-boss
env variable: CALENDAR_PROVIDER_GOOGLE_ENABLED=true
env variable: MESSAGING_PROVIDER_GMAIL_ENABLED=true
env variable: AUTH_GOOGLE_CLIENT_ID, AUTH_GOOGLE_CLIENT_SECRET, AUTH_GOOGLE_APIS_CALLBACK_URL
are these the environment variables that the worker requires or are more needed on top of this? As I presume Server URL/db/redis may be required if in use
I have no issues with overwriting the command on the worker process using
yarn worker:prod
But the command yarn command:prod cron:messaging:gmail-fetch-messages-from-cache Is this command run post setup once the docker server instance is up and running? Aka the docker compose/or other deployment, executes the docker container as normal with yarn start:prod and then you have to start a shell in docker to execute the command or is it a direct replacement for yarn start:prod?
What I am concerned about here is that it suggests the need for a startup script of sortsRegarding Cron, I can also see that active changes are happening: https://github.com/twentyhq/twenty/issues/5497 which I presume IS_GMAIL_SYNC_V2_ENABLED would mean there is no need to execute the cron command?...
GitHub
Implement the new way to synchronise emails from gmail · Issue #549...
Scope & Context Technical inputs Create a feature flag: IS_GMAIL_SYNC_V2_ENABLED Create a new cron FullSyncMessageListFetchCron, which will be executed every 5 minutes and checks whether the sy...
Ok!
1) we have several options for MESSAGE_QUEUE_TYPE: sync | pg-boss | bull-mq. Sync is not recommended for production. I was suggesting pg-boss as it will store jobs in your existing postgres database and worker will leverage it. However if you are familiar with redis this is a better solution (we have a better control on worker with redis behavior).
2) In case you want to use redis, env variables are listed here: https://docs.twenty.com/start/self-hosting/#config
3) We will keep using crons in the future, Cron need to be registered once. they will be stored in postgres or redis depending on what message_queue_type you are using. It does not matter if you register them before or after the worker is started
Welcome | Twenty - Documentation
Twenty is a CRM designed to fit your unique business needs.