Message sync is slow?
I set up v0.20.0 as a fresh install in docker.
updated my docker compose file to the new versions.
Ran
docker exec -it twenty-worker-1 yarn command:prod cron:messaging:messages-import
docker exec -it twenty-worker-1 yarn command:prod cron:messaging:message-list-fetch
I can see the sync in the logs, but it only syncs 20 messages every minute.
Is that normal? can we make it go quicker?
at this rate, it'll sync my emails in about 10 hours š¤£
5 Replies
Unknown Userā¢3mo ago
Message Not Public
Sign In & Join Server To View
just a reference
Hello @paulyili, in self hosting by default pg-boss is used for the cache. Unfortunately, crons can run once every minute with pg-boss at max. If you want to accelerate the import, you'll have to host a redis locally and use bull-mq, and change the environment variables
CACHE_STORAGE_TYPE=redis and MESSAGE_QUEUE_TYPE=bull-mq. The cron will then run once every 5 seconds and import 20 messages every 5 seconds. We put a limit of 20 messages by batch because of google's rate limiting, if we try to retrieve more messages in a batch we have a rate limiting error.
We have to improve the documentation for this, sorry for the issue
If you want to change this value, you can do so in packages/twenty-server/src/modules/messaging/message-import-manager/drivers/gmail/constants/messaging-gmail-users-messages-get-batch-size.constant.tsHey @Raphaƫl oh thanks! I had reddis set up, but forgot to change to bull-mq.
I decided to nuke my instance and try again to see. lightning fast now š
Glad to help š