I currently have a queue setup where the
I currently have a queue setup where the consumer worker initiates a CSV download from an API, continues to poll the API until the CSV is available, downloads the CSV, and then stores it in R2.
Right now only 2 messages are getting sent to the queue at a time (via a cron job at the top of every hour), but only the first one ever gets processed (it's processed to completion).
Could this be a CPU limit issue? I don't see any error logs. These are my current .toml settings:
1 Reply
could you share your queue ID and account ID? Both are safe to share here
max_batch_size = 1 means that your Queue consumer will only consume one message at a time.
Your queue consumers will auto scale, but this auto-scaling kicks in only at the end of processing a batch. So, if it takes a while to poll the API, your queue will be "stuck"