Hey guys, I'm very desperate with a
Hey guys, I'm very desperate with a worker problem that we're fighting with for many weeks now and that the CF support team told me to look on discord from devs for help. Our platform is making a lot of requests to quickbooks API through a cloudflare worker, and we're getting a LOT of 403 errors when trying to fetch quickbooks from the CF worker.
We've escalated this pretty high up with quickbooks, but they've debugged this and insist that our request are not hitting their APIs. I have no idea where the 403 may come from. We use WAF for incoming requests, but this should not touch outgoing requests.
We ONLY see this happening on production. On local development we never get these errors, even when doing the same workload locally.
The error started popping up without explanation, no changes to infra or our code. It just started some day. The error often goes away if we retry, but sometimes it even happens 6 times in a row (often it works after 2-3 retries). It also happens if we reduce concurrency.
We use typescript workers with fetch(), so nothing fancy.
At this point I really don't know where to look at. Is there anything you guys can advice us to check for? My only idea remaining idea right now is to switch production to use something like fly instead of workers, since things seems to work well outside of CF production systems.
Would love to find a way getting it to work on CF, we're really huge believers of the platform.
10 Replies
apologies if its inappropriate to ping you guys directly, but maybe you have an idea @Walshy | Deploying?
Here is example headers we receive as part of the 403
"cf-cache-status": "DYNAMIC",
"cf-ray": "89de5873e20c9d58-DME",
"connection": "keep-alive",
"content-type": "text/html",
"date": "Thu, 04 Jul 2024 10:13:57 GMT",
"server": "cloudflare",
"transfer-encoding": "chunked"
but we can't find anything under this cf-ray id on our cf logs
"cf-cache-status": "DYNAMIC",the fact we see dynamic indicates this came from an origin got a url i can hit?
You mean the quickbooks api we call?
The above request happened on
a url for your worker where you do the fetch and see the 403
also think i saw your ticket..
the worker is
integration-quickbooks.eco.vrp.rocks
so integration-quickbooks.eco.vrp.rocks to quickbooks.api.intuit.comah yep, saw your ticket - it is coming from an origin (guessing quickbooks)
do you think they may be blocking or ratelimitting cloudflare IPs specifically?
it wouldn't surprise me if they blocked the worker IP yeah
hence things work from local or fly.io?
argh
it makes sense, thanks @Walshy | Deploying
no worries, seems they use AWS - make sure they check whatever AWS WAF is