36 Replies
so the errors are all massive HTML responses like:
both staging and prod went down within the same few minutes, and we haven't pushed code since midnight my time
so abt 13 hrs ago
https://dash.cloudflare.com/df681cd207a9e2afa394d260c486fd1e/workers/services/view/htmxpushproxy/production?time-window=1440&versionFilter=all if you can access
What does wrangler tail show?
checking, that's going to explode my console XD
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
so many of these
like
Tail is currently in sampling mode due to the high volume of messages. To prevent messages from being dropped consider adding filters.The production workers.dev having problems is known - we're working on a fix
The issue comes from having a worker named {8hexChars}-{whatever}
Cancelled usually indicates eyeball disconnect. Sure the origin is ok?
for sure
the rest of the system is working fine
it's also happening on our staging version of this service too
Looking into it
appreciate you, lmk if i can provide any info
I isolated it to the presence of a
Content-Length: 0 header in the request and using workers.dev subdomain. Same for you guys?Can you give me a url to repro?
yeah sec
https://htmxpushproxy.tangia.workers.dev/ is the base url, checking the request to get more info on what's needed
HtmxPusherURL + "/creators/twitch:" + creatorTwitchID + "/trigger?trigger=" + trigger is our url construction. getting a final url right now
https://htmxpushproxy.tangia.workers.dev/creators/twitch:75011839/trigger?trigger=interactions-changedAnd it's a GET?
post sorry
sometimes we don't have a body
Info for my repro here
https://discord.com/channels/595317990191398933/779390076219686943/1288953613292671026
I didn't see this thread before I posted
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
this might be the same thing, i don't have the content length of the requests going out unfortunately, I can add that if we need
actually i 100% know, it's an empty body
we're also getting more failures across some of our other services using workers on .dev domains as well
I'm going to start giving heads up tweets, please let me know if there's any more info to this, since it's causing pretty major downtime for us
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
would a short-term fix for us be adding a custom domain?
If you redeploy, you should be back in action.
even with .dev domains?
yes, we're still working to figure out the issue but we'll do that before rolling back out
Sorry for the issue!
ill try deploying real quick
(note: it may take a few mins for it to fully work again on deploy while caches get updated)
Is there a way I can redeploy using the same code from the dashboard? I tried just deploying the current version but i can't tell if that did a redeploy
nvm
yep, errors just went to 0 lol
@Walshy | Deploying when will other services be automatically fixed? Or should I just start rdeploying everyhting? We have a fair few
Once you deploy one Worker, the rest should get fixed
even other workers with different names?
Redeploy has not resolved it for mine
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
yes
what's the route?
@Will Taylor | Workers see info here in reply
Seems resolved now