CD
Cloudflare Developers•2w ago
cinstitute

Cloudflared/Tunnels erroring out every couple of hours - leaves website unreachable

Hey yall I've forwarded a service to the web using Cloudflared/Tunnels at my domain (let's call it service.mywebsite.com). Everything has been running fine for months and then suddenly this morning my status page (hosted elsewhere) notified me that all of my services were not responding. Sure enough I could not reach my website. No error/cloudflare notice page they just never loaded. I restarted my machine and things went back to normal for a little bit then an hour or two later, same thing, everything goes down. I started investigating and noticed the local instances were responding with no issue (simple curl localhost:port/api/health endpoint). So I restarted Cloudflared (I use a docker container). Everything was back online and then an hour or two later exact same thing happens again. This time I grabbed the logs below During normal operation the usual cancelled errors, nothing crazy here. 
2025-02-24T03:05:32Z ERR Request failed error="context canceled" connIndex=2 dest=https://service.mywebsite.com/api/realtime event=0 ip=[IP REDACTED] type=http
2025-02-24T03:05:32Z ERR  error="context canceled" connIndex=1 event=1 ingressRule=4 originService=http://[IP REDACTED]:8090
2025-02-24T03:05:32Z ERR Request failed error="context canceled" connIndex=1 dest=https://service.mywebsite.com/api/realtime event=0 ip=[IP REDACTED] type=http
2025-02-24T03:06:07Z ERR  error="context canceled" connIndex=0 event=1 ingressRule=4 originService=http://[IP REDACTED]:8090
2025-02-24T03:06:07Z ERR Request failed error="context canceled" connIndex=0 dest=https://service.mywebsite.com/api/realtime event=0 ip=[IP REDACTED] type=http
2025-02-24T03:06:25Z ERR  error="context canceled" connIndex=2 event=1 ingressRule=4 originService=http://[IP REDACTED]:8090
2025-02-24T03:06:25Z ERR Request failed error="context canceled" connIndex=2 dest=https://service.mywebsite.com/api/realtime event=0 ip=[IP REDACTED] type=http
2025-02-24T03:05:32Z ERR Request failed error="context canceled" connIndex=2 dest=https://service.mywebsite.com/api/realtime event=0 ip=[IP REDACTED] type=http
2025-02-24T03:05:32Z ERR  error="context canceled" connIndex=1 event=1 ingressRule=4 originService=http://[IP REDACTED]:8090
2025-02-24T03:05:32Z ERR Request failed error="context canceled" connIndex=1 dest=https://service.mywebsite.com/api/realtime event=0 ip=[IP REDACTED] type=http
2025-02-24T03:06:07Z ERR  error="context canceled" connIndex=0 event=1 ingressRule=4 originService=http://[IP REDACTED]:8090
2025-02-24T03:06:07Z ERR Request failed error="context canceled" connIndex=0 dest=https://service.mywebsite.com/api/realtime event=0 ip=[IP REDACTED] type=http
2025-02-24T03:06:25Z ERR  error="context canceled" connIndex=2 event=1 ingressRule=4 originService=http://[IP REDACTED]:8090
2025-02-24T03:06:25Z ERR Request failed error="context canceled" connIndex=2 dest=https://service.mywebsite.com/api/realtime event=0 ip=[IP REDACTED] type=http
continued below:
1 Reply
cinstitute
cinstituteOP•2w ago
Then all of the sudden these start flooding in and the services no longer respond. 
2025-02-24T03:53:37Z ERR  error="stream 10545 canceled by remote with error code 0" connIndex=0 event=1 ingressRule=0 originService=http://[IP REDACTED]:5001
2025-02-24T03:53:37Z ERR Request failed error="stream 10545 canceled by remote with error code 0" connIndex=0 dest=https://service.mywebsite.com/endpoint event=0 ip=[IP REDACTED] type=http
2025-02-24T03:53:37Z ERR  error="stream 2085 canceled by remote with error code 0" connIndex=0 event=1 ingressRule=0 originService=http://[IP REDACTED]:5001
2025-02-24T03:53:37Z ERR Request failed error="stream 2085 canceled by remote with error code 0" connIndex=0 dest=https://service.mywebsite.com/endpoint event=0 ip=[IP REDACTED] type=http
2025-02-24T03:53:40Z ERR  error="stream 10565 canceled by remote with error code 0" connIndex=0 event=1 ingressRule=0 originService=http://[IP REDACTED]:5001
2025-02-24T03:53:40Z ERR Request failed error="stream 10565 canceled by remote with error code 0" connIndex=0 dest=https://service.mywebsite.com/endpoint event=0 ip=[IP REDACTED] type=http
2025-02-24T03:53:37Z ERR  error="stream 10545 canceled by remote with error code 0" connIndex=0 event=1 ingressRule=0 originService=http://[IP REDACTED]:5001
2025-02-24T03:53:37Z ERR Request failed error="stream 10545 canceled by remote with error code 0" connIndex=0 dest=https://service.mywebsite.com/endpoint event=0 ip=[IP REDACTED] type=http
2025-02-24T03:53:37Z ERR  error="stream 2085 canceled by remote with error code 0" connIndex=0 event=1 ingressRule=0 originService=http://[IP REDACTED]:5001
2025-02-24T03:53:37Z ERR Request failed error="stream 2085 canceled by remote with error code 0" connIndex=0 dest=https://service.mywebsite.com/endpoint event=0 ip=[IP REDACTED] type=http
2025-02-24T03:53:40Z ERR  error="stream 10565 canceled by remote with error code 0" connIndex=0 event=1 ingressRule=0 originService=http://[IP REDACTED]:5001
2025-02-24T03:53:40Z ERR Request failed error="stream 10565 canceled by remote with error code 0" connIndex=0 dest=https://service.mywebsite.com/endpoint event=0 ip=[IP REDACTED] type=http
I have no idea what is going on or how to fix this. Cloudflare status page says everything is okay but every hour or so my services go down again and there doesn't seem to be a way for them to recover besides restarting the docker container (which I can't stay up and do all night). Additional logs from Cloudflared start 
2025-02-24T02:53:12Z INF Starting tunnel tunnelID=[REMOVED]
2025-02-24T02:53:12Z INF Version 2025.2.0 (Checksum d7e573b235ef4a1cf45d6b837f61094e4a3403c4ef92e1281a8fec3f7f257691)
2025-02-24T02:53:12Z INF GOOS: linux, GOVersion: go1.22.5-devel-cf, GoArch: amd64
2025-02-24T02:53:12Z INF Settings: map[no-autoupdate:true token:*****]
2025-02-24T02:53:12Z INF Generated Connector ID: [REMOVED]
2025-02-24T02:53:12Z INF Initial protocol quic
2025-02-24T02:53:12Z INF ICMP proxy will use 172.17.0.2 as source for IPv4
2025-02-24T02:53:12Z INF ICMP proxy will use ::1 in zone lo as source for IPv6
2025-02-24T02:53:12Z INF ICMP proxy will use 172.17.0.2 as source for IPv4
2025-02-24T02:53:12Z INF ICMP proxy will use ::1 in zone lo as source for IPv6
2025-02-24T02:53:12Z INF Starting metrics server on [::]:20241/metrics
2025-02-24T02:53:12Z INF Using [CurveID(4588) CurveID(25497) CurveP256] as curve preferences connIndex=0 event=0 ip=[REMOVED]
2025/02/24 02:53:12 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2025-02-24T02:53:12Z INF Registered tunnel connection connIndex=0 connection=[REMOVED] event=0 ip=[REMOVED] location=[REMOVED] protocol=quic
2025-02-24T02:53:12Z INF Using [CurveID(4588) CurveID(25497) CurveP256] as curve preferences connIndex=1 event=0 ip=[REMOVED]
2025-02-24T02:53:12Z INF Updated to new configuration config="[REMOVED]" version=54
2025-02-24T02:53:13Z INF Registered tunnel connection connIndex=1 connection=[REMOVED] event=0 ip=[REMOVED] location=lax05 protocol=quic
2025-02-24T02:53:13Z INF Using [CurveID(4588) CurveID(25497) CurveP256] as curve preferences connIndex=2 event=0 ip=198.41.200.53
2025-02-24T02:53:14Z INF Registered tunnel connection connIndex=2 connection=[REMOVED] event=0 ip=[REMOVED] location=lax01 protocol=quic
2025-02-24T02:53:12Z INF Starting tunnel tunnelID=[REMOVED]
2025-02-24T02:53:12Z INF Version 2025.2.0 (Checksum d7e573b235ef4a1cf45d6b837f61094e4a3403c4ef92e1281a8fec3f7f257691)
2025-02-24T02:53:12Z INF GOOS: linux, GOVersion: go1.22.5-devel-cf, GoArch: amd64
2025-02-24T02:53:12Z INF Settings: map[no-autoupdate:true token:*****]
2025-02-24T02:53:12Z INF Generated Connector ID: [REMOVED]
2025-02-24T02:53:12Z INF Initial protocol quic
2025-02-24T02:53:12Z INF ICMP proxy will use 172.17.0.2 as source for IPv4
2025-02-24T02:53:12Z INF ICMP proxy will use ::1 in zone lo as source for IPv6
2025-02-24T02:53:12Z INF ICMP proxy will use 172.17.0.2 as source for IPv4
2025-02-24T02:53:12Z INF ICMP proxy will use ::1 in zone lo as source for IPv6
2025-02-24T02:53:12Z INF Starting metrics server on [::]:20241/metrics
2025-02-24T02:53:12Z INF Using [CurveID(4588) CurveID(25497) CurveP256] as curve preferences connIndex=0 event=0 ip=[REMOVED]
2025/02/24 02:53:12 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2025-02-24T02:53:12Z INF Registered tunnel connection connIndex=0 connection=[REMOVED] event=0 ip=[REMOVED] location=[REMOVED] protocol=quic
2025-02-24T02:53:12Z INF Using [CurveID(4588) CurveID(25497) CurveP256] as curve preferences connIndex=1 event=0 ip=[REMOVED]
2025-02-24T02:53:12Z INF Updated to new configuration config="[REMOVED]" version=54
2025-02-24T02:53:13Z INF Registered tunnel connection connIndex=1 connection=[REMOVED] event=0 ip=[REMOVED] location=lax05 protocol=quic
2025-02-24T02:53:13Z INF Using [CurveID(4588) CurveID(25497) CurveP256] as curve preferences connIndex=2 event=0 ip=198.41.200.53
2025-02-24T02:53:14Z INF Registered tunnel connection connIndex=2 connection=[REMOVED] event=0 ip=[REMOVED] location=lax01 protocol=quic

Did you find this page helpful?