nomaxx117
nomaxx117
PostsComments
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
Part of what threw me off here was that load was legitimately imbalanced - the lack of a cache meant that there were more RPS for one worker than another hitting everything behind nginx, so things like CPU and memory usage were higher.
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
So, I figured this out. My nginx cache was broken on the node seeing elevated traffic, and my metrics were generated in a proxy which was layered after nginx. I first instrumented cloudflared and saw even RPS. Then I instrumented nginx and saw the same. I then realized that the node with higher traffic saw identical traffic before and after the nginx layer, despite the supposed presence of a cache, so maybe that was the broken node? Tailing the error logs there revealed that there was a classic permissions failure there causing the cache to be circumvented. As far as why only one node got health check alerts, it appears to be due to the nature of the outages I was looking at - the outages were with things like my Redis cluster, not the front-line workers themselves. The cache was preventing the health checks on the second worker from spotting availability issues. I remediated my issues by fixing the permissions issue and disabling caching on the health check endpoint I was using on both workers.
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
will be double checking my logging
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
i'll do more digging, wonder if something is borked on my end
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
but this disagrees with what logs show when i tail journalctl
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
weird
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
Cloudflared tunnel metrics show even rps to each tunnel
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
i'll go check these
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
also, i too am curious about how i got myself into this mess 😂
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
how would one do this?
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
neither did making the tunnels remote
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
i am bamboozled
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
that did not fix the issue
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
i'm just gonna delete the tunnel and make a new one
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
really is lol
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
@Cyb3r-Jok3 i legitimately have no idea how i did this lmao
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
i continue to find novel ways of breaking computers lol
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
how is this possible
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
heavy-worker-1 gets all the monitor traffic
38 replies
CDCloudflare Developers
Created by nomaxx117 on 1/13/2024 in #general-help
Only one tunnel gets checked by the load balancer monitor
from heavy-worker-1 
➜  ~ cloudflared tunnel info heavy-worker-1                                                       
NAME:     heavy-worker-1
ID:       aff69054-<REST>
CREATED:  2023-05-19 22:41:36.32161 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION  ORIGIN IP      EDGE                      
7b612dfc-<REST> 2024-01-14T00:34:27Z linux_arm64  2024.1.2 104.13.171.136 1xdfw01, 1xdfw05, 2xmci01 
➜  ~ cloudflared tunnel info heavy-worker-2                                                       
NAME:     heavy-worker-1
ID:       aff69054-<REST>
CREATED:  2023-05-19 22:41:36.32161 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION  ORIGIN IP      EDGE                      
7b612dfc-<REST> 2024-01-14T00:34:27Z linux_arm64  2024.1.2 104.13.171.136 1xdfw01, 1xdfw05, 2xmci01
➜  ~ cloudflared tunnel info heavy-worker-1                                                       
NAME:     heavy-worker-1
ID:       aff69054-<REST>
CREATED:  2023-05-19 22:41:36.32161 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION  ORIGIN IP      EDGE                      
7b612dfc-<REST> 2024-01-14T00:34:27Z linux_arm64  2024.1.2 104.13.171.136 1xdfw01, 1xdfw05, 2xmci01 
➜  ~ cloudflared tunnel info heavy-worker-2                                                       
NAME:     heavy-worker-1
ID:       aff69054-<REST>
CREATED:  2023-05-19 22:41:36.32161 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION  ORIGIN IP      EDGE                      
7b612dfc-<REST> 2024-01-14T00:34:27Z linux_arm64  2024.1.2 104.13.171.136 1xdfw01, 1xdfw05, 2xmci01
from heavy-worker-2: 
➜  ~ cloudflared tunnel info heavy-worker-1
NAME:     heavy-worker-1
ID:       aff69054-<REST>
CREATED:  2023-05-19 22:41:36.32161 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION  ORIGIN IP      EDGE                      
7b612dfc-<REST> 2024-01-14T00:34:27Z linux_arm64  2024.1.2 104.13.171.136 1xdfw01, 1xdfw05, 2xmci01 
➜  ~ cloudflared tunnel info heavy-worker-2
NAME:     heavy-worker-2
ID:       b7561864-<REST>
CREATED:  2024-01-14 00:49:37.245692 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION  ORIGIN IP      EDGE                      
d537845f-<REST> 2024-01-14T00:51:00Z linux_arm64  2024.1.2 104.13.171.136 1xdfw06, 1xdfw09, 2xmci01
➜  ~ cloudflared tunnel info heavy-worker-1
NAME:     heavy-worker-1
ID:       aff69054-<REST>
CREATED:  2023-05-19 22:41:36.32161 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION  ORIGIN IP      EDGE                      
7b612dfc-<REST> 2024-01-14T00:34:27Z linux_arm64  2024.1.2 104.13.171.136 1xdfw01, 1xdfw05, 2xmci01 
➜  ~ cloudflared tunnel info heavy-worker-2
NAME:     heavy-worker-2
ID:       b7561864-<REST>
CREATED:  2024-01-14 00:49:37.245692 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION  ORIGIN IP      EDGE                      
d537845f-<REST> 2024-01-14T00:51:00Z linux_arm64  2024.1.2 104.13.171.136 1xdfw06, 1xdfw09, 2xmci01
38 replies