Cloudflared HTTP2Origin Issue
Hey guys, just wondering if there's a solution to this issue I'm having accessing a grpc endpoint behind cloudflared: https://github.com/cloudflare/cloudflared/issues/682
GitHub
🐛 http2Origin option and --http2-origin · Issue #682 · cloudflare/c...
Describe the bug @nikr-canva I added the flag mentioned here to my docker container (version 2022.6.3). In the log Settings: map[http2-origin:true loglevel:debug no-autoupdate:true token:*] whi...
88 Replies
I don't think so. It's also marked here:
https://developers.cloudflare.com/network/grpc-connections/
Cloudflare Tunnel currently does not support gRPC.
Cloudflare Docs
gRPC connections · Cloudflare Network settings docs
Cloudflare offers support for gRPC to protect your APIs on any proxied gRPC endpoints. The gRPC protocol helps build efficient APIs with smaller …
So I guess it’s safe to assume this will never get fixed
hmm, actually I was looking at another github issue and it looks like it might work over http2 not but quic
https://github.com/cloudflare/cloudflared/issues/491#issuecomment-1643233485
try that if you need it
GitHub
grpc support · Issue #491 · cloudflare/cloudflared
Are there any plans/eta on supporting GRPC through cloudflared/argo tunnel?
Oh, I did notice that quic is enabled in my cf dashboard, would disabling that fix it?
no, you have to configure your tunnel to not use quic
quic in your dashboard is unrelated
Oh ok, so in my cloudflared config.yaml file?
if you're using local tunnels could do that, or could modify the way you run the tunnel, ex:
cloudflared tunnel --protocol http2 run <UUID or NAME>
Ok great, I’ll give that a shot, thanks mate
hmm ok i ran that command, but I'm seeing in my cloudflared logs
$ cloudflared tunnel --protocol http2 run my-appWhat do see when you started it up? On the line "Registered tunnel connection" -> something
oh, grpc also isn't going to work over http, need https even if self-signed
like in his example:
https://github.com/cloudflare/cloudflared/issues/491#issuecomment-1643233485
nothing obviously wrong, all services look like this:
hmm already had those config values for the grpc service, still got
originService=http://localhost:8080That's http though, not https
ah good point
may be worth mentioning you need to restart the tunnel after a config change
yep always have done this just to be sure
ok I think I need to change how I deploy my service to get https working, a self-signed "insecure" cert should work with this cloudflared config right?
It should yea. I have not tried that setup myself, but a CF employee in that github thread did say that should work for gRPC, at the cost of not being able to do udp proxying in private networks (which I doubt matters to you)
The Quic transport protocol with cloudflared in general has some weird issues/edge cases like that, not surprised it breaks this
ah ok great, just for context this is how I'm deploying the https version: https://zitadel.com/docs/self-hosting/manage/reverseproxy/caddy
(so using this
127.0.0.1.sslip.io domain)
should that be fine? was able to visit the dashboard of this service, just with an "insecure" cert warning in the url bar
but grpcurl --insecure 127.0.0.1.sslip.io:443 zitadel.admin.v1.AdminService/Healthz did return the expected {} resultyea that looks sane to me, its reverse proxying the grpc down via h2c to the actual origin yea? Would need to make sure you set noTLSVerify
" its reverse proxying the grpc down via h2c to the actual origin" i think so, but my networking knowledge is pretty basic so couldn't say for certain
i guess the other question i have is, would grpc work with my hostname (
auth.example.com) once i set this up?that's the idea, only thing to check there would be if you have gRPC enabled on that website/zone in Cloudflare under Network tab
ok cool, think that setting is already enabled in my cf dashboard. alright, i'll redeploy using this
sslip.io domain and see if we have any luck!
ok so this is what I have: config.yaml
then I run $ cloudflared tunnel --protocol http2 run my-app
then i test the sslip.io domain: $ grpcurl --insecure 127.0.0.1.sslip.io:443 zitadel.admin.v1.AdminService/Healthz (which works, returns {})
and finally i test my own subdomain (auth.example.com) and get this output:
i would've thought grpcurl would include application/grpc for it's content type header by default (but maybe that's not the issue)in the post they stated
One final thing, if you get back an obscure error about a missing content-type header e.g. rpc error: code = Unknown desc = failed to query for service descriptor "some.service": unexpected HTTP status code received from server: 302 (Found); malformed header: missing HTTP content-type ... it might be because you have an Access Policy set on your tunnel and here CF is trying to redirect you to your auth page. There's no way to follow this redirect but if you can get your CF auth token then you can pass it as a header with your grpc request and it should work. E.g.Do you have an Access policy on that? perhaps you could use service tokens as well
ah, I don't think so but let me double check
is that under `https://dash.cloudflare.com/:hash/example.com/access ?
i'll post what my cf settings for my domain look like:
from what I can see I don't think so
It'd be under launch zt => Access => Applications or https://one.dash.cloudflare.com/?to=/:account/access/apps
Cloudflare One
Cloudflare One replaces legacy security perimeters with our global edge, making the Internet faster and safer for teams around the world.
not in the normal dash
ah ok, let me check
don't think so?
doesn't look like it
try running grpcurl with -vv?
no difference in output, still
i'll post the curl output, one sec
yea the headers would be helpful
$ curl --insecure https://auth.example.com:443/admin/v1/healthz -vv
* Trying <IP>:443...
* Connected to auth.example.com (<IP>) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
* subject: CN=example.com
* start date: Dec 16 04:35:40 2023 GMT
* expire date: Mar 15 04:35:39 2024 GMT
* issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1P5
* SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: auth.example.com]
* h2 [:path: /admin/v1/healthz]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: /]
* Using Stream ID: 1 (easy handle 0x13100da00)
GET /admin/v1/healthz HTTP/2 Host: auth.example.com User-Agent: curl/8.1.2 Accept: /> < HTTP/2 200 < date: Thu, 08 Feb 2024 04:45:55 GMT < content-length: 0 < alt-svc: h3=":443"; ma=86400 < cf-cache-status: DYNAMIC < report-to: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v3?s=kWpK3unT4gyqxg%2FtIsS1P1CzpXCUtCLEp1E9FE00Wj5BZkVtnj%2Fu4o%2BMYPY8p7Tm8F4eunjimpGzY4mkWoYb5pWKVkDIqx1%2FuAHLJ%2BRv87Vq3ydugnPu6TocmVrqFvp%2B0lXcyx0%3D"}],"group":"cf-nel","max_age":604800} < nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} < server: cloudflare < cf-ray: 852139d6ff29a81f-SYD < * Connection #0 to host auth.example.com left intact
doesn't tell us too much, other then it went down the tunnel
shouldn't need --insecure btw
You can enable a higher logging level in your tunnel: https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/monitor-tunnels/logs/ and try to see the actual request log/response
Cloudflare Docs
Tunnel logs · Cloudflare Zero Trust docs
Tunnel logs record all activity between a cloudflared instance and Cloudflare’s global network, as well as all activity between cloudflared and your …
If the origin service had logs that would be helpful as well to see exactly what it is getting
ok this is what logs (from cloudflared cli output) after i run that curl command:
what about the grpcurl?
will check, one moment (also seeing this? )
think that's just silly stuff that can be ignored. cloudflared can do icmp proxying these days. Some stuff is only logged in debug mode for a reason
ah ok no worries, guessed it wasn't important but just wanted to be sure
ok here's the grpcurl log results:
you missed censoring your domain and ips on those
those look like the origin is http://localhost:8080 directly though
I noticed lol, not a huge issue
if you run
cloudflared tunnel ingress rule https://<your-real-website>, what does it say for 'Using rules from <location>', is it using them from the location you'd expect?I mean I think so, I don't really understand what
sslip.io is doing, what it's resolving, etc but that's what the zitadel example documentation is using: sslip.ioAll it does is return the ip you give it, ex 127.0.0.1. Because some of this tooling/etc needs a hostname and not an IP
oh ok
Still though that looks wrong to me, the origin service shouldn't be that..? You're running the tunnel under your user account, or as a service? if you run it as a service it would copy the config file from local and then use /etc/cloudflared/config.yml forever
just under my user account (i.e. inside a terminal i run
cloudflared tunnel --protocol http2 --loglevel debug run my-app)
(inside my ~./cloudflared directory)
$ cd ~./cloudflared && cloudflared tunnel --protocol http2 run my-app
not sure if it's helpful but these commands are how I'm deploying zitadel locally:
https://raw.githubusercontent.com/zitadel/zitadel/main/docs/docs/self-hosting/manage/reverseproxy/docker-compose.yaml
https://raw.githubusercontent.com/zitadel/zitadel/main/docs/docs/self-hosting/manage/reverseproxy/caddy/docker-compose.yamlwhat's the first line when you launch the tunnel under debugmode? It should spit out the config location as well
I do see this, relevant or no?
2024-02-08T05:12:13Z DBG Fetched protocol: quic
and below it 2024-02-08T05:12:13Z INF Initial protocol http2as long as the actual connections say protocol=http2, shouldn't matter
gotcha
I'm assuming you saying /Users/me/.cloudflared/config.yml above is just you censoring it diferently?
yeah correct, sorry about that
always try to remove identifying information when possible, just incase someone else gets ahold of my pastes
no worries. Yea this is a tunnel config issue of some sort, I'm 99% sure, just trying to understand where. The default for not specifying/unparsable is localhost:8080 iirc
oh ok, so maybe it's that sslip.io domain causing issues?
I was wondering where that locahost:8080 value was coming from
I don't think so, I think this is yaml, one sec. Can you post your config.yml censoring anything needed?
yeah of course
btw my caddyfiles contain this:
that might be where the localhost:8080 is coming from
do you actually have anything running on
http://localhost:8080/ that is even reachable from the tunnel host?well http://localhost:8080 doesn't resolve to the zitadel service if i enter that into the browser
but I should say that localhost:8080 is the port zitadel usually runs on
is caddy/cloudflared somehow redirecting requests from auth.example.com:443 to localhost:8080 possibly?
this isn't relevant right?
< alt-svc: h3=":443"; ma=86400
h3 meaning http3 , i'm guessing that's not an issueit's just a hint to tell your client "hey http3 is here, connect and use it sometime"
I messed around with it a bit more and it looks like that host header thing may be something on CF's end. It looks like for QUIC it shows the right host header/url for the request, but for http2 it's localhost:8080
Not sure it actually matters though
ah ok
think I recall that from https://github.com/cloudflare/cloudflared/issues/682#issuecomment-1440736613
honestly so much of this networking stuff just goes over my head, have no idea how most of it works lol
for you, it works from the same host as the tunnel both to the caddy proxy and to zitadel directly, right?
that comment was about it showing /1.1, but for you and me its showing http/2.0, just with
http://localhost:8080ah ok
er not sure I fully understand, but
https://127.0.0.1.sslip.io does resolve, but https://127.0.0.1 gives ERR_SSL_PROTOCOL_ERROR
sorry if that doesn't answer the question, let me know if there's something i can check to help answer itI think you did answer that further up
if you add
`
does that change anything?
just a last attempt I can think of, if it's failing to get the host header properly from logs, it's probably passing the wrong one too
doesn't seem to unfortunately
otherwise I gotta hop off, sorry for leading you on such a wild goose chase with no solution so far. That weirdness with it showing the wrong incoming url is well weird and complicated the already confusing grpc stuff. I'll see if I can't forward that up if I can reproduce it minimally, it may be related.
As for debugging the actual issue, we can see from logs it is sending with the right headers, and the server is returning content-length 0 with that error. So tunnel -> proxy, something's not being forwarded right? Having caddy dump whatever its receiving including content, if you still had any more will to debug.
It's weird though because it complains about the http content-type, but we know it got to cloudflared. so is it not forwarding to the proxy right?
You could always do gRPC over Private Networking w/ WARP but that would require vpn
nah all good mate, I really appreciate you taking the time to help try to resolve this, wouldn't know where to begin on my own
just trying to get this service working locally so that I can develop with it, but keep hitting roadblocks no matter what I try 😂
Looks like that weirdness with http2 showing the wrong host header is just purely a weird logging thing and the origin gets the correct host header/doesn't matter.
I spun up a quick python grpcio server and it properly works via tunnel forced to http2/with http2 to origin. When QUIC is used instead of http2 it properly doesn't work as expected "Message: "b'/MyService/Echo'" requires exactly one request message.". I tested using reflection as well as gRPC streams, and they all worked. So in theory at least it works with gRPC, but of course could be more specific then that if it's using a specific behavior that is unsupported
ah interesting, so does that mean it's more likely an issue with zitadel itself rather than cloudflared?
I mean you are trying to use a feature which is documented as not supported lol
but yea, that or the proxy
well to be fair they say it's an issue with cloudflared 😂 but again i don't really understand the underlying technical details, was just trying to find a solution
that's super helpful to know though, already discussed this topic with the founder so hopefully he has some insight into what could be going wrong from their end
mate I can't thank you enough for all your help debugging this issue, it's been such a thorn in my side for the last week or so, wouldn't have known where to begin myself
just for context, the founder said this yesterday:
So if that's no longer the problem, there's a good chance that some changes to zitadel itself might make it possible to access that grpc endpoint via
cloudflaredjust in case it's worth mentioning, tunnels aren't forcing http2 to origin. They're just willing to accept it. If the origin doesn't say it supports it in tls connect/ alpn it'll just downgrade silently
keyword in the docs is "attempts" lol
cloudflared is all open source btw, I'm no go programmer but it's not too hard to read:
https://github.com/cloudflare/cloudflared/blob/a9aa48d7a1e5e0d06c30a243802af2b228e73995/ingress/origin_service.go#L355
They just pass it through to ForceAttemptHttp2
// ForceAttemptHTTP2 controls whether HTTP/2 is enabled when a non-zero // Dial, DialTLS, or DialContext func or TLSClientConfig is provided. // By default, use of any those fields conservatively disables HTTP/2. // To use a custom dialer or TLS config and still attempt HTTP/2 // upgrades, set this to true. ForceAttemptHTTP2 bool
ah ok, so zitadel requires the ability to "force" http2 to origin, but cloudflared is only willing to accept it, and doesn't necessitate it?
when you say "forcing" to me, it kinda sounds like they need h2c/http2 clear text
I don't understand why it would require it otherwise, espec with the caddy proxy in the front. It's possible I'm missing something, or they are just saying they need h2c (which cloudflared doesn't support, but I thought that was the whole point of the caddy proxy?)
should I perhaps be using https://zitadel.com/docs/self-hosting/manage/reverseproxy/caddy#tls-mode-external instead of https://zitadel.com/docs/self-hosting/manage/reverseproxy/caddy#tls-mode-enabled?
ZITADEL Docs
You can either setup your environment for TLS mode external or TLS mode enabled.
[TLS mode external] - Caddy terminates TLS and forwards the requests to ZITADEL via unencrypted h2c. This example uses an unsafe self-signed certificate for CaddyBy executing the commands below, you will download the files necessary to run ZITADEL behind Caddy with the following config:
whereas [TLS mode enabled] - Caddy terminates TLS and forwards the requests to ZITADEL via encrypted HTTP/2. This example uses an unsafe self-signed certificate for Caddy and the same for ZITADEL.By executing the commands below, you will download the files necessary to run ZITADEL behind Caddy with the following config:Either should work. The end result is the client (cloudflared) can connect via http/2 encrypted
ah right
H2C is just special. It's not supported by browsers (also why you'll only see http/1.1 when connecting to an insecure site), and not the default golang net package Cloudflare uses. You kind of need prior knowledge too, because you don't have TLS/ALPN to pick the best protocol before connecting, have to rely on upgrade: headers like http3 does and upgrade later, but of course gRPC requires http/2.
Shouldn't matter too much, the only bit worth noting is cloudflared can only do http/2 encrypted. Fun web things though, http/3 doesn't have an unencrypted version at all
ah ok, i think that makes sense... i really need to read up on my networking lol. h2c is basically a way of using http/2 insecurely (without TLS) right?
Yep, h2c just means http/2 Cleartext (which does mean no TLS/SSL)