520: Web server is returning an unknown error
I'm getting a persistent error trying to connect to our API:
<title>PODID-XXXX.proxy.runpod.net | 520: Web server is returning an unknown error</title>
It seems to be a cloudflare error. This is not happening if I browse to the endpoint in Chrome, but it's happening through our application or Postman with almost every request. What's going on here? What can we do to fix it?
POD ID: rxb39edmdni21s
Any guidance would be much appreciated.
@Justin Merrell @Madiator2011 [EU]
49 Replies
I do not know what are tou running so
It doesn't seem like our API is being reached at all. What information can I provide?
It's our own docker image. I've been able to get it up and running without any trouble for some basic testing on a different server in the past.
what kinda of API it is? Is it serverless or normal pods?
Normal pods. It's a ComfyUI based docker image.
Logs indicate that it's running normally:
and what error do you get?
have you tried using tcp port instead og http one?
Not yet, but we do need http working.
I mean tcp port only skip cloudflare proxy
I do seem to be able to reach the server via TCP.
I'm not sure how your docker is build so cant check why you cant reach over cloudlfare url
Actually sorry I tested this: ssh [email protected] -i ~/.ssh/id_ed25519
I'll have to try the direct URL.
Well, what needs to be included in order for it to work correctly?
This is our image base:
ARG IMAGE_BASE="pytorch/pytorch:2.2.0-cuda12.1-cudnn8-runtime"
I would first check logs of the pods and see what causes it to be redjected
I don't see anything there.
This is the last log entry:
2024-02-13T18:11:25.568224329Z Starting server
2024-02-13T18:11:25.568244219Z
2024-02-13T18:11:25.568247790Z To see the GUI go to: http://0.0.0.0:8188
And on the system log:
2024-02-13T18:09:38Z start container
Which other logs should I check?
I'm not sure as im not Comfy UI dev
The request never reaches ComfyUI.
I am comfortable with the codebase, the request never hits our API. It's blocked by Cloudflare.
Admittedly I'm not the Comfy dev either so I might be missing something, but it certainly looks like we never reach the server.
mayby you need change something to ComfyUI to allow remote access
I'll do a bit more testing and get back to you.
It is listening. If we browse to https://rxb39edmdni21s-8188.proxy.runpod.net/ using a browser, it works. If I use Postman or make a get request in any other way, it fails.
It will say "Unauthorized: No token provided" which is correct.
Thank you for your help. I'll try to figure it out. It sounds like your advice is to figure out what's going wrong with our API code and that this isn't an issue with the server or cloudflare. Is that right?
I have an update. It appears that the error is in this line:
decoded_token = auth.verify_id_token(id_token)
In this line we use the Firebase Admin SDK to authenticate our user.
This SDK must be able to communicate with Firebase servers over port 443. I believe this is the problem. How can we get this working on Runpod?
Expose ports | RunPod Documentation
Learn to expose your ports.
You could try this?
Perhaps I'm misunderstanding something, but I'm not sure that this is a viable approach since the Firebase SDK is not something we can modify, and as far as I can tell the exposed ports are proxied and available at unique URLS.
What are you using the firebase SDK for? I use firebase, so im surprised there is an issue
I can't tell exactly what's going wrong, but I can tell that we never get past that line of code on a Runpod instance, and it works everywhere else we've tested.
Authentication.
Like for uploading files?
To authenticate a session? etc?
Yes to authenticate a session.
The user passes their auth token and we validate it, the validation is failing on Runpod.
I guess what do you want to use runpod for?
As a back-end for our application...
Like to generate images? so on?
More or less, yes.
i dont think if u want to use runpod as a backend, that it is best anyways
Like a monolithic backend that is up 24/7
So your advice is to...not use Runpod?
I think what you should do is your service can probably architect itself as:
Frontend
Backend on something like Heroku / fly / whatever else
Runpod serverless for image generation + use the GPU Pod as testing for image generation calls
Yes that's what we're doing...
I see, so why authenticate on runpod then? if you have a backend somewhere else?
you can just forward the request to runpod serverless?
and keep runpod more to just image generation
I understand that this can be architected in different ways, and that we may be able to get around the need to authenticate here through an alternative architecture, but I would still love to understand whether a solution is possible.
At this time we are not intending to rearchitect, we may in the near future, but we want to be able to test a relatively simple setup.
Yeah, I mean, is hard to really debug cause I dont have your code, but the long story short is that it should be able to make any requests. The authentication method is probably not some sort of port connection or something, if it through the SDK from firebase, it probably making a standard http connection
I use firebase for file uploads / use the same SDK for nosql database updates
It's literally these two lines of code:
from firebase_admin import auth
auth.verify_id_token(id_token)
With any ID token it will just fail.
I've also had success in posting to Firebase Storage from Runpod, but this is failing.
It should be using port 443 (standard HTTPS)
Just wondering, do u have some sort of wrapper on this? Cause this is a web authentication, usually right?
How would u be able to make a postman request normally?
Oh got it
they have a firebase admin sdk
Right.
and ur template has the port exposed? I guess?
443? i can assume?
(not a good networking person just running through ideas)
so > the request is getting through to runpod, it just this external library is unable to authenticate
That's what it looks like. I'll keep troubleshooting. I really appreciate your help here!
Ok it might actually be a little bit more complicated than I thought. Is it possible that not all headers are being forwarded?
I added extra logging and our Authorization doesn't seem to be in the request anymore?
To be honest, hard for me to say:
https://www.perplexity.ai/search/authverifyidtok-is-not-usBd8j9ARKubqh8yUwmveg?s=c
So many things could be wrong, from improper sdk initialization, authentication headers, etc.
But I think the main thing is your app is architected wrongly anyways.
Runpod serverless is what you are looking for vs GPU Cloud, would be more for testing the image generation side of your app. If you use Runpod as a backend monolith, you'll have like easily 10x to 100x the cost if you use something like fly.io, or whatever else for monolithic hosting. just 1-2 days of runpod backend hosting, you'll match the cost of a 24/7 backend on heroku.
Second, if you set up your backend somewhere else you'll just have an easier time proxying the request to runpod, and just simplify the image generation part / authorization anyways, since you'll prob need to do JWT tokens / other authentications requiring different databases, and better uptime guarantees.
And finally, by using runpod serverless, you can scale ur image generation + have it in a queue, and webhook back to a standing backend.
I think you certainly could continue to debug this, and all for it, but I think its a bit waste of your time tbh vs just moving ur authentication up one layer, and using runpod to image gen + upload results
And again, I just think the cost structure for running a gpu enabled backend is significantly more than just using GPU CLoud for testing your code + image gen, before then migrating it to runpod serverless
I have a Firebase Functions back-end for most other tasks. This really is for image generation. We haven't written all the handlers yet for runpod serverless, we wanted to do an inital test on GPU cloud first.
Got it
We've previously tested on other servers using this same docker image and expected this test to be seamless.
This was a surprising block.
Yeah, Hmmm, is hard for me to say tbh i guess I would expect it too
But yeah, prob keeping runpod to just a handler.py for image gen is your best bet + also maybe code for file upload to firebase storage is what i use runpod for
I know these two work
and then the auth, i wonder if firebase function is possible
to unload the auth there
or maybe fly.io
is another service with dockerized service
that spins up and down for cpus
But I've confirmed the Authorization header is sent, but not received.
I logged the outgoing request headers both in Postman and through our app, and I logged what was received in the server code. The header is just not there.
Solution
maybe pass it through in the body instead then?
and reassmble the header request
on the runpod side?
Yep, that could work.
Will try.
I've got it working, thanks for your help!
I may have some additional questions when I get up and running with serverless, it sounds like you've got a fair bit of experience there! I really appreciate your patience. Thanks again!
Glad you got it working, well essentially its pretty simple, you can have a handler.py with a function that takes in an event, where the event will have an input key, that you parse off and do anything with
Can read runpod serverless docs some more
But they have the runpod python package, where you just do something like:
runpod.start({handler: yourfunctionname})
and whenever runpod serverless gets a request, it calls the handler function, passes it the request body, you parse it and do anything u want
so if u gets omething working on GPU Cloud, most of the time is a pretty easy transition to Serverless
(the hard part is preloading models into an image usually)
Good to know though that any requests that is coming through the runpod proxy, the headers will be stripped if you try to just directly pass it through. So must assembly it through the body instead.