Gunicorn timeout when sending POST between two apps
Hi there,
My project have two apps, APP_1 for handling everything concerning my website and APP_2 which acts like a proxy for all external API calls (to avoid csrf related issues). Everything run smoothly locally but when I'm in production on Railway, gunicorn seems to time out and crash when request between APP_1 and APP_2 is made.
In this specific case, APP_1 handles a form and send a POST request to APP_2 and it then communicates with Stripe API. The timeout crash happens when APP_1 uses request.post (python library) and POST request never reaches APP_2. (Actually, request does reach App2 but only after a timeout crash ... check logs below ...). I should add that in production, APP_2 online endpoint is working fine and I can reach it with curl command and proper authentification.
Could you help me there ? Or give me some pointers ?
LOGS
=> APP_1 receive form and send post request to APP_2
[2023-08-15 09:57:06,324] [INFO] {views -> post} - Will send request to https://www.mywebsite.com/api/proxy/stripe/
[2023-08-15 09:57:06,330] [INFO] {views -> post} - Membership form is valid
=> Here It's waiting to hear from APP_2 but it timeout
[2023-08-15 09:57:36 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:9)
[2023-08-15 09:57:36 +0000] [9] [INFO] Worker exiting (pid: 9)
[2023-08-15 09:57:36 +0000] [11] [INFO] Booting worker with pid: 11
=> And then, POST request is finally received after App1 has crashed ...
[2023-08-15 09:57:37,212] [INFO] {views -> post} - Stripe proxy is handling form submission ...
[2023-08-15 09:57:37,212] [DEBUG] {views -> post} - request.POST: <QueryDict: {...DATA...}>
[2023-08-15 09:57:37,213] [INFO] {views -> post} - Checking if customer already exists ...
[2023-08-15 09:57:37,418] [INFO] {views -> post} - Getting prices from Stripe ...
[2023-08-15 09:57:37,510] [INFO] {views -> post} - Creating checkout session ...
=> Here it sends back Stripe checkout session url but since APP_1 has crashed noting happens on website.
12 Replies
Project ID:
6b57557e-7c26-4be1-87e8-e6b0cc114cea6b57557e-7c26-4be1-87e8-e6b0cc114cea
Project ID: 6b57557e-7c26-4be1-87e8-e6b0cc114cea
1. are you using internal network?
2. how long until you reach the timeout?
Hey Brody thanks for your answer ! So :
1. Internal network ? Not exactly sure what you mean but both apps have the same root https domain (public).
2. Between form validation "Membership form is valid" at 09:57:06 and "WORKER TIMEOUT " at 09:57:36 so exactly 30 seconds.
1. https://docs.railway.app/reference/private-networking
2. 30 seconds is the default gunicorn timeout, to up the timeout time, add
--timeout 300 to your start command to set the timeout to 5 minutes, for example
1.1. if these two services are in the same project, defiantly look into using private networking as it would speed everything up significantlyOk this is extremely interesting, I wasn't aware of this feature !
I checked and private networking is indeed enabled for my project, raising gunicorn default timeout with start would be the way to go here.
I finally chose to avoid using a proxy altogether, I realized it wasn't really necessary for this specific service. All my calls are now made directly from APP_1 and everything is working smoothly. That said, I can't wrap my head around this internal communication problem, it seems to me that two apps within the same project should be able to send http requests to each other without excessive timeouts 🤔 but surely I'm missing something here ... Anyway, thank you for your answers !
I finally chose to avoid using a proxy altogether, I realized it wasn't really necessary for this specific service. All my calls are now made directly from APP_1 and everything is working smoothly. That said, I can't wrap my head around this internal communication problem, it seems to me that two apps within the same project should be able to send http requests to each other without excessive timeouts 🤔 but surely I'm missing something here ... Anyway, thank you for your answers !
no problem!
Sorry @Brody I have the same problem. Where the code (. - - timeout 300 ) is inserted in my main.py or some config file ?, Thanks
#🛂｜readme #5
as I have previously stated, you add that flag to your start command
Somethink like this: web: gunicorn main:app --timeout 120
if you want 2 minutes, then yes
Yes!! Bro,, thak so much